Stream processing in plain words: change data capture (CDC) and time in streams

← Back to the section

In derived data we agreed: write to the source of truth, and derive caches, indexes, and marts from it as a single stream of changes. This article is about how that stream works in practice.

Stream processing is essentially batch processing that never ends: the same transformations, but not over a finite set of data — over an unbounded stream of events, and at low latency. Let's go through the two things people trip over most: how changes actually flow from the database into derived systems (CDC), and why stream analytics starts to lie when you confuse event time and processing time.

CDC: the database as the source of a change stream

How do you keep a search index always matching the database without sliding into a dual write (two separate writes that silently diverge)? The answer is change data capture (CDC). The idea: make the database the leader, read its change log, and apply those changes to derived systems in exactly the same order they landed in the database.

How it works. The database already has a write-ahead log (WAL) — the same one replication runs off. A CDC tool connects to that log as another "replica" and decodes the change stream. For example, Debezium can read PostgreSQL logical decoding, the MySQL binlog, the MongoDB oplog — and publishes all the changes to a log-based broker like Kafka. From there the search index, the cache, the analytics store are just consumers of one and the same stream; and since the stream preserves the order of changes, they all converge to the database's state.

Two details, without which the CDC picture is incomplete:

• The initial snapshot. The log doesn't keep the whole history (old chunks are dropped), but a new index needs the long-unchanged records too. So the start works like this: first a snapshot of the whole database at a known log position, then the change stream from that position on.
• Log compaction. To avoid keeping the log forever, the broker discards old records for keys that have already been overwritten, keeping only the latest value per key. Then the log turns into a full copy of the current state — and a derived system can be rebuilt from scratch out of it alone.

CDC is asynchronous: derived systems lag behind the database by the replication delay, like ordinary replicas. The upside is that the source of truth barely feels slow consumers — they don't drag it down.

Event time versus processing time

The second trap is subtler. A stream processor often computes something "over the last 5 minutes." And here it suddenly turns out that "time" isn't one notion but two:

• event time — when the action actually happened;
• processing time — when the processor finally got around to that event.

There's a gap between them: events are delayed in queues, on network faults, and especially on a consumer restart (which then burst-processes everything that piled up during the downtime). And here grouping by processing time draws a phantom. The processor stalled for a minute, then on restart processed the accumulated events all at once — and the "request rate" chart grows a spike, though the real rate never changed for a second. You must count by event time.

The difference shows up in a small program — one and the same stream, two counts per minute:

live example

import java.util.Map;
import java.util.TreeMap;

public class WindowDemo {
    record Event(int eventSec, int arrivedSec) {}

    public static void main(String[] args) {
        Event[] stream = {
                new Event(65, 66), new Event(85, 86),
                new Event(125, 185), new Event(145, 186),
                new Event(185, 187), new Event(205, 206)
        };
        Map<Integer, Integer> byEvent = new TreeMap<>();
        Map<Integer, Integer> byArrival = new TreeMap<>();
        for (Event e : stream) {
            byEvent.merge(e.eventSec() / 60, 1, Integer::sum);
            byArrival.merge(e.arrivedSec() / 60, 1, Integer::sum);
        }
        System.out.println("by event time:      " + byEvent);
        System.out.println("by processing time: " + byArrival);
    }
}

In the first row the rate is flat — two events per minute; in the second, minute two is empty and minute three spikes with four.

An analogy: Star Wars did not come out in episode order — first IV, V, VI, then I, II, III. The event date here is the year in the plot, the processing date is when you watched the film. In a stream, order gets scrambled the same way.

A separate pain is straggler events. You seemingly already closed and counted the 37th-minute window, and then an event with timestamp 37:59 flies in, having been stuck in the network. What do you do with it? Two options: either ignore latecomers (and then be sure to watch what fraction you drop), or publish a correction to the already-emitted window. There's no "right" answer — it's a deliberate choice.

Window types

Since we count by event time, we have to decide how to slice time into windows:

• Tumbling — fixed-length windows, and each event lands in exactly one: minute windows 10:03:00–10:03:59, 10:04:00–10:04:59, and so on. The simplest option.
• Hopping — also fixed length, but the windows overlap for smoothing: for example, a 5-minute window with a 1-minute hop.
• Sliding — takes all events that fall within a given width of each other; it has no fixed boundaries.
• Session — no fixed length at all: groups one user's events that occur close in time, and closes when the user goes quiet (say, 30 minutes of silence). A web-analytics classic.

Where this applies

Keeping a derived system fresh in near-real time, or computing metrics over an event stream — that is stream processing.

Where beginners stumble:

• Keeping an index and a cache in sync with a dual write — and they diverge silently: the leader must be the database.
• Counting rate or average by processing time — and, when the processor lags, getting phantom spikes that never happened.
• Treating a window as closed by the wall clock — and silently losing straggler events instead of applying a policy chosen in advance.

In short

• CDC makes the database the leader: derived systems read its log and apply changes in the same order — instead of a dual write, which drifts apart.
• A consumer starts with a snapshot at a known log position plus the change stream from there; a compacted log equals the current state.
• CDC is asynchronous: derived systems lag by the replication delay, but a slow consumer doesn't drag the source of truth down.
• Time-based analytics is counted by event time: by processing time, a lagging processor draws a spike that never happened.
• Windows come as tumbling, hopping, sliding and session; stragglers need a policy chosen up front: drop or correct.

What to read next

• Derived data — why a single change stream is needed at all.
• AMQP and Kafka — the log-based broker CDC and replay rest on.
• Event sourcing — a stream of events as the source of truth.
• Dual writes — the anti-pattern CDC replaces.