In derived data we agreed: write to the source of truth, and derive caches, indexes, and marts from it as a single stream of changes. This article is about how that stream works in practice.
Stream processing is essentially batch processing that never ends: the same transformations, but not over a finished, finite set of data — over an unbounded stream of events, and at low latency. Let's go through the two things people trip over most: how changes actually flow from the database into derived systems (CDC), and why stream analytics starts to lie when you confuse event time and processing time.
CDC: the database as the source of a change stream
How do you keep a search index always matching the database without sliding into a dual write (two separate writes that silently diverge)? The answer is change data capture (CDC). The idea: make the database the leader, read its change log, and apply those changes to derived systems in exactly the same order they landed in the database.
How it works. The database already has a write-ahead log (WAL) — the same one replication runs off. A CDC tool connects to that log as another "replica" and decodes the change stream. For example, Debezium can read PostgreSQL logical decoding, the MySQL binlog, the MongoDB oplog — and publishes all the changes to a log-based broker like Kafka. From there the search index, the cache, the analytics store are just consumers of one and the same stream; and since the stream preserves the order of changes, they all converge to the database's state.
Two details, without which the CDC picture is incomplete:
- The initial snapshot. The log doesn't keep the whole history (old chunks are dropped), but a new index needs the long-unchanged records too. So the start works like this: first a snapshot of the whole database at a known log position, then the change stream from that position on.
- Log compaction. To avoid keeping the log forever, the broker discards old records for keys that have already been overwritten, keeping only the latest value per key. Then the log turns into a full copy of the current state — and a derived system can be rebuilt from scratch out of it alone.
CDC is asynchronous: derived systems lag behind the database by the replication delay, like ordinary replicas. The upside is that the source of truth barely feels slow consumers — they don't drag it down.
Event time versus processing time
The second trap is subtler and nastier. A stream processor often computes something "over the last 5 minutes." And here it suddenly turns out that "time" isn't one notion but two:
- event time — when the action actually happened;
- processing time — when the processor finally got around to that event.
There's a gap between them: events are delayed in queues, on network faults, and especially on a consumer restart (which then burst-processes everything that piled up during the downtime). And here grouping by processing time draws a phantom. The processor stalled for a minute, then on restart processed the accumulated events all at once — and the "request rate" chart grows a spike, though the real rate never changed for a second. You must count by event time.
An analogy. Star Wars did not come out in episode order (first IV, V, VI, then I, II, III). Watch it in release order and the story order is broken. The event's date is the year in the plot, and the processing date is when you watched the film. In a stream, event order gets scrambled the same way, and the algorithm has to account for it.
A separate pain is straggler events. You seemingly already closed and counted the 37th-minute window, and then an event with timestamp 37:59 flies in, having been stuck in the network. What do you do with it? Two options: either ignore latecomers (and then be sure to monitor what fraction you drop), or publish a correction to the already-emitted window. There's no "right" answer — it's a deliberate choice.
Window types
Since we count by event time, we have to decide how to slice time into windows:
- Tumbling — fixed-length windows, and each event lands in exactly one: minute windows 10:03:00–10:03:59, 10:04:00–10:04:59, and so on. The simplest option.
- Hopping — also fixed length, but the windows overlap for smoothing: for example, a 5-minute window with a 1-minute hop.
- Sliding — takes all events that fall within a given width of each other; it has no fixed boundaries.
- Session — no fixed length at all: groups one user's events that occur close in time, and closes when the user goes quiet (say, 30 minutes of silence). A web-analytics classic.
Where this applies
The moment a derived system (an index, a cache, a mart) has to stay fresh in near-real time, or you have to compute metrics over an event stream — you're in stream processing. The practical frame: sync derived systems via CDC and a log-based broker, not dual writes; compute any time-based analytics by event time, not processing time; and for windows, decide honestly, in advance, what you do with latecomers.
Where beginners stumble:
- Keeping an index/cache in sync with a dual write — and they diverge silently. CDC makes the database the leader and derivatives the consumers of its log.
- Counting rate or average by processing time — and, when the processor lags, getting phantom spikes that never happened in reality.
- Treating a window as closed by the wall clock — and losing straggler events. You need a policy chosen in advance: drop (and monitor the fraction) or publish a correction.
- Confusing event-time and processing-time on restart — and replaying the accumulated stream draws an anomaly out of nowhere.
What to read next: derived data — why a single change stream is needed at all; AMQP vs Kafka — the log-based broker CDC and replay rest on; event sourcing — a stream of events as the source of truth; dual writes — the anti-pattern CDC replaces.