In derived data we agreed: write to the source of truth, and derive caches, indexes, and marts from it as a single stream of changes. This article is about how that stream works in practice, and about the two things people trip over most: how changes actually flow from the database into derived systems (CDC) and why stream analytics lies when you confuse event time and processing time. Stream processing is a batch that never ends: the same transformations, but over an unbounded stream, at low latency.
CDC: the database as the source of a change stream
How do you keep a search index always matching the database without sliding into a dual write? The answer is change data capture (CDC): make the database the leader, read its change log, and apply those changes to derived systems in the same order they landed in the database.
The mechanics: the database already has a write-ahead log (WAL) — replication runs off it. A CDC tool connects to it as another "replica" and decodes the change stream: Debezium reads PostgreSQL logical decoding, the MySQL binlog, the MongoDB oplog — and publishes the changes to a log-based broker (Kafka). From there the index, the cache, the analytics store are just consumers of that stream, and since it preserves order, they converge to the database's state.
Two details without which CDC is incomplete:
- The initial snapshot. The log doesn't keep the whole history (old segments are dropped), but a new index needs even the long-unchanged records too. So the start is a snapshot of the whole database at a known log position, followed by the change stream from that position.
- Log compaction. To avoid keeping the log forever, the broker discards old records whose key has been overwritten, keeping only the latest value per key. Then the log = a full copy of the current state, and a derived system can be rebuilt from scratch out of it alone.
CDC is asynchronous: derived systems lag by the replication delay — like ordinary replicas. The upside is that the source of truth barely feels slow consumers.
Event time versus processing time
The second trap is subtler and nastier. A stream processor often computes something "over the last 5 minutes" — and here there are two different times:
- event time — when the action actually happened;
- processing time — when the processor got to the event.
There is a gap between them: events are delayed in queues, on network faults, on a consumer restart (which then burst-processes the backlog). And then grouping by processing time gives a phantom: the processor stalled for a minute, then processed the accumulated events all at once — the "request rate" chart draws a spike, though the real rate never changed. You must count by event time.
Kleppmann's analogy: Star Wars did not come out in episode order (IV, V, VI, then I, II, III). Watch it in release order and the story order is broken. The event's date is the year in the plot; the processing date is when you watched it. In a stream, event order is disrupted the same way, and the algorithm must account for it.
A separate pain is straggler events: you seemingly closed and counted the 37th-minute window, and then an event with timestamp 37:59 arrives, stuck in the network. Two options: ignore latecomers (and monitor the dropped fraction) or publish a correction to the already-emitted window. There is no "right" answer — it is a deliberate choice.
Window types
Since we count by event time, we must decide how to slice time into windows:
- Tumbling — fixed length, each event in exactly one window: minute windows 10:03:00–10:03:59, 10:04:00–10:04:59. The simplest option.
- Hopping — fixed length, but windows overlap for smoothing: a 5-minute window with a 1-minute hop.
- Sliding — all events falling within a given width of each other; no fixed boundaries.
- Session — no fixed length: groups one user's events that occur close in time, and closes when the user goes quiet (say, 30 minutes of silence). A web-analytics classic.
Where this applies
The moment a derived system (index, cache, mart) must stay fresh in near-real time, or you must compute metrics over an event stream — you are in stream processing. The practical frame: sync derived systems via CDC / a log-based broker, not dual writes; compute any time-based analytics by event time, not processing time; for windows, decide honestly what you do with latecomers.
Where beginners stumble:
- Keeping an index/cache in sync with a dual write — they diverge silently. CDC makes the database the leader and derivatives the log's consumers.
- Counting rate/average by processing time — a lagging processor produces phantom spikes that never happened in reality.
- Treating a window as closed by the wall clock — and losing straggler events. You need a policy: drop (and monitor the fraction) or publish a correction.
- Confusing event-time and processing-time on restart — replaying the accumulated stream draws an anomaly out of nowhere.
What to read next: derived data — why a single change stream is needed at all; AMQP vs Kafka — the log-based broker CDC and replay rest on; event sourcing — a stream of events as the source of truth; PostgreSQL replication — the WAL that CDC reads. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 11.