A real application almost never keeps its data in one place. Alongside the main database there usually live a cache, a search index, an analytics store, denormalized tables, read models. It's easy to drown in this zoo and lose track of who owns what and which copy to believe when they disagree.
There's a simple frame that instantly brings order: split every store into two kinds — the system of record and derived data. This is an article "for horizons": it's not about a specific technology but about how to stop getting lost in your own architecture.
The source of truth and its reflections
The system of record (the source of truth) is the store where a fact lives exactly once and in authoritative form. Usually that's your normalized database: a user placed an order — the order is written here. And if the data in the source ever diverges from the data in any other store, the source is by definition right.
Derived data is the result of transforming data from the source, which can always be recomputed. Almost everything else falls here:
- a cache — a fast copy of part of the data;
- a search index (Elasticsearch, GIN) — the same data repacked for text search;
- a materialized view or a denormalized field — a precomputed result;
- a read model in CQRS — data laid out exactly for a specific screen;
- an analytics store (ClickHouse) — the same events, but shaped for scans.
The defining trait of derived data is redundancy: it duplicates what already exists in the source, for the sake of read speed. The same thing can be reflected in different ways, so one source usually has several derived sets.
The key property: derived data is disposable
From this definition follows a practical conclusion that changes how you treat data: derived data is disposable. Lost the cache, the index, or the read model? No disaster: rebuild it from the source, and the system is good as new. But lose the system of record itself — now that's a real disaster, there's nothing to restore from.
Everything else follows from this. Back up and protect the source of truth first. Every derived set must have a from-scratch rebuild procedure (reindex, warm the cache, rebuild the projection). And "the data schema went missing after the rollout" stops being a catastrophe if what went missing was merely a projection — you just rebuild it. Knowing which of your stores is the source and which is derived is already half of a clear architecture.
The three ways to produce derived data
How exactly does data flow from the source into derived sets? There are three processing modes, and it pays to tell them apart:
- Online (a service). Waits for a request and answers as fast as possible; what matters is response time and availability. This is how the application itself works. But online is no good for producing large derived sets.
- Batch. Takes a large set of data as input, grinds on it for minutes or hours, produces a result; what matters here is throughput, and it runs on a schedule (a nightly reindex, a mart recompute). The big-data classic is MapReduce/Spark; in an ordinary backend it's simpler — a queue-based background worker.
- Stream. In between: reacts to events as they arrive, with low latency, but processes a stream of changes rather than the whole set at once. This is how derived data is kept fresh in near-real time.
Batch rebuilds everything whole and is reliable but lagging; stream keeps the derivative fresh but is more complex to build. Choosing between them is a big topic of its own.
The discipline: derive, don't dual-write
The main mistake with derived data is writing to it directly, bypassing the source. The classic example is a dual write: the application writes to both the database and the cache (or the search index) as two separate operations. Sooner or later one operation succeeds and the other fails — and the data silently diverges.
The right principle: write only to the system of record, and derive everything else from it — as a single stream of changes. Then all the derived sets share one order of updates, they're consistent with each other, and they rebuild easily. Technically this is done with events and the outbox pattern, or via CDC (change data capture — reading the database's change log). The same principle underlies event sourcing: the source of truth is the stream of events, and everything else (projections, read models) is derived from it.
Where this applies
The moment a cache, a search index, or a separate mart appears next to the database — you're already managing derived data, whether you called it that or not. The practical frame: explicitly name what's your source of truth and what's derived; give every derived set a rebuild procedure; and never write to a derivative bypassing the source — only derive from a single stream of changes.
Where beginners stumble:
- Dual-writing to the database and the cache/index — two operations without a shared transaction will sooner or later diverge. Write to the source, derive the derivative (events/outbox/CDC).
- Having no rebuild procedure for a derived set — and "reindex from scratch" suddenly turns out to be impossible, while losing the index becomes an incident out of nowhere.
- Backing everything up the same way — not separating the irreplaceable source of truth from disposable derivatives, and spending effort in the wrong place.
- Treating a read model or a denormalized field as a "second truth" — and on a discrepancy starting to guess who's right. The source is always right.
What to read next: the read model in CQRS — derived data shaped for a specific screen; cache invalidation — keeping a derived copy fresh; event sourcing — a stream of events as the source of truth; dual writes — why you can't write to two places at once.