This is the final article of the series on distributed data. In the trouble with distributed systems we went through what you can't trust in such a system: the network, the clocks, your own process. Here we cover the abstractions that hide all of that, and consensus — several nodes agreeing on one question.
The main takeaway will be surprising: half the tasks that look completely different (elect a leader, guarantee a name is unique, commit a distributed transaction) are actually the same task. Let's get to that conclusion step by step.
Linearizability: the illusion of a single copy
In an eventually consistent database, the same query to two replicas can return different things — one managed to update, the other lagged. Linearizability (also called strong consistency) removes this: the system behaves as if there's only one copy of the data and every operation happens instantly and whole. As soon as a client writes a new value, any later reader sees exactly it. This is a recency guarantee — not "it'll converge someday" but "right now it's the latest."
The classic violation. Alice and Bob are watching the final of a match in one room. Alice refreshed the page, saw the score, and shouted it to Bob. Bob refreshes on his end — but his request hit a lagging replica where the match is "still going." The replica's lag turned into a visible bug precisely because there was a second channel between Alice and Bob — her voice.
Linearizability isn't needed everywhere — it's slow (why, see CAP below) — but in a few places you can't do without it:
- Leader election. There must be exactly one leader (otherwise split brain). The nodes have to reach a single opinion on who it is.
- Uniqueness constraints. Two people simultaneously take one name / book one seat / withdraw from one account — exactly one must win.
- Cross-channel dependencies. The example above: two different paths to the data (cache and database, queue and storage) breed a race that linearizability is exactly what closes.
It's important not to confuse linearizability with serializability. Serializability is about transaction isolation (several objects, the order of transactions among themselves). Linearizability is about the recency of read-write on a single object. These are different guarantees, even though the names look alike.
The CAP theorem without the myths
If linearizability is so useful — why not make everything linearizable? The network gets in the way. Picture two data centers with replication between them, and now the link between them has broken. A client on the "cut-off" side wants to write something. The choice is hard:
- Insist on linearizability. A replica in the minority can't confirm its data is current, so it has to refuse — become unavailable — until the link is restored. That's the letter "C": consistency at the price of availability.
- Want availability. Then accept the write locally, but the replicas of the two data centers will inevitably diverge (that's no longer linearizable). That's the letter "A".
This is CAP. But the popular gloss "pick 2 of 3: Consistency, Availability, Partition tolerance" is wrong. A network partition (the letter "P") isn't something you can "pick" or "not pick": it just happens, whether we want it or not. The honest phrasing is shorter: when the network is partitioned — choose between consistency and availability. And while the network is healthy, you get both at once.
Since P (a network partition) is unavoidable in a distributed system, the real choice is a CP or AP side: what to do during a partition — keep consistency (and refuse writes for a while) or availability (and let the replicas diverge). The CA side is a single node in the first place, where there's no partition between nodes. That's exactly why the databases sit along the sides: MongoDB/HBase are CP, Cassandra/DynamoDB are AP.
Besides, CAP is very narrow: it speaks only of one kind of fault (a broken link) and one model (linearizability), and says nothing about delays and dead nodes. So in practice the theorem is of mostly historical interest.
And a separate curious fact: surprisingly few systems are actually linearizable — and they give it up not to survive faults but for speed. Linearizability is always slow (response time trails network latency), and most systems deliberately don't provide it.
Order, causality, and broadcast
Linearizability implies a total order of operations: a single scale on which any two operations can be compared for which came first. But there's a weaker order, often enough — the causal one. It orders only operations linked by the happens-before relation: a question must come before an answer, a row must be created first and only then updated. And operations that know nothing about each other are concurrent, and their order stays undefined. Causal order is partial: it's like the commit history in Git, where branches diverge and later merge.
Causality can be preserved more cheaply than full linearizability. Wall-clock timestamps are no good for ordering (clocks drift), but Lamport timestamps — simple logical counters that grow with each operation and are passed in every message — give a total order consistent with causality. But even they are not enough to settle the question of a unique name: with Lamport timestamps the order becomes known only after the fact, once all operations are gathered together — and "is the name taken right now?" can't be decided that way.
What you really need is to know at what moment the sequence is finally fixed. That's given by total order broadcast: a protocol that guarantees all nodes receive the same messages in the same order, reliably, even under faults. In essence it's the replication log: "deliver a message" = "append to the log," and all replicas, replaying the log in one order, necessarily converge to one state.
Consensus: it's all one task
And here's the climax. It turns out all of the following tasks are equivalent — solve one for real, and you get a solution to the rest:
- total order broadcast;
- a linearizable cell with a "compare and set" operation (compare-and-set);
- leader election;
- a uniqueness constraint;
- atomic commit of a distributed transaction.
All of them are consensus: getting several nodes to agree on one value. And consensus has strict requirements: one decision for everyone, it can't be undone, it must be something actually proposed by one of the nodes, and — termination: the algorithm must eventually reach a decision as long as a majority is alive.
Atomic commit is a familiar special case. The classic protocol is two-phase commit (2PC). The coordinator sends all participants "prepare" (phase 1), collects a "yes" from all, writes the decision to its own log (the point of no return), and sends "commit" (phase 2). Having answered "yes," a participant loses the right to abort on its own. And if the coordinator dies exactly between the phases, the participant is stuck in uncertainty — holding locks and waiting for the coordinator to come back to life. That's why 2PC is called blocking: the coordinator is a single point of failure here. It's essentially bad consensus.
Good consensus is Raft, Paxos, Zab, VSR. They all rest on a majority quorum (two majorities can't fail to overlap → there won't be two contradictory decisions) and on "epochs": in each epoch there's at most one leader, and a decision is made by a majority vote. The theory is scary with the FLP result (in a fully asynchronous system, guaranteed consensus doesn't exist), but in practice partial synchrony or a drop of randomness is enough — and consensus is achievable.
The practical conclusion is single and firm: don't write consensus yourself. Getting it right is nearly impossible. Take a ready-made service — ZooKeeper or etcd: they give you linearizable storage, leader election, distributed locks, and fencing tokens out of the box. That's exactly how HBase, Kafka, and Kubernetes are built inside.
Where this applies
Every time a system needs exactly one leader, a unique name, "atomically claim a resource," or a consistent order of events — you've run into consensus, even if you didn't call it that. The practical frame: don't reach for linearizability and distributed transactions without need (they're slow and fragile — applications route around them with sagas and outbox); and where node agreement is genuinely necessary — don't invent a protocol, hand the coordination to ZooKeeper or etcd.
Where beginners stumble:
- Confusing linearizability and serializability. The first is about the recency of a single object, the second about transaction isolation. Different guarantees.
- Reading CAP as "2 of 3". A network partition isn't chosen; the "consistency or availability" choice arises only at the moment of the partition.
- Ordering events by a Lamport timestamp and thinking they've solved uniqueness. The total order is known only after the fact; to check "now" you need total order broadcast.
- Writing their own distributed lock or leader election. That's consensus, which is nearly impossible to implement correctly. Take ZooKeeper/etcd.
- Reaching for 2PC between microservices. It's a blocking protocol with a coordinator as a single point of failure; in microservices a saga is almost always better.
What to read next: the trouble with distributed systems — partial failures and the fencing token, where consensus begins; replication models — quorums and eventual consistency; distributed transactions — why applications route around 2PC with sagas; ACID and isolation — serializability, which is important not to confuse with linearizability.