How classic leader–follower replication works we cover in practice: streaming replication in PostgreSQL and replica sets in MongoDB. This article is one floor up: what exactly breaks when a replica lags, why some systems keep several leaders at once, and how Dynamo-family databases manage with no leader at all. The three replication models are three different answers to one question: who accepts writes and who resolves conflicts.
Lag anomalies: a vocabulary of guarantees
An asynchronous replica lags behind the leader — usually by fractions of a second, under load by seconds and minutes. The state of "replicas will catch up if writes stop" is called eventual consistency, and the word "eventual" is deliberately non-committal. In practice the lag shows up as three concrete anomalies, each with its own antidote guarantee:
- A user doesn't see their own write. They post a comment (the write goes to the leader), refresh the page (the read lands on a lagging replica) — the comment is gone, which looks like data loss. The guarantee is read-your-writes: a user reads their own data either from the leader or from a replica that has caught up to their write. Practical techniques are in the PostgreSQL replication article.
- Time moves backward. Two consecutive requests hit different replicas: the first has almost caught up, the second lags more — the user sees a comment, refreshes, and the comment disappears. The guarantee is monotonic reads: later reads are never older than earlier ones; the simplest way is to consistently route a user to the same replica (say, by an id hash).
- The answer before the question. In a conversation the replica with the answer catches up before the replica with the question — an observer sees causality inverted. The guarantee is consistent prefix reads: if writes happened in some order, they are read in that order. This hurts most in sharded systems, where partitions have no shared order of writes.
The working rule: when designing reads from replicas, ask yourself which of the three anomalies would hit your scenario — and provide exactly the guarantee you need. Pretending to be a synchronous system while being asynchronous is a source of the most baffling bugs.
A separate rake of the leader–follower model itself is failover: when the leader fails, one replica is promoted, and if replication was asynchronous, the old leader's unreplicated writes are usually discarded. The classic GitHub incident: a promoted, lagging replica started handing out already-used primary keys — and other people's data was shown to the wrong users. The second danger is split brain: two nodes both believe they are the leader and both accept writes.
Multi-leader: several leaders and conflicts
Sometimes a single leader is a bottleneck not by load but by geography and connectivity. Three scenarios where you make several leaders:
- Multiple data centers: a leader in each DC, a write is handled locally and asynchronously replicated to the rest — users don't wait for an intercontinental round trip, and an entire DC failure doesn't stop writes.
- Offline clients: a calendar on a phone and a laptop accepts writes with no network — each device is effectively a leader with its own local database, and sync is asynchronous replication with a lag of hours.
- Collaborative editing: Google Docs is the same model taken to the limit: a local replica in every tab.
The model has one price, and it is serious: write conflicts. Two leaders concurrently accepted a change to the same record, both answered "success" — and when exchanging changes it turned out they are incompatible. Asking the user to sort it out is too late. The database must resolve the conflict convergently — so that all replicas settle on one value:
- Last write wins (LWW): each write has a timestamp, the later one wins. Simple, ubiquitous (the only option in Cassandra) — and it is silent data loss: losing writes, for which the client got an acknowledgment, quietly vanish. Acceptable for a cache; for anything you'd hate to lose, no.
- Merging values: keep both variants and merge them — by application code on the next read, or automatically with CRDTs (data types designed so that concurrent changes merge without loss — counters, sets).
- Prevention: the most practical option is to route all changes to a given record through the same leader (a user's writes always go to their "home" DC). Then for each record the system is effectively single-leader, and there are no conflicts until a failover between DCs happens.
The key to understanding conflicts is the happens-before relation: operation B depends on A if it knew about A. If neither knew about the other — they are concurrent, and that is exactly when a resolution mechanism is needed. Physical clocks are unreliable for this (they are out of sync) — databases track dependency with versions and version vectors.
Leaderless: quorums instead of a leader
The third model drops the leader entirely (Dynamo-style: Cassandra, Riak): the client sends a write to several replicas in parallel and considers it successful after acknowledgment from w of the n nodes; it reads from r nodes at once too, comparing versions. The quorum condition: with w + r > n the write and read sets overlap — at least one of the polled nodes saw the latest write. Typically n=3, w=r=2: one replica can be down while writes and reads continue — failover as a procedure is not needed at all.
Lagging nodes catch up by two mechanisms: read repair (a client, seeing a stale answer, writes the fresh value back) and background anti-entropy (a process compares replicas and rolls the difference forward).
The caveats that keep a quorum from being an absolute guarantee: a sloppy quorum on network faults accepts writes to "foreign" nodes with a subsequent hinted handoff — and then w and r may not overlap; concurrent writes still need conflict resolution (the same LWW with the same losses); monitoring "lag" here is harder than in single-leader, where there is a position in the log. The honest bottom line: quorum databases are tuned for eventual consistency, and w and r control the probability of a stale read, not a guarantee.
Where this applies
Choosing a model is choosing where it hurts: single-leader is simple and gives an order of writes but needs failover and hits one node's ceiling; multi-leader unties geography at the cost of conflicts; leaderless removes failover at the cost of quorum arithmetic and probabilistic guarantees. In a typical backend the right default is single-leader (PostgreSQL) with explicit read guarantees; the other models come in when geography or write availability demand it.
Where beginners stumble:
- Reading everything from replicas — and catching "vanishing comments" and "time going backward." Read guarantees are chosen deliberately, per scenario.
- Trusting LWW. "Last write wins" sounds harmless but means "losing writes disappear silently, even though the client got an acknowledgment."
- Treating w+r>n as an iron guarantee — sloppy quorums, concurrent writes, and recovery from a stale replica leave gaps.
- Enabling multi-leader for "reliability" inside a single DC — you get write conflicts with no upside: within one DC a single leader is the honest choice.
What to read next: PostgreSQL replication — single-leader in practice, including read-after-write and lag monitoring; MongoDB replication and sharding — replica sets and failover; building blocks — where replication sits in the bigger picture. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 5.