← Back to the section

Replication is when a database keeps not one copy but several. The usual setup: one node (the leader, aka master) accepts all writes, and the others (the replicas, aka followers) hold a copy of the same data and serve reads. Why: you can spread reads across replicas to handle more load, and if the leader dies, one of the replicas takes its place.

How this is configured in practice we cover in the articles on replication in PostgreSQL and replica sets in MongoDB. This article is one level up and not tied to a specific database: what actually breaks when there's more than one copy, and what replication models exist. There are three, and they differ by the answer to one question: who accepts writes, and who resolves conflicts. We'll start with the simplest and most common model — one leader, many replicas — because that's where almost all the pitfalls live.

Why a replica lags

When you write something to the leader, the change doesn't appear on the replicas instantly — it has to "travel" to them over the network. Usually that's a fraction of a second and nobody notices. But under load, during a traffic spike, or with network trouble, a replica can fall behind by seconds, sometimes minutes.

This mode — "replicas lag but will catch up if writes stop" — is called eventual consistency. The word "eventual" sounds like a promise, but it doesn't actually commit to anything: nobody says when the replicas will catch up. Usually fast, but there's no guarantee on the timing.

Lag by itself is no disaster. The disaster is that a user can read from a lagging replica and see something strange. There are exactly three strange things, and each has its own "antidote" — a guarantee you can turn on.

The three lag anomalies — and how to fight them

1. "I don't see my own write." You left a comment (the write went to the leader), refreshed the page (the read hit a lagging replica) — and the comment is gone. It looks like the data was lost, though it's actually there, just hasn't reached this replica yet.

The antidote is the read-your-writes guarantee: a user's own data must be read either from the leader or from a replica that has already caught up to the moment of their write. Then the "vanished comment" won't happen. How this is done in PostgreSQL is in the replication article.

2. "Time runs backwards." You refresh the page twice in a row. The first request hit a replica that had almost caught up — the comment is visible. The second hit another replica that lagged more — and the comment disappeared. The data seems to roll back into the past.

The antidote is monotonic reads: each next read must not be "older" than the previous one. The simplest way is to consistently send the same user to the same replica (say, picking the replica by a hash of their id), so they don't bounce between a "fresher" and a "more lagging" one.

3. "The answer before the question." In a conversation you see the reply to a message, but the message itself isn't there yet: the replica with the reply caught up to the leader faster than the replica with the question. Cause and effect swapped places.

The antidote is consistent prefix reads: if writes happened in some order, they must be read in that same order. This is especially painful in systems where data is sliced into parts (shards): the parts have no shared order of writes, and stitching them back into the right sequence is harder.

The working rule: when you decide to read from replicas, ask yourself — which of these three oddities will hit your scenario? — and turn on exactly the guarantee you need. The most baffling bugs are born when the system is asynchronous (replicas lag) but the code is written as if it were instant.

When the leader fails: failover and its traps

Since all writes go through one leader, the question arises: what if it dies? Then one of the replicas is promoted to be the new leader — this is called failover. Sounds simple, but there are two classic traps here.

Lost writes. If replication was asynchronous, the old leader may have had writes it hadn't yet sent to the replicas. When a lagging replica is promoted, those writes are usually thrown away — they simply vanish. A real case from GitHub's practice: a promoted lagging replica started handing out already-used identifiers for records, and data was shown to the wrong users.

Split brain. Sometimes the old leader didn't die but just briefly dropped off the network — and now two nodes think they're the leader and both accept writes. The data diverges and later has to be painfully reconciled. That's why systems invent ways to guarantee there's exactly one leader at any moment.

These traps aren't a reason to fear replication — they're a reason to understand: failover isn't free, and its reliability needs to be tested ahead of time, not during the outage.

Multi-leader: freedom at the price of conflicts

Sometimes a single leader gets in the way not because of load but because of geography. Then you make several leaders. Three typical scenarios:

  • Multiple data centers. Each data center has its own leader: a write is handled locally and then replicated to the others at leisure. The user doesn't have to wait for a request to run to another continent and back, and the failure of a whole data center doesn't stop accepting writes.
  • Offline clients. A calendar on your phone and your laptop accepts writes even with no network. Essentially each device is a little leader with its own local database, and syncing is the same replication, just with a delay of hours.
  • Collaborative editing. Google Docs, where several people edit a document at once, is the same idea taken to the limit: a local copy in every tab.

You pay for this freedom, and the price is steep — write conflicts. Picture it: two leaders simultaneously accepted a change to the same record, both told the user "saved," and when they went to exchange changes, it turned out the edits were incompatible. Asking the user "what did you mean?" is too late. So the database must resolve the conflict itself — and in a way that brings all copies to a single value in the end. There are three ways:

  • Last write wins (LWW). Each write has a timestamp, and the "later" one wins. Simple and widespread (in Cassandra it's the only option). But this way has a nasty price — silent data loss: the losing write, which the client already got a "saved" for, vanishes silently. Fine for a cache; not for anything you'd hate to lose.
  • Merge. Keep both versions and merge them later: either in application code on the next read, or automatically — with special data structures called CRDTs (designed so that simultaneous changes merge without loss: counters, sets, text).
  • Prevent conflicts. The most practical option — make sure all changes to one record always go through the same leader (a user's data always goes to their "home" data center). Then for that record the system effectively becomes "single leader," and there are no conflicts.

It helps to understand what even counts as a conflict. The key notion is happens-before: operation B depends on A if, at the moment of B, A was already known. But if neither operation knew about the other — they are concurrent, and that's exactly when a resolution mechanism is needed. Determining this order by ordinary clocks is unreliable — clocks on different servers drift slightly — so databases track dependencies not with time but with special version counters.

Leaderless: quorums instead of a leader

The third model removes the leader entirely (this is how "Dynamo-style" databases work — Cassandra, Riak). The idea: the client sends a write to several replicas in parallel at once and considers it successful when confirmations arrive from some number of nodes. It reads from several at once too, and compares who has the fresher version.

For this to work, you agree on numbers. Say there are n replicas in total, a write counts as successful after w confirmations, and a read polls r nodes. The magic condition is the quorum: if w + r > n, then the set of nodes we wrote to and the set we read from are guaranteed to overlap in at least one node — which means on a read we'll definitely hit a node with fresh data. A typical set: n=3, w=2, r=2. One replica can be down while writes and reads keep working — and a separate failover procedure isn't needed here at all.

Lagging nodes catch up two ways: read repair (the client, noticing a stale answer at one node, immediately writes the fresh value back there) and a background anti-entropy process that compares replicas against each other and pushes the difference.

Just don't treat a quorum as an iron guarantee — it has caveats:

  • Sloppy quorum: during network trouble a write may be temporarily accepted by the "wrong" nodes (with later delivery to the right ones — hinted handoff). At that moment w and r may fail to overlap.
  • Concurrent writes still need conflict resolution — and often it's the same LWW with the same silent loss.
  • Tracking "how far behind" is harder here than in the single-leader model, where there's a clear position in the log.

The honest bottom line: quorum databases are eventually consistent by nature, and the numbers w and r control not a guarantee but the probability of reading something stale.

Where this applies

Choosing a replication model is choosing where exactly it will hurt:

  • Single-leader — simple and gives a clear order of writes, but requires careful failover and hits a single node's limit on write load.
  • Multi-leader — unties geography and offline, but brings write conflicts.
  • Leaderless — removes failover, but in exchange gives quorum arithmetic and only probabilistic guarantees.

In an ordinary backend the right default is single-leader (PostgreSQL) with deliberately chosen read guarantees. The other models are turned on only when geography or write availability genuinely demand it.

Where beginners stumble:

  • Reading everything from replicas — and catching "vanished comments" and "time backwards." Read guarantees are chosen deliberately, for a specific scenario, not "however it goes."
  • Trusting LWW. "Last write wins" sounds harmless, but in practice it means "losing writes disappear silently, even though the client got a confirmation."
  • Treating w+r>n as an absolute guarantee — sloppy quorums, concurrent writes, and recovery from a stale replica leave gaps.
  • Turning on multi-leader for "reliability" inside a single data center — and getting write conflicts with no benefit: in one data center it's more honest to keep a single leader.

What to read next: PostgreSQL replication — single-leader in practice, including read-your-writes and lag monitoring; MongoDB replication and sharding — replica sets and failover; the building blocks of system design — how replication fits into the bigger picture.