While a program runs on a single computer, everything is predictable: the same operation gives the same result, and if something breaks, it usually breaks all at once and completely. But the moment processes start talking over a network, that certainty is gone. A partial failure appears: some nodes work, others don't, and still others seem to work but nobody knows for sure. This is the defining feature of distributed systems.
Almost this whole article is about what you can't trust in such a system: the network, the clocks, and even the assumption that your own process wasn't put on pause. From here it's a straight road to consensus algorithms, but first you have to understand exactly what they protect against.
Partial failure: the timeout is the only tool
A node sent a request and got no answer. What happened? Many possibilities: the request itself was lost; the node crashed; the node is alive but responding slowly; the node processed everything and the answer got lost. And here's the main trouble — you can't tell these cases apart. The sender has just one fact: "no answer." The only way to make any decision at all is a timeout: wait some amount of time and, if there's still no answer, declare the node non-working.
But what timeout do you choose? A short one — you notice faults quickly, but you risk declaring dead a node that just slowed down under load. Then its work is handed to others — adding load to an already-overloaded system, which is a direct road to a cascading failure. A long timeout — the opposite: you wait a long time before reacting. There's no "correct" value here, because in an ordinary network delays are unbounded: a packet can get stuck in the queue of an overloaded switch, at a busy CPU, at a hypervisor that suspended a virtual machine. (A network with a guaranteed delay is possible in principle — that's how telephony with a dedicated channel per call works — but the internet and data-center networks are optimized for peak traffic via queues, and they pay for it with predictability.) So timeouts are chosen by experiment, and better still — the spread of delays is measured and the timeout tuned dynamically (as Cassandra and Akka do).
Three things you can't trust
The network. Packets are lost and delayed for unpredictable times; a connection can work one way and be silent the other; network faults happen even in carefully managed data centers more often than you'd think. The conclusion is simple: any exchange over the network can fail, and handling those failures must be deliberately designed and tested (in the spirit of Chaos Monkey — deliberately cut the network right in production and see whether the system survives).
Clocks. Every machine has its own internal clock, and they slowly drift apart from each other (this is called clock drift). Importantly, there are two kinds of clock, and confusing them is dangerous.
- Time-of-day clocks (
System.currentTimeMillis()) are synced over the network (via NTP) and can jump backwards during a correction, stumble over "leap seconds," and depend on the unknown accuracy of a time server. They're no good for measuring intervals. - Monotonic clocks (
System.nanoTime()) are guaranteed to go only forward, but their absolute value is meaningless on its own (it's just "some amount from some point"). Durations are measured with these and only these.
Hence the main trap — ordering events by time-of-day timestamps. That's exactly how "last write wins" (LWW) conflict resolution works: two competing writes compare their times, and the "later" one wins. But if two nodes' clocks have drifted by 100 ms, then the write that's "later" by timestamp may actually have happened earlier — and the write the client really made last silently vanishes. Timestamps don't guarantee cause-and-effect order; for that you need logical clocks and versions. (Google Spanner puts GPS receivers and atomic clocks in every data center precisely to squeeze the error down to a couple of milliseconds.)
Process pauses. Your thread can be stopped at any moment for an unpredictable time. There are plenty of causes: an all-encompassing garbage-collector pause (stop-the-world, sometimes for minutes), suspension of a virtual machine while it migrates to another host, "stolen" CPU time, paging a memory page in from disk, even Ctrl+Z. And the sneakiest part — the node doesn't notice it: to the node, an "instant" passed between two adjacent lines of code, when in fact a minute did, and everyone has long since considered it dead.
A majority decides the truth, not the node itself
Out of a pause grows one of the sneakiest bugs. Picture a node holding a distributed lock or believing itself the leader. It checks: "is the lock still mine?" — yes — and goes to write. But between the check and the write a garbage-collector pause froze it. During that minute the lock expired, another node grabbed it, and the awakened first node — still sure it owns it — writes, corrupting the data. Two owners at once.
The moral: a node can't trust its own opinion about its status. In a distributed system the truth is decided by a quorum — a decision of the majority of nodes (usually more than half). If the majority declared a node dead, it's considered dead, even if it's actually working perfectly. A majority decision is safe because two different majorities can't fail to overlap — which means there won't be two contradictory decisions. That's exactly how nodes elect a leader and prevent split brain (when there are two leaders at once).
But a quorum decides who should write — it doesn't stop a latecomer from ruining everything. The practical defense is a fencing token. On each lock grant the lock service returns an ever-increasing number; the client attaches this number to every write; and the storage rejects a write if its number is lower than one already seen. A node that woke up after a pause comes with an old number — and its write is rejected. The key point: the token must be checked by the resource itself (the storage), not the client — because a client that considers itself "fine" is exactly the source of the problem. A detailed walkthrough with Redis and code is in the lock without fencing case.
Byzantine faults and system models
So far we've treated nodes as "honest": they may be silent, slow, or serve stale data — but if they answer, they don't lie. If a node can lie — send arbitrary or forged messages (from memory corruption, a bug, or malice) — it's called a Byzantine fault, and reaching agreement in such an environment is the Byzantine generals problem. Protecting against it is expensive and needed where there's no trust: aerospace (radiation corrupts memory), blockchains (participants don't trust each other). Inside your own data center there usually are no Byzantine faults, and defending against them doesn't pay off — but validating data coming from external clients (validation, injection protection) is always needed.
To reason about correctness, algorithms are described via a system model — a set of assumptions. By timing: synchronous (delays are bounded — unrealistic), partially synchronous (usually behaves well, sometimes not — the realistic default), asynchronous (no assumptions about time at all). By faults: crash-stop, crash-recovery (a node crashes and comes back up, and reliable storage survives the crash), Byzantine. And two kinds of property an algorithm must provide: safety ("nothing bad will happen"; a violation has a specific moment and is irreversible) and liveness ("eventually something good will happen"; an example is that very eventual consistency). Good algorithms hold safety always, and liveness under reasonable assumptions.
Where this applies
The moment you have more than one service and they call each other over the network — you're already in a distributed system, even if you didn't plan for it. Any network call can be lost, hang, or hit a timeout; any node can go silent in the middle of an operation. The practical frame: don't reach for distribution too early (three conditions for when it's genuinely needed), but if it's already there — design for partial failure: timeouts and retries (idempotent, always!), no ordering by wall clocks, decisions via quorum, fencing on shared resources.
Where beginners stumble:
- Assuming "no answer" means "the node crashed." It's indistinguishable from a lost answer or a slow node. Hence double charges and lost data — with not a single error in the logs.
- Ordering events by
currentTimeMillis(). The nodes' clocks have drifted — and LWW silently loses a write the client thought was saved. - Checking a lock and immediately writing. A garbage-collector pause between the check and the write yields two owners. You need a fencing token that the resource itself checks.
- A node trusting its own "I'm still the leader." But during its pause the quorum already elected another. The majority decides the truth.
- Reaching for microservices "for scale." And getting all the problems of distributed systems where a single node would have done.
What to read next: replication models — quorums w+r>n and write conflicts; the building blocks — where queues, replication, and coordination live in a system; a distributed lock without fencing — the fencing-token mechanism with code; when you need distributed patterns — three conditions and three alternatives.