On a single computer a program behaves predictably: the same operation yields the same result, and a failure is usually total — everything goes down at once. The moment processes talk over a network, that certainty is gone. Partial failure appears: some nodes work, others don't, and a third group works but nobody knows for sure. This is the defining trait of distributed systems, and almost all of this article is about what you cannot trust in them: the network, the clock, and even the assumption that your own process was not paused. From here it is a straight road to consensus algorithms, but first you need to understand what they protect against.
Partial failure: the timeout is your only tool
A node sent a request and got no answer — what happened? The request was lost; the node crashed; the node is alive but slow; the node processed the request and the response was lost. These cases are indistinguishable — the sender only has the fact "no answer." The only way to decide anything at all is a timeout: wait some time and declare the node dead.
But which timeout? A short one detects failures fast but risks declaring a node dead when it merely slowed under load; then its work is handed to others, adding load to an already-strained system — the road to a cascading failure. A long one means waiting a long time before reacting. There is no "correct" value, because in an asynchronous network delays are unbounded: a packet can get stuck in the queue of a congested switch, at a busy CPU, at a hypervisor that suspended the VM. A phone network with a guaranteed delay is possible (a dedicated circuit per call), but the internet and datacenter networks are optimized for bursty traffic through queues — at the cost of predictability. So timeouts are tuned experimentally, or better, the delay spread is measured and the timeout adapts dynamically (Phi Accrual in Cassandra and Akka).
Three things you cannot trust
The network. Packets are lost and delayed arbitrarily, a connection can work one way and fail the other, and network faults happen even in managed datacenters more often than you'd think. The takeaway is simple: any network exchange is subject to failure, and handling those failures must be designed and tested (the Chaos Monkey idea — deliberately break the network in production).
Clocks. Every machine has its own quartz clock, and they drift apart (clock drift). There are two kinds of clocks, and confusing them is dangerous:
- Time-of-day clocks (
System.currentTimeMillis()) synchronize via NTP and can jump backward on correction, stumble over leap seconds, and depend on an unknown server error. Useless for measuring intervals. - Monotonic clocks (
System.nanoTime()) are guaranteed to move forward, but their absolute value is meaningless. Durations are measured only with these.
Hence the main trap — ordering events by time-of-day timestamps. That is exactly how last-write-wins (LWW) conflict resolution works: two concurrent writes are compared by timestamp, the later one wins. But if the nodes' clocks drifted by 100 ms, the "later" write may have physically happened earlier, and the one the client actually wrote last silently vanishes. Timestamps do not guarantee causal order — that needs logical clocks and versions (Google Spanner uses GPS and atomic clocks in every datacenter precisely to narrow the error to a few ms).
Process pauses. A thread can be preempted at any moment for an arbitrary time: a stop-the-world garbage collection pause (sometimes minutes), a VM suspend (suspend/live-migration), stolen CPU time, a page swap, even Ctrl+Z. The node does not notice — to it, an "instant" passed between two lines of code, while in reality a minute did, and it was long ago declared dead.
Truth is defined by the majority, not by a node
Out of that pause grows the nastiest bug. Picture a node holding a distributed lock or believing it is the leader. It checks: "is the lock still mine?" — yes, and it goes to write. But between the check and the write a GC pause froze it; during that minute the lock expired, another node grabbed it, and the revived first node — still certain it owns the lock — writes, corrupting the data. Two owners at once.
The moral: a node cannot rely on its own belief about its status. In a distributed system, truth is defined by a quorum — the decision of a majority (usually > half) of nodes. If a quorum declared a node dead, it is dead, even if it works perfectly. Majority quorums are safe because two majorities cannot both exist — no conflicting decisions. This is exactly how nodes decide who is the leader and prevent split brain (two leaders at once).
But a quorum decides who should write; it does not stop a latecomer from wrecking things. The practical defense is a fencing token: on each grant the lock service returns a monotonically increasing number, the client attaches it to every write, and the storage rejects any write with a token lower than one it has already seen. A node revived after a pause arrives with an old number — and its write is refused. Crucially, the token must be checked by the resource itself, not the client (a client that thinks it is "fine" cannot be trusted).
Byzantine faults and system models
So far we have assumed nodes are "honest": they may go silent, lag, or return stale data — but if they answer, they answer truthfully. If a node can lie (send arbitrary or forged messages — from memory corruption, a bug, or malice), that is a Byzantine fault, and agreement in such an environment is the Byzantine generals problem. Defending against it is expensive and needed where there is no trust: aerospace (radiation corrupts memory), blockchains (participants don't trust each other). Inside your own datacenter there are usually no Byzantine faults and the defense doesn't pay off — but validating input from external clients (validation, SQL-injection defense) is always required.
To reason about correctness, algorithms are framed through a system model — a set of assumptions. On timing: synchronous (bounded delays — unrealistic), partially synchronous (usually well-behaved, sometimes not — the realistic default), asynchronous (no timing assumptions). On failures: crash-stop, crash-recovery (a node crashes and comes back, stable storage survives), Byzantine. And two kinds of properties: safety ("nothing bad happens" — a violation has a concrete moment and is irreversible) and liveness ("something good eventually happens" — e.g. eventual consistency). Good algorithms hold safety always, and liveness under reasonable assumptions.
Where this applies
The moment there is more than one service and they call each other over a network — you are already in a distributed system, whether you planned it or not. Any network call may be lost, hang, or hit a timeout; any node may go silent mid-operation. The practical frame: don't reach for distribution prematurely (the three conditions where it's needed), but once you have it, design for partial failure: timeouts and retries (idempotent!), no ordering by wall clocks, decisions through a quorum, fencing on shared resources.
Where beginners stumble:
- Assuming "no answer" means "the node crashed" — it is indistinguishable from a lost response or a slow node. Hence double charges and lost data with not a single error in the logs.
- Ordering events by
currentTimeMillis()— the nodes' clocks drifted, and LWW silently loses a write the client thought was saved. - Checking a lock and immediately writing — a GC pause between the check and the write yields two owners. You need a fencing token checked by the resource.
- A node trusting its own "I'm still the leader" — during its pause the quorum already elected another. The majority defines the truth.
- Reaching for microservices "for scale" — and getting all the trouble of distributed systems where a single node would have sufficed.
What to read next: replication models — w+r>n quorums and write conflicts; building blocks — where queues, replication, and coordination live in a system; when you need distributed patterns — three conditions and three alternatives. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 8.