In the trouble with distributed systems and consistency and consensus we covered the mechanisms: quorums, fencing tokens, linearizability, consensus. This article is about what sits above the mechanisms: how to reason about an application's correctness at all. The key takeaway is non-obvious and reshapes design: reliable lower layers (transactions, TCP, checksums) do not by themselves guarantee that the application is correct. This is a conceptual article "for horizons" — an umbrella over every tactical pattern like idempotency and outbox.
The end-to-end argument: why transactions aren't enough
The classic trap: "we use serializable transactions, therefore the data is correct." No. A serializable transaction guarantees that concurrent transactions don't interfere — but if the application wrote wrong data by mistake (a code bug, a retried request), the transaction will faithfully and durably commit it.
The book's example. A client sends a money transfer, loses the connection before the server's reply, and retries. Each request is a correct transaction on its own, yet twice the money moved:
BEGIN;
UPDATE accounts SET balance = balance + 11 WHERE id = 1234;
UPDATE accounts SET balance = balance - 11 WHERE id = 4321;
COMMIT;
Neither the database nor TCP saves you here. TCP discards duplicates within one connection — but the client reconnected, so to the database it's a brand-new request. The dedup has to be end-to-end — from the client all the way to the store.
This is the end-to-end argument (Saltzer, Reed, Clark, 1984): a correctness function can be implemented completely only with the participation of the system's endpoints. Intermediate layers (the network, the transaction) help performance but do not replace the check at the ends. Hence the solution — an operation identifier the client generates once and threads through every hop:
ALTER TABLE requests ADD UNIQUE (request_id);
BEGIN;
INSERT INTO requests (request_id, ...) VALUES ('0286FD..', ...); -- a retry fails on UNIQUE
UPDATE accounts SET balance = balance + 11 WHERE id = 1234;
UPDATE accounts SET balance = balance - 11 WHERE id = 4321;
COMMIT;
A retried request with the same request_id fails the insert — and the transfer runs exactly once. A relational database enforces that UNIQUE even at weak isolation levels. This is the foundation under all idempotency: "exactly-once" is not broker magic, it's an end-to-end operation key plus dedup at the receiver.
Constraints without global coordination
A strict uniqueness constraint (two people can't grab the same username, the same airplane seat) requires coordination — every competing request must pass through a single point that decides who was first. In a distributed system that's expensive: a single leader, a synchronous quorum, reduced throughput.
But many applications don't need strict uniqueness — a weaker constraint suffices, one you can violate and fix after the fact. Sold more seats than exist? Airline overbooking is a deliberate violation with a compensating transaction: cancel the booking, refund, upgrade. Charged twice? Reverse one payment. The cost of an "apology" is usually low, and the business bakes it into the process. This ties directly to compensation and eventual consistency: don't coordinate everything, resolve conflicts later.
Integrity versus timeliness
The chapter's most useful frame is splitting the vague "consistency" into two distinct requirements:
- Timeliness — the user sees the current state. A timeliness violation is temporary: you read a stale replica, but it converges later. This is linearizability and "read your own writes."
- Integrity — the data is not corrupted: no loss, no contradictions, no "money debited but never received." An integrity violation is permanent: it won't fix itself; it needs an explicit check and repair.
The book's formula: a timeliness violation is "consistency sometimes," an integrity violation is "perpetual inconsistency." And the conclusion: in most applications integrity matters more. A card transaction that hasn't appeared in 24 hours is tolerable (banks reconcile asynchronously). But a balance that doesn't equal the sum of operations, or money that vanished — that's a catastrophe.
The practical meaning: log-based systems (CDC, stream processing, outbox) decouple these two properties. They don't give timeliness by default (consumers are asynchronous), but they hold integrity firmly — via idempotent delivery and dedup by request_id. Often that's the right trade: drop expensive synchronous coordination, keep integrity, accept temporary staleness.
Trust, but verify
The last idea: even correct code runs on imperfect hardware — bits rot on disk, network corruption slips past TCP checksums, databases have bugs in their isolation levels. Mature systems don't trust blindly: HDFS and S3 re-read files in the background and compare against replicas. Hence auditability: systems built on immutable events let you trace the provenance of any derived state and rebuild it from scratch to check, and cryptographic tools (Merkle trees) confirm the data wasn't tampered with. Verify integrity periodically — don't find out about corruption once the backup is already spoiled.
Where this applies
The moment a system has retried requests, multiple stores, or asynchronous consumers, you must reason about correctness separately from "the database is reliable, isn't it." The practical frame: generate an end-to-end operation identifier on the client and dedup at the receiver (that's exactly-once); separate timeliness from integrity and protect integrity first; where strict coordination is expensive, allow a weak constraint with compensation; verify integrity periodically instead of trusting blindly.
Where beginners stumble:
- "We have serializable transactions, so everything is correct" — a transaction won't save you from a retried request or an application bug. You need an end-to-end
request_id. - Relying on TCP/broker dedup — it works on a single hop; when the client reconnects, the duplicate gets through. Dedup must be end-to-end, at the receiver.
- Confusing timeliness and integrity — chasing linearizability where eventual consistency with an integrity guarantee would do, and paying for coordination for nothing.
- Assuming "the database is reliable" means data won't corrupt — without periodic verification, corruption surfaces when it's too late to fix.
What to read next: the trouble with distributed systems — the mechanisms correctness rests on; consistency and consensus — what linearizability is; idempotency — exactly-once tactics in code; compensation — fixing weak constraints after the fact; stream processing — where integrity and timeliness are decoupled. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 12.