Before designing a system step by step, it pays to agree on the words. "Reliable," "scalable," "maintainable" — these labels get stuck onto anything, usually without meaning. Each of the three qualities has precise content, and all of system design is a deliberate trade-off between them. This article is about the concepts themselves; the classic reference here is chapter 1 of Martin Kleppmann's "Designing Data-Intensive Applications."
Reliability: a fault is not a failure
The first distinction that immediately brings order: a fault and a failure are different things.
- A fault is one component deviating from spec: a disk dies, a replica hangs, a person ships a wrong config.
- A failure is the system as a whole no longer doing what the user needs.
Building a fault-free system is impossible. The design goal is different: keep a fault from turning into a failure. A disk dies — the replica keeps serving data; a node hangs — the load balancer takes it out of rotation. Resilience is built not from flawless parts but from ordinary ones whose failures are anticipated.
Faults come from three sources, and each is treated differently:
- Hardware. Disks, memory, power, network. These faults are random and nearly independent: in a cluster of 10,000 disks, one dies on an average day. The cure is redundancy: RAID, replicas, spreading across zones — what the method calls failure domains.
- Software. A systematic bug: a handler crashes on specific input, a process eats a shared resource, a cascade — a small fault in one service takes down its neighbors. Unlike hardware, these faults are correlated: the same bug fires on every node at once. The cure is process isolation, testing, and runtime self-checks of invariants.
- Humans. Studies of large services show the leading cause of outages is operator error during configuration changes; hardware accounts for only 10–25% of cases. The cure is design: interfaces where the right action is the easiest one; sandboxes with real data; fast rollback; gradual rollout to a small share of users.
A counterintuitive technique: since faults are inevitable — cause them on purpose. Randomly kill processes in production and see whether the system survives (that is how Netflix's Chaos Monkey works). This trains not so much the system as the confidence that the resilience machinery actually works instead of existing on paper.
Scalability: describe the load first
"System X is scalable" is a sentence without content. The meaningful question is: "if load grows in this particular way — what are our options?" And for that, load must be described with numbers — load parameters: reads and writes per second, the read-to-write ratio, the working set size, the number of concurrent users. Which parameter matters most depends on the system.
The textbook example is the Twitter timeline. Posts: 4,600 per second; timeline reads: 300,000 per second. Two ways to build a timeline:
- Assemble on read: a tweet is one insert into a global table; the timeline is a query "find everyone I follow and their tweets." Cheap writes, expensive reads.
- Assemble on write: every user has a timeline "mailbox"; posting a tweet fans it out into the mailboxes of all followers. Cheap reads, expensive writes — a tweet by someone with a million followers becomes a million inserts.
Twitter started with the first approach, could not sustain the read load, and switched to the second: with 65× more reads than writes, it is cheaper to pay at write time. And even that is not the end: for celebrities with millions of followers the fan-out is unaffordable, so their tweets are merged in at read time — a hybrid of both approaches. The moral: the key load parameter here is not "tweets per second" but the distribution of followers per user. Without describing the load, this architecture can be neither chosen nor defended.
Hence the main principle: there is no magic scaling sauce. An architecture built for 100,000 small requests per second and one built for 3 large requests per minute are different systems even at identical byte throughput. A good architecture is built on assumptions about which operations will be frequent — that is, on load parameters.
Performance: why the average lies
How do you measure "fast"? For batch processing — throughput (records per second). For online systems — response time. And here is the first trap: response time is not a number but a distribution. The same request takes 20 ms today and 800 ms tomorrow — a context switch, a lost packet, a garbage collection pause.
The arithmetic mean is a poor metric: it does not tell you how many users actually hit a slow response. What you need are percentiles:
- The median (p50) — half the requests are faster than this. The "typical" request.
- p95, p99, p999 — the tail: what 5%, 1%, 0.1% of requests experience.
The tail matters more than it seems. First, the slowest requests often belong to the "heaviest" — that is, the most valuable — users: they have the most data. Amazon states requirements for internal services in terms of p999 — and estimates that an extra 100 ms of latency cuts sales by 1%. Second, there is tail latency amplification: if a page is assembled from calls to 7 backends, each with its own p99, the chance of hitting at least one slow response is far more than 1%. The more parallel calls, the more user requests are stalled by a single slow service.
Another source of tail latency is head-of-line blocking: a few slow requests are enough for fast ones to get stuck behind them in the server's queue. That is why response time is honestly measured on the client side, not the server side.
Percentiles are also what agreements are built on: an SLO (service level objective) and an SLA (agreement with penalties) are phrased like "median under 200 ms, p99 under 1 s, availability 99.9%." How percentiles are computed technically — histograms and aggregation — is covered in the metrics article.
Once the load is described and the metric chosen, the options are well known: vertical scaling (a bigger machine) versus horizontal scaling (many smaller ones). Practice is a pragmatic mix: stateless services multiply easily, while the database stays on one node until the cost of vertical growth or availability requirements force it to become distributed — more in building blocks.
Maintainability: code is read longer than it is written
The main cost of software is not development but maintenance: fixes, operations, adaptation to new scenarios. Three principles make a system livable:
- Operability. A good system makes the operators' routine simple: monitoring and a clear model of "do X — Y happens," automation, independence from individual machines, sane defaults with room for manual override.
- Simplicity. The enemy is accidental complexity: complexity produced by the implementation rather than the problem itself. Symptoms: a state-space explosion, tight coupling, workarounds and special cases. The main weapon against it is abstraction: SQL hides on-disk structures and concurrent access, a high-level language hides machine code. A good abstraction moves complexity behind a clean facade and gets reused.
- Evolvability. Requirements will change — the only question is how expensive each change is for the system. Simple, well-abstracted systems are easier to change than complex ones.
Where this applies
These three words are the skeleton of any architecture conversation. A design discussion with no load parameters, no percentiles, and no failure behavior is a conversation about boxes, not about a system. The system design method starts exactly here: numbers and commitments before the diagram.
Where beginners stumble:
- Calling a system "scalable" without load parameters. Scalability is not a property but an answer to "what do we do when this grows?"
- Watching the average response time. The average hides the tail, and the tail is your most active users. Start with the median and p99.
- Confusing fault and failure — and trying to build a system where "nothing breaks" instead of one that survives breakage.
- Fighting all complexity at once. The problem's own complexity is not going anywhere; what you remove is the accidental kind — the part the implementation added.
What to read next: the system design method — how these concepts turn into a sequence of steps; building blocks — what to answer growing load with; metrics — how to compute percentiles in production. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 1.