"Reliable," "scalable," "maintainable" — these words get slapped on any system, usually with nothing concrete behind them. Yet each of the three qualities has a precise meaning, and almost all of system design is a deliberate trade-off between them: strengthen one, and you usually pay for it somewhere else. Before designing a system step by step, it's worth understanding what these three words actually mean. This article is about the concepts themselves, in plain language.
Reliability: a fault is not yet a failure
The first distinction that instantly clears your head: a fault and a failure are different things.
- A fault is when one component deviates from the norm: a disk dies, a replica hangs, someone pushes a bad config.
- A failure is when the system as a whole stops doing what the user needs.
Building a system with no faults is impossible — parts always break. So that's not the design goal. The goal is to keep a single fault from turning into a failure of the whole system. A disk died, but the replica keeps serving data; a node hung, but the load balancer pulled it out of rotation and the user noticed nothing. Resilience is built not from perfect parts but from ordinary ones whose failures are planned for in advance.
Faults come from three places, and each is treated differently:
- Hardware. Disks, memory, power, network. These faults are random and almost unrelated to each other: in a big cluster of 10,000 disks, on average one dies every day — that's normal, not an emergency. Cured by redundancy: RAID, replicas, spreading across different zones (in the method these are called fault domains).
- Software. A bug in the code: a handler crashes on a certain input, a process slowly eats a shared resource, one small fault cascades and takes down neighboring services. Unlike hardware, these faults are correlated — the same bug fires on all identical nodes at once, so a backup server doesn't save you. Cured by process isolation, tests, and self-checks (the system verifying on the fly that its data hasn't drifted).
- People. Studies of large services show an uncomfortable fact: what most often takes a system down isn't disks but people — operators who changed something wrong in the configuration. Hardware accounts for only 10–25% of outages. This is also cured by design: build interfaces where the right action is the easiest one; provide a sandbox on real data; be able to roll back quickly; roll changes out to a small fraction of users first.
There's a counterintuitive trick: since faults are inevitable anyway — cause them on purpose. For example, randomly kill processes right in production and see whether the system survives (that's how Netflix's Chaos Monkey works). This tests not so much the system as your confidence: the resilience mechanisms actually work rather than just appearing on a diagram.
Scalability: describe the load first
The phrase "system X is scalable" is empty on its own. The meaningful question sounds different: "if the load grows like this — what will we do?". And to answer it, you first have to describe the load in numbers — load parameters: how many reads and writes per second, the ratio of reads to writes, how much data is in the "hot" set, how many concurrent users. Which parameter matters most depends on the specific system.
The classic example is the Twitter timeline. New posts — 4,600 per second, but "show me my timeline" requests — 300,000 per second. There are two ways to build the timeline:
- Assemble on read. A tweet is just one insert into a shared table. The timeline is built by a query: "find everyone I follow and gather their recent tweets." Cheap writes, expensive reads.
- Assemble on write. Each user has a ready-made timeline "mailbox." When someone posts a tweet, it's immediately fanned out into the mailboxes of all their followers. Now reads are cheap (the timeline is pre-assembled), but writes are expensive — one tweet by someone with a million followers becomes a million inserts.
Twitter started with the first way, couldn't handle the read load, and switched to the second: since reads outnumber writes 65 to 1, it pays to spend on writes. But even that isn't the end: for celebrities with millions of followers, fan-out on write is unaffordable, so their tweets are mixed in at read time — a hybrid of the two approaches. The moral: the key load parameter here isn't "tweets per second" at all, but how followers are distributed across users. Without describing the load in numbers, you can neither choose this architecture nor justify it.
Hence the general principle: there's no magic "scaling sauce." A system built for 100,000 small requests per second and a system built for 3 large requests per minute are different architectures, even if they push the same number of bytes. A good architecture always rests on assumptions about which operations will be frequent — that is, on load parameters.
Performance: why average response time lies
How do you measure whether a system is "fast"? For batch processing — by throughput (how many records per second you processed). For online systems — by response time. And here's the first trap: response time isn't one number but a spread. The very same request runs in 20 ms today and 800 ms tomorrow — because of task switching on the server, a lost network packet, or a garbage-collector pause.
Taking the arithmetic mean is a bad idea: it doesn't tell you how many users actually got stuck with a slow response. You need to look at percentiles — they answer "how many requests fit within such-and-such a time":
- The median (p50) — half the requests are faster than this. This is the "typical" request.
- p95, p99, p999 — this is the "tail": the time seen by the unluckiest 5%, 1%, and 0.1% of requests.
The tail matters more than it seems, for two reasons. First, the slowest responses often go to the "heaviest" — and therefore most valuable — users: they have the most data. Amazon states requirements for its services in terms of p999 and calculated that an extra 100 ms of latency drops sales by about 1%. Second, there's tail amplification: if one page is assembled from calls to 7 different services, each with its own p99, then the chance of hitting at least one slow response is already much higher than 1%. The more parallel calls, the more often a single slow service drags down a whole page.
Another source of tail is head-of-line blocking: a couple of slow requests are enough for fast ones to get stuck behind them in the queue. That's why response time is more honestly measured on the client side, not the server — otherwise you can't see how long the request sat in the queue.
Percentiles are also the basis for service-level commitments: an SLO (an internal target) and an SLA (an agreement with the client, sometimes with penalties for violation) are phrased like "median under 200 ms, p99 under 1 second, availability 99.9%." How percentiles are computed technically (histograms, aggregation) is in the metrics article.
Once the load is described in numbers and the metric is chosen, the options for responding to growth are known: vertical scaling (get a more powerful machine) versus horizontal scaling (get many smaller ones). In practice it's a pragmatic mix: stateless services multiply easily, while the database is kept on a single node to the last — until the cost of beefier hardware or availability requirements force it to distribute. More in the building blocks.
Maintainability: code is read longer than it's written
Software eats most of its money not while it's being written, but afterward — in maintenance: fixes, operations, adapting to new requirements. Three things make a system fit for a long life:
- Operability. A good system makes the on-call routine simple: clear monitoring, an obvious "did X → Y will happen" link, automation, independence from any specific machine, sensible defaults with the ability to intervene by hand.
- Simplicity. The main enemy is accidental complexity — the kind born not from the task itself but from the way it's solved. Symptoms: too many states, modules clinging to each other, crutches and special cases everywhere. The main weapon against it is abstraction: SQL hides how data sits on disk and how concurrent access is sorted out; a high-level language hides machine code. A good abstraction tucks complexity behind a clear facade and gets reused many times.
- Evolvability. Requirements will change — the only question is how expensive each change is for the system. Simple, well-abstracted systems are easy to change; tangled ones are painful.
Where this applies
These three words are the skeleton of any conversation about architecture. A design discussion in which load parameters, percentiles, and behavior under faults never came up is a conversation about boxes on a diagram, not about a system. The system-design method starts exactly here: numbers and commitments first, the diagram second.
Where beginners stumble:
- Calling a system "scalable" with no load parameters. Scalability isn't a checkbox property; it's the answer to "what will we do when this particular number grows?"
- Looking at average response time. The average hides the tail, and the tail is your most active users. Start with the median and p99.
- Confusing a fault and a failure — and trying to build a system where "nothing breaks," instead of a system that calmly survives breakage.
- Fighting all complexity at once. The complexity of the task itself won't go away; remove only the accidental kind — the kind the implementation added.
What to read next: the system-design method — how these concepts turn into a sequence of steps; the building blocks — what to answer growing load with; metrics — how to compute percentiles in production.