When a system grows and reaches millions of users, a single server stops coping. Engineers long ago came up with a set of ready-made solutions for common problems — load balancers, caches, queues, and other components. These are called building blocks: there are fewer than twenty of them, and almost any large system is assembled from them.
Knowing the names of the blocks is not enough. Each block gives you something and costs you something. Understanding that cost is the essence of system design.
Stateless services and load balancing
Picture a checkout counter at a store. One counter cannot handle the line — so they open another one. It is the same with services: you run several copies and put a load balancer in front of them that distributes requests between them.
This only works if the service is stateless: each request is processed independently, without any memory of previous ones. The user's session lives in a token, not in the server's memory. Files live in separate storage, not on the process's disk.
What it gives you: you can add copies to handle load; the failure of one copy goes unnoticed — the others keep working.
What it costs: you have to explicitly move any state out — into a database, a cache, or file storage.
Caching
Picture a reference book sitting on your desk. Instead of walking to the library for the same book every time, you keep the pages you need close at hand.
A cache is fast temporary storage (usually in memory, for example Redis). Frequently requested data is placed there, and subsequent requests are answered in milliseconds instead of tens of milliseconds.
What it gives you: read speed and relief for the database.
What it costs: data in the cache can become stale. When the original changes, the cache needs to be cleared or refreshed — this is called invalidation, and doing it correctly is harder than it looks. Another trap: if the cache is empty and thousands of requests arrive at once, they all hit the database at the same time (cache stampede).
A cache appears not "just in case," but when the database genuinely cannot keep up with the read load.
Database replication
What if the main database goes down? Or gets too many requests?
Replication is a mechanism where all changes from the main server (the master) are automatically copied to one or more additional servers (replicas). If the master becomes unavailable, one of the replicas becomes the new master.
What it gives you: fault tolerance; you can read from the replicas without loading the master.
What it costs: replication is not instant — a replica always lags a little behind. If a user just saved something and immediately reads it back, they may get old data. This is solved by routing "read-your-own-writes" to the master.
An important point: replication does not speed up writes — all changes still go through a single master.
Sharding
Replication does not help when there is so much data that it no longer fits on a single server. In that case you apply sharding: the data is split into parts (shards) and each part is stored on a separate server.
For example, users with IDs 1–1,000,000 on one server, and IDs 1,000,001–2,000,000 on another.
What it gives you: data and writes scale horizontally — you can store terabytes and petabytes.
What it costs: choosing the shard key is a hard problem. A wrong choice leads to one shard being overloaded while others sit empty. Queries that touch several shards are slower and more complex. Redistributing data when the sharding scheme changes is a painful operation.
The rule: first vertical growth (a more powerful server), then partitioning within a single database, and only then sharding — when the numbers truly demand it.
Queues and events
Imagine you have an online store. When a customer places an order, you need to: save the order, charge the money, send an email, notify the warehouse. If you do all of this synchronously, the user waits several seconds.
A queue lets you spread this out over time: you save the order, tell the user "accepted," and the rest — the email, the warehouse notification — runs in the background.
Popular tools: Kafka, RabbitMQ.
What it gives you: a fast response to the user; smoothing out load spikes (the queue accumulates tasks during a peak and releases them evenly); a single source of events can be processed by several independent consumers.
What it costs: the result is not available immediately — this is called eventual consistency. A consumer must be able to process the same event twice (which happens during failures) without corrupting data — this is called idempotency. Plus the broker is one more piece of infrastructure that needs to be watched.
Full-text search
An ordinary database can search for exact values: WHERE name = 'John'. But it handles search by meaning poorly: "find everything about cats," accounting for typos, with relevance ranking of results.
For this, specialized search engines are used, most often Elasticsearch. It builds an inverted index: for each word it stores a list of documents where the word appears — and answers very quickly.
What it gives you: full-text search with relevance, facets (filters), suggestions.
What it costs: this is a second store that must be kept in sync with the main database. Data appears in search with a small delay (indexing lag). A substring search over a thousand-row table is no reason to bring in Elasticsearch.
Analytics
A transactional database (PostgreSQL, MySQL) is optimized for fast reads and writes of individual rows. But analytical queries work differently: "how many orders were there last month across all regions" is an aggregation over billions of rows.
For such tasks there are columnar databases, for example ClickHouse. Data in them is stored by columns, which makes it possible to compute aggregates over large volumes very quickly.
What it gives you: answers to analytical queries in seconds instead of hours.
What it costs: a separate pipeline to deliver the data (for example, through Kafka), and data in analytics is updated with a delay.
File storage and CDN
Storing files (images, video, documents) in a database is a bad idea: the database becomes huge, and working with files is inconvenient. For files there is object storage — modeled on Amazon S3. It is designed to store billions of objects of any size.
A CDN (content delivery network) is a distributed network of servers located geographically close to users. When a user in Vladivostok requests an image, it is served from the nearest CDN node rather than traveling from a server in Moscow.
What storage gives you: practically unlimited capacity, convenient work with files through links.
What a CDN gives you: shorter load times for users around the world; relief for the main server.
What it costs: when a file is updated, the old version cached in the CDN may remain for some time (invalidation). Private files require signed links with a limited lifetime.
Rate limiting and overload protection
What if a single user or script sends thousands of requests per second? Without protection, the service goes down.
Rate limiting is a restriction on the number of requests: no more than 100 per minute from a single user. When exceeded, an error 429 (Too Many Requests) is returned.
Inside the system there is a paired mechanism — backpressure: if the task queue is full, new tasks are rejected immediately, instead of piling up until a crash.
What it gives you: protection from abuse and peak load; predictable behavior under pressure.
What it costs: legitimate users who exceed the limit will get a rejection. You need to think about what to limit by: by IP, by user, or by API key.
Consistency in distributed systems
When data is stored on several servers, a fundamental problem arises: what happens if two servers receive different data at the same time and have not managed to synchronize?
There is a well-known theorem, CAP: in a distributed system, during a network failure you cannot simultaneously guarantee availability (the service responds) and consistency (everyone sees the same data). You have to choose.
In practice this is solved in different ways depending on the requirements:
- A single master for writes (PostgreSQL) — a simple and reliable solution.
- Optimistic locking — before an update, we check that the data has not changed since it was read; if it has, we retry.
- Distributed locks (through Redis or ZooKeeper) — expensive and fragile; there is almost always a simpler way.
- Consensus algorithms (for example, Raft) — this is what is used inside Kafka and other distributed systems. They are taken off the shelf, not implemented by hand.
In short
- There are fewer than twenty blocks, and almost any large system is assembled from them.
- Each block solves a specific problem and carries a specific cost — know both.
- Stateless + load balancing is the foundation of horizontal scaling; any state in the process breaks scaling.
- Cache appears when you have measured that the database cannot keep up with reads; without measurements it is needless complexity.
- Replication gives fault tolerance and read scaling, but not write scaling.
- Sharding is a last resort, when the data does not fit on a single node.
- Queues decouple components over time; consumers must be idempotent.
- Search (Elasticsearch) is needed when relevance matters, not just an exact match.
- Analytics (ClickHouse) is a separate database for aggregates over big data.
- CDN reduces latency for geographically distributed users.
- Rate limiting protects against spikes and abuse.
- During a network failure in a distributed system, you choose: availability or consistency.
What to read next
- System design method — how to apply the blocks step by step.
- End-to-end example: a notification system — the blocks in action on a single task.
- Distributed patterns — how to connect the blocks: saga, outbox, idempotency.