How a Notification System Works: Architecture From Scratch

Java · Go · Node · Python

We break down the architecture of a notification system for 10 million users: queues, per-channel isolation, deduplication, an unread counter in Redis, and graceful degradation when a provider fails.

Published 6 июня 2026 updated 10 июля 2026 ~7 min read Author: Вадим Викулин

← Back to the section

Picture this: a user places an order and needs a confirmation — a push to their phone, an email to their inbox, and a message in the app's feed. Who does this? How does it not get lost? How does it avoid being sent twice if something crashes? Let's work through all of this on a concrete system — a marketplace with 10 million users.

Why you even need a separate notification system

You could just send the email straight from the order code — sendEmail(user, "Order accepted"). That's how it starts. Then push shows up, then a "don't duplicate" requirement, then campaigns to a million recipients, then a user wants to unsubscribe from certain categories, and then it turns out the email provider is down and orders are backing up in a queue.

That's why notifications get pulled out into a separate platform with a single API. Product teams say "send an event," and the platform decides on its own: which channel, based on which user preferences, and what to do when a provider fails.

What the platform can do

Let's pin down the task before we draw any diagrams:

Product teams send notifications through a single API: push, email, in-app (the bell).
The user manages subscriptions — they can turn off specific categories on specific channels.
The user sees a notification feed and an unread counter.
The product sees the delivery status.

What we don't do in the first version: a marketing campaign builder, SMS, or ordering guarantees between channels.

How much of everything, and why it matters

Before drawing the diagram, let's estimate the numbers:

10M users, 3M active per day
100M notifications/day → roughly 1200/s on average, peak ×5 during campaigns → 6000/s
90 days of history: ~1 KB × 100M × 90 ≈ 9 TB
Feed: 3M × 8 opens ≈ 280 requests/s, peak ~1000
Unread counter: thousands of lightweight requests/s (on every app screen)
Push provider: ~3000 requests/s at peak — batched sending and rate limiting needed

These numbers immediately reveal three things:

Peaks and the asynchronous nature of sending mean there must be a queue between accepting an event and actually delivering it.
9 TB of history is not for an ordinary table. You need partitioning and moving old data to cold storage.
The unread counter is the hottest read. The database shouldn't be hit for it — you need a cache.

API: we accept an event, not a "letter"

The key decision: the API accepts an event, not a ready-made message. The producer says "this happened for user X," and the platform itself picks the channels based on the user's preferences and builds the text from a template.

POST /v1/notifications
  body: { eventId, userId, category, payload }
  response: 202 { notificationId }

GET  /v1/users/{id}/feed?cursor   — feed with pagination
GET  /v1/users/{id}/unread-count  — unread counter
PUT  /v1/users/{id}/preferences   — subscriptions by category and channel
GET  /v1/notifications/{id}/status — delivery status by channel

The 202 response (accepted, but not processed) is a direct consequence of the numbers: processing is asynchronous, and synchronously we only store the event.

eventId is an identifier from the producer. If it sends the same event twice (for example, on a retry after a network error), we return the same notificationId and don't send a second notification. This is called idempotency — "the same request several times gives the same result."

Where everything is stored

What	Where	Why
Event and delivery status	PostgreSQL, partitioned by month	Streaming writes, pinpoint reads by id
User feed	PostgreSQL, key (user_id, created_at)	Paginated reads of the last N records
Unread counter	Redis	Thousands of reads per second, increment and reset
User preferences	PostgreSQL	Rare reads by user_id
Delivery analytics	ClickHouse	Aggregates by campaign, channel, date

PostgreSQL is the source of truth. The Redis counter can be restored by recomputing it. ClickHouse is filled from delivery events.

The flow: how an event becomes a notification

Producer
  → API service (validation, idempotency by eventId, save to DB + outbox)
  → Kafka (notifications topic)
  → Resolver (reads preferences, picks channels, fills the template)
  → Kafka (separate topics: push / email / feed)
  → Workers:
      push-worker  → FCM / APNs (batched, rate-limited, with retries)
      email-worker → email provider
      feed-worker  → feed in PostgreSQL + counter in Redis + WebSocket notification
  → Delivery statuses → PostgreSQL + ClickHouse

Why separate topics for each channel? If the email provider is down — the email queue piles up, but push notifications go out without delay. The channels are isolated from each other.

The outbox at intake is a table in the same DB where, atomically together with the event, a "move it to Kafka" task is written. A separate process reads it and publishes to Kafka. If the app crashes between saving and publishing — the task stays in the outbox and gets published on recovery. This is the single place where "don't lose it" is guaranteed transactionally.

Deduplication: don't send twice

Kafka guarantees "at-least-once delivery" — a worker may receive the same message twice. So you need two lines of defense:

At intake — a unique index on eventId in the database. A second request with the same eventId returns the already existing notificationId.
In the workers — each worker stores the notificationId + channel pairs it has already processed. Before sending, it checks: has this happened already?

This pair of defenses is the standard approach for any pipeline where delivery is "at least once."

Campaigns to a million recipients

A transactional notification targets one user. A campaign — potentially the entire marketplace.

You can't expand a million rows synchronously at intake — that would take minutes and block everything. So a campaign is a single event that lands in a separate, lower-priority topic. A dedicated fan-out worker reads it and creates individual notifications in batches.

The main rule: campaigns must not delay transactional notifications. A separate topic with a separate pool of consumers — the order confirmation code doesn't wait for the promo emails to be sent out.

The unread counter

The app shows the unread count on every screen — that's thousands of requests per second. Hitting PostgreSQL on every screen open is too expensive.

The scheme is simple:

when the feed-worker adds a record to the feed, it does INCR user:{id}:unread in Redis;
when the user reads it — SET user:{id}:unread 0;
every few minutes a background job reconciles Redis with PostgreSQL (Redis can lose data on restart — the counter must be able to be recomputed).

Reads always come from Redis. PostgreSQL is not involved in this scheme.

What happens on failures

What broke	What happens
Push provider unavailable	The worker retries with pauses, and on a prolonged failure stops trying (circuit breaker); transactional notifications are duplicated into the feed
Kafka unavailable	The API accepts events, the outbox accumulates; once Kafka is back — events are topped up
Redis lost	The counter isn't shown (or is shown without a number), the feed works — PostgreSQL is alive
Analytics lagging	Campaign statistics are delayed — this is declared in the contract; operational statuses from PostgreSQL are fine
A 10× spike (an incident at a producer)	Per-producer rate limiting at the API; the queue smooths the spike; campaigns are throttled first

In short

Notifications get pulled out into a separate platform so product teams don't have to think about channels, preferences, and provider failures.
The API accepts an event, not a "letter" — the platform picks the channel and text based on user preferences.
There is always a queue (Kafka) between intake and delivery — for peaks, asynchrony, and channel isolation.
The outbox at intake is the single place where "don't lose it" is guaranteed transactionally.
Two lines of deduplication: a unique index on eventId at the entrance + a check in every worker.
Campaigns of a million go into a separate topic with lower priority, so they don't delay transactional notifications.
The unread counter is Redis only: thousands of requests per second, recomputed from PostgreSQL on recovery.
On a provider failure: retries with pauses → circuit breaker → degradation (no number on the badge), but nothing is lost.