Every time a shopper places an order on the marketplace, they get an "Order confirmed" email and a push in the app. Sounds simple — take an event, send a letter. In reality there's a separate service behind it, with retries, deduplication, an attempt log, and webhook handling. Let's look at how it works.
Why a separate service
The most obvious option is to send the email straight from the order service. The problem: if Mailgun is down, does the order transaction roll back? Or does the email just get lost? The payment went through, the order was created, but the email vanished — and the user has no idea what's going on.
The second option is to send it within the same transaction through a queue. But then the order service has to know about email and push channels, providers, templates. That's an extra responsibility sitting in the wrong place.
The solution is a separate Notification Service subscribed to order events in Kafka, which does nothing but delivery: it takes an event, picks the channels, renders the template, sends it through a provider, and records the result.
The service's job is not to make business decisions (when to send is decided by the order service, which publishes the event) but to reliably deliver the notification.
Where notifications come from
Notification Service listens to the marketplace.orders.v1 topic in Kafka. That's where Order Service publishes order lifecycle events: OrderConfirmed, OrderPaid, OrderShipped, OrderCancelled, OrderRefunded, OrderDelivered, DisputeOpened, DisputeResolved.
Each event carries an event_id, the event type, the buyer's userId, and the order details. Using userId, the service calls Customer BFF — it requests the user's email and device push tokens, along with their language (locale).
Idempotency. Kafka guarantees at-least-once delivery — the same event can arrive twice. To avoid creating duplicates, every event_id is written to the processed_events table. On reprocessing, INSERT … ON CONFLICT DO NOTHING quietly ignores the duplicate.
Which channels, and when
Each event type maps to specific channels (hardcoded):
OrderConfirmed,OrderPaid,OrderShipped,OrderCancelled,OrderRefunded,DisputeResolved→ email + push to the buyerOrderDelivered→ email onlyDisputeOpened→ push to the seller + email to the support operator
A single event can produce two rows in the log — one per channel. In the log these are two independent rows: the email and the push are tracked separately.
The contact (address, token) is materialized at the moment of sending and stored in the log. If the user later changes their email, the delivery history will still show which address the letter actually went to.
Templates and rendering
The text of the email and the push notification is stored in the templates table. The template key is a combination of the event type and the channel, for example order.confirmed.email or order.confirmed.push. Each template is stored in two languages: ru and en.
A template contains placeholders like ${orderNumber}, ${amount} — they're substituted from the event data before sending.
If there's no template for the needed (event, channel, locale) combination, the notification is not created. An empty email is worse than no email at all, so a missing template fires the notification_template_missing_total metric and an alert. Templates live in the database and are updated without restarting the service (cache TTL 60 seconds).
Delivery statuses
Each notification record passes through several statuses:
| Status | Meaning |
|---|---|
QUEUED | created from an event, waiting to be sent |
SENT | handed off to the provider |
DELIVERED | a Mailgun webhook confirmed delivery (email only) |
BOUNCED | a webhook reported an invalid address |
FAILED | retries exhausted or a permanent error |
Push notifications have no confirmation mechanism at the FCM level — an HTTP 200 from Firebase means "accepted," not "delivered." So a push stays in the SENT status forever. That's an FCM limitation, not a bug.
How retries work
Sending is handled by the DispatchPending scheduler, which every second grabs a batch of records in the QUEUED status (up to 100 of them) and sends them through the provider.
If the provider returns a temporary error (5xx, timeout), the record goes back to QUEUED with a delay: 30 seconds, then 5 minutes, then 30 minutes. After three failed attempts — FAILED. If the error is permanent (4xx, invalid address) — FAILED immediately, with no retries.
Every attempt is recorded in the delivery_attempts table: the date, the attempt number, the result, and a fragment of the provider's response. This is needed for incident analysis.
To avoid overloading the providers, there's a rate limit — no more than 100 emails per second (a token bucket mechanism).
The Mailgun webhook
When Mailgun delivers an email or gets a rejection, it sends a webhook to POST /webhooks/email-events. Using the external_id (the identifier Mailgun returned when the email was sent), the service finds the record and moves it to DELIVERED or BOUNCED.
The webhook is verified by its HMAC signature — without a valid signature the request is rejected (401). Deduplication by (notification_id, webhook_event_id) protects against repeated deliveries of the same event.
What the support operator sees
When a shopper writes "I didn't get my confirmation," the operator opens the log and filters by userId. They see every record: when it was created, which channel, which address, the status, how many attempts, and why the last one failed.
If a notification is in the FAILED status, the operator can hit "Retry" and it goes back into QUEUED. If the address is invalid and a retry is pointless — "Abandon," a final status.
For security reasons, emails and push tokens are encrypted at rest and masked in logs (u***@example.com). Log data is deleted after 90 days (a personal-data retention requirement).
Database schema
Four tables:
notifications— the main one: one row per delivery attempt to a single channel. Stores the materialized contact, the template key, the status, and theexternal_idfor webhook matching.templates— templates (key: event × channel × locale).delivery_attempts— a log of every attempt with its result and the provider's response.processed_events— the idempotency table keyed byevent_id.
Technical stack
Java 21, Spring Boot 3, Spring Kafka — the event consumer. PostgreSQL + jOOQ + Flyway — storage. Spring Web — REST endpoints (webhook + admin log). Spring Security + OAuth2 + an HMAC filter — authorization. Resilience4j — Circuit Breaker and retry when calling Customer BFF. Micrometer/Prometheus, OpenTelemetry — metrics and tracing.
A working example — github.com/remodov/notification-service.
In short
- Notification Service is subscribed to order events in Kafka and does nothing but delivery — the business decision of "when to send" stays in Order Service.
- Idempotency by
event_idprotects against duplicates when Kafka redelivers. - Channels (email/push) are chosen by event type: a single event can produce two rows in the log.
- The contact is materialized at the moment of sending — the history stays accurate even after the address changes.
- Templates in the database are updated without a restart (60-second cache); no template means no notification, and an alert fires.
- Retries: three attempts on temporary errors (30s/5min/30min); on a permanent error — FAILED right away.
- The DELIVERED status is available only for email (via the Mailgun webhook); push stays in SENT — an FCM limitation.
- The log with filtering and manual retry is the operator's main tool when investigating complaints.
What to read next
- Order Service — marketplace architecture — the service that publishes events for Notification.
- Kafka: message broker fundamentals — the basics for understanding consumers and at-least-once.