When you split an application into several services, you lose the main convenience of a monolith — a single database transaction. Before, one database transaction guaranteed: either the order is created, the money is charged and the stock is reserved — or none of it happens. In microservices each service is a separate database, a separate process, a separate network. You cannot roll back someone else's transaction. And the network can fail at the worst possible moment.
This article walks through the patterns that solve exactly this problem — one by one, from simple to complex.
Two-Phase Commit (2PC)
The problem. You need three services to either all commit their changes or all roll back. How do you coordinate that?
The idea. You introduce a coordinator that drives the process through two phases:
- Prepare (voting): the coordinator asks each participant "ready to commit?". The participant does all the work — locks resources, writes to its journal — but does not commit. It answers "yes" or "no".
- Commit or Rollback (decision): if everyone answered "yes" — the coordinator says "commit". If even one answered "no" — everyone rolls back.
Why 2PC is rarely used in microservices:
- Between the phases resources are locked. If the coordinator dies — participants hang with locked rows.
- The coordinator is a single point of failure. If it dies between phases, the system is left in an undefined state.
- Two network round trips — latency grows with every participant.
When it fits: several databases from the same vendor within a single piece of infrastructure (XA transactions). For microservices over the network — almost never.
Three-Phase Commit (3PC) — why it doesn't save you
3PC adds an intermediate PRE_COMMIT phase so that participants can make a decision on their own if the coordinator disappears. Three network round trips instead of two — even more latency. And, crucially: 3PC does not solve the network partition problem. If a participant received PRE_COMMIT, lost its connection, and committed — while another participant rolled back in the meantime — the data is inconsistent.
In practice the problems of 2PC are solved not through 3PC, but through Saga.
Saga — a chain of compensable steps
The problem. You need to coordinate the actions of several services, but without locks and without a single coordinator at the database level.
The idea. Instead of one big transaction — a chain of local transactions. Each step commits on its own. If step N fails — compensating transactions are run for the already completed steps in reverse order.
Step 1: Create the order → Compensation: Cancel the order
Step 2: Charge the money → Compensation: Refund the money
Step 3: Reserve stock → Compensation: Release the reservation
The key difference from 2PC: between steps the system is in a temporarily inconsistent state. This is called eventual consistency — everything will eventually become consistent, but not instantly.
Orchestration
A central orchestrator knows the sequence of steps and which compensation to call on error.
type SagaStep struct {
Name string
Action func(ctx context.Context, s *SagaState) error
Compensate func(ctx context.Context, s *SagaState) error
}
func (s *CreateOrderSaga) Execute(ctx context.Context, req CreateOrderRequest) (OrderResult, error) {
state := NewSagaState(req)
steps := []SagaStep{
{
Name: "create-order",
Action: func(ctx context.Context, st *SagaState) error { return s.orders.Create(ctx, st) },
Compensate: func(ctx context.Context, st *SagaState) error { return s.orders.Cancel(ctx, st.OrderID) },
},
{
Name: "charge-payment",
Action: func(ctx context.Context, st *SagaState) error { return s.payments.Charge(ctx, st) },
Compensate: func(ctx context.Context, st *SagaState) error { return s.payments.Refund(ctx, st.PaymentID) },
},
{
Name: "reserve-inventory",
Action: func(ctx context.Context, st *SagaState) error { return s.inventory.Reserve(ctx, st) },
Compensate: func(ctx context.Context, st *SagaState) error { return s.inventory.Release(ctx, st.OrderID) },
},
}
var completed []SagaStep
for _, step := range steps {
if err := step.Action(ctx, state); err != nil {
s.compensate(ctx, state, completed)
return OrderResult{}, fmt.Errorf("saga step %s: %w", step.Name, err)
}
completed = append(completed, step)
}
return state.Result(), nil
}
Pros: the logic lives in one place, state is easy to track, and it is simple to debug.
Cons: the orchestrator knows about all the services — it can grow into a struct that does everything under the sun.
Choreography
There is no central coordinator. Each service listens to events and reacts by publishing its own:
// PaymentService reacts to the event
func (s *PaymentService) ConsumeOrderEvents(ctx context.Context) error {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: s.brokers,
GroupID: "payment-service",
Topic: "order-events",
})
defer reader.Close()
for {
msg, err := reader.ReadMessage(ctx)
if err != nil {
return err
}
var event OrderCreatedEvent
if err := json.Unmarshal(msg.Value, &event); err != nil {
continue
}
result, err := s.charge(ctx, event.UserID, event.Total)
if err != nil {
s.publish(ctx, PaymentFailedEvent{OrderID: event.OrderID, Reason: err.Error()})
continue
}
s.publish(ctx, PaymentChargedEvent{OrderID: event.OrderID, PaymentID: result.PaymentID})
}
}
Pros: services are loosely coupled and evolve independently.
Cons: the Saga logic is hard to trace — it is smeared across services. It is hard to tell what state the process is in.
Orchestration or choreography — which to choose
Orchestration is better when there are more than three or four steps, when there are branches ("if the payment is partial — a different path"), and when a single monitoring point matters.
Choreography is better when there are two or three steps, the flow is linear without branches, and the services are built by different teams that value their autonomy.
An important note about compensations
A compensation is not a database rollback. Refund() is a new business operation that brings the system to the equivalent of a cancellation. The money is returned in a separate transaction, not by undoing the charge.
A compensation must be idempotent — if you call it twice, the result must be the same:
func (s *PaymentService) Refund(ctx context.Context, paymentID string) error {
payment, err := s.repo.FindByID(ctx, paymentID)
if err != nil {
return err
}
if payment.Status == StatusRefunded {
return nil // already refunded — do nothing
}
if err := s.gateway.Refund(ctx, payment.GatewayID); err != nil {
return err
}
payment.MarkRefunded()
return s.repo.Save(ctx, payment)
}
Transactional Outbox — atomic write of data and events
The problem. A typical mistake: you saved the order to the database, then sent an event to Kafka. Between these two operations the process died — the order exists, the event is lost. The opposite situation is just as bad: the event went out, but the transaction rolled back.
// Dangerous code — not atomic!
repo.Save(ctx, order) // transaction committed
writer.WriteMessages(ctx, orderCreated) // process died — event lost
The solution. The event is saved to the same database, in the same transaction as the business data — into a separate outbox_events table. A separate background process (relay) periodically reads this table and publishes the events to the broker.
CREATE TABLE outbox_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type VARCHAR(255) NOT NULL,
aggregate_id VARCHAR(255) NOT NULL,
event_type VARCHAR(255) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT now(),
published_at TIMESTAMP,
retry_count INT DEFAULT 0
);
// Business code: order and event in one transaction
func (s *OrderService) CreateOrder(ctx context.Context, cmd CreateOrderCommand) (Order, error) {
var order Order
err := pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error {
var err error
order, err = s.orders.SaveTx(ctx, tx, newOrder(cmd))
if err != nil {
return err
}
return s.outbox.SaveTx(ctx, tx,
"Order", order.ID, "OrderCreated", NewOrderCreatedPayload(order))
})
return order, err
}
// Relay: periodically sends unprocessed events
func (r *OutboxRelay) Run(ctx context.Context) {
ticker := time.NewTicker(500 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
r.publishPending(ctx)
}
}
}
func (r *OutboxRelay) publishPending(ctx context.Context) {
events, err := r.repo.FindUnpublished(ctx, 100)
if err != nil {
return
}
for _, event := range events {
err := r.writer.WriteMessages(ctx, kafka.Message{
Topic: topicFor(event),
Key: []byte(event.AggregateID),
Value: event.Payload,
})
if err != nil {
event.RetryCount++
} else {
event.MarkPublished()
}
r.repo.Save(ctx, event)
}
}
At-least-once — why the receiver must be idempotent
Outbox guarantees at-least-once delivery: the event will be sent at least once. The relay can die after sending but before it marked the row as published_at — then on the next run the event goes out again. That is why event consumers must be idempotent (see the Idempotent Consumer section below).
Polling vs CDC
An alternative to a polling relay is Change Data Capture (CDC): an external service reads the database log (WAL) and publishes changes to Kafka directly. But a polling relay inside the service's own code is simpler and more reliable:
- The event contract lives in code, not in the table schema. Changing the structure of
outbox_eventsdoes not break consumers. - CDC is a separate cluster with its own lifecycle and monitoring. When it breaks — that's a separate team and a separate procedure.
- A stuck CDC connector accumulates WAL until the disk fills up. A polling relay creates no such risks.
Polling every 200–500 ms is a delay the business cannot tell apart, and it removes a whole class of operational problems.
Event Sourcing — history as the source of truth
The problem. The orders table stores the current status: status = 'CONFIRMED'. But we don't know when the order was created, when it was paid, or whether it was in a different status before. The history is lost.
The idea. Store not the current state, but the sequence of events that led to it. The current state is computed by replaying all the events.
OrderCreated → PaymentReceived → ItemReserved → OrderConfirmed
The aggregate is rebuilt from its events:
type Order struct {
ID string
Status OrderStatus
uncommittedEvents []DomainEvent
version int64
}
func OrderFromEvents(events []DomainEvent) (*Order, error) {
order := &Order{}
for _, event := range events {
if err := order.apply(event); err != nil {
return nil, err
}
order.version++
}
return order, nil
}
func (o *Order) apply(event DomainEvent) error {
switch e := event.(type) {
case OrderCreated:
o.ID = e.OrderID
o.Status = StatusCreated
case PaymentReceived:
o.Status = StatusPaid
case OrderConfirmed:
o.Status = StatusConfirmed
default:
return fmt.Errorf("unknown event: %T", e)
}
return nil
}
func (o *Order) Confirm() error {
if o.Status != StatusPaid {
return fmt.Errorf("can only confirm PAID orders, current=%s", o.Status)
}
return o.raise(OrderConfirmed{OrderID: o.ID, At: time.Now()})
}
func (o *Order) raise(event DomainEvent) error {
if err := o.apply(event); err != nil {
return err
}
o.uncommittedEvents = append(o.uncommittedEvents, event)
return nil
}
Read projections
Event Sourcing separates writing and reading. Data is written to the Event Store (append only). For reading, projections are built — tables optimized for specific queries. This naturally pairs with CQRS.
func (p *OrderSummaryProjection) OnOrderCreated(ctx context.Context, event OrderCreated) error {
return p.repo.Save(ctx, OrderSummary{
OrderID: event.OrderID,
Status: "CREATED",
CreatedAt: event.CreatedAt,
})
}
func (p *OrderSummaryProjection) OnOrderConfirmed(ctx context.Context, event OrderConfirmed) error {
return p.repo.UpdateStatus(ctx, event.OrderID, "CONFIRMED")
}
When Event Sourcing fits
It fits when you need a complete history of changes (finance, audit, seller settlements), the ability to look at the state at any point in time, or complex domain logic with many status transitions.
It is overkill for simple reference data, CRUD without complex logic, and projects with no history requirements.
Pitfalls
- Projection lag. A projection is updated asynchronously — a user may not see their changes immediately.
- Event versioning. The
OrderCreated_v1event has nocurrencyfield. In_v2it appeared. You need a strategy for migrating old events when reading them. - Event Store size. An aggregate with thousands of events — replaying takes time. The solution is snapshots: periodically save the current state and replay only from the last snapshot.
Idempotent Consumer — protection against duplicates
The problem. In a distributed system messages arrive more than once: broker retries, consumer group rebalancing, at-least-once guarantees. The handler must produce the same result on repeated processing.
The solution. Keep a table of processed events. Before processing, check whether this event has already been handled.
func (p *IdempotentProcessor) Process(ctx context.Context, eventID string,
handler func(ctx context.Context, tx pgx.Tx) error) error {
return pgx.BeginFunc(ctx, p.pool, func(tx pgx.Tx) error {
var exists bool
err := tx.QueryRow(ctx,
`SELECT EXISTS(SELECT 1 FROM processed_events WHERE event_id = $1)`,
eventID).Scan(&exists)
if err != nil {
return err
}
if exists {
return nil // already processed — skip
}
if err := handler(ctx, tx); err != nil {
return err
}
_, err = tx.Exec(ctx,
`INSERT INTO processed_events (event_id, processed_at) VALUES ($1, now())`,
eventID)
return err
})
}
func (c *PaymentEventsConsumer) Run(ctx context.Context) error {
for {
msg, err := c.reader.ReadMessage(ctx)
if err != nil {
return err
}
var event PaymentChargedEvent
if err := json.Unmarshal(msg.Value, &event); err != nil {
continue
}
c.processor.Process(ctx, event.EventID, func(ctx context.Context, tx pgx.Tx) error {
order, err := c.orders.FindByIDTx(ctx, tx, event.OrderID)
if err != nil {
return err
}
order.MarkPaid(event.PaymentID)
return c.orders.SaveTx(ctx, tx, order)
})
}
}
CREATE TABLE processed_events (
event_id VARCHAR(255) PRIMARY KEY,
processed_at TIMESTAMP NOT NULL DEFAULT now()
);
-- Clean up old records
DELETE FROM processed_events
WHERE processed_at < now() - INTERVAL '7 days';
The same principle works for REST APIs — via the Idempotency-Key header. The client generates a unique key, and the server checks whether a request with this key has already been processed:
func (h *PaymentHandler) Charge(w http.ResponseWriter, r *http.Request) {
idempotencyKey := r.Header.Get("Idempotency-Key")
var req ChargeRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
result, err := h.processor.ProcessOrReturn(r.Context(), idempotencyKey,
func(ctx context.Context) (PaymentResult, error) {
return h.payments.Charge(ctx, req)
})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
json.NewEncoder(w).Encode(result)
}
Distributed Lock — protection against concurrent access
The problem. Two instances of the same service process one order at the same time. Both charge the money — the user pays twice.
The solution. A distributed lock via Redis: only one instance can hold the lock on a given resource at a time.
func (l *LockService) ExecuteWithLock(ctx context.Context, lockKey string,
ttl time.Duration, action func(ctx context.Context) error) error {
mutex := l.redsync.NewMutex(lockKey, redsync.WithExpiry(ttl))
if err := mutex.LockContext(ctx); err != nil {
return fmt.Errorf("cannot acquire lock %s: %w", lockKey, err)
}
defer mutex.UnlockContext(ctx)
return action(ctx)
}
func (s *OrderProcessor) ProcessOrder(ctx context.Context, orderID int64) error {
return s.locks.ExecuteWithLock(ctx,
fmt.Sprintf("order-processing:%d", orderID),
30*time.Second,
func(ctx context.Context) error {
order, err := s.orders.FindByID(ctx, orderID)
if err != nil {
return err
}
if err := order.Confirm(); err != nil {
return err
}
return s.orders.Save(ctx, order)
})
}
If Redis is unavailable — you can use SELECT FOR UPDATE in the database:
func (r *OrderRepository) FindByIDForUpdate(ctx context.Context, tx pgx.Tx, id int64) (*Order, error) {
row := tx.QueryRow(ctx, `
SELECT id, status, total
FROM orders
WHERE id = $1
FOR UPDATE SKIP LOCKED`, id)
var order Order
if err := row.Scan(&order.ID, &order.Status, &order.Total); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
return nil, err
}
return &order, nil
}
Locking pitfalls:
- Deadlock. Two processes lock resources in a different order and wait for each other. The solution — always lock in the same order (for example, by ID).
- Stuck lock. A process took a lock, died, and never released it. The solution — a TTL (automatic release on timeout).
- Split-brain in Redis. Redis failed over to a new master while the old one is still alive — two processes hold the same lock. The solution — Redlock (a lock across several independent Redis nodes at once).
How the patterns work together
A real system does not use a single pattern in isolation. They add up into a chain:
OrderService creates the order and saves the event through Outbox — atomically in one transaction. The Polling Relay periodically reads the table and publishes to Kafka. PaymentService processes it through an Idempotent Consumer — duplicates are safe. On concurrent access — a Distributed Lock. The whole process is coordinated by a Saga with compensations if something goes wrong.
In short
- 2PC — strict consistency via a two-phase protocol; only fits within a single piece of infrastructure, and is not used for microservices over the network.
- Saga — a chain of local transactions with compensating steps; orchestration (a central coordinator) or choreography (events between services).
- A compensation is a new business operation, not a database rollback; it must be idempotent.
- Transactional Outbox — event and business data in one transaction; the relay delivers to the broker with an at-least-once guarantee.
- Event Sourcing — storing the history of events instead of the current state; recovery via replay; pairs with CQRS.
- Idempotent Consumer — a table of processed events protects against duplicates on repeated delivery.
- Distributed Lock — a Redis lock (redsync/Redlock) or
SELECT FOR UPDATEprotects against concurrent access.
What to read next
- CQRS — the pattern that separates commands and queries; pairs naturally with Event Sourcing.
- Hexagonal architecture — how to isolate the Saga orchestrator from the infrastructure.