Resilience Patterns: Retry, Circuit Breaker, Bulkhead, DLQ in Go

← Back to the section

When an application talks to a single service or database, failures are rare. But in distributed systems, calls to other services are the norm. Which means it is also normal for something to break: the network drops packets, a service restarts, a database fails over.

The question is not "will it fail?" but "what will happen to the rest of the system when it does?".

Resilience patterns do not prevent failures — they keep one failure from cascading and bringing down everything around it.

Retry — try again on a temporary error

A network call failed. The cause may be random: a pod restart, a brief overload, a garbage-collection pause. Try again in a second and it works.

But if you retry instantly, a hundred clients all pile onto an already sick service at once. If you never retry, you lose the request over a random glitch.

The solution is to retry with growing delays (Exponential Backoff):

Attempt 1: immediately
Attempt 2: after 1 sec
Attempt 3: after 2 sec
Attempt 4: after 4 sec
Attempt 5: after 8 sec (no more than 30 sec)

func retry[T any](ctx context.Context, maxAttempts int, base, max time.Duration,
	call func(ctx context.Context) (T, error)) (T, error) {

	var zero T
	delay := base
	for attempt := 1; ; attempt++ {
		result, err := call(ctx)
		if err == nil {
			return result, nil
		}
		if attempt == maxAttempts || !isRetryable(err) {
			return zero, err
		}
		select {
		case <-ctx.Done():
			return zero, ctx.Err()
		case <-time.After(delay):
		}
		delay = min(delay*2, max)
	}
}

Jitter — so you don't hammer the service all at once

All hundred clients got their error at the same moment. All wait one second. All retry simultaneously. The service goes down again. This is called the thundering herd effect.

Jitter adds a random spread to the delay — clients disperse over time:

delay := min(maxDelay, baseDelay*time.Duration(math.Pow(multiplier, float64(attempt))))
jitter := time.Duration(rand.Int64N(int64(delay / 2)))
time.Sleep(delay + jitter)

Not every request is safe to retry

Safe: operations that can be repeated without side effects — GET, PUT with a fixed key, DELETE by identifier.
Dangerous: POST without an idempotency key. A double submission may create two orders.

For non-idempotent operations, use an Idempotency Key — the client generates a UUID, and the server checks it: if a request with that key has already been processed, it returns the stored result instead of running the operation again.

Circuit Breaker — a "breaker switch" for when a service goes down

The payment gateway is down. A hundred goroutines wait thirty seconds each for a response. Within a minute all resources are tied up waiting. Your service stops answering any requests — even the ones that have nothing to do with payments. This is a cascading failure.

A Circuit Breaker is literally a "switch". It watches the error rate and, if it gets too high, stops sending requests: it returns an error immediately instead of wasting time waiting.

Three states:

CLOSED — normal operation. Requests pass through, and we count the error rate over a sliding window.
OPEN — the breaker has tripped. Requests are rejected instantly without waiting. We wait out the timeout.
HALF_OPEN — we let a few trial requests through. If they succeed, we move to CLOSED. If not, back to OPEN.

Example configuration (sony/gobreaker):

breaker := gobreaker.NewCircuitBreaker[PaymentResult](gobreaker.Settings{
	Name: "paymentService",
	ReadyToTrip: func(counts gobreaker.Counts) bool {
		return counts.Requests >= 10 && // a window of 10 requests
			float64(counts.TotalFailures)/float64(counts.Requests) >= 0.5 // 50% errors → OPEN
	},
	Timeout:     30 * time.Second, // after 30 sec → HALF_OPEN
	MaxRequests: 3,                // trial requests in HALF_OPEN
	IsSuccessful: func(err error) bool {
		return err == nil || errors.Is(err, ErrOrderNotFound) // a business response is not a failure
	},
})

func (s *PaymentService) Charge(ctx context.Context, userID int64, amount Money) (PaymentResult, error) {
	result, err := s.breaker.Execute(func() (PaymentResult, error) {
		return s.client.Charge(ctx, userID, amount.Kopecks())
	})
	if errors.Is(err, gobreaker.ErrOpenState) {
		return PaymentResult{}, ErrServiceTemporarilyUnavailable
	}
	return result, err
}

An important nuance: ErrOrderNotFound (404) is a business response, not a failure. If the Circuit Breaker treats 404 as an error, then a run of "order not found" cases will make it block all requests to the payment gateway. That is why IsSuccessful is a mandatory setting.

Timeout — don't wait forever

The service isn't responding. Without a timeout, the goroutine hangs indefinitely, holding memory and resources. A hundred such goroutines and everything grinds to a halt.

Every network call must be time-bounded:

func (s *PaymentService) ChargeWithTimeout(ctx context.Context, userID int64, amount Money) (PaymentResult, error) {
	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel()
	return s.client.Charge(ctx, userID, amount.Kopecks())
}

Plus a safety net at the HTTP client level:

client := &http.Client{Timeout: 5 * time.Second}

This applies not only to HTTP. Any call to a database, Redis, gRPC, or a message queue — a timeout must be there too (in Go it travels with the context.Context passed as the first argument to every such call).

How Retry, Circuit Breaker, and Timeout fit together

The three patterns work together, and the order matters:

Circuit Breaker → Retry → Timeout → External service

Circuit Breaker on the outside. If the circuit is open, we don't waste time on retries.
Retry in the middle. It repeats on a temporary failure.
Timeout on the inside. Each individual attempt is time-bounded.

Bulkhead — resource isolation

A service calls three external systems: the payment gateway, the warehouse, the insurer. The payment gateway slows down — all two hundred goroutines are busy waiting. Requests to the warehouse and insurer also queue up, even though those systems are working fine.

A Bulkhead is named after the compartments in a ship. Each compartment is watertight: flooding one does not sink the rest.

Two implementation options:

Worker pool — each external call goes through a dedicated fixed-size pool of worker goroutines with a job queue. A hung call doesn't tie up the workers of other systems, and the queue capacity is managed explicitly — but it's more code.

Semaphore — a limit on the number of concurrent calls via a buffered channel or semaphore.Weighted, without a separate pool. Lighter on resources and simpler in code — in Go this is the default choice.

type Bulkhead struct {
	sem chan struct{}
}

func NewBulkhead(maxConcurrent int) *Bulkhead {
	return &Bulkhead{sem: make(chan struct{}, maxConcurrent)}
}

func (b *Bulkhead) Execute(ctx context.Context, maxWait time.Duration, call func() error) error {
	waitCtx, cancel := context.WithTimeout(ctx, maxWait)
	defer cancel()
	select {
	case b.sem <- struct{}{}:
		defer func() { <-b.sem }()
		return call()
	case <-waitCtx.Done():
		return ErrBulkheadFull
	}
}

paymentBulkhead := NewBulkhead(20)   // at most 20 concurrent calls
inventoryBulkhead := NewBulkhead(20)
insuranceBulkhead := NewBulkhead(10)

For HTTP calls with timeouts, a semaphore is usually enough. A dedicated worker pool is needed when timeouts are unreliable or not under your control.

Fallback — a backup response

If the main path is unavailable, use a backup. You don't always need a perfect answer; sometimes "acceptable" beats an error.

Three kinds of backup response:

A default value — when exact data is unavailable, return a reasonable substitute:

func (s *RateService) GetRate(ctx context.Context, currency string) (ExchangeRate, error) {
	rate, err := s.breaker.Execute(func() (ExchangeRate, error) {
		return s.client.GetRate(ctx, currency)
	})
	if err != nil {
		if cached, ok := s.cache.Latest(currency); ok {
			return cached, nil
		}
		return FallbackRate(currency), nil
	}
	return rate, nil
}

A cached response — on failure, return the last known data:

func (s *CatalogService) GetProducts(ctx context.Context, category string) ([]Product, error) {
	products, err := s.breaker.Execute(func() ([]Product, error) {
		return s.client.GetProducts(ctx, category)
	})
	if err != nil {
		return s.cache.Get(category), nil // last known data or an empty list
	}
	s.cache.Put(category, products)
	return products, nil
}

Simplified logic — instead of personalized recommendations, return popular products:

func (s *RecommendationService) GetRecommendations(ctx context.Context, userID int64) ([]Product, error) {
	products, err := s.breaker.Execute(func() ([]Product, error) {
		return s.client.GetPersonalized(ctx, userID)
	})
	if err != nil {
		return s.products.TopByPopularity(ctx, 10)
	}
	return products, nil
}

When a fallback is not appropriate: if the payment gateway is unavailable, you can't charge money "approximately". Here the right answer is an honest "service temporarily unavailable, try again later" error.

Rate Limiter — protection against overload

An external API limits you: 100 requests per second. Exceed it and you're banned for five minutes. Or your own service gets a traffic spike — you need to protect yourself.

limiter := rate.NewLimiter(rate.Limit(50), 50) // 50 requests per second (golang.org/x/time/rate)

func (c *PaymentClient) Register(ctx context.Context, req PaymentRegisterRequest) (PaymentResponse, error) {
	waitCtx, cancel := context.WithTimeout(ctx, 2*time.Second) // wait for a token at most 2 sec
	defer cancel()
	if err := c.limiter.Wait(waitCtx); err != nil {
		return PaymentResponse{}, ErrRateLimitExceeded
	}
	return c.do(ctx, req)
}

The main algorithms:

Fixed Window — the counter resets at the start of each window. Simple, but at the window boundary it may let through up to twice the limit.
Sliding Window — a sliding window, more precise limiting.
Token Bucket — tokens accumulate at a constant rate. It allows short bursts if there are accumulated tokens. This is exactly what golang.org/x/time/rate implements.
Leaky Bucket — requests are processed at a constant rate. It smooths out bursts but adds latency.

Dead Letter Queue — what to do with unprocessable messages

A message in Kafka can't be processed: corrupt data, an unknown format, a failed business validation. Endless retries are pointless — the message is "poisoned" and blocks processing of the ones behind it.

A Dead Letter Queue (DLQ) — after a few failed attempts, the message is moved to a separate queue for manual handling.

func (c *OrderEventsConsumer) Run(ctx context.Context) error {
	for {
		msg, err := c.reader.FetchMessage(ctx) // segmentio/kafka-go
		if err != nil {
			return err
		}
		if err := c.processWithRetry(ctx, msg); err != nil {
			c.sendToDLQ(ctx, msg, err)
		}
		c.reader.CommitMessages(ctx, msg)
	}
}

func (c *OrderEventsConsumer) processWithRetry(ctx context.Context, msg kafka.Message) error {
	var lastErr error
	for attempt := 0; attempt < 3; attempt++ {
		lastErr = c.process(ctx, msg)
		if lastErr == nil {
			return nil
		}
		if !isRetryable(lastErr) { // ValidationError, *json.SyntaxError — no retries
			return lastErr
		}
		time.Sleep(time.Second)
	}
	return lastErr
}

func (c *OrderEventsConsumer) sendToDLQ(ctx context.Context, msg kafka.Message, cause error) {
	c.dlqWriter.WriteMessages(ctx, kafka.Message{
		Topic: msg.Topic + ".dlq",
		Key:   msg.Key,
		Value: msg.Value,
		Headers: append(msg.Headers, kafka.Header{
			Key: "x-error", Value: []byte(cause.Error()),
		}),
	})
}

What to do with a DLQ:

Monitoring — set up an alert for when messages appear in the DLQ.
Re-sending — after fixing the cause, re-send to the main topic. Be careful: if the cause is not fixed, the message will land in the DLQ again.
Manual handling — for business errors that can't be automated.

How the patterns work together

In practice the patterns are combined. The full stack for a single external call looks like this (order from outside → inward):

Rate Limiter — if the limit is exhausted, we don't spend resources further.
Bulkhead — if the semaphore is full, we don't create new calls.
Circuit Breaker — if the service is unavailable, instant rejection or fallback.
Retry — if the call failed, we repeat it with a delay.
Timeout — each attempt is time-bounded.

func (s *PaymentService) Charge(ctx context.Context, req ChargeRequest) (PaymentResult, error) {
	if err := s.limiter.Wait(ctx); err != nil { // 1. Rate Limiter
		return PaymentResult{}, err
	}

	select { // 2. Bulkhead — at most 25 concurrent calls
	case s.sem <- struct{}{}:
		defer func() { <-s.sem }()
	case <-ctx.Done():
		return PaymentResult{}, ctx.Err()
	}

	return s.breaker.Execute(func() (PaymentResult, error) { // 3. Circuit Breaker
		return retry(ctx, 3, time.Second, 30*time.Second, // 4. Retry
			func(ctx context.Context) (PaymentResult, error) {
				callCtx, cancel := context.WithTimeout(ctx, 5*time.Second) // 5. Timeout
				defer cancel()
				return s.client.Charge(callCtx, req)
			})
	})
}

In short

Retry + Exponential Backoff — we survive brief failures; Jitter prevents a simultaneous flood of retries.
Circuit Breaker — we don't spend resources on an unavailable service; three states: CLOSED / OPEN / HALF_OPEN.
Timeout — any network call without a timeout is a potential goroutine leak.
Bulkhead — we isolate resources; one problem service doesn't block the rest.
Fallback — we degrade instead of crashing; not always applicable (payments — an honest error, not an "approximate" answer).
Rate Limiter — we protect ourselves and external APIs from overload.
DLQ — we don't lose "poisoned" messages or get stuck looping on them.
Wrapping order: Rate Limiter → Bulkhead → Circuit Breaker → Retry → Timeout.