Resilience Patterns: Retry, Circuit Breaker, Bulkhead, DLQ in TypeScript

← Back to the section

When an application talks to a single service or database, failures are rare. But in distributed systems, calls to other services are the norm. Which means it is also normal for something to break: the network drops packets, a service restarts, a database fails over.

The question is not "will it fail?" but "what will happen to the rest of the system when it does?".

Resilience patterns do not prevent failures — they keep one failure from cascading and bringing down everything around it.

Retry — try again on a temporary error

A network call failed. The cause may be random: a pod restart, a brief overload, a garbage-collection pause. Try again in a second and it works.

But if you retry instantly, a hundred clients all pile onto an already sick service at once. If you never retry, you lose the request over a random glitch.

The solution is to retry with growing delays (Exponential Backoff):

Attempt 1: immediately
Attempt 2: after 1 sec
Attempt 3: after 2 sec
Attempt 4: after 4 sec
Attempt 5: after 8 sec (no more than 30 sec)

import pRetry from 'p-retry';

const result = await pRetry(() => paymentClient.charge(request), {
  retries: 4,          // 5 attempts in total
  minTimeout: 1_000,
  factor: 2,
  maxTimeout: 30_000,
  shouldRetry: (error) => error instanceof RetryableError,
});

Jitter — so you don't hammer the service all at once

All hundred clients got their error at the same moment. All wait one second. All retry simultaneously. The service goes down again. This is called the thundering herd effect.

Jitter adds a random spread to the delay — clients disperse over time:

import { setTimeout as sleep } from 'node:timers/promises';

const delay = Math.min(maxDelay, baseDelay * multiplier ** attempt);
const jitter = Math.floor(Math.random() * (delay / 2));
await sleep(delay + jitter);

Not every request is safe to retry

Safe: operations that can be repeated without side effects — GET, PUT with a fixed key, DELETE by identifier.
Dangerous: POST without an idempotency key. A double submission may create two orders.

For non-idempotent operations, use an Idempotency Key — the client generates a UUID, and the server checks it: if a request with that key has already been processed, it returns the stored result instead of running the operation again.

Circuit Breaker — a "breaker switch" for when a service goes down

The payment gateway is down. A hundred requests hang waiting thirty seconds each. Within a minute the whole connection pool is exhausted. Your service stops answering any requests — even the ones that have nothing to do with payments. This is a cascading failure.

A Circuit Breaker is literally a "switch". It watches the error rate and, if it gets too high, stops sending requests: it returns an error immediately instead of wasting time waiting.

Three states:

CLOSED — normal operation. Requests pass through, and we count the error rate over a sliding window.
OPEN — the breaker has tripped. Requests are rejected instantly without waiting. We wait out the timeout.
HALF_OPEN — we let a few trial requests through. If they succeed, we move to CLOSED. If not, back to OPEN.

Example configuration (opossum):

import CircuitBreaker from 'opossum';

const paymentBreakerOptions: CircuitBreaker.Options = {
  rollingCountTimeout: 10_000,    // sliding window for the error rate
  errorThresholdPercentage: 50,   // 50% errors → OPEN
  resetTimeout: 30_000,           // after 30 sec → HALF_OPEN
  volumeThreshold: 10,            // minimum calls before it can trip
  errorFilter: (e) => e instanceof OrderNotFoundError, // business response — not a failure
};

const chargeBreaker = new CircuitBreaker(
  (userId: number, amount: Money) => paymentClient.charge(userId, amount.toKopecks()),
  paymentBreakerOptions,
);

chargeBreaker.fallback(() => {
  throw new ServiceTemporarilyUnavailableError('Payment service');
});

export const charge = (userId: number, amount: Money): Promise<PaymentResult> =>
  chargeBreaker.fire(userId, amount);

An important nuance: OrderNotFoundError (404) is a business response, not a failure. If the Circuit Breaker treats 404 as an error, then a run of "order not found" cases will make it block all requests to the payment gateway. That is why errorFilter is a mandatory setting.

Timeout — don't wait forever

The service isn't responding. Without a timeout, the request hangs indefinitely, holding a socket and memory. A hundred such hanging requests and everything grinds to a halt.

Every network call must be time-bounded:

const chargeWithTimeout = new CircuitBreaker(
  (userId: number, amount: Money) => paymentClient.charge(userId, amount.toKopecks()),
  { timeout: 5_000 }, // TimeoutError after 5 seconds
);

const response = await fetch('https://payments.example.com/charge', {
  method: 'POST',
  body: JSON.stringify(request),
  signal: AbortSignal.timeout(5_000),
});

This applies not only to HTTP. Any call to a database, Redis, gRPC, or a message queue — a timeout must be there too.

How Retry, Circuit Breaker, and Timeout fit together

The three patterns work together, and the order matters:

Circuit Breaker → Retry → Timeout → External service

Circuit Breaker on the outside. If the circuit is open, we don't waste time on retries.
Retry in the middle. It repeats on a temporary failure.
Timeout on the inside. Each individual attempt is time-bounded.

Bulkhead — resource isolation

A service calls three external systems: the payment gateway, the warehouse, the insurer. The payment gateway slows down — all two hundred outgoing calls are busy waiting. Requests to the warehouse and insurer also queue up, even though those systems are working fine.

A Bulkhead is named after the compartments in a ship. Each compartment is watertight: flooding one does not sink the rest.

Two implementation options:

Semaphore Isolation — a limit on the number of concurrent calls to each external system. Node.js has no thread pools for I/O — all calls go through a single event loop, so a semaphore is the main isolation tool: a separate concurrency limit per system (p-limit or the capacity option in opossum).

Worker Pool Isolation — a separate worker_threads pool (for example, piscina) for heavy CPU work. Hung CPU work doesn't block the event loop. There is overhead from passing data between threads.

import pLimit from 'p-limit';

const paymentLimit = pLimit(20);   // no more than 20 concurrent calls
const inventoryLimit = pLimit(20);
const insuranceLimit = pLimit(10);

const charge = (req: ChargeRequest): Promise<PaymentResult> =>
  paymentLimit(() => paymentClient.charge(req));

For network calls with timeouts, a semaphore is enough. A worker_threads pool is needed for CPU-heavy work that would otherwise block the event loop.

Fallback — a backup response

If the main path is unavailable, use a backup. You don't always need a perfect answer; sometimes "acceptable" beats an error.

Three kinds of backup response:

A default value — when exact data is unavailable, return a reasonable substitute:

const rateBreaker = new CircuitBreaker(
  (currency: string) => exchangeRateClient.getRate(currency),
  breakerOptions,
);

rateBreaker.fallback(async (currency: string) =>
  (await exchangeRateCache.getLatest(currency)) ?? ExchangeRate.fallback(currency));

export const getRate = (currency: string): Promise<ExchangeRate> =>
  rateBreaker.fire(currency);

A cached response — on failure, return the last known data:

const productsBreaker = new CircuitBreaker(async (category: string) => {
  const products = await catalogClient.getProducts(category);
  await productCache.put(category, products);
  return products;
}, breakerOptions);

productsBreaker.fallback(async (category: string) =>
  (await productCache.get(category)) ?? []);

Simplified logic — instead of personalized recommendations, return popular products:

const recommendationsBreaker = new CircuitBreaker(
  (userId: number) => recommendationClient.getPersonalized(userId),
  breakerOptions,
);

recommendationsBreaker.fallback(() =>
  productRepository.findTopByPopularity(10));

When a fallback is not appropriate: if the payment gateway is unavailable, you can't charge money "approximately". Here the right answer is an honest "service temporarily unavailable, try again later" error.

Rate Limiter — protection against overload

An external API limits you: 100 requests per second. Exceed it and you're banned for five minutes. Or your own service gets a traffic spike — you need to protect yourself.

const register = (request: PaymentRegisterDto): Promise<PaymentResponse> =>
  paymentApiLimiter.schedule(() => paymentClient.register(request));

import Bottleneck from 'bottleneck';

const paymentApiLimiter = new Bottleneck({
  reservoir: 50,                    // 50 requests
  reservoirRefreshAmount: 50,
  reservoirRefreshInterval: 1_000,  // per second
});

The main algorithms:

Fixed Window — the counter resets at the start of each window. Simple, but at the window boundary it may let through up to twice the limit.
Sliding Window — a sliding window, more precise limiting.
Token Bucket — tokens accumulate at a constant rate. It allows short bursts if there are accumulated tokens.
Leaky Bucket — requests are processed at a constant rate. It smooths out bursts but adds latency.

Dead Letter Queue — what to do with unprocessable messages

A message in Kafka can't be processed: corrupt data, an unknown format, a failed business validation. Endless retries are pointless — the message is "poisoned" and blocks processing of the ones behind it.

A Dead Letter Queue (DLQ) — after a few failed attempts, the message is moved to a separate queue for manual handling.

const MAX_ATTEMPTS = 3;

const isRetryable = (e: unknown): boolean =>
  !(e instanceof ValidationError || e instanceof SyntaxError);

await consumer.run({
  eachMessage: async ({ topic, message }) => {
    try {
      await handle(message);
    } catch (e) {
      const attempts = Number(message.headers?.attempts ?? 0) + 1;
      if (attempts >= MAX_ATTEMPTS || !isRetryable(e)) {
        await producer.send({
          topic: `${topic}.dlq`,
          messages: [{
            key: message.key,
            value: message.value,
            headers: { ...message.headers, error: (e as Error).message },
          }],
        });
        return;
      }
      await producer.send({
        topic,
        messages: [{
          key: message.key,
          value: message.value,
          headers: { ...message.headers, attempts: String(attempts) },
        }],
      });
    }
  },
});

What to do with a DLQ:

Monitoring — set up an alert for when messages appear in the DLQ.
Re-sending — after fixing the cause, re-send to the main topic. Be careful: if the cause is not fixed, the message will land in the DLQ again.
Manual handling — for business errors that can't be automated.

How the patterns work together

In practice the patterns are combined. The full stack for a single external call looks like this (order from outside → inward):

Rate Limiter — if the limit is exhausted, we don't spend resources further.
Bulkhead — if the pool is full, we don't create new calls.
Circuit Breaker — if the service is unavailable, instant rejection or fallback.
Retry — if the call failed, we repeat it with a delay.
Timeout — each attempt is time-bounded.

const paymentLimiter = new Bottleneck({
  reservoir: 100,
  reservoirRefreshAmount: 100,
  reservoirRefreshInterval: 1_000,
});

const paymentBulkhead = pLimit(25);

const callWithRetry = (req: ChargeRequest) =>
  pRetry(() => paymentClient.charge(req, { timeout: 5_000 }), {
    retries: 2,
    minTimeout: 1_000,
    factor: 2,
  });

const paymentBreaker = new CircuitBreaker(callWithRetry, {
  rollingCountTimeout: 10_000,
  errorThresholdPercentage: 50,
  resetTimeout: 30_000,
});

export const charge = (req: ChargeRequest): Promise<PaymentResult> =>
  paymentLimiter.schedule(() =>
    paymentBulkhead(() => paymentBreaker.fire(req)));

In short

Retry + Exponential Backoff — we survive brief failures; Jitter prevents a simultaneous flood of retries.
Circuit Breaker — we don't spend resources on an unavailable service; three states: CLOSED / OPEN / HALF_OPEN.
Timeout — any network call without a timeout is a potential resource leak.
Bulkhead — we isolate resources; one problem service doesn't block the rest.
Fallback — we degrade instead of crashing; not always applicable (payments — an honest error, not an "approximate" answer).
Rate Limiter — we protect ourselves and external APIs from overload.
DLQ — we don't lose "poisoned" messages or get stuck looping on them.
Wrapping order: Rate Limiter → Bulkhead → Circuit Breaker → Retry → Timeout.