Resilience Patterns: Retry, Circuit Breaker, Bulkhead, DLQ in Python

← Back to the section

When an application talks to a single service or database, failures are rare. But in distributed systems, calls to other services are the norm. Which means it is also normal for something to break: the network drops packets, a service restarts, a database fails over.

The question is not "will it fail?" but "what will happen to the rest of the system when it does?".

Resilience patterns do not prevent failures — they keep one failure from cascading and bringing down everything around it.

Retry — try again on a temporary error

A network call failed. The cause may be random: a pod restart, a brief overload, a garbage-collection pause. Try again in a second and it works.

But if you retry instantly, a hundred clients all pile onto an already sick service at once. If you never retry, you lose the request over a random glitch.

The solution is to retry with growing delays (Exponential Backoff):

Attempt 1: immediately
Attempt 2: after 1 sec
Attempt 3: after 2 sec
Attempt 4: after 4 sec
Attempt 5: after 8 sec (no more than 30 sec)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, max=30),
    retry=retry_if_exception_type(RetryableError),
)
async def call_payment_gateway(request: PaymentRequest) -> PaymentResult:
    return await payment_client.charge(request)

Jitter — so you don't hammer the service all at once

All hundred clients got their error at the same moment. All wait one second. All retry simultaneously. The service goes down again. This is called the thundering herd effect.

Jitter adds a random spread to the delay — clients disperse over time:

delay = min(max_delay, base_delay * multiplier ** attempt)
jitter = random.uniform(0, delay / 2)
time.sleep(delay + jitter)

# tenacity has a ready-made strategy
@retry(wait=wait_exponential_jitter(initial=1, max=30))
async def call_service() -> None: ...

Not every request is safe to retry

Safe: operations that can be repeated without side effects — GET, PUT with a fixed key, DELETE by identifier.
Dangerous: POST without an idempotency key. A double submission may create two orders.

For non-idempotent operations, use an Idempotency Key — the client generates a UUID, and the server checks it: if a request with that key has already been processed, it returns the stored result instead of running the operation again.

Circuit Breaker — a "breaker switch" for when a service goes down

The payment gateway is down. A hundred threads wait thirty seconds each for a response. Within a minute all threads are busy. Your service stops answering any requests — even the ones that have nothing to do with payments. This is a cascading failure.

A Circuit Breaker is literally a "switch". It watches the error rate and, if it gets too high, stops sending requests: it returns an error immediately instead of wasting time waiting.

Three states:

CLOSED — normal operation. Requests pass through, and we count the error rate over a sliding window.
OPEN — the breaker has tripped. Requests are rejected instantly without waiting. We wait out the timeout.
HALF_OPEN — we let a few trial requests through. If they succeed, we move to CLOSED. If not, back to OPEN.

Example configuration (aiobreaker):

payment_breaker = CircuitBreaker(
    fail_max=5,                              # 5 consecutive errors → OPEN
    timeout_duration=timedelta(seconds=30),  # after 30 sec → HALF_OPEN
    exclude=[OrderError],                    # business errors don't count as failures
)


@payment_breaker
async def charge(user_id: int, amount: Money) -> PaymentResult:
    return await payment_client.charge(user_id, amount.to_kopecks())


async def charge_or_fail_fast(user_id: int, amount: Money) -> PaymentResult:
    try:
        return await charge(user_id, amount)
    except CircuitBreakerError:
        raise ServiceTemporarilyUnavailableError("Payment service")

An important nuance: OrderError.NotFound (404) is a business response, not a failure. If the Circuit Breaker treats 404 as an error, then a run of "order not found" cases will make it block all requests to the payment gateway. That is why exclude is a mandatory setting.

Timeout — don't wait forever

The service isn't responding. Without a timeout, the thread hangs indefinitely, holding memory and resources. A hundred such threads and everything grinds to a halt.

Every network call must be time-bounded:

async def charge_async(user_id: int, amount: Money) -> PaymentResult:
    async with asyncio.timeout(5):
        return await payment_client.charge(user_id, amount.to_kopecks())

# timeouts at the HTTP client level
client = httpx.AsyncClient(
    timeout=httpx.Timeout(5.0, connect=2.0),
)

This applies not only to HTTP. Any call to a database, Redis, gRPC, or a message queue — a timeout must be there too.

How Retry, Circuit Breaker, and Timeout fit together

The three patterns work together, and the order matters:

Circuit Breaker → Retry → Timeout → External service

Circuit Breaker on the outside. If the circuit is open, we don't waste time on retries.
Retry in the middle. It repeats on a temporary failure.
Timeout on the inside. Each individual attempt is time-bounded.

Bulkhead — resource isolation

A service calls three external systems: the payment gateway, the warehouse, the insurer. The payment gateway slows down — all two hundred threads are busy waiting. Requests to the warehouse and insurer also queue up, even though those systems are working fine.

A Bulkhead is named after the compartments in a ship. Each compartment is watertight: flooding one does not sink the rest.

Two implementation options:

Thread Pool Isolation — each external call runs in a separate thread pool. A hung call doesn't tie up the main pool. There is overhead from thread switching.

Semaphore Isolation — a limit on the number of concurrent calls without a separate pool. Lighter on resources, but if a call hangs, a thread from the main pool is tied up too.

# Thread Pool Isolation — a separate pool per system
payment_pool = ThreadPoolExecutor(max_workers=20, thread_name_prefix="payment")
inventory_pool = ThreadPoolExecutor(max_workers=20, thread_name_prefix="inventory")

# Semaphore Isolation — a cap on concurrent calls in asyncio
payment_semaphore = asyncio.Semaphore(20)


async def charge(user_id: int, amount: Money) -> PaymentResult:
    async with payment_semaphore:
        return await payment_client.charge(user_id, amount.to_kopecks())

For HTTP calls with timeouts, Semaphore is usually enough. Thread Pool is needed when timeouts are unreliable or not under your control.

Fallback — a backup response

If the main path is unavailable, use a backup. You don't always need a perfect answer; sometimes "acceptable" beats an error.

Three kinds of backup response:

A default value — when exact data is unavailable, return a reasonable substitute:

@rate_breaker
async def fetch_rate(currency: str) -> ExchangeRate:
    return await exchange_rate_client.get_rate(currency)


async def get_rate(currency: str) -> ExchangeRate:
    try:
        return await fetch_rate(currency)
    except (CircuitBreakerError, ExchangeRateError):
        cached = exchange_rate_cache.get_latest(currency)
        return cached if cached is not None else ExchangeRate.fallback(currency)

A cached response — on failure, return the last known data:

@catalog_breaker
async def fetch_products(category: str) -> list[Product]:
    products = await catalog_client.get_products(category)
    product_cache.put(category, products)
    return products


async def get_products(category: str) -> list[Product]:
    try:
        return await fetch_products(category)
    except (CircuitBreakerError, CatalogError):
        return product_cache.get(category) or []

Simplified logic — instead of personalized recommendations, return popular products:

@recommendation_breaker
async def fetch_personalized(user_id: int) -> list[Product]:
    return await recommendation_client.get_personalized(user_id)


async def get_recommendations(user_id: int) -> list[Product]:
    try:
        return await fetch_personalized(user_id)
    except (CircuitBreakerError, RecommendationError):
        return product_repository.find_top_by_popularity(limit=10)

When a fallback is not appropriate: if the payment gateway is unavailable, you can't charge money "approximately". Here the right answer is an honest "service temporarily unavailable, try again later" error.

Rate Limiter — protection against overload

An external API limits you: 100 requests per second. Exceed it and you're banned for five minutes. Or your own service gets a traffic spike — you need to protect yourself.

payment_api_limiter = AsyncLimiter(max_rate=50, time_period=1.0)


async def register(request: PaymentRegisterDto) -> PaymentResponse:
    async with payment_api_limiter:
        return await payment_client.register(request)

The main algorithms:

Fixed Window — the counter resets at the start of each window. Simple, but at the window boundary it may let through up to twice the limit.
Sliding Window — a sliding window, more precise limiting.
Token Bucket — tokens accumulate at a constant rate. It allows short bursts if there are accumulated tokens.
Leaky Bucket — requests are processed at a constant rate. It smooths out bursts but adds latency.

Dead Letter Queue — what to do with unprocessable messages

A message in Kafka can't be processed: corrupt data, an unknown format, a failed business validation. Endless retries are pointless — the message is "poisoned" and blocks processing of the ones behind it.

A Dead Letter Queue (DLQ) — after a few failed attempts, the message is moved to a separate queue for manual handling.

@retry(
    stop=stop_after_attempt(3),
    wait=wait_fixed(1),
    retry=retry_if_not_exception_type((ValidationError, json.JSONDecodeError)),
)
async def handle(msg: ConsumerRecord) -> None:
    await process_order_event(msg.value)


async def consume_order_events(
    consumer: AIOKafkaConsumer, producer: AIOKafkaProducer,
) -> None:
    async for msg in consumer:
        try:
            await handle(msg)
        except Exception:
            # 3 failures or a non-retryable error → to the DLQ
            await producer.send_and_wait(
                f"{msg.topic}.dlq", msg.value, partition=msg.partition)

What to do with a DLQ:

Monitoring — set up an alert for when messages appear in the DLQ.
Re-sending — after fixing the cause, re-send to the main topic. Be careful: if the cause is not fixed, the message will land in the DLQ again.
Manual handling — for business errors that can't be automated.

How the patterns work together

In practice the patterns are combined. The full stack for a single external call looks like this (order from outside → inward):

Rate Limiter — if the limit is exhausted, we don't spend resources further.
Bulkhead — if the pool is full, we don't create new calls.
Circuit Breaker — if the service is unavailable, instant rejection or fallback.
Retry — if the call failed, we repeat it with a delay.
Timeout — each attempt is time-bounded.

payment_limiter = AsyncLimiter(max_rate=100, time_period=1.0)
payment_semaphore = asyncio.Semaphore(25)
payment_breaker = CircuitBreaker(fail_max=5, timeout_duration=timedelta(seconds=30))


async def call_payment(request: PaymentRequest) -> PaymentResult:
    async with payment_limiter:                     # 1. Rate Limiter
        async with payment_semaphore:               # 2. Bulkhead
            return await charge_protected(request)


@payment_breaker                                    # 3. Circuit Breaker
@retry(
    stop=stop_after_attempt(3),                     # 4. Retry
    wait=wait_exponential(multiplier=1),
    retry=retry_if_exception_type(RetryableError),
)
async def charge_protected(request: PaymentRequest) -> PaymentResult:
    async with asyncio.timeout(5):                  # 5. Timeout
        return await payment_client.charge(request)

In short

Retry + Exponential Backoff — we survive brief failures; Jitter prevents a simultaneous flood of retries.
Circuit Breaker — we don't spend resources on an unavailable service; three states: CLOSED / OPEN / HALF_OPEN.
Timeout — any network call without a timeout is a potential thread leak.
Bulkhead — we isolate resources; one problem service doesn't block the rest.
Fallback — we degrade instead of crashing; not always applicable (payments — an honest error, not an "approximate" answer).
Rate Limiter — we protect ourselves and external APIs from overload.
DLQ — we don't lose "poisoned" messages or get stuck looping on them.
Wrapping order: Rate Limiter → Bulkhead → Circuit Breaker → Retry → Timeout.