Shutdown budgets and observability in FastAPI

← Back to the section

When Kubernetes stops a pod, the application has a limited amount of time to finish its work without losing data. If that time runs out before the application is done, Kubernetes simply kills the process with a SIGKILL signal. Data is lost, transactions are cut off, Kafka messages are left half-processed.

The time budget needs to be planned in advance, not guessed at. And then it needs to be measured: without metrics, the first failure under load turns into an investigation through kubectl logs and guesswork.

What a 60-second budget looks like in FastAPI

By default Kubernetes gives a pod 30 seconds to shut down. For a FastAPI application with Kafka and background tasks this is usually not enough — teams set 60 seconds (terminationGracePeriodSeconds: 60).

These 60 seconds are split between several phases:

An important point: uvicorn graceful drain and lifespan shutdown run in parallel. While uvicorn finishes active HTTP connections, the lifespan block simultaneously stops the Kafka consumer and cancels background tasks. Wall clock = preStop + max(drain, lifespan), not the sum of all phases.

T=0     SIGTERM from Kubernetes
T=0     uvicorn.Server.should_exit = True
T=0     Start in parallel:
        ├── lifespan shutdown block:
        │     ├── readiness_state.is_ready = False  → /health/ready returns 503
        │     ├── asyncio tasks: cancel + waiting for CancelledError (up to 20s)
        │     ├── APScheduler shutdown(wait=True)
        │     ├── aiokafka consumer.stop() / producer.stop() (up to 15s)
        │     └── engine.dispose()  (SQLAlchemy connection pool)
        └── uvicorn graceful drain (--timeout-graceful-shutdown 25s):
              ├── current HTTP requests are finished to completion
              └── new connections are rejected
T≤35s   process exit(0)

What to do if you don't fit

The simple answer is to increase terminationGracePeriodSeconds to 90 or 120 seconds. This is not the best solution: a long shutdown means a long rolling deploy, more time with schema incompatibility, and kubectl drain hangs longer during node maintenance.

The better approach is to reduce the amount of work in each phase:

• aiokafka with max_poll_records=500 → reduce to 100; processing one batch will take 5s instead of 25s;
• an APScheduler job with a heavy iteration → reduce the batch size from 500 to 50;
• an asyncio task with a long cascade → add asyncio.wait_for(task, timeout=15) and exit earlier.

The idea is simple: if an operation doesn't fit into the allotted budget, make it smaller rather than giving it more time.

The app_shutdown_duration_seconds metric

Without a metric it's impossible to understand why a deploy sometimes hangs. We add a simple Gauge via prometheus_client:

import logging
import time
from prometheus_client import Gauge, REGISTRY

logger = logging.getLogger(__name__)

_shutdown_duration = Gauge(
    "app_shutdown_duration_seconds",
    "Duration of graceful shutdown in seconds",
    ["service"],
    registry=REGISTRY,
)


class ShutdownObserver:
    def __init__(self, service_name: str) -> None:
        self._service = service_name
        self._start: float = 0.0

    def on_sigterm(self) -> None:
        self._start = time.monotonic()
        logger.info("received SIGTERM, starting graceful shutdown")

    def on_complete(self) -> None:
        duration = time.monotonic() - self._start
        _shutdown_duration.labels(service=self._service).set(duration)
        logger.info("graceful shutdown completed in %.1fs", duration)

We use it in the lifespan block:

from contextlib import asynccontextmanager
from fastapi import FastAPI

observer = ShutdownObserver(service_name="order-service")


@asynccontextmanager
async def lifespan(app: FastAPI):
    # startup
    yield
    # shutdown — on_sigterm first, before any cleanup
    observer.on_sigterm()
    await consumer.stop()
    await producer.stop()
    observer.on_complete()


app = FastAPI(lifespan=lifespan)

on_sigterm() is called first — we record the moment the signal is received. on_complete() is called after all drain phases — we record the total duration.

In Prometheus you set up an alert that fires early:

# Maximum shutdown duration per service
max by (service) (app_shutdown_duration_seconds)

# Alert: shutdown approaching budget (50s of 60s)
max(app_shutdown_duration_seconds) > 50

An alert at 50s of 60s gives you time to react before applications start receiving SIGKILL.

Why Kubernetes won't tell you why SIGTERM arrived

uvicorn doesn't know the reason for the signal — that's infrastructure-level information. There can be several reasons: a rolling deploy of a new version, an HPA scale-down due to dropping load, a manual kubectl delete pod, the OOM killer, node maintenance.

In the code we record only the fact that it was received:

def on_sigterm(self) -> None:
    self._start = time.monotonic()
    logger.info("received SIGTERM, starting graceful shutdown")

You look for the reason in kubectl describe pod <pod-name>:

Events:
  Type    Reason            Age   From                    Message
  ----    ------            ---   ----                    -------
  Normal  Killing           2m    kubelet                 Stopping container order-service
  Normal  ScalingReplicaSet 10m   deployment-controller   Scaled down replica set order-service-7c8d

Don't try to determine the reason in code — that's not the application's job.

Log levels during shutdown

A common mistake is to log the normal closing of components at the ERROR level.

# Common mistake — every deploy spams alerts
ERROR - SQLAlchemy engine disposed
ERROR - aiokafka consumer stopped
ERROR - APScheduler shut down

The team gets used to ignoring these alerts, and when a real incident happens, they won't notice it right away.

The right approach: a normal shutdown is INFO.

async def _stop_kafka(consumer: AIOKafkaConsumer) -> None:
    await consumer.stop()
    logger.info("aiokafka consumer stopped")


async def _dispose_db(engine: AsyncEngine) -> None:
    await engine.dispose()
    logger.info("SQLAlchemy engine closed")

ERROR during shutdown is only for when something actually went wrong: a force-kill before transactions finish, a lost connection mid-drain, an unhandled exception in lifespan.

In short

• 60s budget: preStop 10s + uvicorn drain (up to 25s) and lifespan shutdown (up to 20s) in parallel; wall clock ≤ 35s, the rest is headroom.
• Lifespan shutdown and HTTP drain run in parallel, wall clock = max(drain, lifespan), not the sum.
• If you don't fit the budget, reduce batch sizes and operation timeouts, don't increase terminationGracePeriodSeconds.
• app_shutdown_duration_seconds — a Gauge from prometheus_client; on_sigterm() first, on_complete() after the drain.
• An alert at shutdown_duration > 50s of 60s fires before SIGKILL.
• The reason for SIGTERM is visible in kubectl describe pod, not in the application code.
• Log a normal shutdown (engine.dispose(), consumer.stop()) at INFO, not ERROR — otherwise every deploy generates false alerts.

What to read next

• uvicorn and lifespan configuration — --timeout-graceful-shutdown, the readiness flag, separate /health/live and /health/ready.
• HTTP drain in FastAPI — uvicorn graceful, preStop sleep, long endpoints via 202 Accepted.
• Kafka shutdown in Python — consumer.stop() with a timeout, manual offset commit.
• Database and persistence — engine.dispose() at the right point in the lifespan shutdown.
• Background tasks and asyncio — APScheduler, CancelledError, the draining flag in the outbox relay.