Kubernetes, load balancers and monitoring constantly ask every service: "are you alive?" and "are you ready to accept requests?". If the service answers incorrectly, traffic goes the wrong way, pods start restarting, and one small database problem turns into a full service outage.
Let's work out how to set these checks up correctly.
Why you need two different endpoints
It seems logical: there's a single /health, it answers "ok" or "not ok". But this creates a serious problem.
Kubernetes uses two different kinds of checks:
- Liveness probe — checks whether the process is alive. If not, Kubernetes kills the pod and starts a new one.
- Readiness probe — checks whether the service is ready to accept traffic. If not, it removes the pod from the load balancer, but doesn't restart it.
If you merge both checks into one and add a database check there, you get a catastrophe: the database slowed down for 30 seconds because of scheduled maintenance → liveness returns an error → Kubernetes kills all the pods → new pods start with the same problematic database → they fail again → loop. The service is completely unavailable because of a temporary lag in the DB.
The right solution: split them.
from fastapi import FastAPI
app = FastAPI()
@app.get("/health/live")
async def liveness() -> dict:
return {"status": "UP"}
@app.get("/health/ready")
async def readiness() -> dict:
checks = await run_readiness_checks()
status = "UP" if all(c["status"] == "UP" for c in checks) else "DOWN"
return {"status": status, "checks": checks}
| Endpoint | What it checks | What Kubernetes does |
|---|---|---|
/health/live | The process responds, the event loop isn't stuck | UP → keep going; DOWN → restarts the pod |
/health/ready | The DB is reachable, dependencies are ready | UP → sends traffic; DOWN → removes from load balancing |
Semantics: if the database is unreachable for 5 seconds — readiness should become DOWN (traffic moves to other replicas), while liveness should stay UP (restarting the pod won't help, it's the same database).
Liveness without external dependencies
Liveness should check only the process itself — whether the event loop is alive, whether the application responds. No external systems.
Here's an example of how not to do it:
# Dangerous: liveness will return 503 on any DB unavailability
@app.get("/health/live")
async def liveness_bad():
result = await check_postgres(postgres_pool)
if result["status"] != "UP":
raise HTTPException(status_code=503, detail=result)
return {"status": "UP"}
With this approach a temporary Postgres lag will trigger a restart of all the pods. The correct way is to check only the application itself:
@app.get("/health/live")
async def liveness() -> dict:
return {"status": "UP"}
Checking external systems with a cache
Readiness checks the real dependencies: the database, Redis, external APIs. But there's another trap here: Kubernetes polls readiness every 5 seconds, and if you have 10 replicas — that's 120 requests per minute just from the health check. For expensive external checks that's a lot.
The solution is a TTL cache: the check result is stored for a few seconds, and repeated requests take it from the cache.
import asyncio
import time
from dataclasses import dataclass
from typing import Optional
import httpx
@dataclass
class CachedCheck:
result: dict
expires_at: float
_payment_cache: Optional[CachedCheck] = None
_payment_lock = asyncio.Lock()
TTL_SECONDS = 10
async def check_payment_api() -> dict:
global _payment_cache
now = time.monotonic()
if _payment_cache and _payment_cache.expires_at > now:
return _payment_cache.result
async with _payment_lock:
if _payment_cache and _payment_cache.expires_at > now:
return _payment_cache.result
result = await _do_check_payment()
_payment_cache = CachedCheck(result=result, expires_at=now + TTL_SECONDS)
return result
async def _do_check_payment() -> dict:
try:
async with httpx.AsyncClient(timeout=2.0) as client:
resp = await client.get("https://api.payment-provider.ru/ping")
resp.raise_for_status()
return {"status": "UP", "component": "payment"}
except Exception as exc:
return {"status": "DOWN", "component": "payment", "error": str(exc)}
The double-check inside asyncio.Lock guards against a race condition: if several coroutines simultaneously discovered a stale cache, only one goes to the external API, the rest wait and take the fresh result.
Checks for PostgreSQL and Redis are simpler — they already have native ping methods:
async def check_postgres(pool: asyncpg.Pool) -> dict:
try:
async with pool.acquire(timeout=1.0) as conn:
await conn.fetchval("SELECT 1")
return {"status": "UP", "component": "postgres"}
except Exception as exc:
return {"status": "DOWN", "component": "postgres", "error": str(exc)}
async def check_redis(redis_client) -> dict:
try:
await redis_client.ping()
return {"status": "UP", "component": "redis"}
except Exception as exc:
return {"status": "DOWN", "component": "redis", "error": str(exc)}
Let's assemble the final readiness endpoint:
from fastapi import APIRouter, Response
health_router = APIRouter()
@health_router.get("/health/ready")
async def readiness(response: Response) -> dict:
checks = await asyncio.gather(
check_postgres(postgres_pool),
check_redis(redis_client),
check_payment_api(),
)
checks_list = list(checks)
all_up = all(c["status"] == "UP" for c in checks_list)
if not all_up:
response.status_code = 503
return {
"status": "UP" if all_up else "DOWN",
"checks": checks_list,
}
An HTTP 503 status on DOWN is mandatory: Kubernetes reads exactly the response code, not the body content.
The /info endpoint with version and git commit
During an incident the first question is "which version of the service is in production right now?". If there's no dedicated endpoint, you'll have to dig into CI logs and match deployment dates.
The solution: at build time write metadata into a file, and FastAPI serves it on request.
In the CI pipeline:
# .github/workflows/build.yml
- name: Write build info
run: |
cat > app/build_info.json << EOF
{
"git_commit": "${{ github.sha }}",
"git_branch": "${{ github.ref_name }}",
"build_time": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"version": "${{ github.run_number }}"
}
EOF
In the application:
import json
from pathlib import Path
from fastapi import APIRouter
info_router = APIRouter()
def _load_build_info() -> dict:
path = Path(__file__).parent / "build_info.json"
if path.exists():
return json.loads(path.read_text())
return {"git_commit": "dev", "version": "local"}
_BUILD_INFO = _load_build_info()
@info_router.get("/info")
async def info() -> dict:
return {
"service": {"name": "order-service"},
"build": _BUILD_INFO,
}
The response looks like this:
{
"service": {"name": "order-service"},
"build": {
"git_commit": "5380f21abc...",
"git_branch": "main",
"build_time": "2026-06-18T14:30:00Z",
"version": "142"
}
}
A separate port for management endpoints
It's better to keep the health endpoints on a separate port — so they aren't reachable through the public load balancer and don't interfere with the main traffic.
import asyncio
import uvicorn
from fastapi import FastAPI
management_app = FastAPI(docs_url=None, redoc_url=None)
management_app.include_router(health_router)
management_app.include_router(info_router)
async def serve_both():
config_business = uvicorn.Config(app, host="0.0.0.0", port=8080)
config_management = uvicorn.Config(management_app, host="0.0.0.0", port=9090)
await asyncio.gather(
uvicorn.Server(config_business).serve(),
uvicorn.Server(config_management).serve(),
)
asyncio.run(serve_both())
In the Kubernetes manifest we specify port 9090 for the probes:
spec:
containers:
- name: order-service
livenessProbe:
httpGet:
path: /health/live
port: 9090
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /health/ready
port: 9090
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2
Common mistakes
Business logic in the health check. If you write if order_count > N: status = "DOWN" — Kubernetes will start restarting the service because of business state. A health check should reflect only the technical state: the process is alive, the database is reachable.
Heavy operations as a probe. Creating a test order or running a full query as part of a probe is a bad idea. A probe should be lightweight: ping, SELECT 1, a simple GET request.
Readiness always 200, but DOWN in the body. Kubernetes looks at the HTTP status, not the body. If readiness returns 200 on DOWN, Kubernetes thinks everything is fine and keeps sending traffic.
Checks without a cache. Without a TTL cache every Kubernetes poll goes straight to the external system. With frequent polling and several replicas that's a noticeable load.
In short
/health/liveand/health/readyare two different endpoints with different semantics. You can't merge them.- Liveness checks only the process itself. External systems aren't included there.
- Readiness checks all dependencies: DB, Redis, external APIs.
- For external API checks use a TTL cache with
asyncio.Lock— otherwise the health check creates extra load. - HTTP 503 on DOWN in readiness is mandatory — Kubernetes reads the status, not the body.
/infowith the git commit and build version greatly simplifies incident analysis.- Management endpoints are better kept on a separate port.
What to read next
- Logging — structlog, contextvars and structured logging
- Metrics — prometheus-client and business metrics
- Tracing — OpenTelemetry and manual spans