When a service receives SIGTERM, uvicorn gives active requests a chance to finish — but only within the timeout (usually 60 seconds). If a long chain of calls doesn't fit, asyncio.CancelledError interrupts it midway. The operation started but didn't reach the end — and on restart the service will try to repeat it.
This is safe only if the operation is idempotent: a repeated call with the same data gives the same result without creating duplicates. That's exactly what this article is about.
What idempotency is and why it matters
Imagine a payments API: the client sent a charge request but didn't get a response — the connection dropped. What to do? Try again? And what if the first request actually went through?
If the payment service is not idempotent, a repeated request creates a second charge. The client pays twice.
Idempotency solves this: if you pass a unique operation key (Idempotency-Key), the server remembers the result of the first call and on repeat simply returns the same response — without performing the action again.
During graceful shutdown it's the same problem: SIGTERM interrupts the request at an arbitrary moment. A new pod gets the same request again. Without idempotency — a duplicate.
HTTP POST with Idempotency-Key in FastAPI
The client sends an Idempotency-Key header — one unique UUID per operation. Even if it repeats the request ten times, the server returns the stored result without performing the action again.
from fastapi import APIRouter, Header
from uuid import UUID
router = APIRouter()
@router.post("/payments", status_code=201)
async def charge_order(
order_id: UUID,
amount: int,
idempotency_key: str = Header(..., alias="Idempotency-Key"),
service: PaymentService = Depends(get_payment_service),
) -> PaymentReceipt:
return await service.charge(order_id, amount, idempotency_key)
PaymentService.charge is arranged like this: first it looks up a record by idempotency_key in the database. If found, it returns the stored response without charging again. If not, it performs the charge and stores the result atomically in a single transaction.
The flow:
Client → POST /payments Idempotency-Key: pay-abc
FastAPI: creates a record, processes it
[SIGTERM, uvicorn doesn't make it — timeout]
Client → retry: POST /payments Idempotency-Key: pay-abc
FastAPI (new pod): finds the record, returns the stored response
No duplicate
Kafka listener with protection against reprocessing
When working with Kafka in manual mode (manual commit), the offset is confirmed explicitly after the event is processed. The risk: if a CancelledError arrives after the side effect but before commit_offsets, the service will get the same event again on restart.
The protection is a processed_event table: a record that the event has already been processed is stored in the same transaction as the action itself. On repeat, a unique-key conflict is detected and processing is skipped.
from aiokafka import AIOKafkaConsumer, TopicPartition
from sqlalchemy.ext.asyncio import AsyncSession
async def handle_order_confirmed(
event: OrderConfirmedEvent,
consumer: AIOKafkaConsumer,
tp: TopicPartition,
offset: int,
session: AsyncSession,
) -> None:
async with session.begin():
already = await session.execute(
select(ProcessedEvent).where(
ProcessedEvent.event_id == event.event_id,
ProcessedEvent.context == "billing",
)
)
if already.scalar_one_or_none():
await consumer.commit({tp: offset + 1})
return
session.add(ProcessedEvent(event_id=event.event_id, context="billing"))
await billing_service.charge(
order_id=event.order_id,
amount=event.total_amount,
idempotency_key=str(event.event_id),
)
await consumer.commit({tp: offset + 1})
An important detail: billing_service.charge also receives an idempotency_key and passes it into the httpx request to the payment provider. This is double protection: both at the Kafka level and at the downstream-call level.
If the transaction rolls back due to CancelledError, the offset isn't confirmed either, and on restart the event will arrive again. A repeated insert into processed_event will fail on the unique constraint — processing is safely skipped.
Outbox relay with a two-phase status
The outbox pattern is used for reliable event publication to Kafka: events are first written to a database table, and a separate relay process reads them from there and sends them.
The problem: if SIGTERM interrupts the relay between sending to Kafka and marking the row as PUBLISHED, on restart the same row will be sent again.
The solution is an intermediate PUBLISHING status:
from sqlalchemy import update
from datetime import datetime, timezone
async def relay_batch(session: AsyncSession, producer: AIOKafkaProducer) -> int:
now = datetime.now(timezone.utc)
result = await session.execute(
update(OutboxEvent)
.where(OutboxEvent.status == "PENDING")
.values(status="PUBLISHING", locked_at=now)
.returning(OutboxEvent.id, OutboxEvent.payload, OutboxEvent.topic)
.limit(50)
)
rows = result.fetchall()
if not rows:
return 0
for row_id, payload, topic in rows:
await producer.send_and_wait(topic, value=payload)
await session.execute(
update(OutboxEvent)
.where(OutboxEvent.id == row_id)
.values(status="PUBLISHED", published_at=now)
)
await session.commit()
return len(rows)
If SIGTERM happens between send_and_wait and the UPDATE, the row hangs in the PUBLISHING status. A separate background task returns such rows back to PENDING after a few minutes. Kafka will receive a repeated send — but the consumer-side processed_event protects against duplication.
The relay loop checks the readiness flag instead of spinning forever:
async def outbox_loop(app_state: AppState) -> None:
while app_state.is_ready:
sent = await relay_batch(session, producer)
if sent == 0:
await asyncio.sleep(1.0)
A common mistake: httpx retry without an Idempotency-Key
# the problematic version
async def charge(order_id: UUID, amount: int) -> dict:
async with httpx.AsyncClient() as client:
for attempt in range(3):
try:
resp = await client.post(
f"{PAYMENT_URL}/charge",
json={"order_id": str(order_id), "amount": amount},
timeout=10.0,
)
resp.raise_for_status()
return resp.json()
except httpx.TransportError:
if attempt == 2:
raise
await asyncio.sleep(0.5)
On SIGTERM during the first attempt: the request went out, the response wasn't received. The retry creates a second request. The provider processes both — a double charge.
The correct way — the Idempotency-Key is generated once before the loop and passed in all attempts:
async def charge(order_id: UUID, amount: int, idempotency_key: str) -> dict:
async with httpx.AsyncClient() as client:
for attempt in range(3):
try:
resp = await client.post(
f"{PAYMENT_URL}/charge",
json={"order_id": str(order_id), "amount": amount},
headers={"Idempotency-Key": idempotency_key},
timeout=10.0,
)
resp.raise_for_status()
return resp.json()
except httpx.TransportError:
if attempt == 2:
raise
await asyncio.sleep(0.5)
CancelledError inside a transaction
asyncio.CancelledError on shutdown can arrive at any await. If it happens inside async with session.begin() after a side effect but before the commit, SQLAlchemy performs a rollback. This is correct: the offset isn't confirmed either, so a retry is safe.
If the code catches CancelledError, you need to either re-raise it or explicitly roll back the transaction:
async def handle_customer_merge(event: CustomerMergeEvent, session: AsyncSession) -> None:
try:
async with session.begin():
session.add(ProcessedEvent(event_id=event.event_id, context="crm"))
await crm_service.merge(event.source_id, event.target_id)
except asyncio.CancelledError:
# the rollback is done by the context manager; re-raise
raise
The main rule: except asyncio.CancelledError: pass inside a transaction is always a mistake.
In short
- Idempotency — a repeated call with the same data gives the same result without duplicates. During graceful shutdown this is a required property for any operation that SIGTERM can interrupt.
- HTTP POST — an
Idempotency-Keyin the header; FastAPI reads it viaHeader(...), and the service stores the result atomically in a single transaction. - Kafka listener —
processed_eventand the side effect are committed in one transaction; the offset is confirmed only after the commit. On rollback — a retry is safe. - Outbox relay — a two-phase status
PENDING → PUBLISHING → PUBLISHED; stuck rows are returned toPENDINGby a background task. - httpx retry — the
Idempotency-Keyis generated once before the retry loop and passed in all attempts. CancelledErrorinside a transaction should be re-raised, not suppressed — otherwise the rollback won't happen.
What to read next
- Kafka: correct consumer and producer shutdown
- Scheduled tasks and the outbox relay
- Database and persistence on shutdown
- HTTP drain and uvicorn timeouts