← Back to the section

Enabling app.enableShutdownHooks() in NestJS is a necessary step, but not a sufficient one. When Kubernetes decides to terminate a pod, there is a window of several seconds between the signal and the cessation of traffic. Without the right k8s configuration, clients get 502s — not because NestJS can't shut down, but because kube-proxy doesn't manage to remove the pod from the list of available addresses in time.

This article is about three settings in the Deployment manifest that close this gap.

Why the default termination timeout is not suitable

When Kubernetes deletes a pod, it gives the process time to terminate on its own — this timeout is set by the terminationGracePeriodSeconds field. By default it equals 30 seconds.

The problem: this budget includes everything: both the kube-proxy delay (more on it below) and the application's own drain. For a NestJS service that finishes HTTP connections, disconnects the Kafka client, and closes the database connection pool, 30 seconds may not be enough — and k8s will send SIGKILL in the middle.

The correct value is 60 seconds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: order-service:2.1.0
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]

Why a preStop sleep is needed

When Kubernetes deletes a pod, two processes start in parallel:

  1. the preStop hook runs (if defined);
  2. the pod is removed from the endpoints list, and the iptables rules are updated via kube-proxy.

The problem without preStop: Kubernetes immediately sends SIGTERM to the process without waiting for the routing update. Kube-proxy updates the rules asynchronously — it takes 5–15 seconds. During all this time new requests still go to the terminating pod and get an error.

The solution is to add preStop: exec: sleep 10. Then:

T=0s   Kubernetes starts deleting the pod:
        — runs preStop (sleep 10)
        — in parallel removes the pod from endpoints
T=10s  preStop finished → Kubernetes sends SIGTERM to the process
T=10s  NestJS receives SIGTERM and starts terminating:
        — the readiness probe starts returning 503
        — the Kafka clients are closed
        — the HTTP connections finish
        — the database pool is closed
T=60s  If the process is still alive → SIGKILL

After the preStop delay the routing is already updated, new requests don't reach the pod, and NestJS can terminate calmly.

An important nuance: preStop counts toward terminationGracePeriodSeconds. With a 60-second budget and a 10-second preStop, NestJS has 50 seconds left for the drain — that is enough.

Two different probes for two different goals

Kubernetes has two kinds of pod health checks:

  • readinessProbe — Kubernetes removes the pod from endpoints if it does not pass. The pod stays alive, it just doesn't receive traffic.
  • livenessProbe — Kubernetes restarts the pod if it does not pass.

This is fundamentally different behavior, and when terminating a pod you need to use them differently.

spec:
  containers:
    - name: app
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 3000
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 2
      livenessProbe:
        httpGet:
          path: /health/live
          port: 3000
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
        initialDelaySeconds: 30

On the NestJS side, the endpoints are set up by the @nestjs/terminus library:

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly db: TypeOrmHealthIndicator,
    private readonly shutdownState: ShutdownStateService,
  ) {}

  @Get('live')
  @HealthCheck()
  liveness() {
    return this.health.check([]);
  }

  @Get('ready')
  @HealthCheck()
  readiness() {
    if (this.shutdownState.isDraining()) {
      throw new ServiceUnavailableException('draining');
    }
    return this.health.check([
      () => this.db.pingCheck('postgres'),
    ]);
  }
}

ShutdownStateService — one service that holds the termination state:

@Injectable()
export class ShutdownStateService implements BeforeApplicationShutdown {
  private draining = false;

  isDraining(): boolean {
    return this.draining;
  }

  beforeApplicationShutdown(): void {
    this.draining = true;
  }
}

How this works during termination

On receiving SIGTERM, NestJS calls beforeApplicationShutdown() — the draining flag becomes true. The next readinessProbe poll (once every 5 seconds) will get a 503, and after two failed requests (failureThreshold: 2) — a total of 10 seconds — Kubernetes finally removes the pod from endpoints.

Liveness meanwhile keeps returning 200: the pod is alive and terminating correctly, there is no need to restart it.

A common mistake: one endpoint for both checks

If /health is used for both readiness and liveness, then when the pod terminates (when it needs to return 503 for readiness) liveness will also get a 503 — and Kubernetes will restart the pod instead of terminating it correctly. This destroys the entire drain logic.

A startup delay for liveness

initialDelaySeconds: 30 on the livenessProbe gives NestJS time to start up. A DI container with TypeORM and migrations can take 10–20 seconds to start. Without the delay, liveness may fire before the application is ready and cause an endless restart loop.

Readiness meanwhile is already active from the first second — it's just that while the application is not ready, no traffic goes to it.

Rolling deploy without losing requests

By default, during an update Kubernetes may terminate the old pod before the new one becomes ready. This leads to a brief shortage of capacity and possible errors on loaded services.

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  • maxSurge: 1 — Kubernetes creates an additional pod during the update (4 instead of 3). The new pod accepts traffic before the old one terminates.
  • maxUnavailable: 0 — no pod goes out of service without a replacement.

The update sequence:

Start: 3 v1 pods

1. A v2 pod is created. Total: 3×v1 + 1×v2
2. The v2 pod passed the readinessProbe → started accepting traffic
3. One v1 pod starts terminating:
   — preStop sleep 10s
   — SIGTERM → draining = true
   — /health/ready → 503 → removed from endpoints
   — NestJS finishes connections and closes resources
   — exit 0
   Total: 2×v1 + 1×v2 (3 active)
4. A second v2 pod is created. Total: 2×v1 + 2×v2
... and so on for each pod

For a large number of replicas (50+), instead of maxSurge: 1 you use maxSurge: 25%.

Common mistakes

No preStop — in the 5–15 second window after SIGTERM, clients get 502s because kube-proxy has not yet updated the routing.

terminationGracePeriodSeconds: 30 — with a 10-second preStop and a 20-second NestJS drain, the budget runs out with no margin. Any slow Kafka client or heavy database query goes over the limit, and SIGKILL tears apart unfinished connections.

One /health for both probes — during termination liveness gets a 503 and restarts the pod instead of draining it correctly.

livenessProbe checks the database — if the database is unavailable, Kubernetes will restart the pod. But a restart won't help: the database is still unavailable. Liveness should check only that the process is alive. External dependencies go into the readinessProbe.

maxUnavailable: 1 — Kubernetes may terminate the old pod before the new one is ready. For an update without downtime you need maxUnavailable: 0.

Your own let shuttingDown = false flag instead of a single ShutdownStateService — the state spreads across several places, terminus doesn't see it and returns 200 when a 503 is needed.

In short

  • The default terminationGracePeriodSeconds: 30 is not suitable — set 60.
  • preStop: sleep 10 gives kube-proxy time to remove the pod from endpoints before SIGTERM. Without it — guaranteed 502s in the transition window.
  • preStop counts toward the terminationGracePeriodSeconds budget: with 60s and a 10s preStop, NestJS has 50s left for the drain.
  • readinessProbe and livenessProbe are different endpoints with different behavior on failure: the first removes the pod from traffic, the second restarts it.
  • On termination you need readiness 503, not liveness 503 — otherwise Kubernetes will restart the pod instead of draining it.
  • maxSurge: 1, maxUnavailable: 0 — the new pod accepts traffic before the old one terminates.
  • One ShutdownStateService with a draining flag — a single source of state for all checks.
  • HTTP drain in NestJS — how to close connections and set a force deadline.
  • NestJS configuration for graceful shutdown — enableShutdownHooks, ShutdownStateService, the force timer.
  • Kafka shutdown in NestJS — disconnecting the consumer and producer with a timeout.
  • Shutdown budget and observability — how to distribute the 60 seconds across phases.