← Back to the section

Enabling app.enableShutdownHooks() in NestJS is a necessary step, but not a sufficient one. When Kubernetes decides to terminate a pod, there is a window of several seconds between the signal and the cessation of traffic. Without the right k8s configuration, clients get 502s — not because NestJS can't shut down, but because kube-proxy doesn't manage to remove the pod from the list of available addresses in time.

This article is about three settings in the Deployment manifest that close this gap.

Why the default termination timeout is not suitable

When Kubernetes deletes a pod, it gives the process time to terminate on its own — this timeout is set by the terminationGracePeriodSeconds field. By default it equals 30 seconds.

The problem: this budget includes everything: both the kube-proxy delay (more on it below) and the application's own drain. For a NestJS service that finishes HTTP connections, disconnects the Kafka client, and closes the database connection pool, 30 seconds may not be enough — and k8s will send SIGKILL in the middle.

The correct value is 60 seconds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: order-service:2.1.0
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]

Why a preStop sleep is needed

When Kubernetes deletes a pod, two processes start in parallel:

  1. the preStop hook runs (if defined);
  2. the pod is removed from the endpoints list, and the iptables rules are updated via kube-proxy.

The problem without preStop: Kubernetes immediately sends SIGTERM to the process without waiting for the routing update. Kube-proxy updates the rules asynchronously — it takes 5–15 seconds. During all this time new requests still go to the terminating pod and get an error.

The solution is to add preStop: exec: sleep 10. Then:

T=0s Kubernetes starts deleting the pod T=0s preStop runs: sleep 10 T=0s in parallel the pod leaves the endpoints T=10s preStop is over — the process gets SIGTERM T=10s the readiness probe starts returning 503 T=10s the Kafka clients are closed T=10s open HTTP connections finish T=10s the database pool is closed T=60s still alive — SIGKILL arrives

Two things happen at once at second zero: the application still accepts requests while the routing is only being rebuilt. The preStop pause exists exactly so that SIGTERM arrives after the pod has left the routing. It costs ten seconds out of the 60-second budget — 50 are left for the drain.

After the preStop delay the routing is already updated, new requests don't reach the pod, and NestJS can terminate calmly.

An important nuance: preStop counts toward terminationGracePeriodSeconds. With a 60-second budget and a 10-second preStop, NestJS has 50 seconds left for the drain — that is enough.

Two different probes for two different goals

Kubernetes has two kinds of pod health checks:

  • readinessProbe — Kubernetes removes the pod from endpoints if it does not pass. The pod stays alive, it just doesn't receive traffic.
  • livenessProbe — Kubernetes restarts the pod if it does not pass.

This is fundamentally different behavior, and when terminating a pod you need to use them differently.

spec:
  containers:
    - name: app
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 3000
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 2
      livenessProbe:
        httpGet:
          path: /health/live
          port: 3000
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
        initialDelaySeconds: 30

On the NestJS side, the endpoints are set up by the @nestjs/terminus library:

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly db: TypeOrmHealthIndicator,
    private readonly shutdownState: ShutdownStateService,
  ) {}

  @Get('live')
  @HealthCheck()
  liveness() {
    return this.health.check([]);
  }

  @Get('ready')
  @HealthCheck()
  readiness() {
    if (this.shutdownState.isDraining()) {
      throw new ServiceUnavailableException('draining');
    }
    return this.health.check([
      () => this.db.pingCheck('postgres'),
    ]);
  }
}

ShutdownStateService — one service that holds the termination state:

@Injectable()
export class ShutdownStateService implements BeforeApplicationShutdown {
  private draining = false;

  isDraining(): boolean {
    return this.draining;
  }

  beforeApplicationShutdown(): void {
    this.draining = true;
  }
}

How this works during termination

On receiving SIGTERM, NestJS calls beforeApplicationShutdown() — the draining flag becomes true. The next readinessProbe poll (once every 5 seconds) will get a 503, and after two failed requests (failureThreshold: 2) — a total of 10 seconds — Kubernetes finally removes the pod from endpoints.

Liveness meanwhile keeps returning 200: the pod is alive and terminating correctly, there is no need to restart it.

A common mistake: one endpoint for both checks

If /health is used for both readiness and liveness, then when the pod terminates (when it needs to return 503 for readiness) liveness will also get a 503 — and Kubernetes will restart the pod instead of terminating it correctly. This destroys the entire drain logic.

A startup delay for liveness

initialDelaySeconds: 30 on the livenessProbe gives NestJS time to start up. A DI container with TypeORM and migrations can take 10–20 seconds to start. Without the delay, liveness may fire before the application is ready and cause an endless restart loop.

Readiness meanwhile is already active from the first second — it's just that while the application is not ready, no traffic goes to it.

Rolling deploy without losing requests

By default, during an update Kubernetes may terminate the old pod before the new one becomes ready. This leads to a brief shortage of capacity and possible errors on loaded services.

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  • maxSurge: 1 — Kubernetes creates an additional pod during the update (4 instead of 3). The new pod accepts traffic before the old one terminates.
  • maxUnavailable: 0 — no pod goes out of service without a replacement.

The update sequence:

StepWhat happensPods serving traffic
startthree v1 pods serve traffic3×v1
1a v2 pod is created3×v1 + 1×v2
2the v2 pod passed readinessProbe and started accepting traffic3×v1 + 1×v2
3one v1 pod terminates: preStop 10s → SIGTERM → draining = true → /health/ready returns 503 → removed from the endpoints → connections finish → exit 02×v1 + 1×v2, three active
4a second v2 pod is created2×v1 + 2×v2
onwardsthe same for every remaining podnever fewer than three active

For a large number of replicas (50+), instead of maxSurge: 1 you use maxSurge: 25%.

Common mistakes

No preStop — in the 5–15 second window after SIGTERM, clients get 502s because kube-proxy has not yet updated the routing.

terminationGracePeriodSeconds: 30 — with a 10-second preStop and a 20-second NestJS drain, the budget runs out with no margin. Any slow Kafka client or heavy database query goes over the limit, and SIGKILL tears apart unfinished connections.

One /health for both probes — during termination liveness gets a 503 and restarts the pod instead of draining it correctly.

livenessProbe checks the database — if the database is unavailable, Kubernetes will restart the pod. But a restart won't help: the database is still unavailable. Liveness should check only that the process is alive. External dependencies go into the readinessProbe.

maxUnavailable: 1 — Kubernetes may terminate the old pod before the new one is ready. For an update without downtime you need maxUnavailable: 0.

Your own let shuttingDown = false flag instead of a single ShutdownStateService — the state spreads across several places, terminus doesn't see it and returns 200 when a 503 is needed.

In short

  • The default terminationGracePeriodSeconds: 30 is not suitable — set 60.
  • preStop: sleep 10 gives kube-proxy time to remove the pod from endpoints before SIGTERM. Without it — guaranteed 502s in the transition window.
  • preStop counts toward the terminationGracePeriodSeconds budget: with 60s and a 10s preStop, NestJS has 50s left for the drain.
  • readinessProbe and livenessProbe are different endpoints with different behavior on failure: the first removes the pod from traffic, the second restarts it.
  • On termination you need readiness 503, not liveness 503 — otherwise Kubernetes will restart the pod instead of draining it.
  • maxSurge: 1, maxUnavailable: 0 — the new pod accepts traffic before the old one terminates.
  • One ShutdownStateService with a draining flag — a single source of state for all checks.