← Back to the section

When Kubernetes stops a container, it sends the process a SIGTERM signal. If the application did not handle this signal, it dies immediately, tearing apart all active HTTP requests. The client gets a 502 Bad Gateway.

Graceful shutdown is the application's ability to correctly finish already-started requests before exiting. In NestJS this does not work on its own: you need to explicitly enable several mechanisms.

Why the process dies instantly

By default, on receiving SIGTERM Node.js simply calls process.exit(). NestJS does not intercept this signal and does not run any shutdown hooks.

For NestJS to start listening to operating-system signals, you need to call app.enableShutdownHooks() before starting the server:

// main.ts
async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  app.enableShutdownHooks(); // without this line — instant exit

  await app.listen(3000);
}
bootstrap();

After this, on receiving SIGTERM, SIGINT, or SIGHUP, Nest will call app.close(), which goes through the lifecycle phases: first beforeApplicationShutdown on all modules, then server.close(), then onApplicationShutdown.

Why server.close() is not enough

server.close() stops accepting new connections and waits for the current ones to close. That sounds right — but there is a problem: if a client holds a keep-alive connection open, server.close() will wait forever.

In Kubernetes the container must terminate within terminationGracePeriodSeconds (usually 60 seconds). If the drain hangs, Kubernetes will forcibly kill the process with SIGKILL, and active requests will be interrupted anyway.

The solution is to wrap the drain in a Promise.race with a timeout:

// main.ts
async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.enableShutdownHooks();

  const server = app.getHttpServer();

  const originalClose = app.close.bind(app);
  app.close = async () => {
    // Node >= 18.2: immediately close empty keep-alive connections
    server.closeIdleConnections();

    await Promise.race([
      originalClose(),
      new Promise<void>((resolve) => {
        const t = setTimeout(resolve, 30_000); // 30 seconds — a working balance
        t.unref(); // does not block the event loop from finishing
      }),
    ]);
  };

  await app.listen(3000);
}

A few notes on the timeout:

  • Less than 20 seconds is too little for a loaded service: requests with an execution time of 5+ seconds will be interrupted.
  • More than 45 seconds is risky: it won't fit within the standard 60-second Kubernetes budget.
  • 30 seconds is a working balance for ordinary REST services.

closeIdleConnections() appeared in Node.js 18.2 and immediately closes keep-alive connections with no active requests. Without it the drain drags on: server.close() waits for the client to close the empty socket itself.

How Kubernetes learns that the pod is leaving

The order matters when stopping a pod: Kubernetes must remove the pod from load balancing before the application stops accepting requests. Otherwise part of the traffic will arrive at an already-terminating instance.

Kubernetes uses a readiness probe — a periodic request to /health/ready. If it returns 503, the pod is excluded from the list of active endpoints.

The algorithm:

  1. The pod receives SIGTERM.
  2. The application immediately switches readiness to "not ready".
  3. Kubernetes sees a 503 on /health/ready and removes the pod from load balancing.
  4. New traffic stops arriving at this pod.
  5. The application finishes the already-started requests.
  6. The process terminates.

For this you need a single source of the drain state — ShutdownStateService:

// shutdown-state.service.ts
import { Injectable, BeforeApplicationShutdown, Logger } from '@nestjs/common';

@Injectable()
export class ShutdownStateService implements BeforeApplicationShutdown {
  private readonly logger = new Logger(ShutdownStateService.name);
  private draining = false;

  isDraining(): boolean {
    return this.draining;
  }

  beforeApplicationShutdown(signal: string): void {
    this.logger.log(`Received ${signal}, starting graceful shutdown`);
    this.draining = true;
    // from this moment /health/ready will return 503
  }
}

BeforeApplicationShutdown is a NestJS interface. The beforeApplicationShutdown method is called first on app.close(), even before server.close(). This is exactly where you need to switch the state, while the HTTP server still accepts requests from the probe.

Wiring up the readiness probe with Terminus

@nestjs/terminus is the official library for health checks in NestJS. It provides HealthCheckService and the @HealthCheck decorator.

We wire ShutdownStateService into the module and the controller:

// health.module.ts
import { Module } from '@nestjs/common';
import { TerminusModule } from '@nestjs/terminus';
import { HealthController } from './health.controller';
import { ShutdownStateService } from '../shutdown/shutdown-state.service';

@Module({
  imports: [TerminusModule],
  controllers: [HealthController],
  providers: [ShutdownStateService],
  exports: [ShutdownStateService],
})
export class HealthModule {}

// health.controller.ts
import { Controller, Get } from '@nestjs/common';
import { HealthCheck, HealthCheckService, HealthCheckResult, HealthCheckError } from '@nestjs/terminus';
import { ShutdownStateService } from '../shutdown/shutdown-state.service';

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly shutdownState: ShutdownStateService,
  ) {}

  @Get('live')
  @HealthCheck()
  liveness(): Promise<HealthCheckResult> {
    return this.health.check([]); // process is alive — always 200
  }

  @Get('ready')
  @HealthCheck()
  readiness(): Promise<HealthCheckResult> {
    return this.health.check([
      () =>
        this.shutdownState.isDraining()
          ? Promise.reject(new HealthCheckError('draining', { readiness: { status: 'down' } }))
          : Promise.resolve({ readiness: { status: 'up' } }),
    ]);
  }
}

Two different endpoints is not a coincidence. Liveness and readiness mean different things:

  • /health/live — the process is running and not hung. Kubernetes restarts the pod if liveness returns an error. You must not switch it to 503 during draining — Kubernetes will treat it as a failure and restart the pod, killing everything unfinished.
  • /health/ready — the pod is ready to accept traffic. Kubernetes removes the pod from load balancing on a 503 but does not restart it. This is exactly the endpoint that ShutdownStateService switches.

Kubernetes configuration:

# fragment of deployment.yaml
livenessProbe:
  httpGet:
    path: /health/live
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 5

Common mistakes

A shuttingDown flag right in the service. A local variable does not integrate with the NestJS lifecycle hooks. Terminus does not know about it, the readiness probe won't switch, and Kubernetes won't remove the pod from load balancing.

pool.end() in beforeApplicationShutdown. The database connection must be closed only after the HTTP drain — in onApplicationShutdown. If you close the pool earlier, requests that are still being processed will lose the database connection and finish with an error.

process.exit(0) in your own SIGTERM handler. This bypasses the entire NestJS lifecycle. You don't need to call process.exit yourself — enableShutdownHooks does it correctly and at the right moment.

In short

  • app.enableShutdownHooks() is mandatory: without it NestJS does not intercept SIGTERM and the lifecycle hooks do not run.
  • server.close() waits forever with open keep-alive connections — you need a Promise.race with a 30-second timeout.
  • closeIdleConnections() (Node ≥ 18.2) speeds up the drain by immediately freeing empty keep-alive sockets.
  • ShutdownStateService implements BeforeApplicationShutdown and switches readiness to 503 as the first thing on SIGTERM.
  • readiness=503 tells Kubernetes to remove the pod from load balancing; liveness=503 signals a failure and triggers a restart — they must not be confused.
  • The database connection is closed in onApplicationShutdown, after the HTTP drain finishes, not earlier.
  • HTTP drain in NestJS — what happens to active requests, closeIdleConnections, keep-alive.
  • Kubernetes and graceful shutdown — preStop sleep, terminationGracePeriodSeconds, maxUnavailable.
  • Closing the database — pool.end() in the right phase.
  • Background tasks and Kafka — consumer.disconnect(), producer.disconnect() with a timeout.