HTTP drain in NestJS: how not to lose requests on restart

← Back to the section

The service restarts on every deployment. If some user was making a request at that moment, they will get a 502 error. It looks like a random glitch, but in reality it is a predictable and solvable problem.

Let's look at why this happens and how to configure NestJS so that the service correctly finishes current requests before stopping.

Why 502s appear during a deployment

When Kubernetes decides to stop a pod (for example, during an update), it sends the process a SIGTERM signal. By default, a Node.js process terminates immediately — together with all unfinished requests. A client that was waiting for a response gets a broken connection — the browser or API client shows a 502.

Graceful shutdown is the service's ability to wait for current requests to finish before turning off. Instead of "die immediately" — "finish what you started, then turn off".

For this you need to:

1. Configure NestJS to listen for SIGTERM.
2. On receiving the signal, stop accepting new requests but wait for the current ones to finish.
3. Add a small delay in Kubernetes so that traffic has time to switch to other pods.

Step 1: enable SIGTERM handling in NestJS

By default, NestJS does not listen for SIGTERM. One line changes this:

// main.ts
async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.enableShutdownHooks(); // without this SIGTERM = instant death
  await app.listen(3000);
}
bootstrap();

enableShutdownHooks() subscribes NestJS to SIGTERM and runs the lifecycle hooks when the signal is received. Don't add a manual process.once('SIGTERM', ...) alongside it — two handlers will create a race.

Step 2: close the HTTP server correctly

When NestJS has received SIGTERM, it calls app.close(), which stops the HTTP server. But there is a subtlety here.

server.close() stops accepting new connections but waits for active requests to finish — with no time limit. If some request hangs, the server will wait forever.

closeIdleConnections() (Node.js 18.2+) closes empty keep-alive connections. Without this, browsers and HTTP clients that hold a connection open "for the future" will block the server from shutting down.

Let's implement both with a time limit:

// http-drain.service.ts
import { Injectable, BeforeApplicationShutdown } from '@nestjs/common';
import { HttpAdapterHost } from '@nestjs/core';
import * as http from 'http';

@Injectable()
export class HttpDrainService implements BeforeApplicationShutdown {
  private readonly DRAIN_TIMEOUT_MS = 25_000;

  constructor(private readonly httpAdapterHost: HttpAdapterHost) {}

  async beforeApplicationShutdown(): Promise<void> {
    const server: http.Server = this.httpAdapterHost.httpAdapter.getHttpServer();
    server.closeIdleConnections();

    await Promise.race([
      new Promise<void>((resolve) => server.close(() => resolve())),
      new Promise<void>((resolve) =>
        setTimeout(resolve, this.DRAIN_TIMEOUT_MS).unref(),
      ),
    ]);
  }
}

Here Promise.race picks whichever happens first: all requests finished, or 25 seconds passed. .unref() means the timer does not keep the process alive — if the requests finish earlier, Node.js exits calmly.

Step 3: tell Kubernetes the service is preparing to stop

On shutdown you need to quickly remove the pod from rotation — so that Kubernetes stops routing new requests to it. A readiness probe is used for this.

We create a service that tracks the state:

@Injectable()
export class ShutdownStateService implements BeforeApplicationShutdown {
  private draining = false;

  isDraining(): boolean {
    return this.draining;
  }

  beforeApplicationShutdown(): void {
    this.draining = true;
  }
}

And use it in the health endpoint:

@Controller('health')
export class HealthController {
  constructor(
    private readonly health: HealthCheckService,
    private readonly shutdownState: ShutdownStateService,
  ) {}

  @Get('ready')
  @HealthCheck()
  readiness() {
    return this.health.check([
      () =>
        this.shutdownState.isDraining()
          ? Promise.reject(new Error('draining'))
          : { ready: { status: 'up' } },
    ]);
  }

  @Get('live')
  @HealthCheck()
  liveness() {
    return this.health.check([]);
  }
}

As soon as NestJS receives SIGTERM, draining becomes true, and /health/ready starts returning 503. Kubernetes sees this and removes the pod from the list of active ones. New requests no longer arrive.

It is important to have two separate endpoints: /health/live (is the process alive) and /health/ready (is it ready to accept requests). If the service hangs, we restart it. If it is preparing to stop, we remove it from routing but do not restart it.

Step 4: preStop sleep in Kubernetes

Even with a correctly configured NestJS there is a problem at the Kubernetes level.

When a pod is deleted, Kubernetes does two things at the same time:

• Sends the process SIGTERM.
• Removes the pod from the Service endpoints.

But updating iptables on other nodes is an asynchronous operation. It takes several seconds. During this interval, other pods may still send traffic to the stopped pod — and get errors.

The solution is a preStop hook. It is a delay before SIGTERM is sent:

spec:
  containers:
    - name: order-service
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
  terminationGracePeriodSeconds: 60

What happens with preStop:

T=0    Kubernetes decided to delete the pod
T=0+   preStop starts (sleep 10)
       Kubernetes begins removing the pod from endpoints and updating iptables
T=10s  preStop finished — now SIGTERM is sent
T=10s+ NestJS begins graceful shutdown
       By this point iptables on other nodes are already updated

10 seconds is enough for most clusters: the kube-proxy update takes 1–5 seconds, the load balancer update — another 1–3 seconds. During the preStop time the service keeps working normally — SIGTERM has not arrived yet.

Long requests

Sometimes an endpoint runs 30–40 seconds: a data export, complex processing. With the usual approach such a request will simply be cut off during a deployment — the force timer expires earlier.

The solution: don't make long synchronous requests. Instead of "wait 40 seconds" — "get a job and check the status":

@Controller('orders')
export class OrderController {
  constructor(private readonly dispatcher: OrderUseCaseDispatcher) {}

  @Post('export')
  async requestExport(
    @Headers('Idempotency-Key') idempotencyKey: string,
    @Body() dto: ExportOrdersDto,
  ): Promise<ExportRequestResponse> {
    const jobId = await this.dispatcher.dispatch(
      new RequestOrderExportCommand(idempotencyKey, dto),
    );
    return { jobId, status: 'QUEUED' }; // returns in <100ms
  }

  @Get('export/:jobId')
  async getExport(@Param('jobId') jobId: string): Promise<ExportStatusResponse> {
    return this.dispatcher.dispatch(new GetOrderExportQuery(jobId));
  }
}

POST /orders/export queues the job and immediately returns an ID. The client periodically asks GET /orders/export/:jobId and gets the status. During a deployment the short POST finishes in a fraction of a second — the drain is not blocked.

The Idempotency-Key header lets the client safely retry the request if the connection was interrupted.

If an endpoint is slow not because of architecture but because of N+1 queries to the database — this is worth fixing: an optimized query usually fits within 2–3 seconds and does not create problems.

Common mistakes

process.exit(0) in the SIGTERM handler. This kills the process immediately and tears active connections apart. Use app.close() — it will run the lifecycle hooks and wait for completion.

server.closeAllConnections() at the start of the drain. This tears apart all connections, including active requests. Use closeIdleConnections() — it closes only empty keep-alive connections.

No preStop sleep. Traffic keeps going to the pod for several more seconds after SIGTERM, until iptables updates. Without the delay — guaranteed 502s on every deployment.

No app.enableShutdownHooks(). Without it NestJS does not listen for SIGTERM at all. The process dies instantly.

In short

• During a deployment Kubernetes sends SIGTERM — without configuration the process dies immediately, active requests are cut off.
• app.enableShutdownHooks() in main.ts is mandatory; without it SIGTERM is ignored.
• server.close() waits for active requests to finish, closeIdleConnections() frees empty keep-alive sockets.
• Set a force timer (25 seconds) — otherwise server.close() may wait forever.
• ShutdownStateService + /health/ready → 503 immediately on receiving SIGTERM, so that Kubernetes removes the pod from rotation.
• preStop sleep 10 in the manifest is mandatory: it gives kube-proxy time to update iptables before SIGTERM.
• Convert long synchronous requests (>10 seconds) to the 202 Accepted + polling scheme.

What to read next

• Budgets and observability — how to compute the budget by phases and add shutdown-time metrics.
• Database and persistence — in what order to close the connection pool.
• Background tasks and outbox — how to correctly stop BullMQ workers.
• Kafka shutdown — disconnect with a timeout and commit semantics.