Shutdown budgets and observability in NestJS

Node Backend

How to correctly allocate the 60-second shutdown budget in NestJS: preStop, HTTP drain, Kafka, background tasks — and how to measure shutdown duration with a prom-client metric.

Published 28 июня 2026 ~5 min read Author: Вадим Викулин

← Back to the section

When Kubernetes deletes a pod, the application has a limited amount of time to finish its work and shut down. This time is called the shutdown budget. If you exceed it, Kubernetes forcibly kills the process, and some operations are cut off mid-flight.

The task is to distribute the budget across phases correctly and make it visible where exactly things got stuck when there is a problem.

Where the 60 seconds come from

terminationGracePeriodSeconds: 60 is the standard value in Kubernetes. It is the total limit from the start of pod deletion to the forced SIGKILL.

But the first 10 seconds go to the preStop delay: it is needed so that kube-proxy has time to remove the pod from the load balancer before the pod stops accepting traffic. After preStop, the process has 50 seconds left for all phases.

What a typical breakdown looks like

NestJS runs shutdown hooks sequentially, but some of the work inside a hook can be done in parallel.

T=0     SIGTERM received, NestJS starts shutdown
T=0     beforeApplicationShutdown():
        ├── readiness probe → 503 (new requests stop coming in)
        ├── kafkajs consumer.disconnect() — up to 20s  ─┐ in parallel
        └── app.close() → HTTP drain         — up to 25s ─┘
T=25s   HTTP drain and Kafka finished
T=25s   onApplicationShutdown():
        ├── BullMQ worker.close()
        └── pool.end() (PostgreSQL)
T=45s   exit(0)   ← within the 50s budget

Stage	Duration	Mechanism
preStop sleep	10s	`lifecycle.preStop` (part of the total 60s)
HTTP drain + Kafka disconnect	up to 25s	in parallel in `beforeApplicationShutdown`
BullMQ + PostgreSQL	up to 20s	`onApplicationShutdown`
Total after SIGTERM	up to 45s	within 50s

An important point: the kafkajs disconnect and HTTP drain run in parallel, so their ceiling is not the sum (20+25) but the maximum: 25 seconds.

What to do if you don't fit

The first instinct is to raise terminationGracePeriodSeconds to 90 or 120. That is a bad idea: a long shutdown increases deployment time and widens the window during which the old and new versions of the service run at the same time.

The right path is to reduce the amount of work:

Kafka: lower maxBytesPerPartition and maxWaitTimeInMs so that batches are smaller and the handler finishes faster.
BullMQ: split heavy jobs into short steps via chaining instead of one long job.
Scheduled tasks: reduce the iteration size (50 records instead of 500).

The app_shutdown_duration_seconds metric

Without a metric, investigating "why the deployment hung" boils down to manually clicking through the logs of dozens of pods. With a metric, you can see immediately: the shutdown took 47 seconds — almost the entire budget.

Implementation with prom-client:

import { Injectable, Logger, OnApplicationShutdown, BeforeApplicationShutdown } from '@nestjs/common';
import { Gauge, Registry } from 'prom-client';

@Injectable()
export class ShutdownObserverService implements BeforeApplicationShutdown, OnApplicationShutdown {
  private readonly logger = new Logger(ShutdownObserverService.name);
  private readonly shutdownDuration: Gauge<string>;
  private shutdownStartMs = 0;

  constructor(registry: Registry) {
    this.shutdownDuration = new Gauge({
      name: 'app_shutdown_duration_seconds',
      help: 'Duration of graceful shutdown in seconds',
      labelNames: ['service'],
      registers: [registry],
    });
  }

  beforeApplicationShutdown(): void {
    this.shutdownStartMs = Date.now();
    this.logger.log('received SIGTERM, starting graceful shutdown');
  }

  onApplicationShutdown(): void {
    const durationMs = Date.now() - this.shutdownStartMs;
    this.shutdownDuration.set(
      { service: process.env.SERVICE_NAME ?? 'order-service' },
      durationMs / 1000,
    );
    this.logger.log(`graceful shutdown completed in ${durationMs}ms`);
  }
}

beforeApplicationShutdown is the first hook: we capture the start moment and write the log before any cleanup. onApplicationShutdown is the last hook: we record the final gauge value once all phases are complete.

Useful Prometheus queries:

# Maximum shutdown duration per service
max by (service) (app_shutdown_duration_seconds)

# Alert: approaching the budget (more than 50s out of 60s)
max(app_shutdown_duration_seconds) > 50

Why log SIGTERM

It is important to record the very fact of receiving the signal — this is the first line in the investigation of any incident. NestJS passes the signal name into the hook:

beforeApplicationShutdown(signal: string): void {
  this.logger.log(`received ${signal}, starting graceful shutdown`);
}

The application does not know the reason for the signal (deployment, scale-down, manual pod deletion) — that is infrastructure-level information. You find it in kubectl describe pod <pod-name>: the Events section there shows who initiated the termination and why.

The right log level during shutdown

A common mistake is to log connection closures at the ERROR level:

// Bad — every deployment generates false alerts
[OrderService] ERROR - pg pool ended
[OrderService] ERROR - kafkajs consumer disconnected

Closing the pool and disconnecting Kafka are normal shutdown events. If you write them as ERROR, the team quickly gets used to ignoring alerts, and a real error goes unnoticed.

The right way:

@Injectable()
export class DatabaseModule implements OnApplicationShutdown {
  private readonly logger = new Logger(DatabaseModule.name);

  async onApplicationShutdown(): Promise<void> {
    await this.pool.end();
    this.logger.log('pg pool closed'); // INFO
  }
}

ERROR at the shutdown stage is only for when something actually broke: a forced close before a transaction finishes, an unhandled exception in a shutdown hook, a lost connection during draining.

In short

Total budget: 60s, of which 10s is preStop. After preStop, the process has 50 seconds left.
The kafkajs disconnect and HTTP drain run in parallel in beforeApplicationShutdown — there is no need to sum them.
If you don't fit, reduce the amount of work rather than increasing terminationGracePeriodSeconds.
app_shutdown_duration_seconds via a prom-client Gauge is the minimal metric for investigating deployment problems.
The log of the SIGTERM fact is written first, before cleanup, in beforeApplicationShutdown.
A normal close (pool.end(), consumer.disconnect()) is the INFO level, not ERROR.