Shutdown budgets and graceful shutdown observability

← Back to the section

When graceful shutdown took too long and the pod was force-killed — how do you figure out what exactly did not finish in time? Without metrics and logs you can only guess. Let's work through how to calculate a realistic time budget and what to add to the code so you notice problems before an incident.

Where the 60 seconds come from

When deleting a pod, Kubernetes waits at most terminationGracePeriodSeconds seconds and then kills the process. The default value is 60 seconds. What manages to happen in that time?

Shutdown consists of several steps:

Add up the maximums and you get 70 seconds, which is more than the 60-second budget. In practice this is not a problem: Spring runs the HTTP drain, the scheduler and Kafka in parallel, not one after another. The real wall clock is 25–40 seconds. A 60-second budget leaves headroom for slower shutdowns under high load.

Parallel execution diagram:

T=0     SIGTERM
T=0     Spring publishes ContextClosedEvent
T=0     In parallel, the following start:
        ├── Kafka listener.stop() (up to 15s)
        ├── Scheduler shutdown() (up to 20s)
        └── Tomcat graceful drain (up to 25s)
T=25s   All three phases are done
T=25s   ApplicationContext.close() — closing DataSource and the rest
T=30s   Process terminated

What to do if you do not fit

The first instinct is to raise terminationGracePeriodSeconds to 90 or 120 seconds. That is a mistake.

A long shutdown lengthens the rolling deploy. While the old pod is still running, the new one is already taking traffic: both versions of the code work against the same database. The longer this window, the higher the chance of schema-compatibility problems. On top of that, kubectl drain waits 30 seconds by default: if a pod does not finish in time, the drain hangs.

The right path is to reduce the amount of work, not increase the budget:

• Kafka max.poll.records: 500 → 100: the listener finishes processing in 5 seconds instead of 25.
• @Async tasks with long chains → break them into short steps.
• Heavy @Scheduled tasks → shrink the batch size (50 records instead of 500).

Shutdown duration metric

Without a metric, the only way to know how long a shutdown took is to scroll through the logs of every pod by hand. With a metric it is immediately visible on a chart.

We add a gauge that gets updated throughout the whole shutdown:

@Component
@Slf4j
public class ShutdownObserver {

    private final MeterRegistry meterRegistry;
    private final long terminationGracePeriodSeconds;
    private volatile long shutdownStartMs;

    public ShutdownObserver(MeterRegistry meterRegistry,
                            @Value("${terminationGracePeriodSeconds:60}") long terminationGracePeriodSeconds) {
        this.meterRegistry = meterRegistry;
        this.terminationGracePeriodSeconds = terminationGracePeriodSeconds;
    }

    @EventListener(ContextClosedEvent.class)
    public void onShutdown() {
        shutdownStartMs = System.currentTimeMillis();
        log.info("Graceful shutdown started, deadline={}s", terminationGracePeriodSeconds);
        meterRegistry.gauge(
            "app_shutdown_duration_seconds",
            this,
            obs -> (System.currentTimeMillis() - obs.shutdownStartMs) / 1000.0
        );
    }

    @PreDestroy
    public void onPreDestroy() {
        var durationMs = System.currentTimeMillis() - shutdownStartMs;
        log.info("Graceful shutdown completed in {}ms", durationMs);
    }
}

Useful Prometheus queries:

# How long the shutdown took per service
max by (service) (app_shutdown_duration_seconds)

# Warning when we approach the budget
max(app_shutdown_duration_seconds) > 50

An alert at 50 seconds (out of a 60-second budget) gives you time to spot the problem before force-kills of pods begin.

Why we got SIGTERM — how to find out

Spring does not know the reason the termination signal arrived. There can be several causes:

• an ordinary rolling deploy;
• the HPA scaled the number of replicas down;
• someone manually deleted the pod via kubectl delete pod;
• the OOM killer terminated the process due to a memory shortage;
• maintenance on a cluster node.

There is no point trying to determine the cause in the code — that is infrastructure-level information, not application-level. The application only records the fact:

@EventListener(ContextClosedEvent.class)
public void onShutdown() {
    log.info("SIGTERM received, starting graceful shutdown");
}

You look for the cause through kubectl describe pod <name> — the Events section shows who deleted the pod and why:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  Killing            2m    kubelet                    Stopping container app
  Normal  ScalingReplicaSet  10m   deployment-controller      Scaled down replica set

For a deep investigation there is the Kubernetes audit log.

Routine shutdown events are not ERROR

A common mistake: in the default configuration some libraries write shutdown messages at the ERROR level.

ERROR HikariPool-1 - Shutdown initiated...
ERROR Closing JPA EntityManagerFactory for persistence unit 'default'
ERROR org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down...

These are normal events that happen on every deploy. At the ERROR level they land in the alert channel — Slack or PagerDuty. The team gets dozens of false alerts on every deploy and learns to ignore them. When a real failure occurs, the reaction is late.

The fix is to set the correct level explicitly in logback-spring.xml:

<logger name="com.zaxxer.hikari" level="INFO"/>
<logger name="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" level="INFO"/>
<logger name="org.springframework.kafka.listener.KafkaMessageListenerContainer" level="INFO"/>

Only real problems should remain at the ERROR level: a forced termination with data loss, a connection dropped at an unexpected moment, exceptions that should not happen during a normal shutdown.

Common mistakes

• Raising terminationGracePeriodSeconds to 90+ — lengthens the version-incompatibility window and blocks kubectl drain. Better to reduce the amount of work.
• No duration metric — when a deploy has problems you cannot tell which phase did not finish in time.
• No log about receiving SIGTERM — the shutdown starts, but the logs have no starting point.
• Setting all timeouts to the maximum at once — every phase lasts 25 seconds, effectively turning parallel execution into sequential.
• Trying to determine the SIGTERM cause in code — look at kubectl describe pod instead.

In short

• The 60-second budget is made up of four phases, but three of them run in parallel — the real shutdown time is 25–40 seconds.
• If you do not fit, reduce the amount of work (fewer records per batch), do not increase the budget.
• The app_shutdown_duration_seconds metric and a log about the start of shutdown are the minimum for investigating deploy problems.
• The application does not know the SIGTERM cause — look at kubectl describe pod.
• Routine shutdown events (HikariPool, EntityManagerFactory) are INFO level, not ERROR.

Further reading

• JVM and Spring configuration — timeout-per-shutdown-phase and other parameters.
• HTTP drain — preStop sleep and Tomcat drain up to 25 seconds.
• Kafka shutdown — how the Kafka listener finishes its work.
• Scheduler and @Async — waiting for scheduled tasks.
• Kubernetes — terminationGracePeriodSeconds and pod configuration.