Shutdown budgets and graceful shutdown observability

← Back to the section

When graceful shutdown took too long and the pod was force-killed — how do you figure out what exactly did not finish in time? Without metrics and logs you can only guess. Let's work through how to calculate a realistic time budget and what to add to the code so you notice problems before an incident.

Where the 60 seconds come from

When deleting a pod, Kubernetes waits at most terminationGracePeriodSeconds seconds and then kills the process. The default value is 60 seconds. What manages to happen in that time?

Shutdown consists of several steps:

Add up the maximums and you get 70 seconds, which is more than the 60-second budget. This matters, because of how Spring actually shuts the application down: it does not stop everything at once but goes group by group, and spring.lifecycle.timeout-per-shutdown-phase applies to each group separately. Queue consumers and the scheduler stop first, the web server last. So the budgets add up instead of overlapping.

The same 60 seconds on one scale — first with every timeout at its maximum:

What to do if you do not fit

The first instinct is to raise terminationGracePeriodSeconds to 90 or 120 seconds. That is a mistake.

A long shutdown lengthens the rolling deploy. While the old pod is still running, the new one is already taking traffic: both versions of the code work against the same database. The longer this window, the higher the chance of schema-compatibility problems. On top of that, kubectl drain waits 30 seconds by default: if a pod does not finish in time, the drain hangs.

The right path is to reduce the amount of work, not increase the budget:

• Kafka max.poll.records: 500 → 100: the listener finishes processing in 5 seconds instead of 25.
• @Async tasks with long chains → break them into short steps.
• Heavy @Scheduled tasks → shrink the batch size (50 records instead of 500).

Shutdown duration metric

Without a metric, the only way to know how long a shutdown took is to scroll through the logs of every pod by hand. With a metric it is immediately visible on a chart.

We add a gauge that gets updated throughout the whole shutdown:

@Component
@Slf4j
public class ShutdownObserver {

    private final MeterRegistry meterRegistry;
    private final long terminationGracePeriodSeconds;
    private volatile long shutdownStartMs;

    public ShutdownObserver(MeterRegistry meterRegistry,
                            @Value("${terminationGracePeriodSeconds:60}") long terminationGracePeriodSeconds) {
        this.meterRegistry = meterRegistry;
        this.terminationGracePeriodSeconds = terminationGracePeriodSeconds;
    }

    @EventListener(ContextClosedEvent.class)
    public void onShutdown() {
        shutdownStartMs = System.currentTimeMillis();
        log.info("Graceful shutdown started, deadline={}s", terminationGracePeriodSeconds);
        meterRegistry.gauge(
            "app_shutdown_duration_seconds",
            this,
            obs -> (System.currentTimeMillis() - obs.shutdownStartMs) / 1000.0
        );
    }

    @PreDestroy
    public void onPreDestroy() {
        var durationMs = System.currentTimeMillis() - shutdownStartMs;
        log.info("Graceful shutdown completed in {}ms", durationMs);
    }
}

Useful Prometheus queries:

# How long the shutdown took per service
max by (service) (app_shutdown_duration_seconds)

# Warning when we approach the budget
max(app_shutdown_duration_seconds) > 50

An alert at 50 seconds (out of a 60-second budget) gives you time to spot the problem before force-kills of pods begin.

Why we got SIGTERM — how to find out

Spring does not know the reason the termination signal arrived. There can be several causes:

• an ordinary rolling deploy;
• the HPA scaled the number of replicas down;
• someone manually deleted the pod via kubectl delete pod;
• the OOM killer terminated the process due to a memory shortage;
• maintenance on a cluster node.

There is no point trying to determine the cause in the code — that is infrastructure-level information, not application-level. The application only records the fact:

@EventListener(ContextClosedEvent.class)
public void onShutdown() {
    log.info("SIGTERM received, starting graceful shutdown");
}

You look for the cause through kubectl describe pod <name> — the Events section shows who deleted the pod and why:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  Killing            2m    kubelet                    Stopping container app
  Normal  ScalingReplicaSet  10m   deployment-controller      Scaled down replica set

For a deep investigation there is the Kubernetes audit log.

Routine shutdown events are not ERROR

A common mistake: in the default configuration some libraries write shutdown messages at the ERROR level.

ERROR HikariPool-1 - Shutdown initiated...
ERROR Closing JPA EntityManagerFactory for persistence unit 'default'
ERROR org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down...

These are normal events that happen on every deploy. At the ERROR level they land in the alert channel — Slack or PagerDuty. The team gets dozens of false alerts on every deploy and learns to ignore them. When a real failure occurs, the reaction is late.

The fix is to set the correct level explicitly in logback-spring.xml:

<logger name="com.zaxxer.hikari" level="INFO"/>
<logger name="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" level="INFO"/>
<logger name="org.springframework.kafka.listener.KafkaMessageListenerContainer" level="INFO"/>

Only real problems should remain at the ERROR level: a forced termination with data loss, a connection dropped at an unexpected moment, exceptions that should not happen during a normal shutdown.

Common mistakes

• Raising terminationGracePeriodSeconds to 90+ — lengthens the version-incompatibility window and blocks kubectl drain. Better to reduce the amount of work.
• No duration metric — when a deploy has problems you cannot tell which phase did not finish in time.
• No log about receiving SIGTERM — the shutdown starts, but the logs have no starting point.
• Setting all timeouts to the maximum at once — the groups are stopped one after another, so the maximums add up and the last phase is cut off by SIGKILL.
• Trying to determine the SIGTERM cause in code — look at kubectl describe pod instead.

In short

• The 60-second budget is split between four phases, and Spring stops them one after another: preStop 10 + Kafka 15 + scheduler 20 + drain 25 = 70 seconds, which is over budget.
• If you do not fit, reduce the amount of work (fewer records per batch), do not increase the budget.
• The app_shutdown_duration_seconds metric and a log about the start of shutdown are the minimum for investigating deploy problems.
• The application does not know the SIGTERM cause — look at kubectl describe pod.
• Routine shutdown events (HikariPool, EntityManagerFactory) are INFO level, not ERROR.

Further reading

• JVM and Spring configuration — timeout-per-shutdown-phase and other parameters.
• HTTP drain — preStop sleep and Tomcat drain up to 25 seconds.
• Kafka shutdown — how the Kafka listener finishes its work.
• Scheduler and @Async — waiting for scheduled tasks.
• Kubernetes — terminationGracePeriodSeconds and pod configuration.