← Back to the section

When you roll out a new version of a service, Kubernetes stops the old pod and starts a new one. Do it carelessly and some requests will fail with a 502 error. The user sees a failure, monitoring fires an alert. Let's look at how to avoid this.

What happens when a pod is stopped

Suppose the service is handling a user request. At that moment Kubernetes decides to stop the pod — for example, during a rolling update. Without any special configuration, the process receives a SIGTERM signal and terminates. The request is interrupted midway. The client sees a 502 or a dropped connection.

Spring Boot knows how to avoid this — the mode is called graceful shutdown. It's enabled with a single line:

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

With this enabled, here's what happens after SIGTERM:

T=0    Spring receives SIGTERM
T=0+   Readiness status → OUT_OF_SERVICE (returns 503)
T=0+   Tomcat stops accepting new connections
       Active requests keep being processed
T=5s   Kubernetes sees the 503, removes the pod from load balancing
T=N    All active requests have finished
T=N    Spring shuts down

An active request that's already being processed is not interrupted. Spring waits for it to finish — up to the timeout-per-shutdown-phase value. This is exactly what HTTP drain means: "drain" everything already in flight into responses.

Why Spring graceful isn't enough — you need preStop

Here lies a non-obvious problem. Kubernetes runs on many nodes, and when a pod is marked for deletion, that information doesn't propagate across the cluster instantly.

Here's what happens: Kubernetes simultaneously sends SIGTERM to the process and issues the command to remove the pod from the Service's endpoints list. But kube-proxy on other nodes updates its iptables rules with a delay — usually 1–5 seconds, and on large clusters (thousands of nodes) up to 20 seconds. During all that time, new requests are still routed to the pod that has already started shutting down and is no longer accepting connections.

The result: even with correct Spring graceful, there will be 502s in this window.

The solution is a preStop hook: make Kubernetes wait before sending SIGTERM.

spec:
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
      terminationGracePeriodSeconds: 60

With preStop the sequence changes:

T=0    Kubernetes decides to delete the pod
T=0+   The preStop hook starts — the process sleeps for 10 seconds
       In parallel: the pod is removed from the Service endpoints
       In parallel: kube-proxy on the nodes updates iptables
T=10s  preStop finished
T=10s  Kubernetes sends SIGTERM
T=10s+ Spring graceful drain: waits for active requests to finish

During those 10 seconds of sleep, kube-proxy has time to update. By the time Spring receives SIGTERM and stops accepting connections, new traffic is no longer routed to this pod. Spring drain deals only with the requests that were already being processed.

An important point: terminationGracePeriodSeconds is the total budget (preStop + Spring drain + everything else). A value of 60 seconds is usually enough: 10s preStop + up to 30s Spring drain + headroom.

Ten seconds is a good value for most clusters. On very large ones (thousands of nodes) you can increase it to 15–20 seconds.

Long-running synchronous operations

Suppose there's an endpoint that generates a report — synchronously, in 30 seconds. When the pod stops, five such requests are in progress. Spring will wait for them to finish, but timeout-per-shutdown-phase: 30s won't let it wait forever. Some requests will be interrupted.

The problem isn't in Spring or in Kubernetes — the problem is in the endpoint's design. A long-running synchronous operation over HTTP fits poorly with any way of stopping the service.

Option 1: 202 Accepted + polling

Instead of keeping the connection open, the endpoint immediately returns a 202 and a task identifier. The client polls the status periodically.

@RestController
@RequiredArgsConstructor
public class ReportController {

    private final ReportService reportService;

    @PostMapping("/reports")
    public ResponseEntity<ReportStarted> start(
        @RequestHeader("Idempotency-Key") String key,
        @RequestBody @Valid ReportRequest req
    ) {
        var reportId = reportService.enqueue(key, req);
        return ResponseEntity.accepted()
            .header("Location", "/reports/" + reportId)
            .body(new ReportStarted(reportId, "QUEUED"));
    }

    @GetMapping("/reports/{id}")
    public ReportStatus get(@PathVariable Long id) {
        return reportService.getStatus(id);
    }
}

The POST returns a response in milliseconds. The actual generation runs in a background thread. When the pod stops, the short POST doesn't block the drain, while the background process can finish its work within its own budget.

Option 2: @Async with the right settings

If the API has to be synchronous by interface, the heavy work can be offloaded to a thread pool with graceful shutdown configured:

@PostMapping("/process")
public ResponseEntity<ProcessStarted> process(
    @RequestHeader("Idempotency-Key") String key,
    @RequestBody @Valid ProcessRequest req
) {
    asyncProcessor.startAsync(key, req);
    return ResponseEntity.accepted().body(new ProcessStarted(key, "STARTED"));
}

asyncProcessor uses a ThreadPoolTaskExecutor with waitForTasksToCompleteOnShutdown = true — then, on shutdown, Spring will wait for the tasks in this pool to finish.

Option 3: optimization

Sometimes an endpoint is slow not by the nature of the operation, but because of extra database queries or unoptimized code. It's worth checking: maybe those 30 seconds are several N+1 queries, and the operation itself could take 2 seconds.

Common mistake: disabling graceful through Tomcat customization

Sometimes developers add a custom Tomcat setting and accidentally break graceful:

// This will break graceful shutdown
@Bean
public WebServerFactoryCustomizer<TomcatServletWebServerFactory> tomcatCustomizer() {
    return factory -> factory.addConnectorCustomizers(connector -> {
        ((AbstractProtocol<?>) connector.getProtocolHandler()).setKeepAliveTimeout(0);
        connector.setAsyncTimeout(0);
    });
}

Setting awaitTermination(0, SECONDS) or zeroing out timeouts at the connector level forces Tomcat to shut down immediately on SIGTERM — Spring graceful stops working, requests are dropped. If you need to customize Tomcat, verify that none of the parameters zero out the shutdown wait time.

In short

  • Spring graceful shutdown (server.shutdown: graceful) drains active HTTP requests to a response before stopping.
  • Spring graceful alone isn't enough: kube-proxy on other nodes updates 1–15 seconds after SIGTERM, and during that window new traffic reaches a pod that is already shutting down.
  • preStop: sleep 10 in the k8s manifest solves this: the pod sleeps for 10 seconds, kube-proxy has time to update, and only then does Kubernetes send SIGTERM.
  • terminationGracePeriodSeconds must cover preStop + Spring drain with headroom; a value of 60s is usually enough.
  • Long-running synchronous operations (over 10 seconds) fit poorly with the drain — move them to 202 Accepted + polling or @Async with waitForTasksToCompleteOnShutdown.
  • Customizing Tomcat with zeroed-out timeouts breaks graceful — check your settings.
  • Graceful shutdown in Spring Boot — overview — a full overview of all the phases.
  • Kubernetes and terminationGracePeriodSeconds — pod configuration details.
  • Scheduled tasks and @Async at shutdown — how to configure thread pools.