← Back to the section

When Kubernetes decides to remove a pod — to update the version or to scale — it sends the process a SIGTERM signal. If the application simply exits on this signal, all the requests that were in flight at that moment will be cut off midway. The client will get an error.

To prevent this, you need to properly "drain" the HTTP server: wait for the current requests to finish and only then exit. This is called HTTP drain.

Why a simple exit breaks requests

Imagine: a client sent a request, the server started processing it — and at that moment the process received SIGTERM and exited. The connection was cut off in the middle, the client saw a 502 or a connection reset.

In Go, for a correct server stop there is http.Server.Shutdown(ctx). Unlike Close(), which immediately tears down all connections, Shutdown does three things:

  1. Stops accepting new connections.
  2. Waits until all active handlers return a response.
  3. Completes only after all requests have run to the end.

How a correct shutdown is structured

The minimal structure looks like this:

// internal/server/server.go

func Run(ctx context.Context, srv *http.Server, appState *health.State, cfg Config) error {
    sigC := make(chan os.Signal, 1)
    signal.Notify(sigC, syscall.SIGTERM, syscall.SIGINT)
    defer signal.Stop(sigC)

    errC := make(chan error, 1)
    go func() { errC <- srv.ListenAndServe() }()

    select {
    case sig := <-sigC:
        slog.InfoContext(ctx, "received SIGTERM, starting graceful shutdown",
            "signal", sig.String())
    case err := <-errC:
        return err
    }

    appState.SetNotReady()

    shutCtx, cancel := context.WithTimeout(context.Background(), cfg.ShutdownTimeout)
    defer cancel()

    if err := srv.Shutdown(shutCtx); err != nil {
        return fmt.Errorf("http shutdown: %w", err)
    }
    return nil
}

Order matters: appState.SetNotReady() is called first, and only then srv.Shutdown. This switches the health endpoint to the "not ready" state before the server starts to refuse anything.

Why you need a health endpoint

When Kubernetes decides whether to send traffic to a pod, it checks the readiness probe — a dedicated endpoint that the service registers itself.

// chi routes

r.Get("/health/live", func(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK)
})
r.Get("/health/ready", func(w http.ResponseWriter, _ *http.Request) {
    if !appState.IsReady() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

/health/live always responds — it says "the process is alive". /health/ready returns 503 after SetNotReady() is called — this is a signal for Kubernetes to remove the pod from the list of load-balancing addresses and stop sending new requests.

// internal/health/state.go

type State struct{ ready atomic.Bool }

func NewState() *State {
    s := &State{}
    s.ready.Store(true)
    return s
}

func (s *State) SetNotReady() { s.ready.Store(false) }
func (s *State) IsReady() bool { return s.ready.Load() }

Note: atomic.Bool is used, not an ordinary variable. The health endpoint is called from several goroutines in parallel, so a thread-safe operation is needed.

The kube-proxy problem and why you need a preStop sleep

Even with a correctly written Shutdown, there is a hidden trap in Kubernetes.

When the kubelet decides to remove a pod, it does several things at once:

  • sends SIGTERM to the process,
  • tells the rest of the cluster components that the pod must be removed from load balancing.

But "remove from load balancing" is not instant. On every node of the cluster there is a kube-proxy that updates the iptables rules. This takes a few seconds. If the cluster is large (hundreds of nodes), it can take 10–15 seconds before all nodes learn that the pod is unavailable.

As a result, the process has already received SIGTERM and started Shutdown (new connections are not accepted), while other pods in the cluster are still sending requests to it through the old iptables rules. Those requests will get an error.

The solution is the preStop hook in the Kubernetes manifest:

# k8s/deployment.yaml
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: order-service
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]

Kubernetes runs preStop before sending SIGTERM. While sleep 10 is running, the application works normally and accepts traffic. During those 10 seconds, kube-proxy on all nodes manages to update the rules. Only after that does SIGTERM arrive — and by that point no one is sending traffic to this pod anymore.

The sequence with preStop:

T=0    kubelet decides to remove the pod
T=0+   preStop started (sleep 10)
       In parallel: kube-proxy updates iptables on the nodes
T=10s  preStop finished
T=10s  kubelet sends SIGTERM
T=10s+ the application catches the signal, calls SetNotReady() and Shutdown()
T=N    all in-flight requests are finished, the process exits

Long operations are a separate story

Server.Shutdown waits until all active HTTP handlers finish. If an operation takes 20–30 seconds (for example, complex business logic with several steps), this creates a problem: on SIGTERM, several such requests in processing may not fit within the shutdown time budget.

202 Accepted + polling

A good solution for long operations is to immediately return 202 Accepted and queue the task:

// internal/handler/order_handler.go

func (h *OrderHandler) Submit(w http.ResponseWriter, r *http.Request) {
    orderID := chi.URLParam(r, "id")
    idempotencyKey := r.Header.Get("Idempotency-Key")

    taskID, err := h.tasks.Enqueue(r.Context(), SubmitOrderTask{
        OrderID:        orderID,
        IdempotencyKey: idempotencyKey,
    })
    if err != nil {
        httperr.Write(w, r, err)
        return
    }

    w.Header().Set("Location", "/orders/"+orderID+"/status")
    w.WriteHeader(http.StatusAccepted)
    _ = json.NewEncoder(w).Encode(map[string]string{"taskId": taskID})
}

func (h *OrderHandler) SubmitStatus(w http.ResponseWriter, r *http.Request) {
    orderID := chi.URLParam(r, "id")
    status, err := h.orders.GetSubmitStatus(r.Context(), orderID)
    if err != nil {
        httperr.Write(w, r, err)
        return
    }
    _ = json.NewEncoder(w).Encode(status)
}

The POST responds in milliseconds. The client periodically checks GET /orders/{id}/status. When the server stops, there are no long goroutines in HTTP — Shutdown completes quickly.

Background goroutine with a WaitGroup

If you need to continue the computation in the background and wait for it on shutdown:

// internal/handler/product_handler.go

func (h *ProductHandler) Reindex(w http.ResponseWriter, r *http.Request) {
    catalogID := chi.URLParam(r, "catalogID")

    h.wg.Add(1)
    go func() {
        defer h.wg.Done()
        if err := h.catalog.Reindex(context.Background(), catalogID); err != nil {
            slog.Error("reindex catalog", "catalog_id", catalogID, "error", err)
        }
    }()

    w.WriteHeader(http.StatusAccepted)
}

h.wg.Wait() is called in the shutdown sequence after srv.Shutdown — the long operation manages to finish within the overall time budget.

Shutdown order in main

// cmd/order-service/main.go

func main() {
    ctx := context.Background()

    pool, err := pgxpool.New(ctx, cfg.DatabaseURL)
    if err != nil {
        slog.Error("pgxpool init", "error", err)
        os.Exit(1)
    }

    appState := health.NewState()

    r := chi.NewRouter()
    r.Get("/health/live", liveHandler())
    r.Get("/health/ready", readyHandler(appState))
    // ... business routes

    srv := &http.Server{Addr: cfg.Addr, Handler: r}

    if err := server.Run(ctx, srv, appState, cfg); err != nil {
        slog.Error("server run", "error", err)
    }

    slog.Info("closing pool")
    pool.Close()
}

pool.Close() comes after server.Run — that is, after Shutdown has finished and all goroutines from the WaitGroup have completed. This is important: if you close the database pool earlier, the requests still being processed will lose their connection.

Common mistakes

srv.Close() instead of srv.Shutdown(ctx) — Close() immediately cuts off all connections without waiting for handlers. Always use Shutdown with a timeout context (20–25 seconds).

No preStop sleep — even with a correct Shutdown, without preStop, in the 5–15 second window after SIGTERM new requests will still land on a pod that no longer accepts connections. The bug shows up in rolling deploys and is hard to reproduce.

preStop sleep shorter than 5 seconds — on large clusters (1000+ nodes) the propagation of iptables updates can take up to 20 seconds. The minimum is 10 seconds.

A synchronous long endpoint without async — operations longer than 10 seconds should be moved to 202 Accepted + polling, otherwise several such requests in flight may not fit within the shutdown time budget.

pool.Close() before srv.Shutdown — the database is closed last, after HTTP and all background goroutines.

In short

  • http.Server.Shutdown(ctx) waits for all active requests to finish; Close() cuts them off immediately — these are different things.
  • Before Shutdown, call SetNotReady() — the health endpoint will start returning 503, and Kubernetes will stop sending traffic.
  • preStop: sleep 10 in the manifest is mandatory: it gives kube-proxy time to update iptables before SIGTERM arrives.
  • Move long operations (>10 seconds) to 202 Accepted + polling or a background goroutine with a WaitGroup.
  • The database pool is closed last — after the HTTP drain and all background tasks.
  • Budgets and observability — how to distribute the 60 seconds between the shutdown phases.
  • DB and persistence — the pgxpool closing order and transactions on SIGTERM.
  • Kafka shutdown — stopping the consumer and a correct offset commit.
  • Kubernetes — terminationGracePeriodSeconds, probes, maxUnavailable: 0.