Kubernetes and graceful shutdown in Go: terminationGracePeriodSeconds, probes, rolling update

← Back to the section

A Go service can carefully finish processing requests on shutdown — but that is not enough. Without the right Kubernetes configuration, clients will still get 502 errors: kube-proxy does not manage to remove the pod from the list of active ones before the process stops responding. Three parameters in the Deployment manifest close this gap.

Why a 502 on restart is a configuration problem

When Kubernetes stops a pod, it does two things at once: it removes the pod from the Service endpoints (so that new traffic does not go there) and sends the process a SIGTERM signal. The problem is that updating the routing tables via kube-proxy takes several seconds. In this window, the load balancer is still directing requests to a pod that has already begun to shut down.

The solution is to give kube-proxy time to synchronize before SIGTERM reaches the process. For this, the preStop hook is used.

terminationGracePeriodSeconds and preStop

terminationGracePeriodSeconds is the overall time budget that Kubernetes allots to the entire pod termination sequence. If the process has not finished within this time, Kubernetes sends SIGKILL.

The default value is 30 seconds. This is often not enough: preStop takes 10 seconds, and the Go process is left with only 20 seconds to finish all goroutines, HTTP connections, and to close the DB pool.

The correct configuration:

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: order-service
          image: order-service:2.1.0
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]

How the termination sequence works:

It is important to understand: terminationGracePeriodSeconds is counted from T=0, that is, from the very beginning. The preStop sleep of 10 seconds is part of this budget, not a separate time on top of it.

Without preStop: Kubernetes will send SIGTERM immediately, kube-proxy is still directing traffic to the pod, the process is no longer responding — clients get 502s for 5–15 seconds.

Two separate health endpoints

A typical mistake is one /health endpoint for both checks. Kubernetes uses two different types of checks with different behavior on failure:

• readinessProbe: if the pod is not ready, Kubernetes removes it from endpoints. The pod keeps running.
• livenessProbe: if the pod does not respond, Kubernetes restarts it.

During graceful shutdown, different behavior is needed: readiness must return 503 (so that Kubernetes stops sending traffic), while liveness must stay 200 (so that Kubernetes does not restart the pod in the middle of shutting down).

The readiness state via an atomic flag:

// internal/health/state.go
package health

import "sync/atomic"

type State struct{ ready atomic.Bool }

func NewState() *State {
    s := &State{}
    s.ready.Store(true)
    return s
}

func (s *State) SetNotReady() { s.ready.Store(false) }
func (s *State) IsReady() bool { return s.ready.Load() }

Two separate routes:

// internal/server/routes.go
func RegisterHealthRoutes(r chi.Router, state *health.State) {
    r.Get("/health/live", func(w http.ResponseWriter, _ *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    r.Get("/health/ready", func(w http.ResponseWriter, _ *http.Request) {
        if !state.IsReady() {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })
}

On receiving SIGTERM, SetNotReady() is called first — before srv.Shutdown. Kubernetes checks readiness every 5 seconds, and after two consecutive 503s it removes the pod from endpoints. New traffic stops arriving; requests already accepted by the server are drained via srv.Shutdown.

atomic.Bool matters here: the flag is read and written from different goroutines (HTTP handlers and the shutdown goroutine), so an atomic operation is needed — a plain bool without synchronization gives undefined behavior.

Configuring probes in the Deployment

containers:
  - name: order-service
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3

Go services start quickly — usually in under 5 seconds. So initialDelaySeconds: 15 for liveness is enough. After startup, Kubernetes will start checking readiness after 5 seconds: if the service came up, the pod is immediately added to endpoints.

The SIGTERM handler and the shutdown order

// internal/server/server.go
func Run(
    ctx context.Context,
    srv *http.Server,
    cfg Config,
    appState *health.State,
    cancelConsumerCtx context.CancelFunc,
    consumerDone <-chan struct{},
    schedulerDone <-chan struct{},
    closePool func(),
) error {
    sigC := make(chan os.Signal, 1)
    signal.Notify(sigC, syscall.SIGTERM, syscall.SIGINT)
    defer signal.Stop(sigC)

    errC := make(chan error, 1)
    go func() { errC <- srv.ListenAndServe() }()

    select {
    case sig := <-sigC:
        slog.InfoContext(ctx, "received SIGTERM, starting graceful shutdown", "signal", sig.String())
    case err := <-errC:
        return err
    }

    appState.SetNotReady() // readiness → 503 first thing

    cancelConsumerCtx()  // signal the background goroutines to stop
    <-consumerDone       // wait for the current batch to finish
    <-schedulerDone      // wait for the scheduler's current iteration

    shutCtx, cancel := context.WithTimeout(context.Background(), cfg.ShutdownTimeout)
    defer cancel()
    if err := srv.Shutdown(shutCtx); err != nil {
        slog.ErrorContext(ctx, "http shutdown error", "error", err)
    }

    closePool() // the connection pool — last
    return nil
}

A few important details:

srv.Shutdown(ctx), not srv.Close() — Shutdown waits for all active requests to finish, Close cuts off connections immediately.

context.WithTimeout(context.Background(), ...) — not the parent ctx: by the time of shutdown the parent context is already cancelled, and we need to give the HTTP server time to finish the requests.

The connection pool (pgxpool) is closed last: the Kafka consumer goroutines and the scheduler may still access the DB during their shutdown.

Rolling update without traffic loss

When deploying a new version, Kubernetes by default may stop the old pod before the new one has managed to accept traffic. To prevent this:

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

maxUnavailable: 0 — Kubernetes does not turn off the old pods until it has brought up the new ones. The number of active pods never drops below the declared count.

maxSurge: 1 — allows temporarily having one extra pod. The new pod starts, passes the readinessProbe, is added to endpoints — and only after that does the shutdown of one old pod begin.

The deploy sequence with three replicas:

Common mistakes

No preStop — SIGTERM arrives before kube-proxy has removed the pod from endpoints. Clients get 502s for several seconds after the shutdown begins.

terminationGracePeriodSeconds: 30 (the default) with a preStop of 10 seconds — the Go process is left with only 20 seconds to finish. If shutdown takes longer, Kubernetes kills the process forcibly.

One /health for both probes — during graceful shutdown, liveness returns 503, Kubernetes decides the pod is stuck and restarts it in the middle of shutting down.

srv.Close() instead of srv.Shutdown(ctx) — Close cuts off connections immediately, and clients get errors.

Closing the DB pool before the goroutines finish — if pgxpool is closed while goroutines are still trying to run queries, they get errors instead of a normal shutdown.

maxUnavailable: 1 in production — during a deploy the number of active pods temporarily drops, the load on the remaining ones rises, and the SLO may be violated.

Liveness depends on DB availability — if the DB is unavailable, liveness returns 503, and Kubernetes restarts the pods. But a restart will not fix the DB, and the pods will begin to restart in a loop. Liveness must always return 200; put the DB availability check into readiness.

In short

• preStop: sleep 10 gives kube-proxy time to remove the pod from endpoints before SIGTERM reaches the process. Without it, 502s on every deploy are guaranteed.
• terminationGracePeriodSeconds: 60 — explicitly in the Deployment. The default (30) does not leave enough time with a preStop of 10 seconds.
• /health/live always returns 200; /health/ready returns 503 from the very first moment of shutdown — this is the signal for Kubernetes to remove the pod from endpoints.
• atomic.Bool for the readiness flag — the flag is read from HTTP goroutines and written from the shutdown goroutine at the same time.
• The shutdown order: readiness → 503, background goroutines, srv.Shutdown, DB pool last.
• maxSurge: 1, maxUnavailable: 0 — a new pod is added to rotation before the old one begins to shut down.

What to read next

• HTTP drain and in-flight requests — how srv.Shutdown waits for active connections.
• Background goroutines and outbox — context.Context and sync.WaitGroup for background tasks.
• DB and persistence — the order of closing pgxpool and transactions on shutdown.