Shutdown budgets and observability in Go

← Back to the section

When an application takes unexpectedly long to shut down, it is hard to tell what exactly "got stuck". Without a metric and a proper log it is a black box: the deploy hangs, alerts fire, but where exactly is unclear. In this article we will look at how much time a Go application has to shut down, how to break that time down by phases, and how to make the shutdown visible.

How much time the application has

Kubernetes terminates pods according to a specific sequence. First the preStop hook runs (for example, sleep 10s), then the pod is sent SIGTERM, and the terminationGracePeriodSeconds countdown begins. The standard value is 60 seconds. If the process has not exited — SIGKILL arrives.

An important detail: the preStop hook runs before SIGTERM. That means the Go process still has 60 seconds after receiving the signal, not 60 minus preStop.

A typical budget for a Go service with Kafka and PostgreSQL:

The sum of the phases can exceed 60 seconds, because not all maximums happen at the same time. Under real load, shutdown usually takes 15–35 seconds. But if something goes wrong — you need the headroom.

In Go, shutdown is an explicit sequence in main, not parallel phases as in Spring. Wall clock is the sum of the time of each step in order.

What to do if you do not fit the budget

The first impulse is to increase terminationGracePeriodSeconds to 90 seconds. That is a mistake:

• During a rolling deploy, both generations of code work against the same DB schema for longer.
• kubectl drain by default waits only 30 seconds — with a long budget it hangs.

The right way is to reduce the amount of work in each phase:

• Reduce MinBytes/MaxBytes on the kafka-go reader — the consumer processes less per iteration.
• Reduce the outbox batch size: LockOutboxBatch(ctx, 100) → LockOutboxBatch(ctx, 20).
• In heavy goroutines, split the work into short iterations with a ctx.Done() check after each step.

The shutdown-time metric

To know how long shutdown actually took on each deploy, a Prometheus gauge app_shutdown_duration_seconds is set up. It needs to be recorded after all phases are complete — conveniently done via defer.

// internal/server/server.go
package server

import (
    "context"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var shutdownDuration = promauto.NewGauge(prometheus.GaugeOpts{
    Name: "app_shutdown_duration_seconds",
    Help: "Duration of graceful shutdown in seconds",
})

func Run(ctx context.Context, srv *http.Server, cfg Config, shutdownFns []func()) error {
    sigC := make(chan os.Signal, 1)
    signal.Notify(sigC, syscall.SIGTERM, syscall.SIGINT)
    defer signal.Stop(sigC)

    errC := make(chan error, 1)
    go func() { errC <- srv.ListenAndServe() }()

    select {
    case sig := <-sigC:
        slog.InfoContext(ctx, "received SIGTERM, starting graceful shutdown",
            "signal", sig.String())
    case err := <-errC:
        return err
    }

    start := time.Now()
    defer func() {
        dur := time.Since(start).Seconds()
        shutdownDuration.Set(dur)
        slog.InfoContext(ctx, "graceful shutdown complete", "duration_s", dur)
    }()

    for _, fn := range shutdownFns {
        fn()
    }
    return nil
}

shutdownFns is an explicit list in main in the required order:

// cmd/order-service/main.go
shutdownFns := []func(){
    func() { appState.SetNotReady() },       // readiness → 503
    func() { cancelConsumer() },             // signal the consumer to stop
    func() { consumerWg.Wait() },            // wait for the offset commit
    func() { schedulerWg.Wait() },           // wait for the outbox batch
    func() { srv.Shutdown(shutCtx) },        // drain in-flight HTTP
    func() {
        pool.Close()
        slog.InfoContext(ctx, "pgxpool closed")
    },
}

In Prometheus you can then look across services and set up an alert:

# Maximum shutdown time per service
max by (service) (app_shutdown_duration_seconds)

# Alert: shutdown took more than 50 of 60 seconds
max(app_shutdown_duration_seconds) > 50

The first thing to log

Immediately upon receiving the signal — before any action — the SIGTERM fact needs to be recorded:

case sig := <-sigC:
    slog.InfoContext(ctx, "received SIGTERM, starting graceful shutdown",
        "signal", sig.String())
    appState.SetNotReady()

The Go process does not know the reason for SIGTERM — it is a deploy, a scale-down (HPA scale-down), or something else. There is no need to determine the reason in code: os.Signal does not carry that information. The reason is found in kubectl describe pod <pod-name> under the Events field.

A common mistake: closing the pool at Error level

On application shutdown, pgxpool is closed as a routine operation — this is normal, not an error. If you log it as slog.Error, every rolling deploy will generate alerts in monitoring.

// Correct: closing the pool is Info
pool.Close()
slog.InfoContext(ctx, "pgxpool closed")

// Correct: an error closing the kafka writer is a different matter
if err := producer.Close(); err != nil {
    slog.ErrorContext(ctx, "kafka writer close error", "error", err)
} else {
    slog.InfoContext(ctx, "kafka writer closed")
}

The team gets used to ignoring false Error messages on deploys — and at some point misses a real incident. Separate "normal shutdown" (Info) from "something broke" (Error).

What a full shutdown looks like

T=0       SIGTERM (after preStop sleep 10s)
T=0       slog: "received SIGTERM, starting graceful shutdown"
T=0       appState.SetNotReady()  → /health/ready → 503
T=0       cancelConsumer()        → consumer receives ctx.Done()
T=0..15   consumerWg.Wait()       → offset committed, reader closed
T=15..35  schedulerWg.Wait()      → outbox batch finishes its iteration
T=35..50  srv.Shutdown(shutCtx)   → in-flight HTTP requests drained
T=50      pool.Close()            → slog: "pgxpool closed"
T=50      shutdownDuration.Set(50.0)
T=50      slog: "graceful shutdown complete", duration_s=50

Under real load, order-service fits into 15–25 seconds: the consumer drains quickly, the outbox batch is small, HTTP requests are short.

In short

• The Go process has 60 seconds after SIGTERM; the preStop hook runs before the signal and is not part of that budget.
• Shutdown in Go is an explicit sequence of steps: consumer → goroutines → HTTP → database. Wall clock is the sum of the phases.
• If you do not fit — reduce the size of batches and iterations, do not increase terminationGracePeriodSeconds.
• app_shutdown_duration_seconds via promauto.NewGauge + defer after the last step is the standard metric for tracking the budget.
• The first action on receiving SIGTERM is to log the fact. The reason for the signal is found via kubectl describe pod.
• A normal shutdown (closing the pool, the consumer) is slog.Info, not slog.Error. Error — only if something actually broke.

What to read next

• HTTP drain in Go — how http.Server.Shutdown waits for in-flight requests.
• Kafka shutdown in Go — kafka-go reader, offset commit, and closing the writer.
• DB and persistence in Go — pgxpool in the right phase, active transactions.
• Kubernetes and graceful shutdown — terminationGracePeriodSeconds, probes, maxUnavailable.