← Back to the section

Background goroutines — schedulers, workers, the outbox-relay — are the place where graceful shutdown most often breaks silently. The goroutine keeps working after receiving the stop signal, the database pool is closed before the transaction has finished. As a result, on a deploy some operations go out without a commit — the state diverges.

Let us look at how to correctly organize the shutdown of background goroutines: what to check, in what order to close resources, and why the outbox-relay requires special attention.

Why goroutines do not stop by themselves

A goroutine in Go is an independent thread of execution. When the application receives SIGTERM, the goroutines do not learn about it automatically. Without an explicit stop signal, a goroutine keeps looping forever.

The standard tool is context.Context. When the main code calls cancel(), all goroutines that watch ctx.Done() get the signal: "time to finish".

The second tool is sync.WaitGroup. It lets the main code wait until all goroutines have finished their current iteration, before closing the database connections.

The structure of a background goroutine

Any background goroutine gets two arguments: ctx context.Context and wg *sync.WaitGroup. The caller registers the goroutine in the WaitGroup before starting it.

type OutboxRelay struct {
    pool     *pgxpool.Pool
    queries  *db.Queries
    producer *kafka.Writer
    interval time.Duration
}

func (r *OutboxRelay) Run(ctx context.Context, wg *sync.WaitGroup) {
    defer wg.Done()
    ticker := time.NewTicker(r.interval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            if err := r.processOneBatch(ctx); err != nil {
                slog.WarnContext(ctx, "outbox relay batch failed", "error", err)
            }
        }
    }
}

A few important details:

  • defer wg.Done() — first, so that the counter is released on any exit from the function.
  • select checks ctx.Done() before each tick, not inside processOneBatch. This means "do not start a new iteration", not "abort the current one".
  • If SIGTERM arrived while processOneBatch was running, the goroutine waits for it to finish and only on the next turn sees <-ctx.Done() and exits.

A common mistake is to write for { processOneBatch(ctx) } without select and ctx.Done(). Such a goroutine ignores the stop signal, and the WaitGroup will never be released.

The shutdown order in main

The order of closing resources is critical. Violating the order leads to a panic: if you close the database pool before the goroutines have finished their transactions, pgx panics on the attempt to acquire a connection from a closed pool.

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    pool, err := pgxpool.New(ctx, cfg.DatabaseURL)
    if err != nil {
        slog.Error("pgxpool connect", "error", err)
        os.Exit(1)
    }

    relay := &OutboxRelay{
        pool:     pool,
        queries:  db.New(pool),
        producer: &kafka.Writer{Addr: kafka.TCP(cfg.KafkaBroker), Topic: "order-events"},
        interval: 500 * time.Millisecond,
    }

    var wg sync.WaitGroup
    wg.Add(1)
    go relay.Run(ctx, &wg)

    srv := &http.Server{Addr: cfg.Addr, Handler: buildRouter()}

    sigC := make(chan os.Signal, 1)
    signal.Notify(sigC, syscall.SIGTERM, syscall.SIGINT)
    defer signal.Stop(sigC)

    select {
    case sig := <-sigC:
        slog.Info("received SIGTERM, starting graceful shutdown", "signal", sig.String())
    case err := <-func() chan error {
        c := make(chan error, 1)
        go func() { c <- srv.ListenAndServe() }()
        return c
    }():
        slog.Error("server error", "error", err)
        cancel()
        return
    }

    appState.SetNotReady()    // readiness → 503 first

    cancel()                  // signal to the goroutines: do not start new iterations

    shutCtx, shutCancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer shutCancel()
    if err := srv.Shutdown(shutCtx); err != nil {
        slog.Error("http shutdown", "error", err)
    }

    wg.Wait()                 // wait for all goroutines to finish

    if err := producer.Close(); err != nil {
        slog.Error("kafka writer close", "error", err)
    }
    pool.Close()              // pool — last
    slog.Info("graceful shutdown complete")
}

The rule is simple: pool.Close() always comes strictly after wg.Wait().

Outbox-relay: why the transaction cannot be interrupted

The outbox-relay is a goroutine that takes events from the outbox_event table and publishes them to Kafka. Its distinctive feature: an unfinished transaction means a potential duplicate on publish.

That is why, inside processOneBatch, the transaction is opened on context.Background(), not on the parent ctx. Even if ctx is already cancelled (SIGTERM arrived), the transaction is carried through to commit or rollback.

func (r *OutboxRelay) processOneBatch(ctx context.Context) error {
    tx, err := r.pool.Begin(context.Background()) // a separate ctx for the transaction
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback(context.Background())

    qtx := r.queries.WithTx(tx)

    events, err := qtx.LockOutboxBatch(context.Background(), db.LockOutboxBatchParams{
        Limit: 50,
    })
    if err != nil {
        return fmt.Errorf("lock outbox batch: %w", err)
    }
    if len(events) == 0 {
        return nil
    }

    for _, e := range events {
        msg := kafka.Message{
            Key:   []byte(e.AggregateID),
            Value: e.Payload,
        }
        if err := r.producer.WriteMessages(context.Background(), msg); err != nil {
            return fmt.Errorf("publish event %s: %w", e.ID, err)
        }
        if err := qtx.MarkDispatched(context.Background(), e.ID); err != nil {
            return fmt.Errorf("mark dispatched %s: %w", e.ID, err)
        }
    }

    if err := tx.Commit(context.Background()); err != nil {
        return fmt.Errorf("commit outbox batch: %w", err)
    }
    return nil
}

The SQL query to lock the rows (sqlc):

-- name: LockOutboxBatch :many
SELECT id, aggregate_id, topic, payload
FROM outbox_event
WHERE dispatched_at IS NULL
ORDER BY created_at
LIMIT @limit
FOR UPDATE SKIP LOCKED;

FOR UPDATE SKIP LOCKED solves the problem of parallel runs during a rolling update. If two instances of the application are running at the same time, they do not take the same rows: each takes its own and does not wait for the others. When the old instance finishes a batch and commits, the rows are marked dispatched_at IS NOT NULL — the new one will skip them. If the old one crashed without a commit — the rows are unlocked, and the new one will pick them up.

For ordinary goroutines without critical transactions (for example, deleting stale records) you can pass ctx directly: interruption is acceptable, and the next run will finish the job.

Several goroutines — one WaitGroup

If the service starts several background tasks, they are all registered in a single WaitGroup:

var wg sync.WaitGroup

wg.Add(1)
go outboxRelay.Run(ctx, &wg)

wg.Add(1)
go productConsumer.Run(ctx, &wg)

wg.Add(1)
go expiredOrderCleaner.Run(ctx, &wg)

// on shutdown — one shared call
cancel()
wg.Wait()
pool.Close()

All goroutines finish in parallel, and shutdown waits for the slowest. If OutboxRelay finishes a batch in 3 seconds and ProductConsumer in 8 seconds, the total wait is 8 seconds, not 11.

The time budget

Graceful shutdown has an overall limit — usually 60 seconds, of which Kubernetes leaves about 10 seconds for preStop. Background goroutines must fit within ~20 seconds.

If a batch of 50 events takes too long — reduce the batch size to 10–15 events. There is no need to increase the budget: the batch should be small by design.

An approximate breakdown:

preStop sleep            10s
http.Server.Shutdown    ≤25s  (in parallel with cancelling the goroutines)
goroutines WaitGroup    ≤20s
pool.Close               ~1s
────────────────────────────
total                   ≤56s < 60s

In practice, the HTTP drain finishes before the outbox completes its batch — they partially overlap.

In short

  • Goroutines do not stop by themselves on SIGTERM — they need a context.Context with cancel().
  • ctx.Done() is checked in the select before each tick: "do not start a new iteration", not abort the current one.
  • defer wg.Done() — as the first line in the goroutine, wg.Add(1) — before go in the calling code.
  • pool.Close() strictly after wg.Wait() — otherwise a panic on an open transaction.
  • Inside the critical section of the outbox-relay, the transaction is opened on context.Background(), not on the cancelled ctx.
  • FOR UPDATE SKIP LOCKED protects against double-processing of the same rows during a rolling update.
  • Several goroutines — one shared WaitGroup, finishing in parallel.
  • Keep the batch size small: background goroutines must fit within ~20 seconds.
  • HTTP drain in Go — http.Server.Shutdown and coordination with preStop.
  • DB and persistence on shutdown — the order of pool.Close() and transactions.
  • Kafka shutdown in Go — writer.Close() and offset commit on context cancellation.
  • Idempotency on shutdown — protection against duplicates on SIGTERM.
  • Kubernetes and graceful shutdown — terminationGracePeriodSeconds, preStop, rolling update.