When an application receives SIGTERM, it must stop carefully: finish the operations it has started, not leave a transaction half-done, and only then release the database connections. In Go this order is set by hand — there is no framework that knows what to close first and what last.
Why you cannot close the pool immediately
If you call pool.Close() before the background goroutines have finished, you get a problem: the scheduler or outbox-relay reaches its next pool.Acquire() and gets a panic or an error — because the pool is already closed while the business operation was in full swing.
The rule is simple: the connection pool is closed last — after all goroutines have run to completion.
In main this looks like an explicit sequence of steps:
func run(ctx context.Context, cfg Config) error {
pool, err := pgxpool.New(ctx, cfg.DatabaseURL)
if err != nil {
return fmt.Errorf("pgxpool.New: %w", err)
}
appState := health.NewState()
consumerCtx, cancelConsumer := context.WithCancel(ctx)
var consumerWg sync.WaitGroup
schedulerCtx, cancelScheduler := context.WithCancel(ctx)
var schedulerWg sync.WaitGroup
queries := db.New(pool)
consumer := consumer.NewOrderConsumer(queries)
relay := scheduler.NewOutboxRelay(queries, pool)
consumerWg.Add(1)
go func() { defer consumerWg.Done(); consumer.Run(consumerCtx) }()
schedulerWg.Add(1)
go func() { defer schedulerWg.Done(); relay.Run(schedulerCtx, &schedulerWg) }()
srv := buildServer(cfg, appState, queries)
go srv.ListenAndServe()
sigC := make(chan os.Signal, 1)
signal.Notify(sigC, syscall.SIGTERM, syscall.SIGINT)
defer signal.Stop(sigC)
<-sigC
slog.InfoContext(ctx, "received SIGTERM, starting graceful shutdown")
// 1. readiness → 503: k8s removes the pod from endpoints
appState.SetNotReady()
// 2. consumer: stop signal + wait for the current message
cancelConsumer()
consumerWg.Wait()
// 3. scheduler: stop signal + wait for the current iteration
cancelScheduler()
schedulerWg.Wait()
// 4. HTTP: drain in-flight requests
shutCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(shutCtx); err != nil {
slog.WarnContext(ctx, "http shutdown", "error", err)
}
// 5. DB pool — last
pool.Close()
slog.InfoContext(ctx, "pgxpool closed")
return nil
}
The consumer and the scheduler hold connections from the pool. Steps 2 and 3 guarantee that they finished their work before step 5.
How to finish active transactions
The mechanism depends on who holds the transaction.
HTTP handler
sqlc queries in the handler run within the pool; srv.Shutdown lets them finish. The request either manages to perform the operation and return a response, or fits within the shutdown timeout — in both cases the database state stays consistent:
func (h *OrderHandler) Create(w http.ResponseWriter, r *http.Request) {
var req CreateOrderRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
httperr.Write(w, r, apperr.NewValidation("decode request"))
return
}
order, err := h.queries.CreateOrder(r.Context(), db.CreateOrderParams{
CustomerID: req.CustomerID,
Amount: req.Amount,
})
if err != nil {
httperr.Write(w, r, fmt.Errorf("create order: %w", err))
return
}
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(order)
}
Background goroutine
Here there is an important point. If the context is already cancelled (ctx.Done() has arrived) but the transaction has already started — you cannot interrupt it. That is why, for transactions outside HTTP, context.Background() is used:
func (s *PaymentSettler) settle(ctx context.Context, orderID uuid.UUID) error {
// ctx may be cancelled — open the transaction on Background,
// so that SIGTERM does not interrupt a write that has started
tx, err := s.pool.Begin(context.Background())
if err != nil {
return fmt.Errorf("begin tx: %w", err)
}
defer tx.Rollback(context.Background())
q := db.New(tx)
order, err := q.LockOrderForUpdate(context.Background(), orderID)
if err != nil {
return fmt.Errorf("lock order %s: %w", orderID, err)
}
if err := q.UpdateOrderStatus(context.Background(), db.UpdateOrderStatusParams{
ID: order.ID,
Status: db.OrderStatusPaid,
}); err != nil {
return fmt.Errorf("update order status %s: %w", orderID, err)
}
if err := tx.Commit(context.Background()); err != nil {
return fmt.Errorf("commit payment settle %s: %w", orderID, err)
}
return nil
}
The signal via ctx.Done() means "do not start the next iteration", not "abort right now". The goroutine checks this before a new turn of the loop:
func (s *PaymentSettler) Run(ctx context.Context, wg *sync.WaitGroup) {
defer wg.Done()
ticker := time.NewTicker(s.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
slog.InfoContext(ctx, "payment settler: shutting down")
return
case <-ticker.C:
if err := s.processNextOrder(ctx); err != nil {
slog.WarnContext(ctx, "payment settler", "error", err)
}
}
}
}
Outbox-relay
The relay captures a batch of events atomically via FOR UPDATE SKIP LOCKED. A batch that has started must be carried through to the end:
func (r *OutboxRelay) processOneBatch(ctx context.Context) error {
tx, err := r.pool.Begin(context.Background())
if err != nil {
return fmt.Errorf("begin outbox tx: %w", err)
}
defer tx.Rollback(context.Background())
q := db.New(tx)
events, err := q.LockOutboxBatch(context.Background(), batchSize)
if err != nil {
return fmt.Errorf("lock outbox batch: %w", err)
}
for _, e := range events {
if err := r.producer.Publish(context.Background(), e); err != nil {
return fmt.Errorf("publish event %s: %w", e.ID, err)
}
if err := q.MarkDispatched(context.Background(), e.ID); err != nil {
return fmt.Errorf("mark dispatched %s: %w", e.ID, err)
}
}
if err := tx.Commit(context.Background()); err != nil {
return fmt.Errorf("commit outbox batch: %w", err)
}
return nil
}
Migrations run only at startup
A common misconception: "you need to clean something up in the database on shutdown". This is almost never needed. The database schema is not rolled back when the service stops.
golang-migrate runs once — at startup, before creating the pool:
func applyMigrations(ctx context.Context, databaseURL string) error {
m, err := migrate.New("file://migrations", databaseURL)
if err != nil {
return fmt.Errorf("migrate.New: %w", err)
}
defer m.Close()
if err := m.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
return fmt.Errorf("migrate up: %w", err)
}
slog.InfoContext(ctx, "migrations applied")
return nil
}
On shutdown there is nothing to do with migrations — m.Close() was already called via defer at startup.
Common mistakes
pool.Close() before WaitGroup.Wait() — goroutines will hit a closed pool in the middle of a transaction. First wait for all goroutines to finish, then close the pool.
pool.Close() in a separate goroutine without synchronization — the shutdown order becomes nondeterministic. Sequential shutdown steps in main are the only reliable way.
tx.Begin(r.Context()) in the critical section of a background task — if the context is already cancelled, the transaction will fail at the start. For background tasks outside HTTP, use context.Background().
slog.Error on a normal pool.Close() — a normal pool closure does not warrant the Error level. Use slog.Info.
In short
- The connection pool is closed last — after
WaitGroup.Wait()for all goroutines. - Cancelling the context means "do not start the next step", not "abort right now".
- Transactions in background goroutines are opened on
context.Background()— so that SIGTERM does not interrupt a write that has already started. - HTTP transactions are drained via
srv.Shutdownwith a timeout. golang-migrate— only at startup; on shutdown there is nothing to do with the database.- A normal
pool.Close()is logged at the Info level, not Error.
What to read next
- HTTP drain —
srv.Shutdown, in-flight requests, long endpoints. - Background tasks and outbox —
sync.WaitGroup,ctx.Done(), the outbox-relay loop. - Kafka shutdown — consumer,
CommitMessages,writer.Close(). - Budgets and observability — structured logging and shutdown metrics.