SLO and alerts in Go: error budget and multi-window burn rate

← Back to the section

When a service starts to slow down or fail, you need to know about it before users do. In this article we'll look at what an SLO is, why a single "alert on every error" doesn't work, and how to set up alerting properly with Prometheus in a Go service.

Why "error → alert" doesn't work

Many people's first idea is this: every time an ERROR appears in the logs — send a notification. It seems reliable. In practice — the team starts ignoring the alert channel within a week.

The problem is that isolated errors are normal. A user made a mistake in a request, an external service was briefly unavailable, the network flickered. If a notification arrives on every such error, it's just noise. And the signal of a real outage drowns in the noise.

The right approach: an alert fires not on a single error, but on the rate at which the allowance of acceptable errors is being spent.

SLO and error budget in plain terms

SLO (Service Level Objective) is a promise about the level of service. Not "the service always works", but a concrete figure: "99.9% of requests succeed over the last 30 days".

The difference is fundamental. If we take 99.9% as the target, we get an error budget:

• 99.9% success → 0.1% can be spent on errors
• Over 30 days that's ≈ 43 minutes of acceptable downtime

This budget is not just a technical figure. It answers the question: "can we ship a risky release right now?". If the budget is almost untouched — yes. If 5% is left — first deal with reliability.

A target of 99.99% (52 minutes a year) means a completely different level of cost: multiple regions, active replication. 99.9% (8.7 hours a year) is achievable in a single region with a normal on-call rotation.

Metrics from the chi middleware

The SLO is computed from the metrics the middleware writes on every request. The two key counters are the number of requests by status and their duration:

// internal/platform/middleware/metrics.go
var (
    httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    }, []string{"method", "path", "status_class"})

    httpRequestDurationSeconds = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency",
        Buckets: []float64{0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0},
    }, []string{"method", "path", "status_class"})
)

func Metrics(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ww := middleware.NewWrapResponseWriter(w, r.ProtoMajor)
        start := time.Now()
        next.ServeHTTP(ww, r)
        status := statusClass(ww.Status())
        path := chi.RouteContext(r.Context()).RoutePattern()
        httpRequestsTotal.WithLabelValues(r.Method, path, status).Inc()
        httpRequestDurationSeconds.WithLabelValues(r.Method, path, status).Observe(time.Since(start).Seconds())
    })
}

func statusClass(code int) string {
    switch {
    case code < 400:
        return "success"
    case code < 500:
        return "client_error"
    default:
        return "server_error"
    }
}

An important detail: path comes from chi.RouteContext(...).RoutePattern() — that's the template /orders/{orderID}, not the actual URL /orders/ord-123. If you record concrete IDs in the label, a separate time series is created for every order, and Prometheus will soon crash out of memory.

SLO targets for endpoints

Different endpoints matter differently. A payment endpoint demands higher availability than search:

The targets are chosen together with product: the technical side says what it takes to reach each level, the business side decides whether the cost is justified.

PromQL for the SLI

The SLI (Service Level Indicator) is the actually measured value used to judge whether the SLO is met:

# Availability of POST /orders over the last 30 days
sum(rate(http_requests_total{path="/orders",method="POST",status_class="success"}[30d]))
  /
sum(rate(http_requests_total{path="/orders",method="POST"}[30d]))

# p99 latency over 30 days
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{path="/orders",method="POST"}[30d]))
)

For latency use p99 (the 99th percentile), not the average. The average hides the "tail": if 1% of requests take 10 seconds, the average may look perfectly fine.

Multi-window burn rate

A single alert "SLO violated over 30 days" arrives too late — the budget is already spent. You need to warn earlier, while the outage is just flaring up.

The idea: compute the budget-burn rate (burn rate) over several time windows.

burn_rate = error_rate_in_window / acceptable_error_rate

For an SLO of 99.9% the acceptable error rate = 0.1%. If 1.44% errors are observed over an hour — that's a burn rate of 14.4×: at that pace the whole monthly budget will run out in 20 hours.

This approach is called multi-window multi-burn-rate and is described in the Google SRE Workbook. The gist: the short window reacts quickly to acute outages, the long one to slow degradation.

# ops-repo/alerts/order-slo.yaml
groups:
  - name: order-slo
    rules:
      - alert: OrdersAvailabilityFastBurn
        expr: |
          (
            sum(rate(http_requests_total{path="/orders",method="POST",status_class="server_error"}[1h]))
            /
            sum(rate(http_requests_total{path="/orders",method="POST"}[1h]))
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Orders SLO: fast error-budget burn (1h window)"
          runbook_url: "https://runbooks.internal/orders-slo-fast-burn"

      - alert: OrdersAvailabilitySlowBurn
        expr: |
          (
            sum(rate(http_requests_total{path="/orders",method="POST",status_class="server_error"}[6h]))
            /
            sum(rate(http_requests_total{path="/orders",method="POST"}[6h]))
          ) > (6 * (1 - 0.999))
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Orders SLO: slow error-budget burn (6h window)"
          runbook_url: "https://runbooks.internal/orders-slo-slow-burn"

The for: 2m on the fast alert means: the condition must hold continuously for 2 minutes before the notification arrives. This filters out brief spikes. The for: 15m on the slow one means a sustained degradation, not a random burst.

A latency alert

- alert: OrdersLatencyP99High
  expr: |
    histogram_quantile(0.99,
      sum by (le) (rate(http_request_duration_seconds_bucket{path="/orders",method="POST"}[5m]))
    ) > 0.5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Orders p99 latency > 500ms"
    runbook_url: "https://runbooks.internal/orders-latency-high"

An alert for budget exhaustion

A separate alert for when less than 10% of the budget remains over 30 days. This isn't a reason to wake someone at night — it's a signal for planning the next sprint.

A recording rule computes the remaining budget in advance, so the alert doesn't spend resources on a heavy query every minute:

groups:
  - name: order-slo-recording
    rules:
      - record: job:orders_availability_sli:rate30d
        expr: |
          sum(rate(http_requests_total{path="/orders",method="POST",status_class="success"}[30d]))
            /
          sum(rate(http_requests_total{path="/orders",method="POST"}[30d]))

      - record: job:orders_error_budget_remaining:ratio
        expr: |
          1 - ((1 - job:orders_availability_sli:rate30d) / (1 - 0.999))

  - name: order-slo-alerts
    rules:
      - alert: OrdersErrorBudgetExhausted
        expr: job:orders_error_budget_remaining:ratio < 0.1
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Orders: less than 10% of the error budget remaining"
          description: |
            Risky releases are paused until the budget recovers.
            Next sprint — work on reliability.
          runbook_url: "https://runbooks.internal/orders-error-budget-exhausted"

Alerts outside the SLO

The SLO reflects the user experience — the success of requests. But there are signals worth tracking separately, with different reaction criteria.

Business metrics

Business errors (a declined payment, a failed validation) don't necessarily show up as 5xx. They should be counted separately:

// internal/order/metrics.go
var (
    ordersFailedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "orders_failed_total",
        Help: "Orders that failed business validation or processing",
    }, []string{"reason"})

    paymentDeclinedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "payment_declined_total",
        Help: "Payments declined by payment provider",
    }, []string{"decline_code"})
)

The label values are only low-cardinality: insufficient_funds, card_expired, fraud_suspected. No order_id or customer_id in labels — that's an explosive growth of time series.

Process resources

Goroutines and the database connection pool:

// cmd/server/main.go
prometheus.MustRegister(collectors.NewGoCollector())
prometheus.MustRegister(collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))

A custom collector for pgxpool:

// internal/platform/metrics/pgpool.go
type pgPoolCollector struct {
    pool          *pgxpool.Pool
    emptyAcquires *prometheus.Desc
    idle          *prometheus.Desc
}

func NewPgPoolCollector(pool *pgxpool.Pool) prometheus.Collector {
    return &pgPoolCollector{
        pool: pool,
        emptyAcquires: prometheus.NewDesc("pgxpool_empty_acquires_total",
            "Cumulative number of acquisitions that found the pool empty", nil, nil),
        idle: prometheus.NewDesc("pgxpool_idle_connections",
            "Idle connections in the pool", nil, nil),
    }
}

func (c *pgPoolCollector) Collect(ch chan<- prometheus.Metric) {
    stat := c.pool.Stat()
    ch <- prometheus.MustNewConstMetric(c.emptyAcquires, prometheus.CounterValue,
        float64(stat.EmptyAcquireCount()))
    ch <- prometheus.MustNewConstMetric(c.idle, prometheus.GaugeValue,
        float64(stat.IdleConns()))
}

func (c *pgPoolCollector) Describe(ch chan<- *prometheus.Desc) {
    ch <- c.emptyAcquires
    ch <- c.idle
}

An alert on pool saturation — via rate() over the counter of empty connection-acquire attempts:

- alert: PgPoolEmptyAcquiresHigh
  expr: rate(pgxpool_empty_acquires_total[1m]) > 5
  for: 1m
  labels:
    severity: warning
  annotations:
    runbook_url: "https://runbooks.internal/pgpool-saturation"

A summary map of alerts

The runbook — a mandatory part of the alert

Every alert must contain annotations.runbook_url. A notification at 3 a.m. with no instructions isn't a call to action, it's a riddle. The on-call spends time on an investigation that someone has already done.

A good runbook is short and concrete. For OrdersAvailabilityFastBurn:

# Orders SLO Fast Burn — Runbook

## Symptoms
Burn rate > 14.4× over the last hour.

## Diagnosis (in order)
1. `rate(http_requests_total{path="/orders",status_class="server_error"}[5m])` — the absolute 5xx rate.
2. `pgxpool_waiting_connections > 0` — not enough database connections.
3. `circuit_breaker_state{service="payment-provider"} == 1` — the external service is unavailable.
4. Grafana → traces with `status=ERROR` on `/orders` over the last 30 minutes.

## Actions
- If #2: increase the number of replicas or the connection pool size.
- If #3: acknowledge the alert, wait for recovery; enable the fallback path if there is one.
- Otherwise: escalate to `#order-service-oncall`.

The Go code contains no runbook — it only supplies the metrics. The runbook is kept in the ops repository, and the link to it is in the YAML alert rule.

Common mistakes

An alert on every log record with the ERROR level. The result — the team turns off notifications. The right way: count the aggregated error rate, not individual events.

SLO = 100%. Mathematically this means a zero budget: any error violates the target. Realistic values are 99.9% or 99.95%.

An alert without for:. It fires on any brief spike. Even for: 2m removes most of the noise.

A single alert for the whole 30-day window. It arrives too late. You need a short window (1h) for acute outages and a long one (6h) for slow degradation.

The average instead of a percentile for latency. The average hides slow requests. Use p99 (or p95).

In short

• An SLO is a promise about the level of service in numbers: "99.9% of requests succeed over 30 days".
• The error budget is the acceptable allowance of failures. For 99.9% that's ≈ 43 minutes a month.
• You should alert not on individual errors, but on the budget-burn rate.
• Multi-window: a short window (1h, burn > 14.4×) — wake someone immediately; a long one (6h, burn > 6×) — create a task.
• For latency — p99, not the average; without for: the alert fires on noise.
• Business metrics, process resources and queues — separate alerts with their own runbooks.
• Every alert contains a runbook_url. A notification without instructions is a riddle for the on-call.
• The Go code only supplies metrics; the alert rules and runbooks are in the ops repository.

What to read next

• Metrics in Go — chi middleware, RED metrics, the pgxpool Collector.
• Tracing in Go — a detailed incident breakdown via OTel traces.
• Health checks in Go — liveness/readiness and why they don't replace SLOs.
• Logging in Go — slog, the OTel bridge, the log → trace link.