SLO and alerts in Node.js: error budget, burn rate and how not to miss an incident

← Back to the section

The on-call engineer receives two hundred alerts over the night. By morning they start muting channels and ignoring notifications. At some point a real incident arrives — and no one reacts.

This is classic "alert fatigue". To avoid it, you need a system where every alert means a real problem and every alert explains what to do. This is exactly what SLOs exist for.

What an SLO is and why you need it

SLO (Service Level Objective) is a concrete promise of a service level. Not "the service should work well", but "99.9% of requests to POST /orders should complete successfully within a 30-day window".

Why formalize this? Because from an SLO an error budget automatically follows — the number of errors the service can "spend" over a period and still meet the promise.

An SLO of 99.9% means an error budget of 0.1% over 30 days. Translated into minutes — that's about 43 minutes of allowed downtime per month. This budget becomes the measure that links reliability and development speed: when the budget runs out, risky releases are paused until it recovers.

SLI (Service Level Indicator) is what measures fulfillment of the SLO. For a web service this is usually the fraction of successful requests (availability) and latency.

An important point: an SLO must not be 100%. 100% is a different architecture (multi-region, immediate failover), different costs. For most services 99.9% or 99.95% is a reasonable choice, discussed with the product owner.

How to measure an SLI in NestJS

The SLI is computed from metrics. In NestJS it's convenient to collect them via prom-client using an interceptor that measures every HTTP request.

// metrics/http-metrics.interceptor.ts
@Injectable()
export class HttpMetricsInterceptor implements NestInterceptor {
  private readonly histogram: Histogram<string>;

  constructor() {
    this.histogram = new Histogram({
      name: 'http_server_requests_seconds',
      help: 'HTTP request duration',
      labelNames: ['method', 'route', 'status_class'] as const,
      buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
    });
  }

  intercept(ctx: ExecutionContext, next: CallHandler): Observable<unknown> {
    const req = ctx.switchToHttp().getRequest<Request>();
    const route = req.route?.path ?? 'unknown';
    const method = req.method;
    const end = this.histogram.startTimer();

    return next.handle().pipe(
      tap({
        next: () => {
          const status = ctx.switchToHttp().getResponse<Response>().statusCode;
          end({ method, route, status_class: statusClass(status) });
        },
        error: (err: unknown) => {
          const code = err instanceof HttpException ? err.getStatus() : 500;
          end({ method, route, status_class: statusClass(code) });
        },
      }),
    );
  }
}

function statusClass(code: number): string {
  if (code < 400) return 'success';
  if (code < 500) return 'client_error';
  return 'server_error';
}

Note the route label — it's the template /orders/:id, not the raw URL with a real order identifier. If you use the raw URL, the number of unique label values grows to millions and Prometheus starts consuming a huge amount of memory.

Next, the SLI is computed in Prometheus:

# Availability SLI: fraction of successful POST /orders requests over 30 days
sum(rate(http_server_requests_seconds_count{route="/orders",method="POST",status_class="success"}[30d]))
  /
sum(rate(http_server_requests_seconds_count{route="/orders",method="POST"}[30d]))

# Latency SLI: 95th percentile of latency
histogram_quantile(0.95,
  sum by (le) (rate(http_server_requests_seconds_bucket{route="/orders",method="POST"}[30d]))
)

Latency is computed via histogram_quantile (p95 or p99), never via the average — the average hides the tails, and that's exactly where the slow requests that annoy users live.

Multi-window alerts: fast and slow burn

A simple alert "errors exceeded N%" works poorly in practice: it's either too sensitive (wakes you at night because of a short-lived spike) or too dumb (notices the problem only when the budget is already almost burned).

The solution is the approach from the Google SRE Workbook: count not just errors, but the budget burn rate.

The formula:

burn rate = current error rate / (1 - SLO target)

For an SLO of 99.9% the denominator equals 0.001. If over the last hour errors were 1.44% — the burn rate equals 14.4. This means that at such a rate the entire monthly budget will be gone in about 2 days.

In practice two windows are used:

Fast burn (window 1 hour, burn rate threshold > 14.4) — 5% of the budget in an hour, this is a catastrophe. Wakes the on-call immediately.
Slow burn (window 6 hours, burn rate threshold > 6) — a sustained degradation. Creates a ticket to be dealt with in the morning.

# Alertmanager — alerts for POST /orders, SLO 99.9%
- alert: OrdersSloFastBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{route="/orders",method="POST",status_class="server_error"}[1h]))
      /
      sum(rate(http_server_requests_seconds_count{route="/orders",method="POST"}[1h]))
    ) > (14.4 * (1 - 0.999))
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "POST /orders fast burn — SLO at risk"
    runbook: "https://runbooks.internal/orders-slo-fast-burn"

- alert: OrdersSloSlowBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{route="/orders",method="POST",status_class="server_error"}[6h]))
      /
      sum(rate(http_server_requests_seconds_count{route="/orders",method="POST"}[6h]))
    ) > (6 * (1 - 0.999))
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "POST /orders slow burn — degradation"
    runbook: "https://runbooks.internal/orders-slo-slow-burn"

for: 2m on fast burn excludes instant spikes — the alert won't fire if the problem passed on its own within a minute. for: 15m on slow burn makes sure the degradation is sustained, not random.

When the budget is almost gone

A separate alert fires when the remaining error budget drops below 10%. This is not a signal to fix something at night — it's a signal to the product owner and the team: the next sprint should be focused on reliability, and risky releases paused.

- alert: OrdersErrorBudgetExhausted
  expr: <budget_remaining_expr> < 0.1
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Less than 10% error budget left on POST /orders"
    description: |
      The team switches from new features to reliability.
      Risky releases are paused until the budget recovers.
    runbook: "https://runbooks.internal/orders-error-budget"

The remaining-budget formula:

1 - (
  (1 - sum(rate(http_server_requests_seconds_count{route="/orders",method="POST",status_class="success"}[30d]))
       / sum(rate(http_server_requests_seconds_count{route="/orders",method="POST"}[30d])))
  / (1 - 0.999)
)

Business metrics alongside the SLO

HTTP metrics tell you about technical failures, but not always about business problems. A request may return 200 OK, but the order wasn't created because of a logic error.

Domain counters in NestJS help track this:

// metrics/order.metrics.ts
export const orderCreatedTotal = new Counter({
  name: 'order_created_total',
  help: 'Orders successfully created',
  labelNames: ['type'] as const,
});

export const orderFailedTotal = new Counter({
  name: 'order_failed_total',
  help: 'Orders failed during processing',
  labelNames: ['reason'] as const,
});

// use-cases/create-order/create-order.handler.ts
async handle(cmd: CreateOrderCommand): Promise<OrderId> {
  try {
    const order = await this.orderRepository.save(Order.create(cmd));
    orderCreatedTotal.inc({ type: cmd.type });
    return order.id;
  } catch (err) {
    orderFailedTotal.inc({ reason: classifyReason(err) });
    throw err;
  }
}

The reason label is a category (validation_error, payment_declined, inventory_unavailable), not a raw error message. Raw messages are again high-cardinality, which breaks Prometheus.

Infrastructure alerts: early signs

These alerts aren't about the SLO directly, but they warn about a problem before it affects users.

Event loop lag — if Node.js slows down in the event loop, latencies will grow within 1–2 minutes:

- alert: NodeEventLoopLag
  expr: nodejs_eventloop_lag_seconds > 0.1
  for: 3m
  annotations:
    summary: "Event loop lag > 100ms"
    runbook: "https://runbooks.internal/node-event-loop-lag"

Memory shortage — when the heap is more than 85% occupied:

- alert: NodeHeapSaturation
  expr: nodejs_heap_used_bytes / nodejs_heap_size_limit > 0.85
  for: 5m
  annotations:
    summary: "Node.js heap is 85%+ full"

Job queue — if BullMQ accumulates jobs faster than it processes them:

- alert: OrderEventsQueueLag
  expr: bullmq_queue_waiting_total{queue="order-events"} > 500
  for: 5m
  annotations:
    summary: "BullMQ order-events queue lag > 500"
    runbook: "https://runbooks.internal/bullmq-order-events-lag"

Database connection pool metrics are convenient to collect via setInterval:

@Injectable()
export class PgPoolMetricsService implements OnModuleInit {
  private readonly poolWaiting: Gauge<string>;

  constructor(private readonly pool: Pool) {
    this.poolWaiting = new Gauge({
      name: 'pg_pool_waiting_count',
      help: 'pg pool waiting connections',
      labelNames: [],
    });
  }

  onModuleInit(): void {
    setInterval(() => {
      this.poolWaiting.set(this.pool.waitingCount);
    }, 5000);
  }
}

Runbook — an instruction for each alert

An alert without an explanation of what to do is a call at 3 a.m. without an explanation of why. The on-call opens Slack, sees "something broke" and starts from scratch: looking for graphs, recalling the architecture, guessing the cause.

Every alert must contain annotations.runbook — a link to a page with a clear order of actions: what to check first, how to diagnose, whom to call if the problem isn't solved within N minutes.

A good runbook answers the questions:

What does this alert mean?
Which graphs to look at?
What to do, step by step?
When to escalate?

Common mistakes

An alert on every event in the logs. logger.error(...) is not an alert. An alert is an anomaly in the rate. Use rate(app_errors_total[5m]) > threshold grouped by category.

SLO 100%. It's unachievable without enormous costs and it blocks the team: any release becomes risky. Discuss a realistic target with the product owner.

One broad alert "something broke". Separate categories — SLO burn, infrastructure, domain, queues — help quickly understand where to dig.

No for: in the alert. Without it the alert fires on any instant spike. At minimum: for: 2m for critical, for: 5m for warnings.

In short

SLO — a concrete promise (for example, 99.9% of successful requests over 30 days). Error budget — the allowed number of errors that follows from the SLO.
SLI is computed via http_server_requests_seconds with the status_class label, latency via histogram_quantile (p95/p99), not via the average.
Multi-window burn rate: fast burn (1 hour, rate > 14.4) — wakes the on-call, slow burn (6 hours, rate > 6) — creates a ticket.
The budget-exhaustion alert (< 10%) — not to fix at night, but a signal for the team to switch to reliability.
Infrastructure alerts (event loop lag, heap, connection pool, queues) — a separate category, early signs of problems.
Business metrics (order_failed_total) — separate counters in handlers, labels — categories, not raw messages.
Every alert contains annotations.runbook. Without it an alert is a call with no explanation.