SLO and Alerts in Java/Spring Boot

← Back to the section

A team sets up monitoring, alerts fire constantly — and at some point nobody reacts to them anymore. Then, when a real incident happens, it's noticed an hour later. This is the classic problem of "blind monitoring": lots of noise, little signal.

SLO is a way to bring order: agree on what exactly counts as normal operation, and alert only when that norm is violated.

Thresholds and formulas come later in the article — here is the whole picture on one scale up front: the same errors, but the decision is made not by their share, but by the speed at which they eat the budget.

What an SLO is and why you need it

SLO (Service Level Objective) is a numeric target for the service level. Not "everything works," but "99.9% of requests succeed over the last 30 days."

Where the number comes from: the business, together with the team, decides how much loss is acceptable. 99.9% means that over 30 days the service can "spend" 43 minutes on failures — this is the error budget. As long as the budget is not exhausted, the team can ship releases, including risky ones. Once it's exhausted — pause for stabilization.

SLI (Service Level Indicator) is what actually gets measured. Usually two kinds:

• Availability: the share of requests without 5xx errors.
• Latency: p95 or p99 of the response time.

Why p95 and not the average? The average easily "smears out" problems: if 5% of requests take 10 seconds to answer, the average may still look perfectly acceptable. A percentile shows what's actually happening to real users.

Example SLO for several endpoints:

Choose an SLO realistically: 99.99% (52 minutes for the whole year) requires an entirely different infrastructure — multiple regions, immediate failover. That's an order of magnitude more expensive than 99.9% (8.7 hours per year).

How to compute an SLI in Prometheus

Spring Boot with Micrometer automatically publishes the http_server_requests_seconds metric. The SLI is computed from it:

# Availability SLI: share of successful requests over 30 days
sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status!~"5.."}[30d]))
  /
sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[30d]))

# Latency SLI: p95 over 30 days
histogram_quantile(0.95,
  sum by (le) (rate(http_server_requests_seconds_bucket{uri="/orders",method="POST"}[30d]))
)

The result is a number between 0 and 1. If the availability SLI = 0.999, then 99.9% of requests succeed — the SLO is met.

Multi-window burn rate: how to alert correctly

Computing the SLI over 30 days is fine for reports, but for alerts it's too slow. If the service is failing right now, learning about it a week later is useless.

The other extreme: alerting on every single error. That leads to constant noise — the team stops paying attention.

The solution from the Google SRE Workbook is the burn rate (the speed at which the budget is consumed).

How it works. An SLO of 99.9% gives a budget of 0.1% over 30 days. If in 1 hour there are so many errors that at that pace the budget would run out in 20 hours — that's an emergency. The number that expresses this:

burn rate = (current error rate) / (1 - SLO_target)

For an SLO of 99.9%:

• burn rate > 14.4 over 1 hour — the budget will run out in ~20 hours. React immediately.
• burn rate > 6 over 6 hours — the budget will run out in ~5 days. Investigate before the end of the working day.
• burn rate around 1 — normal consumption, everything is fine.

Fast burn and slow burn are two separate alerts, each with its own window and its own threshold: 14.4 over 1 hour and 6 over 6 hours. An event doesn't need to cross both thresholds at once: a forty-minute failure at a 6% error rate gives a burn rate of 40 over an hour and only 6.7 over six hours — the first alert is the one that wakes the on-call engineer.

In Google's full recipe each alert also gets a short confirmation window (5 minutes for fast burn, 30 minutes for slow burn), so that the alert clears as soon as the spike is over instead of hanging around for another hour. In the example below, for plays that role.

- alert: OrdersSloFastBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status=~"5.."}[1h]))
      /
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[1h]))
    ) > (14.4 * (1 - 0.999))
  for: 2m
  annotations:
    runbook: https://runbooks.internal/orders-slo-fast-burn

- alert: OrdersSloSlowBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status=~"5.."}[6h]))
      /
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[6h]))
    ) > (6 * (1 - 0.999))
  for: 15m
  annotations:
    runbook: https://runbooks.internal/orders-slo-slow-burn

Fast burn wakes the on-call engineer immediately — a potentially serious failure. Slow burn creates a ticket in the tracker — the degradation isn't critical but needs investigation.

Alert on budget exhaustion

When less than 10% of the error budget remains, it's a signal to the team: the next month is not the time for risky releases, it's time to focus on resilience.

How to compute the remaining budget:

# How much budget is left: 1 = all free, 0 = all consumed
1 - (
  (1 - <availability_sli_30d>) / (1 - <slo_target>)
)

Alert if less than 10% remains:

- alert: OrdersErrorBudgetExhausted
  expr: <budget_remaining_expression> < 0.1
  for: 1h
  annotations:
    summary: "Only 10% of the error budget remains"
    description: |
      The team switches from features to reliability.
      Releases of risky changes are paused until the budget recovers.

This is not "we must fix it urgently at night" — it's a planned signal for planning the next sprint.

Alerts beyond the SLO

The SLO measures what the user sees. But there are signals that warn about a problem earlier, before it affects users:

For example, a Circuit Breaker in the open state means that requests to an external service are going down a fallback path — the SLO is still healthy, but the problem already exists and may get worse.

Common mistakes

Alerting on every error in the logs

One client keeps trying to send an invalid request 1000 times a minute — 1000 log.error(...) calls turn into 1000 alerts. The team starts ignoring them. Then a real incident happens, and it's noticed too late.

The right approach is to alert on the rate of errors of a specific type, not on every individual case:

- alert: HighErrorRate
  expr: sum by (exception) (rate(app_errors_total[5m])) > 1
  for: 5m
  annotations:
    runbook: https://runbooks.internal/high-error-rate

A single ValidationException is normal for bad input, not an alert. But if there are 100 such exceptions per minute — it's worth investigating.

An SLO of 100%

100% means "we can never make a mistake." In practice, every release becomes a source of anxiety — any regression is an immediate SLO violation, an urgent incident. The error budget is zero — there's no room to maneuver.

99.9% gives 43 minutes a month of "legitimate" time for failures. The team can ship a risky change, confirm the problem, and roll back — all without violating the SLO.

An alert without instructions

PagerDuty wakes the on-call engineer at three in the morning. The alert: "p95 latency on /orders > 1s." Without instructions, the on-call opens Grafana, doesn't know what to look at, doesn't know whom to call. The result — a call to the team lead at three in the morning without any preparation.

A good runbook for the same alert:

1. Check hikaricp_connections_pending — if it's above zero, scale the application or the database.
2. Check external_calls_duration_seconds{system="payment-provider"} — if it's above 2 seconds, the incident is on the payment side; you can acknowledge it and wait.
3. If neither the first nor the second — escalate to #order-service-oncall.

A runbook is a mandatory part of every alert. Without it, the alert is incomplete.

In short

• SLO is a numeric target (for example, 99.9% of requests succeed). SLI is what is actually measured in Prometheus.
• Error budget is the "allowed" amount of failures. 99.9% gives 43 minutes a month. As long as the budget isn't exhausted, the team can take risks.
• Burn rate shows how fast the budget is being consumed. Fast burn (1 hour, threshold 14.4) — immediate reaction. Slow burn (6 hours, threshold 6) — planned investigation.
• Use two windows per alert: if only the short window exceeds the threshold, it may be a spike, not a trend.
• An alert on budget exhaustion (< 10%) is a signal for planning, not for the on-call engineer.
• Infrastructure alerts (JVM memory, DB pool, Circuit Breaker) are separate from the SLO — they warn earlier.
• Don't alert on every error — alert on the rate of errors of a specific type.
• A 100% SLO is a trap: every release becomes a threat of an incident.
• Every alert must have a runbook: what to check, what to do, whom to call.

What to read next

• Metrics in Spring Boot — configuring http_server_requests_seconds to compute an SLI.
• Tracing in Spring Boot — a detailed look at problematic requests through traces.
• Health checks — why kubernetes probes don't replace an SLO.
• Google SRE Workbook — Alerting on SLOs — the primary source for multi-window burn rate.