SLO and Alerts in Java/Spring Boot

Java Backend

SLO (Service Level Objective) in plain terms: what an error budget is, how to compute an SLI in Prometheus, multi-window burn rate from the Google SRE Workbook — fast burn over 1 hour and slow burn over 6 hours. How to avoid drowning in false alerts and why every alert needs a runbook.

Published 28 июня 2026 updated 10 июля 2026 ~7 min read Author: Вадим Викулин

← Back to the section

A team sets up monitoring, alerts fire constantly — and at some point nobody reacts to them anymore. Then, when a real incident happens, it's noticed an hour later. This is the classic problem of "blind monitoring": lots of noise, little signal.

SLO is a way to bring order: agree on what exactly counts as normal operation, and alert only when that norm is violated.

What an SLO is and why you need it

SLO (Service Level Objective) is a numeric target for the service level. Not "everything works," but "99.9% of requests succeed over the last 30 days."

Where the number comes from: the business, together with the team, decides how much loss is acceptable. 99.9% means that over 30 days the service can "spend" 43 minutes on failures — this is the error budget. As long as the budget is not exhausted, the team can ship releases, including risky ones. Once it's exhausted — pause for stabilization.

SLI (Service Level Indicator) is what actually gets measured. Usually two kinds:

Availability: the share of requests without 5xx errors.
Latency: p95 or p99 of the response time.

Why p95 and not the average? The average easily "smears out" problems: if 5% of requests take 10 seconds to answer, the average may still look perfectly acceptable. A percentile shows what's actually happening to real users.

Example SLO for several endpoints:

Endpoint	Availability	Latency
`POST /orders`	99.9% non-5xx	p95 < 500ms
`POST /payments`	99.95% non-5xx	p95 < 1s
`GET /orders/{id}`	99.95% non-5xx	p95 < 200ms

Choose an SLO realistically: 99.99% (52 minutes for the whole year) requires an entirely different infrastructure — multiple regions, immediate failover. That's an order of magnitude more expensive than 99.9% (8.7 hours per year).

How to compute an SLI in Prometheus

Spring Boot with Micrometer automatically publishes the http_server_requests_seconds metric. The SLI is computed from it:

# Availability SLI: share of successful requests over 30 days
sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status!~"5.."}[30d]))
  /
sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[30d]))

# Latency SLI: p95 over 30 days
histogram_quantile(0.95,
  sum by (le) (rate(http_server_requests_seconds_bucket{uri="/orders",method="POST"}[30d]))
)

The result is a number between 0 and 1. If the availability SLI = 0.999, then 99.9% of requests succeed — the SLO is met.

Multi-window burn rate: how to alert correctly

Computing the SLI over 30 days is fine for reports, but for alerts it's too slow. If the service is failing right now, learning about it a week later is useless.

The other extreme: alerting on every single error. That leads to constant noise — the team stops paying attention.

The solution from the Google SRE Workbook is the burn rate (the speed at which the budget is consumed).

How it works. An SLO of 99.9% gives a budget of 0.1% over 30 days. If in 1 hour there are so many errors that at that pace the budget would run out in 20 hours — that's an emergency. The number that expresses this:

burn rate = (current error rate) / (1 - SLO_target)

For an SLO of 99.9%:

burn rate > 14.4 over 1 hour — the budget will run out in ~20 hours. React immediately.
burn rate > 6 over 6 hours — the budget will run out in ~5 days. Investigate before the end of the working day.
burn rate around 1 — normal consumption, everything is fine.

For reliability, use two windows: if the short window shows a fire but the long one is calm — it may be a brief spike rather than a trend. The alert fires only when both windows exceed the threshold.

- alert: OrdersSloFastBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status=~"5.."}[1h]))
      /
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[1h]))
    ) > (14.4 * (1 - 0.999))
  for: 2m
  annotations:
    runbook: https://runbooks.internal/orders-slo-fast-burn

- alert: OrdersSloSlowBurn
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST",status=~"5.."}[6h]))
      /
      sum(rate(http_server_requests_seconds_count{uri="/orders",method="POST"}[6h]))
    ) > (6 * (1 - 0.999))
  for: 15m
  annotations:
    runbook: https://runbooks.internal/orders-slo-slow-burn

Fast burn wakes the on-call engineer immediately — a potentially serious failure. Slow burn creates a ticket in the tracker — the degradation isn't critical but needs investigation.

Alert on budget exhaustion

When less than 10% of the error budget remains, it's a signal to the team: the next month is not the time for risky releases, it's time to focus on resilience.

How to compute the remaining budget:

# How much budget is left: 1 = all free, 0 = all consumed
1 - (
  (1 - <availability_sli_30d>) / (1 - <slo_target>)
)

Alert if less than 10% remains:

- alert: OrdersErrorBudgetExhausted
  expr: <budget_remaining_expression> < 0.1
  for: 1h
  annotations:
    summary: "Only 10% of the error budget remains"
    description: |
      The team switches from features to reliability.
      Releases of risky changes are paused until the budget recovers.

This is not "we must fix it urgently at night" — it's a planned signal for planning the next sprint.

Alerts beyond the SLO

The SLO measures what the user sees. But there are signals that warn about a problem earlier, before it affects users:

What to monitor	Metric	What it means
JVM memory	`jvm_memory_used / max > 0.85`	GC pauses or OutOfMemory coming soon
DB connection pool	`hikaricp_connections_pending > 0`	Requests are waiting for a connection — latency is growing
Business errors	`order_failed_total` rate > 100/min	Something changed in the data or the logic
Circuit Breaker	`resilience4j_circuitbreaker_state{state="open"}`	An external service is unavailable, falling back
Cache hits	`cache_hits / (hits+misses) < 0.7`	The cache barely helps — DB load is growing
Kafka lag	`kafka_consumer_lag_max > 10000`	Consumers can't keep up with producers

For example, a Circuit Breaker in the open state means that requests to an external service are going down a fallback path — the SLO is still healthy, but the problem already exists and may get worse.

Common mistakes

Alerting on every error in the logs

One client keeps trying to send an invalid request 1000 times a minute — 1000 log.error(...) calls turn into 1000 alerts. The team starts ignoring them. Then a real incident happens, and it's noticed too late.

The right approach is to alert on the rate of errors of a specific type, not on every individual case:

- alert: HighErrorRate
  expr: sum by (exception) (rate(app_errors_total[5m])) > 1
  for: 5m
  annotations:
    runbook: https://runbooks.internal/high-error-rate

A single ValidationException is normal for bad input, not an alert. But if there are 100 such exceptions per minute — it's worth investigating.

An SLO of 100%

100% means "we can never make a mistake." In practice, every release becomes a source of anxiety — any regression is an immediate SLO violation, an urgent incident. The error budget is zero — there's no room to maneuver.

99.9% gives 43 minutes a month of "legitimate" time for failures. The team can ship a risky change, confirm the problem, and roll back — all without violating the SLO.

An alert without instructions

PagerDuty wakes the on-call engineer at three in the morning. The alert: "p95 latency on /orders > 1s." Without instructions, the on-call opens Grafana, doesn't know what to look at, doesn't know whom to call. The result — a call to the team lead at three in the morning without any preparation.

A good runbook for the same alert:

Check hikaricp_connections_pending — if it's above zero, scale the application or the database.
Check external_calls_duration_seconds{system="payment-provider"} — if it's above 2 seconds, the incident is on the payment side; you can acknowledge it and wait.
If neither the first nor the second — escalate to #order-service-oncall.

A runbook is a mandatory part of every alert. Without it, the alert is incomplete.

In short

SLO is a numeric target (for example, 99.9% of requests succeed). SLI is what is actually measured in Prometheus.
Error budget is the "allowed" amount of failures. 99.9% gives 43 minutes a month. As long as the budget isn't exhausted, the team can take risks.
Burn rate shows how fast the budget is being consumed. Fast burn (1 hour, threshold 14.4) — immediate reaction. Slow burn (6 hours, threshold 6) — planned investigation.
Use two windows per alert: if only the short window exceeds the threshold, it may be a spike, not a trend.
An alert on budget exhaustion (< 10%) is a signal for planning, not for the on-call engineer.
Infrastructure alerts (JVM memory, DB pool, Circuit Breaker) are separate from the SLO — they warn earlier.
Don't alert on every error — alert on the rate of errors of a specific type.
A 100% SLO is a trap: every release becomes a threat of an incident.
Every alert must have a runbook: what to check, what to do, whom to call.