When a service goes to production, one question comes up: "is everything working fine right now?". Logs show events after the fact, and traces show individual requests. Metrics are a continuous picture: how many requests per second, what percentage fail, how much memory is used, whether the connection pool is exhausted. Metrics are the first to signal an anomaly, even before users start complaining.
How the Micrometer + Prometheus pairing works
In the past, every monitoring tool required its own library. Prometheus needed one dependency, Datadog another. You would have to rewrite code every time you switched the backend.
Micrometer solves this the way SLF4J does for logs: a unified API for recording metrics, with the concrete backend plugged in separately. You write counter.increment() and Micrometer forwards the value to Prometheus, Datadog, or any other configured registry.
Prometheus is a time-series storage system. It comes to the service on a schedule (usually every 15 seconds) and pulls metrics from the /actuator/prometheus endpoint in text format. Then you build dashboards and configure alerts in Grafana.
The flow: service → /actuator/prometheus → Prometheus scraper → Prometheus TSDB → Grafana.
Setup
Two dependencies in build.gradle.kts:
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")
Expose the endpoint in application.yml:
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus
After startup GET /actuator/prometheus returns hundreds of lines with JVM, HTTP, and connection-pool metrics — all automatically, without a single line of code.
Standard tags for all metrics
A common problem: you see metrics on the dashboard, but it is unclear — is this production or staging? Version 1.2.3 or 1.3.0?
The solution is to add the service, env, and version tags once in the configuration, and they will automatically appear on every metric:
spring:
application:
name: order-service
management:
metrics:
tags:
service: ${spring.application.name}
env: ${ENV:dev}
version: ${BUILD_VERSION:unknown}
Now in Grafana you can filter: service="order-service", env="prod" — and see only what you need. And compare deployments: version="1.2.3" versus version="1.2.4".
Important: you do not need to add .tag("service", "order-service") manually on every metric — global tags are applied automatically. Duplicating one triggers IllegalArgumentException: duplicate tag.
The RED method for HTTP: what to look at first
RED is three questions about any request-driven service:
- Rate — how many requests per second?
- Errors — what percentage fail?
- Duration — how long do requests take?
Spring Boot Actuator + Micrometer collect this automatically through the http_server_requests_seconds metric. PromQL queries for Grafana:
# Rate — requests per second by endpoint
sum(rate(http_server_requests_seconds_count[5m])) by (uri, method)
# Errors — share of 5xx errors
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (uri)
# Duration — p95 latency
histogram_quantile(0.95, sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m])))
To see p95/p99 latency, you need to enable the histogram:
management:
metrics:
distribution:
percentiles-histogram:
http.server.requests: true
slo:
http.server.requests: 100ms,500ms,1s,5s
slo sets the bucket boundaries: Prometheus will count how many requests fit within 100ms, 500ms, and so on. This lets you formulate an SLO: "95% of requests faster than 500ms".
The USE method for resources: memory, threads, connections
USE is three questions about any resource:
- Utilization — how busy is the resource?
- Saturation — is there a queue of waiters?
- Errors — are there errors at the resource level?
Spring Boot automatically exports JVM and infrastructure metrics:
| Metric | What it shows |
|---|---|
jvm_memory_used_bytes{area="heap"} | used heap memory |
jvm_memory_max_bytes{area="heap"} | maximum heap |
jvm_gc_pause_seconds_sum | total GC pause time |
executor_active_threads{name="taskExecutor"} | active threads in the pool |
hikaricp_connections_active | active database connections |
hikaricp_connections_pending | requests waiting for a connection |
Example alerts:
- alert: HeapMemoryHigh
expr: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} > 0.85
for: 10m
- alert: HikariPoolSaturated
expr: hikaricp_connections_pending > 0
for: 5m
hikaricp_connections_pending > 0 means requests are queued for a database connection — a sure sign of pool saturation.
Your own business metrics through MeterRegistry
The built-in metrics are enough for infrastructure. But business events — "order created", "payment processed", "cart total" — you have to add yourself.
Micrometer offers four instruments:
- Counter — a monotonically increasing counter. Good for events: order created, payment declined.
- Gauge — a current value that can go both up and down. Good for states: queue size, number of active users.
- Timer — operation duration with a histogram. Good for measuring processing time.
- DistributionSummary — arbitrary numbers with a histogram. Good for cart totals, file sizes.
@Component
public class OrderMetrics {
private final Counter orderCreatedCounter;
private final Timer paymentProcessingTimer;
private final DistributionSummary orderAmountSummary;
public OrderMetrics(MeterRegistry registry) {
this.orderCreatedCounter = Counter.builder("order_created_total")
.description("Total orders created")
.tag("channel", "web")
.register(registry);
this.paymentProcessingTimer = Timer.builder("payment_processing_seconds")
.publishPercentiles(0.5, 0.95, 0.99)
.register(registry);
this.orderAmountSummary = DistributionSummary.builder("order_amount_rubles")
.baseUnit("rubles")
.register(registry);
}
public void orderCreated() { orderCreatedCounter.increment(); }
public void recordPaymentDuration(Duration duration) {
paymentProcessingTimer.record(duration);
}
public void recordOrderAmount(BigDecimal amount) {
orderAmountSummary.record(amount.doubleValue());
}
}
Usage in a service:
@Service
@RequiredArgsConstructor
public class CreateOrderService {
private final OrderRepository orderRepository;
private final OrderMetrics metrics;
@Transactional
public Order create(CreateOrderCommand command) {
var order = orderRepository.save(Order.create(command));
metrics.orderCreated();
metrics.recordOrderAmount(order.amount());
return order;
}
}
How to name metrics
Prometheus follows a convention: snake_case, unit of measurement in the name.
Correct:
order_created_total— a counter with the_totalsuffixpayment_processing_seconds— a timer with_secondsorder_amount_rubles— a summary with a currency unitqueue_size— a gauge, number of items
Incorrect:
orderCreatedCount— camelCase, no_totalpaymentTime— no unit of measurementorder_processing_ms— Prometheus prefers seconds, not milliseconds
Low tag cardinality — the most important rule
Prometheus stores a separate time series for each unique combination of tag values. So if the user_id tag takes a million values, the metric spawns a million time series — and that is just for one metric.
Good tags are categories with a small number of values:
// Correct — 3-10 values per tag
counter.tag("channel", "web") // web, mobile, api
.tag("payment_method", "card") // card, sbp, crypto
.increment();
Bad tags are unique identifiers:
// Wrong — a million values = a million time series
counter.tag("user_id", String.valueOf(userId)) // unique for each user
.tag("order_id", String.valueOf(orderId)) // unique for each order
.increment();
A million user_id × a million order_id × several environment tags = billions of time series. Prometheus will not cope, and the scraper will crash with OOM.
If you need detail on a specific user or request, that is a job for tracing (Tempo/Jaeger), not metrics. Each span in a trace carries arbitrary attributes with no cardinality limits.
Common mistakes
Micrometer without a Prometheus registry. If you add only micrometer-core, metrics are collected into the built-in SimpleMeterRegistry and kept in memory only. Everything is lost on restart, and Prometheus sees nothing. You need micrometer-registry-prometheus.
An exposed /actuator/prometheus endpoint. This endpoint can leak sensitive data: revenue totals, payment counters, business KPIs. In production its access is restricted with network policies — only the Prometheus scraper from the monitoring namespace. A handy trick is to move the actuator to a separate port:
management:
server:
port: 8081
Then business traffic goes to 8080 and actuator traffic to 8081, which is not exposed externally through the Ingress.
Non-standard environment tag names. If one service writes app=foo and another service_name=foo, cross-service dashboards in Grafana break. Agree at the configuration level (management.metrics.tags) and stick to one standard across all services.
In short
- Micrometer is a facade for metrics (like SLF4J for logs), Prometheus is the storage backend, and
/actuator/prometheusis the collection point. - The
service,env, andversiontags are configured once throughmanagement.metrics.tagsand applied globally. - RED (Rate, Errors, Duration) — three questions for HTTP;
http_server_requests_secondsis collected automatically. - USE (Utilization, Saturation, Errors) — three questions for resources; JVM, Hikari, and thread pools are collected automatically.
- Your own metrics:
Counterfor events,Gaugefor states,Timerfor duration,DistributionSummaryfor numeric distributions. - Metric names:
snake_case, unit in the name (_seconds,_total,_bytes). - High-cardinality tags (
user_id,order_id) blow up Prometheus — use only categorical values with a small number of options. - Keep the
/actuator/prometheusendpoint off public access: a separate port or network policies.
What to read next
- Request tracing — when you need detail on a specific request rather than an aggregate.
- Logging — structured JSON logs and context via MDC.
- Observability configuration — management port, Logback profiles.
- Health checks — liveness vs readiness and custom indicators.