← Back to the section

When a service goes to production, one question comes up: "is everything working fine right now?". Logs show events after the fact, and traces show individual requests. Metrics are a continuous picture: how many requests per second, what percentage fail, how much memory is used, whether the connection pool is exhausted. Metrics are the first to signal an anomaly, even before users start complaining.

How the Micrometer + Prometheus pairing works

In the past, every monitoring tool required its own library. Prometheus needed one dependency, Datadog another. You would have to rewrite code every time you switched the backend.

Micrometer solves this the way SLF4J does for logs: a unified API for recording metrics, with the concrete backend plugged in separately. You write counter.increment() and Micrometer forwards the value to Prometheus, Datadog, or any other configured registry.

Prometheus is a time-series storage system. It comes to the service on a schedule (usually every 15 seconds) and pulls metrics from the /actuator/prometheus endpoint in text format. Then you build dashboards and configure alerts in Grafana.

The flow: service → /actuator/prometheus → Prometheus scraper → Prometheus TSDB → Grafana.

Setup

Two dependencies in build.gradle.kts:

implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")

Expose the endpoint in application.yml:

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

After startup GET /actuator/prometheus returns hundreds of lines with JVM, HTTP, and connection-pool metrics — all automatically, without a single line of code.

Standard tags for all metrics

A common problem: you see metrics on the dashboard, but it is unclear — is this production or staging? Version 1.2.3 or 1.3.0?

The solution is to add the service, env, and version tags once in the configuration, and they will automatically appear on every metric:

spring:
  application:
    name: order-service

management:
  metrics:
    tags:
      service: ${spring.application.name}
      env: ${ENV:dev}
      version: ${BUILD_VERSION:unknown}

Now in Grafana you can filter: service="order-service", env="prod" — and see only what you need. And compare deployments: version="1.2.3" versus version="1.2.4".

Important: you do not need to add .tag("service", "order-service") manually on every metric — global tags are applied automatically. Duplicating one triggers IllegalArgumentException: duplicate tag.

The RED method for HTTP: what to look at first

RED is three questions about any request-driven service:

  • Rate — how many requests per second?
  • Errors — what percentage fail?
  • Duration — how long do requests take?

Spring Boot Actuator + Micrometer collect this automatically through the http_server_requests_seconds metric. PromQL queries for Grafana:

# Rate — requests per second by endpoint
sum(rate(http_server_requests_seconds_count[5m])) by (uri, method)

# Errors — share of 5xx errors
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (uri)

# Duration — p95 latency
histogram_quantile(0.95, sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m])))

To see p95/p99 latency, you need to enable the histogram:

management:
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true
      slo:
        http.server.requests: 100ms,500ms,1s,5s

slo sets the bucket boundaries: Prometheus will count how many requests fit within 100ms, 500ms, and so on. This lets you formulate an SLO: "95% of requests faster than 500ms".

The USE method for resources: memory, threads, connections

USE is three questions about any resource:

  • Utilization — how busy is the resource?
  • Saturation — is there a queue of waiters?
  • Errors — are there errors at the resource level?

Spring Boot automatically exports JVM and infrastructure metrics:

MetricWhat it shows
jvm_memory_used_bytes{area="heap"}used heap memory
jvm_memory_max_bytes{area="heap"}maximum heap
jvm_gc_pause_seconds_sumtotal GC pause time
executor_active_threads{name="taskExecutor"}active threads in the pool
hikaricp_connections_activeactive database connections
hikaricp_connections_pendingrequests waiting for a connection

Example alerts:

- alert: HeapMemoryHigh
  expr: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} > 0.85
  for: 10m

- alert: HikariPoolSaturated
  expr: hikaricp_connections_pending > 0
  for: 5m

hikaricp_connections_pending > 0 means requests are queued for a database connection — a sure sign of pool saturation.

Your own business metrics through MeterRegistry

The built-in metrics are enough for infrastructure. But business events — "order created", "payment processed", "cart total" — you have to add yourself.

Micrometer offers four instruments:

  • Counter — a monotonically increasing counter. Good for events: order created, payment declined.
  • Gauge — a current value that can go both up and down. Good for states: queue size, number of active users.
  • Timer — operation duration with a histogram. Good for measuring processing time.
  • DistributionSummary — arbitrary numbers with a histogram. Good for cart totals, file sizes.
@Component
public class OrderMetrics {

    private final Counter orderCreatedCounter;
    private final Timer paymentProcessingTimer;
    private final DistributionSummary orderAmountSummary;

    public OrderMetrics(MeterRegistry registry) {
        this.orderCreatedCounter = Counter.builder("order_created_total")
            .description("Total orders created")
            .tag("channel", "web")
            .register(registry);
        this.paymentProcessingTimer = Timer.builder("payment_processing_seconds")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry);
        this.orderAmountSummary = DistributionSummary.builder("order_amount_rubles")
            .baseUnit("rubles")
            .register(registry);
    }

    public void orderCreated() { orderCreatedCounter.increment(); }

    public void recordPaymentDuration(Duration duration) {
        paymentProcessingTimer.record(duration);
    }

    public void recordOrderAmount(BigDecimal amount) {
        orderAmountSummary.record(amount.doubleValue());
    }
}

Usage in a service:

@Service
@RequiredArgsConstructor
public class CreateOrderService {

    private final OrderRepository orderRepository;
    private final OrderMetrics metrics;

    @Transactional
    public Order create(CreateOrderCommand command) {
        var order = orderRepository.save(Order.create(command));
        metrics.orderCreated();
        metrics.recordOrderAmount(order.amount());
        return order;
    }
}

How to name metrics

Prometheus follows a convention: snake_case, unit of measurement in the name.

Correct:

  • order_created_total — a counter with the _total suffix
  • payment_processing_seconds — a timer with _seconds
  • order_amount_rubles — a summary with a currency unit
  • queue_size — a gauge, number of items

Incorrect:

  • orderCreatedCount — camelCase, no _total
  • paymentTime — no unit of measurement
  • order_processing_ms — Prometheus prefers seconds, not milliseconds

Low tag cardinality — the most important rule

Prometheus stores a separate time series for each unique combination of tag values. So if the user_id tag takes a million values, the metric spawns a million time series — and that is just for one metric.

Good tags are categories with a small number of values:

// Correct — 3-10 values per tag
counter.tag("channel", "web")          // web, mobile, api
       .tag("payment_method", "card")  // card, sbp, crypto
       .increment();

Bad tags are unique identifiers:

// Wrong — a million values = a million time series
counter.tag("user_id", String.valueOf(userId))   // unique for each user
       .tag("order_id", String.valueOf(orderId)) // unique for each order
       .increment();

A million user_id × a million order_id × several environment tags = billions of time series. Prometheus will not cope, and the scraper will crash with OOM.

If you need detail on a specific user or request, that is a job for tracing (Tempo/Jaeger), not metrics. Each span in a trace carries arbitrary attributes with no cardinality limits.

Common mistakes

Micrometer without a Prometheus registry. If you add only micrometer-core, metrics are collected into the built-in SimpleMeterRegistry and kept in memory only. Everything is lost on restart, and Prometheus sees nothing. You need micrometer-registry-prometheus.

An exposed /actuator/prometheus endpoint. This endpoint can leak sensitive data: revenue totals, payment counters, business KPIs. In production its access is restricted with network policies — only the Prometheus scraper from the monitoring namespace. A handy trick is to move the actuator to a separate port:

management:
  server:
    port: 8081

Then business traffic goes to 8080 and actuator traffic to 8081, which is not exposed externally through the Ingress.

Non-standard environment tag names. If one service writes app=foo and another service_name=foo, cross-service dashboards in Grafana break. Agree at the configuration level (management.metrics.tags) and stick to one standard across all services.

In short

  • Micrometer is a facade for metrics (like SLF4J for logs), Prometheus is the storage backend, and /actuator/prometheus is the collection point.
  • The service, env, and version tags are configured once through management.metrics.tags and applied globally.
  • RED (Rate, Errors, Duration) — three questions for HTTP; http_server_requests_seconds is collected automatically.
  • USE (Utilization, Saturation, Errors) — three questions for resources; JVM, Hikari, and thread pools are collected automatically.
  • Your own metrics: Counter for events, Gauge for states, Timer for duration, DistributionSummary for numeric distributions.
  • Metric names: snake_case, unit in the name (_seconds, _total, _bytes).
  • High-cardinality tags (user_id, order_id) blow up Prometheus — use only categorical values with a small number of options.
  • Keep the /actuator/prometheus endpoint off public access: a separate port or network policies.
  • Request tracing — when you need detail on a specific request rather than an aggregate.
  • Logging — structured JSON logs and context via MDC.
  • Observability configuration — management port, Logback profiles.
  • Health checks — liveness vs readiness and custom indicators.