← Back to the section

"Observability" in a Spring service is built from three pillars: metrics (what and how much), traces (how a single request travels across services), logs (what happened, in detail). Spring Boot combines them into one stack: Actuator + Micrometer + Micrometer Tracing + structured logs.

The most expensive mistake in this stack is a mistake in the metrics: the number of series in Prometheus is set not by the flow of orders, but by the set of tag values.

One metric orders.created: as many series as tag combinationsa series = one combination of tag values; Prometheus holds every series in memory counter("orders.created", "category", c, "status", s)8 categories × 3 statuses 24 series in memorythe same 24 at 100 orders a day and at 10 million + "user_id", order.userId()50,000 customers a month 8 × 3 × 50,000 = 1,200,000 seriesa new customer means a new series, and it stays24 series per metric became 1.2M — because of one tag 01234 GBPrometheus memory24 series72 KB — indistinguishable at this scale 3.6 GB — 3 KB per active series1.2M series 3.6 GB for one metric — memory runs out in the service and in Prometheus A tag is a category with a dozen values, not a record keya per-user breakdown is a log with traceId, where there are no series

The number of series is set by tags, not by traffic: 24 series hold at a hundred orders and at ten million, while a single user_id tag turns them into 1.2M series and 3.6 GB of memory. A breakdown by a specific user belongs in a log with traceId, not in a metric.

Spring Boot Actuator

Standard endpoints for management and observation. To add it:

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-actuator")
}

By default /actuator/health and /actuator/info are exposed. Everything else is closed and must be exposed explicitly:

management.endpoints.web.exposure.include=health,info,metrics,prometheus,env,loggers
management.endpoint.health.show-details=when_authorized

/actuator/health

The health check tells you the service is "alive". By default the state is an aggregate of all registered HealthIndicators (datasource, mongo, redis, kafka...).

{
  "status": "UP",
  "components": {
    "db": {"status": "UP", "details": {...}},
    "diskSpace": {"status": "UP"},
    "ping": {"status": "UP"}
  }
}

In Kubernetes it is used in readiness and liveness probes:

livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }

/health/liveness means "the process is alive". /health/readiness means "ready to accept traffic". The difference matters: a liveness failure restarts the pod, while a readiness failure removes the pod from the service but keeps it running (for example, during warmup).

Custom HealthIndicator

@Component
public class PricingServiceHealthIndicator implements HealthIndicator {

    private final PricingClient client;

    @Override
    public Health health() {
        try {
            client.ping();
            return Health.up().build();
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

It shows up under /actuator/health/pricingService automatically.

Security for Actuator

Actuator endpoints expose a lot of information (env, beans, mappings, configprops), so locking them down is mandatory:

@Bean
public SecurityFilterChain actuatorChain(HttpSecurity http) throws Exception {
    return http
        .securityMatcher("/actuator/**")
        .authorizeHttpRequests(auth -> auth
            .requestMatchers("/actuator/health/**", "/actuator/info").permitAll()
            .anyRequest().hasRole("ACTUATOR"))
        .httpBasic(Customizer.withDefaults())
        .build();
}

The health endpoint is usually open; everything else sits behind authentication.

Micrometer — an abstraction over metrics

Micrometer is a facade library for metrics (like SLF4J for logs). One codebase, different backends: Prometheus, Datadog, CloudWatch, New Relic, and so on.

There are three main metric types:

Counter — a value that only grows

@Component
@RequiredArgsConstructor
public class OrderMetrics {

    private final MeterRegistry registry;

    public void onCreated(String category) {
        registry.counter("orders.created", "category", category).increment();
    }
}

Questions like "how many orders in the 'sweets' category over the last hour" become a Prometheus query: rate(orders_created{category="sweets"}[1h]).

Timer — measuring duration

@Bean
public Timer.Builder orderProcessingTimer() {
    return Timer.builder("order.processing");
}

@Component
public class OrderService {

    private final MeterRegistry registry;

    public void process(Order order) {
        Timer.Sample sample = Timer.start(registry);
        try {
            // ... processing
        } finally {
            sample.stop(registry.timer("order.processing", "status", "ok"));
        }
    }
}

Questions: p50/p95/p99 latency, average duration, histogram buckets.

Gauge — the current value

@Component
public class QueueMetrics {

    public QueueMetrics(MeterRegistry registry, MessageQueue queue) {
        Gauge.builder("queue.size", queue, MessageQueue::size)
            .register(registry);
    }
}

Used for "number of connections" or "queue size" — values that go both up and down.

Standard metrics out of the box

Without any code, Actuator + Micrometer publish:

  • jvm.memory.used, jvm.gc.pause — memory and GC.
  • process.cpu.usage, system.cpu.usage — CPU.
  • http.server.requests — every HTTP request with tags (uri, method, status, exception).
  • jdbc.connections.active, hikaricp.connections.idle — the database connection pool.
  • kafka.consumer.records-lag — Kafka consumer lag (with spring-kafka).
  • rabbitmq.connections.active — RabbitMQ.

/actuator/prometheus exposes everything in Prometheus format, ready for scraping.

Distributed tracing

When a request passes through 3-5 services, plain logs are useless — events from different services are interleaved. You need a trace_id that flows through every call.

In Spring Boot 3 the standard is Micrometer Tracing on top of OpenTelemetry or Brave (Zipkin):

dependencies {
    implementation("io.micrometer:micrometer-tracing-bridge-otel")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
}

management.tracing.sampling.probability=1.0           # in prod use 0.1 or less
management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces

Once configured:

  • Every incoming HTTP request gets a traceId (generated if absent).
  • Outgoing RestClient / WebClient / Feign calls automatically add a traceparent header.
  • Kafka/AMQP messages carry traceparent in their headers.
  • The logging MDC already contains traceId and spanId.
# in the logs:
2026-05-18 10:23:45 INFO  [trace=abc12345 span=def67890] Processing order 42

In Jaeger / Tempo / Honeycomb you see the whole path of the request, from input all the way to every downstream service.

Custom spans

Sometimes you need a span around a specific business operation (not just HTTP/JDBC):

@Component
@RequiredArgsConstructor
public class OrderService {

    private final ObservationRegistry registry;

    public void process(Order order) {
        Observation.createNotStarted("order.process", registry)
            .lowCardinalityKeyValue("category", order.category().name())
            .observe(() -> {
                // your code
            });
    }
}

Observation is a high-level abstraction that, under the hood, creates a trace span plus tags for metrics.

Structured logging

JSON logs are the production standard: they are parsed by ELK / Loki / Splunk without any regexes.

<!-- logback-spring.xml -->
<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>correlationId</includeMdcKeyName>
        </encoder>
    </appender>

    <springProfile name="prod">
        <root level="INFO">
            <appender-ref ref="JSON"/>
        </root>
    </springProfile>
    <springProfile name="!prod">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

Spring Boot 3.4+ has built-in support for structured logging without the logstash encoder:

logging.structured.format.console=ecs   # ECS (Elastic Common Schema) format

What NOT to measure

Careless metrics can end up costing more than the processing itself.

  • Don't use a tag with a high-cardinality value (user_id, order_id, correlation_id) — Prometheus creates a separate time series for each. Thousands of users → millions of time series → out of memory.
  • Business data (balance, item count) can be measured, but through a Counter/Gauge with low-cardinality tags (category, region, status).

The principle: tag = category, value = what you measure. If you want to measure per user, that's a job for logs, not metrics.

Further reading