← Back to the section

A metric says "p95 climbed to 3 seconds." A log says "lots of errors in payment-service." But neither the metric nor the log shows what actually happened to a specific client request — which services it passed through and where it lost time.

Distributed tracing solves exactly this: it records the path of every request through all the services in the system. You see the full route — POST /orders → auth-service → payment-service → notification-service — with the time spent at each step.

What a span and a trace are

Tracing used to be implemented by every company in its own way: Zipkin, Jaeger, AWS X-Ray — each with its own client and format. Switching backends was painful.

Today there is OpenTelemetry — an open standard and a set of SDKs that works with any backend (Jaeger, Tempo, Datadog, Honeycomb). Write once, run against any storage.

Two key concepts:

  • Span — a single operation: an incoming HTTP request, a database query, a call to another service. A span has a name, a start and end time, tags (attributes), and a reference to its parent span.
  • Trace — a tree of related spans. All the spans of a single request across all services form one trace with a shared traceId.

When order-service calls payment-service, it passes the traceId in an HTTP header. payment-service sees it and creates a child span — that's how spans link into a tree.

Getting started: automatic spans with no code

Add the starter to build.gradle:

implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")
implementation("io.opentelemetry.instrumentation:opentelemetry-logback-mdc-1.0")

After that, Spring Boot automatically creates spans for:

  • incoming HTTP requests (Spring MVC) — with the attributes http.method, http.url, http.status_code;
  • outgoing HTTP requests (RestClient, WebClient) — propagating the traceId onward;
  • every SQL query through JDBC;
  • sending and receiving Kafka messages;
  • cache operations through Spring Cache.

Without a single line of your own code you already see the full picture: "HTTP → SQL → Kafka → outgoing HTTP".

Configure the collector endpoint in application.yml:

otel:
  exporter:
    otlp:
      endpoint: http://otel-collector:4317
  traces:
    sampler: parentbased_traceidratio
    sampler.arg: 0.1

How the traceId travels between services

The W3C Trace Context standard defines the format of the traceparent header:

traceparent: 00-5e92c8a3b1f4d2e6a7c8e9f0a1b2c3d4-1f2e3d4c5b6a7980-01

It encodes: the protocol version, the traceId (16 bytes), the spanId (8 bytes), and flags (for example, "this request is being traced").

OpenTelemetry does this automatically: on an incoming request it extracts the traceparent from the headers and attaches it to the current trace; on an outgoing HTTP call or when sending a Kafka message it adds the header. Services write no code for this.

Adding business context to a span

Automatic spans contain technical details — the URL, the status, the table name. To later find traces in Jaeger by orderId, you need to add business attributes manually.

@Service
@RequiredArgsConstructor
public class ConfirmOrderHandler {

    private final OrderRepository orderRepository;
    private final Tracer tracer;

    @Transactional
    public Order handle(ConfirmOrderCommand command) {
        var span = tracer.spanBuilder("confirmOrder")
            .setAttribute("order.id", command.orderId())
            .startSpan();
        try (var scope = span.makeCurrent()) {
            var order = orderRepository.findById(command.orderId())
                .orElseThrow();
            order.confirm();
            span.setAttribute("order.status", order.status().name());
            return orderRepository.save(order);
        } catch (Exception e) {
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

Important: span.end() in the finally block is mandatory. If the span is not closed, the collector will never receive the data — the trace will hang as unfinished.

If you don't need attributes and recordException, an annotation is simpler:

@WithSpan("confirmOrder")
public Order handle(ConfirmOrderCommand command) { ... }

What to put in attributes and what not to

Span attributes are stored separately — in Tempo, Jaeger, Honeycomb — often with different access permissions and retention. Personal data must not go there.

You may include:

  • internal identifiers: order.id, customer.id, payment.id;
  • enumerations and statuses: order.status, payment.method;
  • technical labels: external.system="sber", circuit_breaker.state="open".

You must not include:

  • email, phone, customer name;
  • card number, IBAN, passport;
  • the entire request body.

Sampling: how many traces to keep

Recording 100% of traces in production is expensive. A moderately loaded service (1000 requests/s) at 100% sampling produces terabytes of data per day.

The standard approach is parentbased_traceidratio at 1-10%:

  • if the incoming request is already marked as "traced" (the flag in traceparent) — the service also takes part in the trace;
  • otherwise — a random fraction of requests is traced.

On the collector side you configure tail-based sampling: 100% of traces with errors are kept regardless of the base ratio. This gives you a complete set of failing traces without overloading storage.

For low-load services (under 10 requests/s), 100% sampling is fine.

Tracing in logs: linking a log entry to a trace

opentelemetry-logback-mdc-1.0 automatically adds traceId and spanId to the MDC (Mapped Diagnostic Context). If you write JSON logs, these fields end up in every entry:

{
  "@timestamp": "2026-05-25T22:30:00Z",
  "level": "ERROR",
  "message": "Failed to charge payment: orderId=12345",
  "traceId": "5e92c8a3b1f4d2e6a7c8e9f0a1b2c3d4",
  "spanId": "1f2e3d4c5b6a7980"
}

In Grafana you click the traceId in Loki — Tempo opens with the full trace of that request. You see which log corresponds to which span in which service.

Common mistakes

The trace breaks on @Async. When Spring runs a method on another thread via @Async or CompletableFuture.runAsync(...), the OTel context is not propagated automatically — the trace breaks. Solution: a TaskDecorator that copies the OTel context into the new thread.

A span without try-finally. If you create a span manually but span.end() isn't guaranteed (for example, an exception is thrown before it) — the span will never reach the collector.

// Don't do this — on an exception the span won't be closed
var span = tracer.spanBuilder("foo").startSpan();
doWork();
span.end();

// Correct
var span = tracer.spanBuilder("foo").startSpan();
try (var scope = span.makeCurrent()) {
    doWork();
} finally {
    span.end();
}

In short

  • Distributed tracing shows the path of a specific request through all services with the time spent at each step.
  • OpenTelemetry is the industry standard; opentelemetry-spring-boot-starter gives you automatic spans for HTTP, JDBC, Kafka, and cache with no code.
  • traceparent (W3C) is the header through which the traceId travels between services; OTel propagates it automatically.
  • Manual spans are needed for business operations with attributes; span.end() in finally is mandatory.
  • Attributes may hold internal identifiers and statuses; personal data must not.
  • Sampling at 1-10% in production plus tail-based 100% for errors balances completeness against storage cost.
  • opentelemetry-logback-mdc-1.0 automatically puts traceId/spanId into the MDC — every log entry becomes a clickable link to a trace.
  • @Async and CompletableFuture.runAsync break the trace without a TaskDecorator.

Further reading

  • Logging in Java — structured logs and linking them with the traceId.
  • Metrics in Java — Micrometer, Prometheus, and why the traceId is a poor fit for labels.
  • Health checks in Java — liveness, readiness, and custom checks.