← Back to the section

When a request passes through several services and something is slow or fails, logs and metrics answer only partially: "there was an error in payment-service" and "p95 grew". But where exactly the time was lost and in which service the chain broke is hard to figure out.

Distributed tracing solves this problem: it records a specific request from the entry into the system to the exit, step by step. In this article we'll work out how to wire tracing into a FastAPI service through OpenTelemetry.

What a span and a trace are

Imagine a POST /orders request as a call tree. Each node of the tree is a span: a named interval of time with a start, an end and attributes. All the spans of one request are joined into a trace by a shared trace_id.

POST /orders  (100ms)
├── SELECT * FROM orders  (5ms)
├── POST https://payment-service/charge  (80ms)
│   └── SELECT * FROM payments  (3ms)
└── INSERT INTO order_events  (2ms)

In such a picture you immediately see: 80% of the time went into the external HTTP call.

OpenTelemetry is a library and a standard responsible for creating spans, passing trace_id between services and sending data to storage (Grafana Tempo, Jaeger). For Python it replaced the outdated Zipkin and Brave clients.

Installation and setup

You need several packages: the SDK itself, instrumentations for FastAPI, SQLAlchemy and the HTTP client, and an exporter to the collector:

opentelemetry-sdk
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-sqlalchemy
opentelemetry-instrumentation-httpx
opentelemetry-exporter-otlp-proto-grpc

Then we create an initialization function:

# app/telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor


def configure_tracing(app, engine, *, sample_rate: float = 0.1) -> None:
    provider = TracerProvider(sampler=ParentBasedTraceIdRatio(sample_rate))
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
    )
    trace.set_tracer_provider(provider)

    FastAPIInstrumentor.instrument_app(app)
    SQLAlchemyInstrumentor().instrument(engine=engine)
    HTTPXClientInstrumentor().instrument()

# app/main.py
app = FastAPI()
configure_tracing(app, engine, sample_rate=0.1)

After that, without a single line in the business code, spans already appear for every incoming HTTP request, SQL query and outgoing HTTP call.

When a request arrives at a second service, OTel reads the traceparent header from the incoming HTTP request and continues the same trace, creating a child span. When the service itself makes an HTTP call, OTel automatically adds this header to the outgoing request:

traceparent: 00-5e92c8a3b1f4d2e6a7c8e9f0a1b2c3d4-1f2e3d4c5b6a7980-01
              │     │                                  │                 │
              │     trace-id (shared across all services)  span-id       flags
              version

This is the W3C Trace Context format. HTTPXClientInstrumentor sets it automatically — no explicit code is needed.

A manual span for an important business operation

Auto-instrumentation covers HTTP and SQL, but sometimes you need to mark a specific business operation as a separate span. For this you use the start_as_current_span context manager:

# app/orders/confirm_order.py
import structlog
from opentelemetry import trace
from opentelemetry.trace import StatusCode

log = structlog.get_logger(__name__)
tracer = trace.get_tracer(__name__)


class ConfirmOrderHandler:
    def __init__(self, order_repo: OrderRepository) -> None:
        self._order_repo = order_repo

    async def handle(self, command: ConfirmOrderCommand) -> Order:
        with tracer.start_as_current_span("confirm_order") as span:
            span.set_attribute("order.id", str(command.order_id))
            span.set_attribute("customer.id", str(command.customer_id))

            order = await self._order_repo.find_for_update(command.order_id)
            if order is None:
                span.set_status(StatusCode.ERROR, "order_not_found")
                raise OrderNotFoundError(command.order_id)

            order.confirm()
            span.set_attribute("order.status", order.status.value)

            await self._order_repo.save(order)
            log.info("order_confirmed", order_id=str(command.order_id))
            return order

with tracer.start_as_current_span(...) guarantees that the span closes on exit from the block — even on an exception. On an error OTel automatically calls span.record_exception(exc) and marks the span as ERROR.

What to put in span attributes

Span attributes are extra context that helps you find the right trace among thousands. Add business identifiers: order.id, customer.id, enum values (order.status, payment.method).

Important: tracing data is stored in Tempo or Jaeger with a different access and retention mode than operational databases. Users' personal data — email, phone, card number — must not be put there.

with tracer.start_as_current_span("publish_product") as span:
    span.set_attribute("product.id", str(command.product_id))
    span.set_attribute("product.category", command.category.value)
    span.set_attribute("seller.id", str(command.seller_id))
    # Wrong:
    # span.set_attribute("product.description", command.description)
    # span.set_attribute("seller.email", seller.email)

Sampling: how many traces to keep

Recording 100% of traces in a loaded service is expensive. Usually head-based sampling is used: record a random 10% of requests.

ParentBasedTraceIdRatio works like this: if the incoming traceparent is already marked as "record", the child spans are recorded too. Otherwise — a random 10%.

provider = TracerProvider(
    sampler=ParentBasedTraceIdRatio(rate=0.1)  # 10%
)

Through environment variables (without changing the code):

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

On the collector side (Grafana Alloy, OTel Collector) you can configure tail-based sampling: record 100% of traces that had an error, and 10% of all the rest. This way no failure is missed.

If the service receives fewer than 10 requests per second, you can skip configuring sampling — the storage won't be overwhelmed.

trace_id in logs

When you see an error in a log in Grafana, you want to open the full trace for that request with one click. For this, every log record needs to contain a trace_id.

The OTel–structlog processor does this automatically — it adds trace_id and span_id to every record while a request is being processed:

# app/logging.py
import structlog
from opentelemetry import trace as otel_trace


def add_otel_context(logger, method, event_dict):
    span = otel_trace.get_current_span()
    ctx = span.get_span_context()
    if ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict


structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        add_otel_context,
        structlog.processors.JSONRenderer(),
    ]
)

The result in the log:

{
  "timestamp": "2026-06-19T10:15:30Z",
  "level": "error",
  "event": "payment_failed",
  "order_id": "ord-9912",
  "trace_id": "5e92c8a3b1f4d2e6a7c8e9f0a1b2c3d4",
  "span_id": "1f2e3d4c5b6a7980"
}

In Grafana we click on the trace_id → Tempo → we see the whole distributed trace step by step.

Context in asyncio and a thread pool

In asyncio the OTel context is carried across await automatically — you don't need to do anything:

async def process_order(order_id: UUID) -> None:
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", str(order_id))
        await validate_order(order_id)   # the span is active here too
        await charge_payment(order_id)   # and here

If part of the work runs in a separate thread through run_in_executor, the context must be copied manually — otherwise the trace breaks:

import asyncio
import contextvars

loop = asyncio.get_event_loop()
ctx = contextvars.copy_context()

result = await loop.run_in_executor(
    None,
    ctx.run,
    sync_heavy_computation,
    order_id,
)

Common mistakes

100% sampling in production — under high load this quickly overflows the storage. Use ParentBasedTraceIdRatio(0.1) plus tail-based sampling on the collector for errors.

Personal data in attributes — customer.email, card.number must not land in tracing storage. Only internal identifiers and enum values.

tracer.start_span(...) without finishing it — if you don't use a context manager and don't call span.end() in a finally block, the span will never close. The preferred option is with tracer.start_as_current_span(...).

run_in_executor without copy_context() — the trace breaks at the thread boundary: spans in the executor don't see the parent span.

In short

  • A distributed trace is a tree of spans for one request. Each span is a named interval of time with attributes.
  • The traceparent header links spans across services. HTTPXClientInstrumentor sets it automatically.
  • Auto-instrumentation of FastAPI, SQLAlchemy and httpx gives the "HTTP → SQL → HTTP out" picture without manual code.
  • Manual spans are created through with tracer.start_as_current_span(...) — the context manager guarantees closing.
  • In span attributes — only internal identifiers and enums. No personal data.
  • 10% sampling in production + 100% of errors on the collector side.
  • The OTel–structlog processor adds trace_id to every log record automatically.
  • In asyncio the context carries across await by itself; for run_in_executor you need contextvars.copy_context().
  • Metrics in Python — Prometheus, cardinality, RED metrics.
  • Logging in Python — structlog JSON, PII hygiene.
  • Tracing in Java — a similar article for Spring Boot.
  • Tracing in Go — OpenTelemetry from scratch for a Go service.