← Back to the section

Imagine: a request arrives at your service, goes through the database, calls an external API — and somewhere in the middle something slows down. The metrics say "p95 latency went up", the logs say "there was an error somewhere". But neither shows what exactly happened to this specific request.

Distributed tracing solves this problem. It follows the path of a single request through all the services and shows: here the HTTP server accepted the call, here it went to the database and spent 180 ms there, here it called an external API and got a timeout. All of it — in one view, with timestamps.

Java has a javaagent that instruments the bytecode automatically. Go has nothing like it, but the libraries from the OpenTelemetry ecosystem (otelhttp, otelpgx, otelchi) give almost the same thing — with a single line to wire them in.

Setting up the TracerProvider

It all starts with the TracerProvider — the central object that knows where to send the data and what percentage of requests to record.

// internal/platform/tracing/setup.go
package tracing

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

type Config struct {
    OTLPEndpoint string
    ServiceName  string
    Version      string
    Env          string
    SampleRate   float64
}

func Setup(ctx context.Context, cfg Config) (func(context.Context) error, error) {
    exp, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(cfg.OTLPEndpoint),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, fmt.Errorf("tracing exporter: %w", err)
    }

    res, _ := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(cfg.ServiceName),
            semconv.ServiceVersion(cfg.Version),
            semconv.DeploymentEnvironmentName(cfg.Env),
        ),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithSampler(
            sdktrace.ParentBased(sdktrace.TraceIDRatioBased(cfg.SampleRate)),
        ),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

The function returns shutdown — it must be called when the application terminates, so that an unfilled buffer has time to reach the collector.

ParentBased is an important detail: if the incoming request already carries a traceparent header marked "record", your service will record a span too. If not — SampleRate decides. This means an external gateway can control which requests are traced along the entire chain.

In main.go it looks like this:

shutdown, err := tracing.Setup(ctx, tracing.Config{
    OTLPEndpoint: cfg.OTLPEndpoint,
    ServiceName:  "order-service",
    Version:      version,
    Env:          cfg.AppEnv,
    SampleRate:   0.01, // 1% in production
})
if err != nil {
    slog.Error("tracing setup failed", "error", err)
    os.Exit(1)
}
defer func() { _ = shutdown(ctx) }()

Auto-instrumenting the chi router

Without extra code the chi router doesn't create spans. A single line with otelchi.Middleware fixes this:

import (
    "github.com/go-chi/chi/v5"
    "go.opentelemetry.io/contrib/instrumentation/github.com/go-chi/chi/v5/otelchi"
)

router := chi.NewRouter()
router.Use(otelchi.Middleware("order-service", otelchi.WithChiRoutes(router)))
// ... the rest of the middleware and routes

After this every HTTP request automatically gets a span with the route name, HTTP method, response code and execution time.

For outgoing requests to other services — wrap the transport:

func New() *http.Client {
    return &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
        Timeout:   10 * time.Second,
    }
}

otelhttp.NewTransport automatically adds a traceparent header to every outgoing request. The receiving service sees this header and creates a child span — the trace assembles into a single chain with no extra code.

Instrumenting the database via otelpgx

Every SQL query is a separate operation with latency worth seeing. otelpgx plugs into the pgx pool with a single line:

// internal/platform/postgres/pool.go
import "go.opentelemetry.io/contrib/instrumentation/github.com/jackc/pgx/v5/otelpgx"

func NewPool(ctx context.Context, dsn string) (*pgxpool.Pool, error) {
    cfg, err := pgxpool.ParseConfig(dsn)
    if err != nil {
        return nil, fmt.Errorf("parse dsn: %w", err)
    }
    cfg.ConnConfig.Tracer = otelpgx.NewTracer()
    return pgxpool.NewWithConfig(ctx, cfg)
}

After this, in Tempo (or Jaeger) you'll see child spans for every SQL query with the attributes db.statement, db.operation, db.name — and you'll immediately understand how long each query takes in the context of the whole call.

Manual spans for key operations

Automatic spans cover HTTP and the database. But the business logic between them stays invisible. For this you add manual spans:

func (h *ConfirmOrderHandler) Handle(ctx context.Context, cmd ConfirmOrderCommand) error {
    ctx, span := otel.Tracer("order").Start(ctx, "ConfirmOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id", cmd.OrderID),
        attribute.String("payment.method", string(cmd.PaymentMethod)),
    )

    order, err := h.orders.Load(ctx, cmd.OrderID)
    if err != nil {
        span.RecordError(err, trace.WithStackTrace(true))
        span.SetStatus(codes.Error, err.Error())
        return fmt.Errorf("load order: %w", err)
    }

    if err = order.Confirm(); err != nil {
        span.RecordError(err, trace.WithStackTrace(true))
        span.SetStatus(codes.Error, err.Error())
        return fmt.Errorf("confirm order: %w", err)
    }

    span.SetAttributes(attribute.String("order.status", string(order.Status)))

    if err = h.orders.Save(ctx, order); err != nil {
        span.RecordError(err, trace.WithStackTrace(true))
        span.SetStatus(codes.Error, err.Error())
        return fmt.Errorf("save order: %w", err)
    }

    return nil
}

Two important points:

defer span.End() right after Start — this is the Go idiom, the analog of try-finally in Java. defer guarantees the span is closed on any execution path, including a panic. Forgetting span.End() means losing the span entirely.

The ctx with the active span is passed further into h.orders.Load(ctx, ...) — this is exactly how child operations (SQL queries via otelpgx) become child spans. The context is the only mechanism for the link.

What to put in span attributes, and what not to

Span attributes are the tags by which traces are later searched and filtered. Here it's important to understand one constraint: tracing stores (Tempo, Jaeger) have broader access and different retention than the main database. Personal data in attributes is a compliance violation.

Good: identifiers (order.id, customer.id), statuses and enum values (order.status, payment.method), technical parameters (search.limit, circuit_breaker.state).

Bad: email, phone, card number, the full JSON of the request body. Data from which personal information can be reconstructed.

Internal UUID identifiers (order.id, customer.id) are fine: they're needed to navigate to a record in the main database and reveal nothing by themselves.

Sampling: how many traces to record

Recording 100% of requests in production is too expensive under any load. The standard approach: 1% via TraceIDRatioBased, but 100% for errors.

The sampling percentage is set in the TracerProvider configuration (the SampleRate: 0.01 parameter above). At 100 requests per second that's 1 trace/s, about 86,000 traces a day — Tempo handles it at no cost.

For 100% retention of error traces, tail-based sampling is configured in the OTel Collector — outside the application code:

# otel-collector/config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

errors-policy guarantees: a trace with an error is always retained, even if it was caught by the 1% sampling. Slow requests (longer than 1 second) are also retained in full.

Trace ID in logs

When you see an error in a trace, the next step is to look at that request's logs. For this, each log line needs a trace_id.

If you pass ctx into the slog methods (InfoContext, WarnContext, ErrorContext) and configure the OTel-slog bridge, trace_id and span_id are added to the log records automatically:

h.log.InfoContext(ctx, "order_confirmed",
    slog.String("order_id", cmd.OrderID),
    slog.String("customer_id", order.CustomerID),
)

In Loki the record will contain "trace_id": "5e92c8a3b1f4d2e6…" — you click it, jump to Tempo, and see the whole request path. Adding trace_id manually via ctx.Value(...) is unnecessary: the bridge does it more reliably and consistently.

Goroutines and a broken trace

A common problem: a goroutine starts with context.Background() and creates a separate root span — the link to the parent trace is lost.

// Bad — the trace is broken
go func() {
    h.notify(context.Background(), order.CustomerID)
}()

// Good — pass the parent context as an argument
notifyCtx, notifyCancel := context.WithTimeout(ctx, 5*time.Second)
defer notifyCancel()
go func(ctx context.Context) {
    if err := h.notify(ctx, order.CustomerID); err != nil {
        h.log.WarnContext(ctx, "customer_notify_failed",
            slog.String("customer_id", order.CustomerID),
            slog.String("error", err.Error()),
        )
    }
}(notifyCtx)

The context is passed as an argument (not captured through the closure) — this protects against the case where the parent goroutine cancels ctx before the child one finishes.

In short

  • TracerProvider — configured once at startup, registered globally via otel.SetTracerProvider. Returns shutdown for a clean termination.
  • otelchi + otelhttp.NewTransport — automatic spans for incoming and outgoing HTTP with no code in the handlers.
  • otelpgx — one cfg.ConnConfig.Tracer = otelpgx.NewTracer() makes all SQL queries visible in the trace.
  • A manual span: ctx, span := otel.Tracer("...").Start(ctx, "...") + defer span.End() right away. Pass the updated ctx further — this links the child operations.
  • Span attributes — internal IDs and enum values. Personal data (email, card, phone) — don't put it in.
  • Sampling: 1% (TraceIDRatioBased(0.01)) in production + tail-based in the OTel Collector for 100% of errors.
  • A goroutine with context.Background() breaks the trace. Pass the parent ctx as an argument.
  • Logging in Go — slog, the OTel-slog bridge, linking trace_id to a log record.
  • Context propagation in Go — context.Context as the only propagation mechanism; goroutines and fan-out.
  • Metrics in Go — prometheus/client_golang, why high cardinality isn't for metrics.
  • SLO and alerts — multi-window burn-rate alerts, error budget.