Observability configuration in Python: management port, logs and metrics

← Back to the section

When a service starts up in production for the first time, the first question arrives: "Where do I look at metrics? Where do logs go? What does DOWN on a health check mean?" The answers depend on how observability is configured at startup. This article is about three parts of such a setup in FastAPI: a separate management port, logging through structlog and Prometheus metrics.

Why metrics and the health check should live on a separate port

The simplest approach is to add /metrics and /health right onto the main server. It works, but it creates inconveniences:

• Scraping traffic from Prometheus and Kubernetes probes get mixed with business requests in the same event loop.
• You can't restrict /metrics to internal traffic only with a network policy without touching the main port.
• Swagger UI (/docs) and the API schema (/openapi.json) end up available at the same address as the metrics — they have to be disabled separately.

The solution is two ASGI applications: business traffic on the main port (8080), management (/metrics, /health/*, /info) on a separate one (8081):

# app/platform/metrics/management.py
from fastapi import FastAPI
from prometheus_client import make_asgi_app

def build_management_app(settings, ready_check) -> FastAPI:
    mgmt = FastAPI(
        title="management",
        docs_url=None,
        redoc_url=None,
        openapi_url=None,
    )

    mgmt.mount("/metrics", make_asgi_app())

    @mgmt.get("/health/live")
    async def liveness():
        return {"status": "UP"}

    @mgmt.get("/health/ready")
    async def readiness():
        ok = await ready_check()
        if not ok:
            from fastapi.responses import JSONResponse
            return JSONResponse({"status": "DOWN"}, status_code=503)
        return {"status": "UP"}

    @mgmt.get("/info")
    async def info():
        return {
            "service": settings.service_name,
            "version": settings.version,
            "env":     settings.app_env,
        }

    return mgmt

# app/main.py
import uvicorn, asyncio
from app.config import Settings
from app.platform.metrics.management import build_management_app
from app.app import build_app

async def main():
    settings = Settings()
    app  = build_app(settings)
    mgmt = build_management_app(settings, ready_check=app.state.ready_check)

    config_biz  = uvicorn.Config(app,  host="0.0.0.0", port=settings.port,            log_config=None)
    config_mgmt = uvicorn.Config(mgmt, host="0.0.0.0", port=settings.management_port, log_config=None)

    await asyncio.gather(
        uvicorn.Server(config_biz).serve(),
        uvicorn.Server(config_mgmt).serve(),
    )

if __name__ == "__main__":
    asyncio.run(main())

Now Kubernetes probes go to 8081, the Prometheus scraper does too — while the Ingress publishes only 8080.

All parameters in one Settings class

When settings are scattered across separate variables in different files, it's easy to miss that LOG_LEVEL is named differently on different servers. It's more convenient to gather everything in one place through pydantic-settings:

# app/config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="APP_", case_sensitive=False)

    app_env:         str   = Field("development", description="production | staging | development")
    service_name:    str   = "order-service"
    version:         str   = Field("unknown", alias="BUILD_VERSION")
    port:            int   = 8080
    management_port: int   = 8081

    log_level:       str   = "INFO"
    otel_endpoint:   str   = Field("", alias="OTEL_EXPORTER_OTLP_ENDPOINT")
    sampling_ratio:  float = Field(0.1, description="0.01–1.0; 1.0 only on dev")

BUILD_VERSION comes from CI as an environment variable — the same way as in services written in other languages. All parameters are read from the environment automatically, with the APP_ prefix.

Logs: readable locally, JSON in production

In development it's nice to see lines like 10:42:03 [info] order_confirmed order_id=ORD-9912. In production the same format is a headache for Loki and Datadog: you need to write a regex to parse it.

structlog solves this by switching the renderer depending on the environment:

# app/platform/log/setup.py
import logging, sys
import structlog
from structlog.types import EventDict, WrappedLogger

def add_service_info(settings):
    def processor(logger: WrappedLogger, method: str, event_dict: EventDict) -> EventDict:
        event_dict["service"] = settings.service_name
        event_dict["env"]     = settings.app_env
        event_dict["version"] = settings.version
        return event_dict
    return processor

def configure_logging(settings) -> None:
    is_prod = settings.app_env == "production"

    shared_processors = [
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt="iso"),
        add_service_info(settings),
    ]

    renderer = structlog.processors.JSONRenderer() if is_prod else structlog.dev.ConsoleRenderer(colors=True)

    structlog.configure(
        processors=[*shared_processors, renderer],
        wrapper_class=structlog.make_filtering_bound_logger(
            logging.getLevelName(settings.log_level)
        ),
        context_class=dict,
        logger_factory=structlog.PrintLoggerFactory(sys.stdout),
        cache_logger_on_first_use=True,
    )

Locally the output looks like this:

10:42:03 [info     ] order_confirmed    order_id=ORD-9912 customer_id=C-441

In production it's JSON, which log aggregators parse without extra configuration:

{"event": "order_confirmed", "level": "info", "service": "order-service", "env": "production",
 "order_id": "ORD-9912", "customer_id": "C-441", "request_id": "0193a8f3-...", "timestamp": "2026-06-19T10:42:03Z"}

structlog.contextvars.merge_contextvars at the start of the chain adds request_id and user_id from the request context — they are set by middleware, not by the handler itself. trace_id and span_id appear once the OTel processor is connected — more on that in the tracing article.

Important: configure_logging is called once at startup, not on every request. A repeated structlog.configure call overwrites the settings.

Histogram buckets for latency measurement

By default a Prometheus histogram uses standard bucket boundaries: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 seconds. If a service has an SLO of "p99 < 500ms", the standard buckets give an inaccurate histogram_quantile — the 0.5 boundary exists, but there are no intermediate values near it.

Buckets are set when the metric is registered; you can't change them afterward:

# app/platform/metrics/http.py
from prometheus_client import Histogram, Counter

HTTP_REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["method", "path", "status_class"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],
)

HTTP_REQUESTS_TOTAL = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status_class"],
)

The standard labels service, env, version are better extracted into a factory function so you don't repeat them on every metric:

# app/platform/metrics/common.py
from prometheus_client import Counter

def make_business_counter(name: str, description: str, extra_labels: list[str]) -> Counter:
    return Counter(name, description, ["service", "env", "version", *extra_labels])

# app/order/metrics.py
from app.platform.metrics.common import make_business_counter
from app.config import Settings

_orders_created = make_business_counter(
    "orders_created_total",
    "Orders successfully created",
    ["payment_method"],
)

class OrderMetrics:
    def __init__(self, settings: Settings) -> None:
        self._service = settings.service_name
        self._env     = settings.app_env
        self._version = settings.version

    def order_created(self, payment_method: str) -> None:
        _orders_created.labels(
            service=self._service,
            env=self._env,
            version=self._version,
            payment_method=payment_method,
        ).inc()

The path label: the route template, not the actual URL

A typical mistake when first wiring up metrics is to use request.url.path as the value of the path label. For the endpoint /orders/{order_id} every order creates a separate time series: /orders/ORD-9912, /orders/ORD-9913, /orders/ORD-9914… After a few days there are tens of thousands of them.

In FastAPI the route template is available through request.scope["route"]:

# app/platform/metrics/middleware.py
import time
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from app.platform.metrics.http import HTTP_REQUEST_DURATION, HTTP_REQUESTS_TOTAL

class MetricsMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        start = time.perf_counter()
        response = await call_next(request)
        duration = time.perf_counter() - start

        route = request.scope.get("route")
        path  = route.path if route else "unknown"
        status_class = _status_class(response.status_code)

        HTTP_REQUESTS_TOTAL.labels(method=request.method, path=path, status_class=status_class).inc()
        HTTP_REQUEST_DURATION.labels(method=request.method, path=path, status_class=status_class).observe(duration)

        return response


def _status_class(code: int) -> str:
    if code < 400:
        return "success"
    if code < 500:
        return "client_error"
    return "server_error"

route.path returns /orders/{order_id} — a fixed number of time series regardless of the number of orders.

Tracing: configuration at startup

OTel is configured once in lifespan, and the sampling ratio comes from Settings:

# app/platform/tracing/setup.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

def configure_tracing(settings, app) -> None:
    if not settings.otel_endpoint:
        return

    provider = TracerProvider(
        sampler=ParentBasedTraceIdRatio(settings.sampling_ratio),
        resource=Resource.create({
            "service.name":           settings.service_name,
            "service.version":        settings.version,
            "deployment.environment": settings.app_env,
        }),
    )
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint=settings.otel_endpoint))
    )
    trace.set_tracer_provider(provider)

    FastAPIInstrumentor.instrument_app(app)
    SQLAlchemyInstrumentor().instrument()
    HTTPXClientInstrumentor().instrument()

# app/app.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from app.config import Settings
from app.platform.log.setup import configure_logging
from app.platform.tracing.setup import configure_tracing

def build_app(settings: Settings) -> FastAPI:
    configure_logging(settings)

    @asynccontextmanager
    async def lifespan(app: FastAPI):
        configure_tracing(settings, app)
        yield

    return FastAPI(
        title=settings.service_name,
        lifespan=lifespan,
        docs_url="/docs"         if settings.app_env != "production" else None,
        openapi_url="/openapi.json" if settings.app_env != "production" else None,
    )

docs_url=None in production removes Swagger UI and the API schema from the business port — the API structure is not exposed to external traffic.

Common mistakes

One port for business traffic and management. Prometheus scraping and K8s probes go to the main server. You need to run two uvicorn.Server instances.

make_asgi_app() on the business router. /metrics ends up on the business port and is reachable through the Ingress. The Prometheus scraper should go to the management port directly, bypassing the Ingress.

Standard histogram buckets with a strict SLO. DEFAULT_BUCKETS don't cover the needed range precisely. Buckets are set once at registration — they can't be changed afterward.

A raw URL in the path label. /orders/ORD-9912, /orders/ORD-9913 are separate time series. Use request.scope["route"].path.

Entity values as metric labels. user_id=U-441 or order_id=ORD-9912 as a label is a cardinality explosion. Such values belong in traces (OTel span attributes), not in metrics.

structlog.configure() in the request handler. The configuration is overwritten on every request. Only one call at startup.

sampling_ratio=1.0 in production on a loaded service. 100% tracing puts heavy load on the exporter and storage. On dev/staging it's fine, in production it's 0.01–0.1.

In short

• Two ASGI applications: business traffic on 8080, management (/metrics, /health/*, /info) on 8081. This way the Prometheus scraper and K8s probes don't interfere with the main event loop.
• pydantic-settings gathers all observability parameters in one class; environment variables with the APP_ prefix.
• structlog switches the renderer by APP_ENV: ConsoleRenderer for development, JSONRenderer for production — no regex in the log aggregators.
• structlog.configure() is called once at startup. merge_contextvars goes first in the processor chain.
• Histogram buckets are set at registration for a specific SLO threshold. They can't be changed later.
• The path label is the route template (/orders/{order_id}), not the actual URL. Otherwise every order creates a separate time series.
• OTel is configured once in lifespan. sampling_ratio comes from Settings, it isn't hardcoded.
• /docs and /openapi.json in production are None: the API structure is not available to external traffic.

What to read next

• Context propagation — request_id in middleware, bind_contextvars, copy_context for offloading to threads.
• Health checks — liveness, readiness with a TTL cache, asyncpg ping.
• Logging — structlog processors, OTel bridge, masking of sensitive data.
• Metrics — RED middleware, business counters, PrometheusInstrumentator.
• Tracing — OTel setup, FastAPIInstrumentor, manual spans.