Metrics in Python — prometheus-client, RED, USE and business indicators

← Back to the section

When a service runs under load, the questions become concrete: how many requests per second? how many of them fail? how much memory does the process use? Logs answer "what happened", while metrics answer "how are things right now". Without metrics you have to guess at these questions.

In the Python ecosystem this is done with prometheus-client and Prometheus — a system for collecting and storing time series. FastAPI is connected in a single line through prometheus-fastapi-instrumentator.

What Prometheus is and how it works

Prometheus doesn't wait for data to be sent to it — it comes and takes it itself. Every 15 seconds the scraper does a GET /metrics to your service and gets a text list of current values. This is called the pull model.

Your service stores metrics in memory and serves them on request. The prometheus-client library takes care of both storage and formatting the response.

Wiring it up

Install the dependencies:

prometheus-client>=0.20
prometheus-fastapi-instrumentator>=7.0

Initialization in main.py:

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

Instrumentator(
    should_group_status_codes=True,
    should_ignore_untemplated=True,
    excluded_handlers=["/health/live", "/health/ready", "/metrics"],
).instrument(app).expose(app, endpoint="/metrics", include_in_schema=False)

After startup GET /metrics serves a text list of metrics. The Prometheus scraper picks it up every 15 seconds.

should_group_status_codes=True groups 200/201/204 into the class 2xx — this reduces the number of unique label combinations and lowers the load on Prometheus.

Standard service labels

By default Prometheus only knows the address it took the data from. To distinguish services, environments (prod/staging) and versions, you add identity labels through Info:

import os
from prometheus_client import Info

SERVICE_INFO = Info("service", "Service identity labels")
SERVICE_INFO.info({
    "service": os.getenv("SERVICE_NAME", "order-service"),
    "env":     os.getenv("APP_ENV", "dev"),
    "version": os.getenv("BUILD_VERSION", "unknown"),
})

This exports a single metric service_info{service="order-service",env="prod",version="1.4.2"} 1. Grafana pulls these values in through group_left and adds them to any chart. This way you don't have to duplicate these labels on every counter.

RED for HTTP — automatically

RED is three questions about the state of an HTTP service:

• Rate — how many requests per second
• Errors — what fraction ends in an error
• Duration — how long a request takes to process

Instrumentator collects all of this without extra code and exports:

http_requests_total{handler, method, status_code}
http_request_duration_seconds_bucket{handler, method, le}
http_request_size_bytes_bucket{handler, method, le}

Prometheus queries for a dashboard:

# Rate — requests per second per endpoint
sum(rate(http_requests_total[5m])) by (handler, method)

# Errors — the share of 5xx
sum(rate(http_requests_total{status_code="5xx"}[5m])) by (handler)
  /
sum(rate(http_requests_total[5m])) by (handler)

# Duration p95
histogram_quantile(
  0.95,
  sum by (le, handler) (rate(http_request_duration_seconds_bucket[5m]))
)

If you need specific boundaries for an SLO (for example, 100ms/500ms/1s):

from prometheus_fastapi_instrumentator import Instrumentator, metrics

Instrumentator().instrument(app).add(
    metrics.latency(buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0])
)

USE for resources — also automatically

USE is three questions about resources:

• Utilization — how loaded the resource is
• Saturation — whether there's a queue
• Errors — whether there are errors at the resource level

prometheus-client exports process and GC metrics with no configuration:

An example of alerts on resource saturation:

groups:
  - name: order-service
    rules:
      - alert: MemoryHigh
        expr: >
          process_resident_memory_bytes{service="order-service"}
          > 400 * 1024 * 1024
        for: 10m
        labels:
          severity: warning

      - alert: CPUHigh
        expr: >
          rate(process_cpu_seconds_total{service="order-service"}[5m]) > 0.8
        for: 5m
        labels:
          severity: warning

Connection pools (asyncpg, Redis) don't export metrics themselves — they need to be instrumented manually through Gauge (more on that below).

Business metrics

HTTP and resources are infrastructure. Business metrics answer a different question: what's happening in the domain? How many orders were created? What's the distribution of amounts? How many carts are active right now?

For this, prometheus-client has three types of objects:

• Counter — only grows (number of events, errors)
• Histogram — distribution of values (amounts, durations)
• Gauge — the current value, can grow and fall (active carts, pool size)

Move metrics into a separate module:

# app/metrics/order_metrics.py
from prometheus_client import Counter, Histogram, Gauge

order_created_total = Counter(
    "order_created_total",
    "Total orders created",
    ["channel"],
)

order_amount_rubles = Histogram(
    "order_amount_rubles",
    "Order amount distribution",
    ["channel"],
    buckets=[100, 500, 1000, 5000, 10000, 50000],
)

payment_processing_seconds = Histogram(
    "payment_processing_seconds",
    "Payment processing duration",
    ["payment_method", "status_class"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],
)

cart_active_total = Gauge(
    "cart_active_total",
    "Currently active shopping carts",
)

Usage in the handler:

from app.metrics.order_metrics import order_created_total, order_amount_rubles

class CreateOrderHandler:
    def __init__(self, order_repo: OrderRepository) -> None:
        self._order_repo = order_repo

    async def handle(self, command: CreateOrderCommand) -> Order:
        order = await self._order_repo.save(Order.create(command))

        order_created_total.labels(channel=command.channel).inc()
        order_amount_rubles.labels(channel=command.channel).observe(
            float(order.amount)
        )
        return order

Histogram.observe() automatically distributes the value across buckets and accumulates _sum/_count — from these Prometheus computes quantiles without extra code.

For timing, the Histogram.time() context manager is convenient — or perf_counter by hand if you need a label based on the result:

import time
from app.metrics.order_metrics import payment_processing_seconds

class ProcessPaymentHandler:
    async def handle(self, command: ProcessPaymentCommand) -> PaymentResult:
        start = time.perf_counter()
        try:
            result = await self._payment_client.charge(command)
            status = "success"
            return result
        except PaymentClientError:
            status = "client_error"
            raise
        except Exception:
            status = "server_error"
            raise
        finally:
            duration = time.perf_counter() - start
            payment_processing_seconds.labels(
                payment_method=command.payment_method,
                status_class=status,
            ).observe(duration)

An example of a Gauge for the asyncpg pool:

from prometheus_client import Gauge

db_pool_size = Gauge("db_pool_size", "asyncpg pool total connections")
db_pool_active = Gauge("db_pool_active", "asyncpg pool acquired connections")

async def update_pool_metrics(pool: asyncpg.Pool) -> None:
    db_pool_size.set(pool.get_size())
    db_pool_active.set(pool.get_size() - pool.get_idle_size())

Call it from a background task every N seconds — not from a handler, otherwise the metric only updates when there's traffic.

How to name metrics correctly

The Prometheus convention: snake_case, with the unit of measurement in the name.

order_created_total          # Counter — the _total suffix
payment_processing_seconds   # duration Histogram — _seconds
order_amount_rubles          # amount Histogram — unit in the name
cart_active_total            # Gauge — current count

Common mistakes:

orderCreatedCount            # camelCase — not accepted
paymentTime                  # no unit — unclear, seconds? milliseconds?
order_amount                 # no unit

A Counter always ends in _total. A Histogram carries the unit (_seconds, _bytes, _rubles) — Prometheus automatically adds _bucket, _sum, _count with the correct suffixes.

Low label cardinality — important

Labels in Prometheus aren't just text. Each unique combination of labels creates a separate time series in the database. A million unique values = a million series = Prometheus crashes from lack of memory.

The rule: a label is a category with a small number of values, not a unique identifier.

# Good — 3–10 values per label
order_created_total.labels(
    channel="web",           # web / mobile / api
).inc()

payment_processing_seconds.labels(
    payment_method="SBP",    # CARD / SBP / CRYPTO
    status_class="success",  # success / client_error / server_error
).observe(duration)

# Bad — a million time series, OOM Prometheus
order_created_total.labels(
    user_id=str(command.user_id),    # millions of values
    order_id=str(order.id),          # millions of values
).inc()

If you need to observe specific entities — that's a job for tracing, not for metrics. Metrics give an aggregated picture, traces give the details of a specific request.

/metrics on a separate port

The /metrics endpoint shouldn't be visible from the outside. In FastAPI it's isolated through a separate ASGI application on a different port:

import uvicorn
import os
from prometheus_client import make_asgi_app

metrics_app = make_asgi_app()

async def run_metrics_server() -> None:
    config = uvicorn.Config(
        metrics_app,
        host="0.0.0.0",
        port=int(os.getenv("METRICS_PORT", "9090")),
    )
    server = uvicorn.Server(config)
    await server.serve()

Running it in parallel with the main application:

import asyncio

async def main() -> None:
    await asyncio.gather(
        run_app(),
        run_metrics_server(),
    )

Port 9090 is closed off at the level of Ingress rules and opened only for the monitoring namespace through a NetworkPolicy.

In short

• prometheus-client stores metrics in memory, Prometheus takes them by the pull model every 15 seconds.
• prometheus-fastapi-instrumentator automatically collects RED metrics (rate/errors/duration) for HTTP endpoints.
• USE process metrics (memory, CPU, GC) are exported with no configuration — process_* and python_gc_*.
• Business events — Counter (grows), Histogram (distribution), Gauge (current value).
• Identity labels (service, env, version) are set once through Info, not duplicated on every metric.
• Metric names: snake_case + unit of measurement (_seconds, _bytes, _total).
• Labels are categories with a small number of values. user_id as a label = a time-series explosion and a Prometheus crash.
• /metrics — on a separate port through make_asgi_app(), closed off from external traffic.

What to read next

• Tracing in Python — OTel, manual spans, per-entity observability without a cardinality explosion.
• Logging in Python — structlog, JSON in production, context fields.
• SLO and alerts in Python — error budget, multi-window burn rate, runbook.
• Health checks in Python — liveness vs readiness, custom checks.