Metrics in Go: prometheus/client_golang, RED, USE and business counters

← Back to the section

When a service behaves strangely under load — responds slowly, fails with errors — without metrics you're left guessing. Metrics are the numbers a service reports about itself: how many requests per second, what fraction ends in an error, how long a response takes. Prometheus collects these numbers, Grafana draws the graphs. In the Go stack, prometheus/client_golang and the helper library promauto handle this.

Two ports: business and management

The main mistake when first wiring this up is to mount /metrics on the same port as the business API. That's dangerous: anyone who knows the address will see the service's internal statistics. The right approach is a separate management server on a different port (usually :9090) that only the infrastructure can see.

// internal/platform/metrics/server.go
package metrics

import (
    "net/http"

    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func StartManagement(addr string) *http.Server {
    mux := http.NewServeMux()
    mux.Handle("/metrics", promhttp.Handler())
    mux.HandleFunc("/health/live", liveHandler)
    mux.HandleFunc("/health/ready", readyHandler)
    return &http.Server{Addr: addr, Handler: mux}
}

In main.go we start both servers at once via errgroup:

businessSrv := &http.Server{Addr: cfg.Addr, Handler: router}
managementSrv := metrics.StartManagement(cfg.ManagementAddr) // :9090

g, ctx := errgroup.WithContext(ctx)
g.Go(func() error { return businessSrv.ListenAndServe() })
g.Go(func() error { return managementSrv.ListenAndServe() })
if err := g.Wait(); err != nil {
    log.ErrorContext(ctx, "server_stopped", slog.String("error", err.Error()))
}

The Prometheus scraper hits /metrics on the management port every 15 seconds and stores the data in its own database. It never sees the business port.

Standard labels once

Every service has standard attributes: name (service), environment (env), version (version). They need to be added to all metrics — but not copied into every With call. We create them once at startup via prometheus.Labels and apply them via MustCurryWith:

// internal/platform/metrics/common.go
var commonLabels = prometheus.Labels{
    "service": env.ServiceName,
    "env":     env.AppEnv,
    "version": env.Version,
}

var ordersCreatedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "orders_created_total",
    Help: "Orders successfully created",
}, []string{"service", "env", "version", "payment_method"})

// fix the common labels once:
var ordersCreated = ordersCreatedTotal.MustCurryWith(commonLabels)

// in the handler — only the business label:
ordersCreated.With(prometheus.Labels{"payment_method": "CARD"}).Inc()

This way each call has no repetition of service/env/version — only what changes meaningfully.

RED metrics via middleware

RED is three questions about the health of an HTTP service:

• Rate — how many requests per second?
• Errors — what percentage ends in an error?
• Duration — how long does the response take?

The best way to collect them is a single middleware that wraps all routes. An important detail: the path must come from the chi route pattern (/orders/{id}), not from the raw URL (/orders/abc123). Otherwise every unique ID creates a separate time series, and within a week there'll be millions of them.

// internal/platform/middleware/metrics.go
var (
    httpRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method, path and status class",
    }, []string{"method", "path", "status_class"})

    httpRequestDurationSeconds = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "path", "status_class"})
)

func Metrics(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ww := chimiddleware.NewWrapResponseWriter(w, r.ProtoMajor)
        start := time.Now()
        next.ServeHTTP(ww, r)

        path := chi.RouteContext(r.Context()).RoutePattern()
        status := statusClass(ww.Status())
        httpRequestsTotal.WithLabelValues(r.Method, path, status).Inc()
        httpRequestDurationSeconds.WithLabelValues(r.Method, path, status).Observe(time.Since(start).Seconds())
    })
}

func statusClass(code int) string {
    switch {
    case code < 400:
        return "success"
    case code < 500:
        return "client_error"
    default:
        return "server_error"
    }
}

The middleware is wired into the router after chi has parsed the path:

r := chi.NewRouter()
r.Use(RequestID)
r.Use(otelhttp.Middleware("order-service"))
r.Use(Metrics) // after otelhttp — the span already exists
r.Use(chimiddleware.Logger)

PromQL queries for the dashboard:

# RPS by route
sum(rate(http_requests_total[5m])) by (path, method)

# 5xx error rate
sum(rate(http_requests_total{status_class="server_error"}[5m])) by (path)
  / sum(rate(http_requests_total[5m])) by (path)

# p95 latency
histogram_quantile(0.95,
  sum by (le, path) (rate(http_request_duration_seconds_bucket[5m]))
)

USE metrics — runtime and connection pool

USE is three questions about resources:

• Utilization — how busy is the resource?
• Saturation — is there a queue of waiters?
• Errors — are there failures?

For the Go runtime (goroutines, GC, heap) and the process (CPU, file descriptors) everything is already in the standard collectors:

// internal/platform/metrics/setup.go
func RegisterCollectors() {
    prometheus.MustRegister(
        collectors.NewGoCollector(),       // goroutines, GC pauses, heap
        collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), // CPU, FD
    )
}

For the database connection pool (pgx) there's no collector out of the box, but writing one is easy:

type pgxPoolCollector struct {
    pool       *pgxpool.Pool
    acquired   *prometheus.Desc
    idle       *prometheus.Desc
    totalConns *prometheus.Desc
}

func NewPgxPoolCollector(pool *pgxpool.Pool, service string) *pgxPoolCollector {
    labels := prometheus.Labels{"service": service}
    return &pgxPoolCollector{
        pool:       pool,
        acquired:   prometheus.NewDesc("pgx_pool_acquired_conns", "Acquired connections", nil, labels),
        idle:       prometheus.NewDesc("pgx_pool_idle_conns", "Idle connections", nil, labels),
        totalConns: prometheus.NewDesc("pgx_pool_total_conns", "Total connections", nil, labels),
    }
}

func (c *pgxPoolCollector) Collect(ch chan<- prometheus.Metric) {
    stat := c.pool.Stat()
    ch <- prometheus.MustNewConstMetric(c.acquired, prometheus.GaugeValue, float64(stat.AcquiredConns()))
    ch <- prometheus.MustNewConstMetric(c.idle, prometheus.GaugeValue, float64(stat.IdleConns()))
    ch <- prometheus.MustNewConstMetric(c.totalConns, prometheus.GaugeValue, float64(stat.TotalConns()))
}

Key metrics from the collectors:

Business metrics next to the handler

System metrics answer the question "how is the service running", but not "what's happening in the business". How many orders were created? What percentage of confirmations failed? For this you add business metrics — counters and histograms that live next to the code of a specific module:

// internal/order/metrics.go
var (
    ordersCreatedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "orders_created_total",
        Help: "Orders successfully created",
    }, []string{"payment_method"})

    orderAmountRub = promauto.NewHistogram(prometheus.HistogramOpts{
        Name:    "order_amount_rub",
        Help:    "Order amount in rubles",
        Buckets: []float64{100, 500, 1000, 5000, 10000, 50000},
    })

    orderConfirmFailedTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "order_confirm_failed_total",
        Help: "Order confirmation failures by reason",
    }, []string{"reason"})
)

Usage in the handler:

func (h *CreateOrderHandler) Handle(ctx context.Context, cmd CreateOrderCommand) (*Order, error) {
    order, err := domain.NewOrder(cmd)
    if err != nil {
        return nil, fmt.Errorf("create order: %w", err)
    }
    if err := h.orders.Save(ctx, order); err != nil {
        return nil, fmt.Errorf("save order: %w", err)
    }

    ordersCreatedTotal.WithLabelValues(string(cmd.PaymentMethod)).Inc()
    orderAmountRub.Observe(float64(order.AmountMinor) / 100)
    return order, nil
}

Four metric types in Prometheus:

• Counter — only grows: orders_created_total, payment_failed_total.
• Gauge — a current value: queue size, active sessions.
• Histogram — a distribution of values: order amounts, processing time.
• CounterVec / HistogramVec — the same, but sliced by labels.

Metric names: snake_case and a unit of measurement

Prometheus expects names in snake_case, with the unit of measurement as a suffix. This is needed for compatibility with ready-made dashboards and alerts:

orders_created_total          — Counter (the _total suffix is mandatory)
payment_duration_seconds      — Histogram (time in seconds)
order_amount_rub              — Histogram (currency unit)
pgx_pool_acquired_conns       — Gauge (not a Counter — no _total)

Common mistakes:

orderCreatedCount    — violation: camelCase, no _total
paymentTime          — violation: no unit of measurement
orderAmount          — violation: no unit of measurement

Low cardinality in labels

Prometheus stores a separate time series for every unique combination of label values. If a user's UUID or an order ID ends up in a label — within a week there are millions of time series and the Prometheus server crashes on memory.

The rule is simple: a label is a category, not a unique identifier.

Correct:

ordersCreatedTotal.WithLabelValues("CARD").Inc()          // payment_method: CARD/SBP/CRYPTO
httpRequestsTotal.WithLabelValues("GET", "/orders", "success").Inc() // chi route pattern
orderConfirmFailedTotal.WithLabelValues("insufficient_stock").Inc()  // reason: a fixed set

Incorrect:

ordersCreatedTotal.WithLabelValues(cmd.OrderID).Inc()      // a unique UUID — OOM
httpRequestsTotal.WithLabelValues("GET", r.RequestURI, "success").Inc() // /orders/SKU-123456

If you need to trace a specific order or user — that's what tracing is for (distributed tracing via OTel spans). A span is stored once, not as a separate time series in the metrics database.

In short

• /metrics — only on a separate management port, not on the business port.
• Standard labels service/env/version — once via MustCurryWith, don't copy into every With.
• RED for HTTP — via chi middleware; the path comes from the route pattern, not the raw URL.
• USE for the runtime — collectors.NewGoCollector() + collectors.NewProcessCollector(...).
• Business metrics live next to the module's handler, not in a central file.
• promauto.NewCounterVec registers the metric automatically — no explicit Register needed.
• Names: snake_case, a suffix with the unit (_total, _seconds, _rub).
• A label is a category with dozens of values at most; unique IDs → OOM.

What to read next

• Observability configuration in Go — the management port, the log level at runtime.
• Health checks in Go — liveness/readiness on the management port.
• Tracing in Go — OTel spans for high-cardinality data.
• SLO and alerts in Go — multi-window burn-rate alerts on RED metrics.