Observability configuration in Go: management port, logs and metrics

Go Backend

How to set up observability in a Go service from scratch: a separate management port for /metrics and /health, log format via APP_ENV, histogram buckets tuned to your SLO, and changing the log level dynamically without a restart.

Published 28 июня 2026 ~6 min read Author: Вадим Викулин

← Back to the section

Once a service is running in production, you need to keep track of what it's doing: read the logs, collect metrics, check its state. All of this is called observability. In this article we'll look at how to set it up properly in a Go service: where to mount /metrics and /health, how the log format depends on the environment, and why a histogram with the wrong bucket boundaries lies to you about your SLO.

Why you need a separate management port

Imagine you have a single port :8080 for everything. Prometheus knocks on it for metrics every 15 seconds, Kubernetes checks /health every 5 seconds — and all of it runs in the same pool as real user requests.

The problem is twofold. First — load on the business server's handler pool. Second — security: /pprof (the memory profiler) must not be reachable from the same address that receives public traffic.

The solution: two http.Server instances in one process.

// cmd/server/main.go
func run(ctx context.Context, cfg Config) error {
    router := buildRouter(deps)
    businessSrv := &http.Server{
        Addr:    cfg.Addr,   // :8080 — for external traffic
        Handler: otelhttp.NewHandler(router, cfg.ServiceName),
    }
    managementSrv := platform.StartManagement(cfg.ManagementAddr, deps) // :8081

    g, gCtx := errgroup.WithContext(ctx)
    g.Go(func() error { return businessSrv.ListenAndServe() })
    g.Go(func() error { return managementSrv.ListenAndServe() })
    g.Go(func() error {
        <-gCtx.Done()
        shutCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        _ = businessSrv.Shutdown(shutCtx)
        _ = managementSrv.Shutdown(shutCtx)
        return nil
    })
    return g.Wait()
}

errgroup starts both servers in parallel and shuts them down cleanly on a termination signal.

The management server mounts exactly four routes:

// internal/platform/metrics/server.go
func StartManagement(addr string, deps Deps) *http.Server {
    mux := http.NewServeMux()
    mux.Handle("GET /metrics", promhttp.Handler())
    mux.HandleFunc("GET /health/live", liveHandler)
    mux.Handle("GET /health/ready", deps.ReadyHandler)
    mux.HandleFunc("GET /info", infoHandler(deps.Version, deps.Commit, deps.BuildTime))
    mux.HandleFunc("PUT /log-level", logLevelHandler(deps.LogLevel))
    return &http.Server{Addr: addr, Handler: mux}
}

In Kubernetes a network policy lets Prometheus connect on 8081, while the Ingress publishes only 8080. Scraping traffic and health probes never hit the business server's handler pool.

An important rule: do not mount /pprof on the management port in production without mTLS or a network policy. Memory profiles expose the service's internal data.

Log format by APP_ENV

During local development you want readable logs. In production — structured JSON that Loki or Datadog can parse without regular expressions.

A single switch on an environment variable handles this:

// internal/platform/log/setup.go
func New(env string, level *slog.LevelVar) *slog.Logger {
    opts := &slog.HandlerOptions{Level: level}
    if env == "production" {
        return slog.New(slog.NewJSONHandler(os.Stdout, opts))
    }
    return slog.New(slog.NewTextHandler(os.Stdout, opts))
}

Locally (APP_ENV=dev) the output looks like this:

10:42:03.211 INFO order_confirmed order_id=ORD-9912 customer_id=C-441

In production (APP_ENV=production) — JSON with a trace identifier:

{"time":"2026-06-19T10:42:03.211Z","level":"INFO","msg":"order_confirmed","order_id":"ORD-9912","customer_id":"C-441","trace_id":"4b3e..."}

The logger is created once at startup and passed into handlers through the constructor — slog.SetDefault(...) is not used inside packages. This lets tests pass in any logger without touching global state.

Dynamic log level

A common production situation: something is going wrong, but at the INFO level the logs only show symptoms. You want to switch to DEBUG — and back to INFO once you've figured it out. Without restarting the service.

slog.LevelVar is a mutex-protected level variable. It can be changed at any moment, and all subsequent messages will be filtered by the new threshold:

// cmd/server/main.go
var logLevel slog.LevelVar // defaults to 0 = INFO
log := platform.New(cfg.AppEnv, &logLevel)

The management endpoint accepts a PUT request and changes the level:

// internal/platform/metrics/server.go
func logLevelHandler(level *slog.LevelVar) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        var body struct{ Level string `json:"level"` }
        if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
            http.Error(w, "bad request", http.StatusBadRequest)
            return
        }
        var l slog.Level
        if err := l.UnmarshalText([]byte(body.Level)); err != nil {
            http.Error(w, "unknown level", http.StatusBadRequest)
            return
        }
        level.Set(l)
        w.WriteHeader(http.StatusNoContent)
    }
}

Switching: curl -X PUT :8081/log-level -d '{"level":"DEBUG"}'.

Histogram buckets and the SLO

Prometheus measures latency with a histogram: every request falls into one of the predefined buckets. Percentiles — p99, for example — are then computed from these buckets.

The problem with prometheus.DefBuckets is that it defines generic boundaries from 5ms to 10s. If your SLO is "p99 < 500ms", the boundary at 0.5 does exist in the standard buckets, but its neighbors are 0.25 and 1.0. Interpolation will produce an inaccurate result. You need to define the buckets explicitly with a boundary exactly at the SLO threshold:

// internal/platform/metrics/http.go
var httpRequestDurationSeconds = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "HTTP request latency",
    Buckets: []float64{0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0},
}, []string{"method", "path", "status_class"})

The 0.5 bucket gives an exact histogram_quantile(0.99, ...) with no interpolation for the SLO "p99 < 500ms".

The standard labels service, env, version are added once via MustCurryWith — not on every .WithLabelValues(...) call:

// internal/platform/metrics/common.go
func CommonLabels(cfg Config) prometheus.Labels {
    return prometheus.Labels{
        "service": cfg.ServiceName,
        "env":     cfg.AppEnv,
        "version": cfg.Version,
    }
}

// internal/order/metrics.go
type OrderMetrics struct {
    created *prometheus.CounterVec
}

func NewOrderMetrics(cfg platform.Config) *OrderMetrics {
    return &OrderMetrics{
        created: ordersCreatedTotal.MustCurryWith(platform.CommonLabels(cfg)),
    }
}

func (m *OrderMetrics) OrderCreated(paymentMethod string) {
    m.created.WithLabelValues(paymentMethod).Inc()
}

Cardinality: why the metric's path must not be a raw URL

If you record a value like /orders/ORD-9912 in the path label, Prometheus creates a separate time series for every order. A thousand orders — a thousand time series. This is called a cardinality explosion, and it kills Prometheus.

The right way: store the route template in the label — /orders/{orderID}. Chi provides it through RouteContext:

// internal/platform/middleware/metrics.go
func Metrics(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ww := middleware.NewWrapResponseWriter(w, r.ProtoMajor)
        start := time.Now()
        next.ServeHTTP(ww, r)
        routeCtx := chi.RouteContext(r.Context())
        path := "unknown"
        if routeCtx != nil && routeCtx.RoutePattern() != "" {
            path = routeCtx.RoutePattern()
        }
        status := statusClass(ww.Status())
        httpRequestsTotal.WithLabelValues(r.Method, path, status).Inc()
        httpRequestDurationSeconds.WithLabelValues(r.Method, path, status).
            Observe(time.Since(start).Seconds())
    })
}

chi.RouteContext(r.Context()).RoutePattern() returns /orders/{orderID} — a fixed number of time series regardless of the number of orders.

The same logic applies to user_id, order_id and other identifiers: they must not be label values. To track a specific request, use traces (OTel span attributes), not metrics.

Build information via /info

A handy endpoint: when something goes wrong in production, you can immediately see which version of the service is running.

The values are injected at build time via -ldflags:

// cmd/server/version.go
var (
    version   = "dev"
    commit    = "none"
    buildTime = "unknown"
)

# Makefile
LDFLAGS=-ldflags "-X main.version=$(VERSION) -X main.commit=$(COMMIT) -X main.buildTime=$(BUILD_TIME)"
go build $(LDFLAGS) -o bin/order-service ./cmd/server

func infoHandler(version, commit, buildTime string) http.HandlerFunc {
    payload, _ := json.Marshal(map[string]string{
        "version":    version,
        "commit":     commit,
        "build_time": buildTime,
    })
    return func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        _, _ = w.Write(payload)
    }
}

In short

Two http.Server instances in one process: business traffic on :8080, management (/metrics, /health/*, /info) on :8081.
errgroup starts both servers and shuts them down cleanly on a signal.
APP_ENV=production → slog.NewJSONHandler, otherwise slog.NewTextHandler.
slog.LevelVar lets you change the log level via a PUT request without a restart.
Histogram buckets are defined at registration with an explicit boundary at the SLO threshold, not via prometheus.DefBuckets.
Standard labels service/env/version — once via MustCurryWith, not on every call.
The path label holds the route template (/orders/{orderID}), not a raw URL — otherwise a cardinality explosion.
/pprof in production only behind mTLS or a network policy.