Observability

A service in production is a black box. A user complains that "everything is slow," and you stare at a running process without understanding which request is slow, which service it got stuck on, or what failed. Observability is the set of signals that let you understand, from inside a running system, what is happening and why — without attaching a debugger.

People usually talk about three pillars:

• Logs — what happened: lines of events. Structured (in JSON) and carrying a shared trace_id, so they can be searched and correlated.
• Metrics — how much and how fast: numbers over time (requests per second, latency, errors, resource usage). Cheap, aggregatable, and the basis for graphs and alerts.
• Tracing — where exactly: the path of a single request through all services, with the timing of every step. It answers the question "which of the ten services lost 800 ms."

This section breaks observability down by topic:

• Logging — structured logs, levels, what and how to write (and what must never end up in logs).
• Metrics — which ones to collect (RED/USE), how to name them without exploding cardinality.
• Tracing — distributed tracing, spans, OpenTelemetry.
• Context propagation — how trace_id and context travel between services over HTTP and message brokers.
• Health checks — liveness/readiness, how they differ and why the orchestrator needs them.
• SLOs and alerts — what to alert on, so alerts wake you for real issues and not for noise.
• Configuration — how to turn all of this on and set it up in a service.

Articles come in variants for different languages and stacks — pick your own.