← back to the section

The payment failed, the screen says «Something went wrong». You attach a screenshot, and a day later the task comes back marked «cannot reproduce». The screenshot proved that the user was in pain, but it said nothing about where the pain started, or why.

While a request travels through the system, every service leaves a trail — lines in a work journal everyone calls the log. Finding your own request in there is the cheapest way to turn a guess into a fact.

one click on «Pay» — three services — three log lines order-apitook POST /payments stock-apireserves the item payment-apicharges the card req=7f3areq=7f3areq=7f3a 10:42:07INFOorder-apireq=7f3aorder 42 created 10:42:07INFOstock-apireq=7f3aitem reserved 10:42:37ERRORpayment-apireq=7f3agateway timeout, 30 s the screen said «something went wrong»the log says which service, which second, and why

One click passes through three services, and each writes its own line carrying the shared req=7f3a. That identifier stitches three logs into one thread: two INFO milestones went through, the third line is an ERROR from payment-api after thirty seconds of waiting. The screen called it «something went wrong»; the log named the service, the second and the cause.

What a log gives you that a screen cannot

A screen shows the outcome, a log shows the road to it. «Payment does not go through» and «payment-api gave up on the card gateway after thirty seconds at 10:42:37» are the same finding at two levels of proof, and the second one does not come back marked «cannot reproduce». A log line does not replace reproduction steps: it answers a different question — not «how do I repeat this», but «what was happening inside».

On a single test machine the log is a file read with tail -f and grep right in the terminal (the minimum set of commands). Where there are many services nobody reads files: lines from every machine flow into one searchable store.

Levels: DEBUG, INFO, WARN, ERROR

Write everything down and the log grows by gigabytes a day with nothing findable in it. Write only crashes and you cannot see what led up to them. So every line carries a level of importance, and the service is started with a setting for which level to write from.

DEBUG holds the developer's details: variable values, request bodies, answers from neighbouring systems. Test machines usually have it on, production usually does not — partly because of volume, partly because other people's personal data leaks into such lines. INFO marks the milestones of a scenario: «order 42 created», «payment sent»; they show how far the request got. WARN means «odd, but we are alive»: a retry, a slower answer than usual, an empty field replaced by a default. It is the most underrated level, because WARN often appears minutes before the first ERROR. ERROR means the operation did not happen, and next to it sits a long stack of calls: attach the whole thing to the report, but read the first line — it holds the error name and the message.

Levels nest. Set INFO and the DEBUG lines are not filtered out, they are never written. That is why «turn DEBUG on for this test machine and repeat» is a reasonable request.

A level in the log and an error on the screen are two different things. An ERROR line is not necessarily yours: other people's scenarios run alongside. The other way round happens too — the screen shows an error while no ERROR line exists anywhere, which is what a routine refusal looks like when a form check did not pass.

Finding your own request in the stream

You are not alone on a test machine: a neighbouring team, automated tests and background jobs write thousands of lines a minute into the same log. Reading its tail with your eyes is a fine way to find somebody else's error and file the defect in the wrong place.

Three handles narrow the stream. Time: write down the second you clicked, not «sometime after lunch»; here is the trap — servers usually live in UTC while your clock is local, so a click at 12:05 in Tokyo sits in the log at 03:05. You work the offset out once. Your own data: the account's email or the order number, grep "order-42" app.log. The third handle, the most precise one, is the request identifier.

The identifier that runs through every service

A click almost never lives inside one service: one creates the order, another reserves the item, a third takes the money, and each keeps its own log. Having found your line in the first, you are back in the crowd in the second. Hence the correlation identifier — a random string minted by the first service that accepts the request and passed on in a header, which every other service then writes into each of its lines. Names vary — request id, correlation id, trace id; in headers it shows up as X-Request-Id or the standard traceparent. The point is the same: a shared key that stitches the logs into one story in time order.

You pick it up in the Network tab: open the request, look at the response headers. That copied string is your pass into the logs, and it always belongs in the bug report. No identifier in the response is a finding of its own.

Kibana in plain terms: where collected logs live

Files on ten machines are unreadable, and after a container restart the file disappears along with its history. So the lines are shipped into one searchable store — the classic bundle is called ELK: Elasticsearch keeps and searches, Logstash (or the lightweight Filebeat) picks lines up and splits them into fields, Kibana gives you the screen to search on. Other bundles exist — Grafana Loki, OpenSearch — and the habits carry over.

The time window in the top right corner decides everything: while it says «last 15 minutes», yesterday's error will never be found and the screen will honestly show nothing. Set the window before you search.

Next comes searching by field rather than by raw text. A collected line is split into level, service, message, request_id, so a query reads request_id: "7f3a" or service: "payment-api" and level: ERROR. Plain word search works too, but it is noisy. Expand the line you found to see every field, and look at its neighbours.

Metrics and dashboards: what Grafana adds

A log answers questions about one request. It does not answer «is this worse than it used to be» — that needs numbers taken regularly: requests per second, the share of 5xx answers, response time at the p95 mark, memory in use, queue length. Such a number is a metric, a set of graphs over them is a dashboard, and Grafana usually draws them on top of a metric store such as Prometheus or Zabbix.

The split is simple: a metric shows the shape of a problem and its moment, a log shows the cause. A spike of errors at 12:05 on the graph sends you to Kibana with the window 12:04–12:06 and level ERROR. Manual work needs a dashboard on long runs: does response time creep up hour by hour, does memory leak.

When the log says the defect is elsewhere

A defect bounces between teams until somebody names the boundary it appeared behind. Logs name it: the last INFO tells you who finished, the first ERROR tells you who could not.

  • No line carries your identifier at all. The request never arrived — that is a question about the machine and the network, not about the code.
  • An ERROR in your service wrapping somebody else's refusal: «call timed out», «502 from a neighbour». The defect sits on the boundary, and both lines go into the report.
  • An error in the log but status 200 for the user. That is a second defect: the failure was hidden.

In short

  • A log answers «what was happening inside», not «how to repeat it»; a line with a timestamp and a service name kills «cannot reproduce».
  • Levels nest: with INFO set, DEBUG lines do not exist — turn them on beforehand.
  • Read WARN as carefully as ERROR: it often lands minutes before the failure.
  • Log time is usually UTC — work the offset out once.
  • The identifier from the response header stitches the logs of all services into one thread.
  • In Kibana set the time window first and search by field; Grafana shows the shape of a problem, logs show the cause.