← Back to the section

From the outside every Kubernetes incident looks the same — "the service is down." But the causes differ: the process itself crashed, memory ran out, the image failed to pull, a health check didn't pass, routing broke. This article is a set of commands and a step-by-step walkthrough of typical situations: from a symptom we find the cause and understand what to do.

The basic set of commands

Before we dig into specific symptoms — a dozen commands you'll need constantly. All examples use the namespace payments; substitute your own.

kubectl -n payments get pods                                  # overall state: STATUS, RESTARTS, AGE
kubectl -n payments describe pod order-service-xxx           # pod details: events, reasons
kubectl -n payments logs order-service-xxx                   # logs of the current container
kubectl -n payments logs order-service-xxx -p                # logs of the PREVIOUS run (before the restart!)
kubectl -n payments logs deploy/order-service --tail=200 -f  # logs of all replicas of the deployment
kubectl -n payments get events --sort-by=.lastTimestamp      # the namespace event feed
kubectl -n payments exec -it order-service-xxx -- sh         # a shell inside the container
kubectl -n payments port-forward svc/order-service 8080:80   # the service to your localhost
kubectl -n payments top pods                                  # actual CPU and memory
kubectl -n payments get endpoints order-service              # who really stands behind the Service
kubectl -n payments rollout status deploy/order-service      # how the rollout is going

Two commands are underrated more than the rest. logs -p is the only way to see what happened before the restart: the current logs start from zero after a restart. get events --sort-by=.lastTimestamp — Kubernetes spells out the reasons in plain text: the node ran out of memory, it couldn't pull the image, a health check failed.

CrashLoopBackOff: the pod restarts over and over

The pod starts, crashes, Kubernetes waits (the pause grows with every attempt) and restarts it again. This isn't a separate problem — it's a symptom: the application is exiting. The cause is inside.

The order of investigation:

  1. logs -p — what did the application say before it crashed? A stack trace? A "port already in use" message? A config read error?
  2. describe pod — look at the exit code. Code 1 — the process crashed on its own, the reason is in the logs. Code 137 — the process was killed from the outside, see OOMKilled below.
  3. If there are no logs at all — the application didn't reach the logger: check the image's entrypoint and the logging configuration.

One separate, common case is being killed by the liveness probe. The application takes 90 seconds to start, but the liveness probe begins checking after 30 — the pod is killed again and again, never managing to start. The fix is a startup probe with an honest failureThreshold. More on probes in the article Spring Boot in Kubernetes.

OOMKilled: the container was killed because of memory

The container exceeded its memory limit — and the operating system kernel killed it. Instantly, with no stack trace and no farewell messages in the logs. In the describe pod output you'll see: Last State: Terminated, Reason: OOMKilled.

The exit code in this case is 137 (128 + the SIGKILL signal).

For applications on a virtual machine — Java (JVM), .NET, Kotlin — the cause is almost always one of two: the runtime doesn't know about the container's limit and allocates the heap proportionally to all of the node's memory rather than the limit. Modern JVMs can read cgroup limits automatically starting with Java 10 — make sure the version is suitable and the required flags are set. More in resource settings.

Don't confuse them: an OutOfMemoryError in the logs is not OOMKilled. That means the heap ran out inside a live process. It's fixed differently: look at what's holding memory and either increase the heap or hunt for a leak.

ImagePullBackOff: the image won't pull

The node can't download the image from the registry. Three causes cover almost everything:

  • A typo in the tag, or the tag was never pushed — CI built the image, but the push failed.
  • No access to the registry: the imagePullSecret expired or wasn't specified at all.
  • The registry is unreachable from the node.

describe pod will show the exact registry error in the events section. This is the one incident of all where the application code is guaranteed to be irrelevant.

Pending: the pod hangs and goes nowhere

The pod is created, but Kubernetes hasn't assigned it to any node. describe pod → events → the reason will be there in words.

The most common ones:

  • Insufficient memory / Insufficient cpu — no single node has that much free requests. Scheduling goes by requests, not by actual consumption. The cluster can be "full" even at low real usage.
  • A mismatch in nodeSelector, affinity, or taints — the pod is looking for a node with certain labels, but there are none.

What a developer should do: check whether the service's requests are set too high. If not — a conversation about capacity with the team that manages the cluster.

Flapping readiness: neighbors get 503s

The symptom: the service "works," but consumers periodically hit 503s or timeouts. The pod keeps entering and dropping out of rotation.

The cause is an unstable readiness probe. When the pod fails the check, Kubernetes removes it from the endpoints: no traffic goes to it. When it passes — it's put back.

Typical scenarios:

  • An unstable external dependency is included in readiness. When it flaps, the whole service flaps.
  • GC pauses or overload make the probe miss its timeout.
  • Overly aggressive periodSeconds and failureThreshold.

The check: run kubectl get endpoints several times in a row — the list keeps shrinking and recovering. In describe pod — Readiness probe failed in the events.

The fix: take out of readiness the dependencies without which the service can still respond in a degraded mode, and adjust the thresholds.

"The request doesn't arrive": check the chain

From the outside you get 502 or 504, but the pods are green. Check the chain along the request path top to bottom, each link separately:

kubectl -n payments get ingress api                               # 1. Is the Ingress there, host/path correct?
kubectl -n payments get endpoints order-service                  # 2. Are there pods behind the Service?
kubectl -n payments port-forward svc/order-service 8080:80       # 3. Does the Service respond?
kubectl -n payments port-forward pod/order-service-xxx 8080:8080 # 4. Does the pod itself respond?

Where the chain breaks is where the problem lies:

  • Step 4 doesn't work → the problem is in the application.
  • Steps 2–3 → selector or readiness.
  • Step 1 → Ingress, DNS, or TLS.

Empty endpoints with live pods is almost always readiness or a typo in the selector (the pod's labels don't match the service's selector — this happens after renaming manifests). It's also worth checking the NetworkPolicy: "the request doesn't arrive" between namespaces may be a policy, not a breakage.

The pod restarted "on its own"

RESTARTS > 0 without a deploy — a restart always has a cause, and it's recorded. describe pod → Last State. The options:

  • OOMKilled — see above.
  • Liveness probe failed — check whether it's probing an external database or another service.
  • The node went into drain or was updated — this is a normal situation. Pods must survive relocation: for that you need PodDisruptionBudget and at least two replicas.
  • Evicted — the node ran out of disk space or another resource and evicted the pod.

What should be in place before an incident

Everything described above takes minutes to sort out when the service has basic hygiene:

  • Structured logs to stdout — the platform collects them. Files inside the container die together with it.
  • Metrics in Prometheus format — the de facto standard in Kubernetes.
  • Alerts on restarts and on endpoints being unavailable.
  • A dashboard with memory and CPU per pod compared against limits.

Without this, debugging turns into guessing. More in the Observability Style Guide.

In short

  • logs -p shows the logs before the restart — it's the first thing to look at on CrashLoopBackOff.
  • get events --sort-by=.lastTimestamp — Kubernetes states in plain text what went wrong.
  • CrashLoopBackOff — the application is crashing; exit code 1 → look at the logs, exit code 137 → OOMKilled.
  • OOMKilled: the runtime doesn't see the cgroup limit or the limit is too small; there's no trace in the logs.
  • ImagePullBackOff: the tag doesn't exist, no access to the registry, or the registry is unreachable.
  • Pending: requests don't fit on the node; scheduling goes by requests, not by actuals.
  • Flapping readiness → the pod drops out of endpoints → 503s for consumers.
  • 502/504 with live pods: check the Ingress → Service → endpoints → pod chain.
  • Spring Boot in Kubernetes — probes and resources: preventing half of these situations.
  • Networking and traffic — endpoints, Ingress, and NetworkPolicy.
  • Deployment and configuration — rollout status and rollbacks.
  • Observability Style Guide — the logs, metrics, and alerts without which debugging is blind.