← Back to the section

← Back to the section

Kubernetes takes care of starting containers, scheduling them across nodes, and restarting crashed pods. But there are four things nobody does for the developer: setting up health checks correctly, honestly reserving memory for the JVM, shutting down cleanly without dropping requests, and passing configuration in from outside the image. Without these, the service will work "almost fine" — right up until the first deploy under load.

The difference between the three probes shows up in a single incident: the database is unreachable for two minutes, and everything after that depends on which group its check sits in.

Three probes: why Kubernetes keeps asking "are you alive?"

Kubernetes has no idea what is happening inside your container. To figure out whether it should send traffic there and whether the pod needs a restart, it asks three questions — probes. Each question is handled in its own way:

• startup probe — "have you started yet?" Until the application answers "yes," the other two questions aren't asked. This protects slow-starting applications: without a startup probe, Kubernetes may decide the pod is stuck and kill it right in the middle of the first startup.
• readiness probe — "can I send you traffic?" If "no," the pod is removed from load balancing but keeps running. It's a pause, not a verdict: this is useful while caches warm up or a database connection is being re-established.
• liveness probe — "is the process responding at all?" If "no," the pod is restarted. This is a last resort: assign it only to conditions that a restart actually fixes (a deadlock, a hung thread), not to temporary trouble.

Spring Boot Actuator supports this separation out of the box. You need to enable it:

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: readinessState, db

Then /actuator/health/liveness and /actuator/health/readiness are two independent endpoints. In the Kubernetes manifest they are declared separately:

containers:
  - name: app
    startupProbe:
      httpGet: { path: /actuator/health/liveness, port: 8081 }
      failureThreshold: 30
      periodSeconds: 2
    readinessProbe:
      httpGet: { path: /actuator/health/readiness, port: 8081 }
      periodSeconds: 5
    livenessProbe:
      httpGet: { path: /actuator/health/liveness, port: 8081 }
      periodSeconds: 10
      failureThreshold: 3

The most common mistake: putting a database check inside the liveness probe. It sounds logical — "if the database is unreachable, let's restart." In reality: the database goes down → the liveness probe fails → all pods of the service restart in a loop → on top of the database incident you now have a nonstop storm of restarts, JVM warmups, and lost connections. The rule is simple: liveness checks only the internal state of the process (the default livenessState); dependencies on external services belong in readiness at most.

Memory: why the JVM doesn't fit inside the limit

When you set a memory limit on a container, it feels obvious: set 512 MiB and it won't use more. With the JVM that's not how it works.

A JVM process's memory is more than just heap. Besides heap there is metaspace (classes, code), thread stacks, the JIT compiler, direct buffers — together another few hundred megabytes. If you give the whole limit to heap, there's nothing left for the rest, and the container gets killed by the operating system — no warning, no log entry, just exit 137.

A modern JVM can see the container's limit and pick the heap size itself. This is controlled by a flag:

-XX:MaxRAMPercentage=60 -XX:+ExitOnOutOfMemoryError

60–75% of the limit for heap is a working range. The remainder goes to JVM overhead. Example: with a 768 MiB limit, heap gets ~460 MiB, and the rest is for metaspace and everything else.

Two different symptoms of running out of memory:

• OOMKilled (exit 137, silence in the logs) — the container is killed by the OS kernel: the process as a whole exceeded the limit. The cause is usually non-heap consumption. Fix: raise the limit or lower MaxRAMPercentage.
• OutOfMemoryError in the logs — the heap inside the JVM ran out. Fix: hunt for a leak or increase heap.

Practical rules for resources:

resources:
  requests: { cpu: "500m", memory: "768Mi" }
  limits: { memory: "768Mi" }

For memory: set the limit equal to the request — then there are no surprises from noisy neighbors on the node. For CPU: better not to set a limit, or set it high. CPU throttling hurts the garbage collector and increases latency — workload isolation is provided by requests, not limits.

Graceful shutdown: don't drop requests during a deploy

When Kubernetes stops a pod (during a deploy or scaling down), the following happens: the pod is marked for deletion → removed from load balancing → the container is sent SIGTERM → after terminationGracePeriodSeconds — SIGKILL.

Two places where you can drop requests:

First: removing the pod from load balancing doesn't happen instantly. For a few seconds after SIGTERM, traffic can still arrive at the stopping pod. The solution is a short pause before shutdown begins:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]
terminationGracePeriodSeconds: 30

Second: in-flight requests need to be allowed to finish. Spring Boot can complete current requests before shutting down:

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 20s

The budget must add up: preStop (5s) + time to finish requests (20s) < terminationGracePeriodSeconds (30s). If it doesn't, SIGKILL will cut requests off halfway.

Configuration: one image, settings from outside

The container principle: the exact same image runs on dev, staging, and prod. The differences between environments live outside the image, in Kubernetes objects:

• ConfigMap — ordinary settings (URLs, timeouts, topic names).
• Secret — passwords, tokens, keys.

In Spring Boot they come in through environment variables:

envFrom:
  - configMapRef: { name: order-service-config }
  - secretRef: { name: order-service-secrets }

Kubernetes automatically maps the variable APP_DATASOURCE_URL to the property app.datasource.url — Spring reads it with no extra configuration.

A few rules:

• Secrets don't go in a ConfigMap, and certainly not in the image.
• The profile (SPRING_PROFILES_ACTIVE) is set by the environment, not the Dockerfile.
• Required settings are best validated at startup via @Validated @ConfigurationProperties — failing at deploy time is better than getting a surprise at runtime.

Autoscaling (HPA)

The Horizontal Pod Autoscaler watches metrics and adds or removes pods:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: order-service }
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

A few specifics for JVM services:

• A new replica doesn't help instantly: starting the JVM, warming up the JIT and connection pools takes tens of seconds. On a sudden spike in load, the HPA is too late. That's why minReplicas should provide spare capacity up front.
• CPU is a poor metric for services that spend most of their time waiting on a database or an API (I/O-bound). Scaling by request count or queue length is more honest, but that requires custom metrics.

In short

• startup probe protects against being killed on a slow start; readiness takes the pod out of load balancing without a restart; liveness restarts, only for internal failures.
• Including the database in liveness is dangerous: the failure of a single dependency triggers a restart storm across the whole service.
• JVM memory > heap: metaspace, stacks, JIT. MaxRAMPercentage=60-75 leaves room for overhead.
• OOMKilled (exit 137) means the OS kernel killed the process for exceeding the limit; OutOfMemoryError means the heap ran out inside the JVM.
• server.shutdown: graceful + a 5-second preStop pause = requests aren't dropped during a deploy.
• One image for all environments; config in a ConfigMap, secrets in a Secret, the profile set by the environment.
• @Validated @ConfigurationProperties — bad config fails the startup instead of breaking the runtime.
• An HPA using a CPU metric is too late on spikes; minReplicas should provide a buffer.

What to read next

• Deployment and configuration — how manifests make their way to the cluster.
• Operations and debugging — OOMKilled and CrashLoopBackOff in practice.
• Spring Actuator and Micrometer — health groups and metrics.