MiddleDebuggingVery commonNot answered yet

A Pod is in CrashLoopBackOff — how do you tell an app crash from a failing liveness probe?

A Pod is in CrashLoopBackOff and restarting. Explain what the status does and does not tell you, then use the artifact to decide whether this is an application crash or a failing liveness probe — and how you would tell the two apart in general.

$ kubectl get pod worker-84c
NAME         READY   STATUS             RESTARTS   AGE
worker-84c   0/1     CrashLoopBackOff   5          4m

$ kubectl describe pod worker-84c
    Last State:  Terminated
      Reason:    Error
      Exit Code: 1
    Restart Count: 5
Events:
  Warning  BackOff  kubelet  Back-off restarting failed container

Diagnose and fix.

CrashLoopBackOff only means the container keeps dying and the kubelet backs off — not why. Check logs --previous and lastState.terminated: Reason Error, Exit 1 is an app crash. A liveness kill logs Liveness probe failed; exit 137 is a SIGKILL.

✗Treating CrashLoopBackOff as a root cause rather than a symptom
✗Reading live logs instead of logs --previous for the dead container
✗Assuming every restart is an app bug, ignoring a liveness-probe kill

→What exit code would you expect from an OOM kill, and how does that differ here?
→How does a too-aggressive initialDelaySeconds on liveness cause a crash loop?

Contents

Solution

1. The status is a symptom, not a cause

CrashLoopBackOff only means: the container starts, dies, the kubelet waits (10s, 20s, 40s… up to 5m) and retries. Why it dies is a separate lookup.

2. The key discriminator

$ kubectl logs worker-84c --previous     # stdout of the dead instance
    ...
    panic: connect tcp 10.0.0.9:5432: connection refused

$ kubectl describe pod worker-84c | sed -n '/Last State/,/Exit Code/p'
    Last State: Terminated  Reason: Error  Exit Code: 1

logs --previous. Here the app crashed itself (could not reach the DB).

restarts by the kubelet's decision, not because the app exited.

App crash: lastState.terminated with a non-zero code (1, 2); the reason is in
Liveness-probe kill: the Events show Liveness probe failed: ..., and the container
Exit Code 137 = 128 + 9 (SIGKILL) — OOM or a probe kill; check Reason.

3. Fix

Here it is the app: fix the unreachable dependency or add retries/wait. Had it been the probe, you would relax initialDelaySeconds / failureThreshold.