Kubernetes Troubleshooting
Diagnosing CrashLoopBackOff, Pending, OOMKilled, Evicted and ImagePull errors, node NotReady, stuck rollouts and Ingress 502s from kubectl output.
12 questions
MiddleDebuggingVery commonA Pod is in CrashLoopBackOff — how do you tell an app crash from a failing liveness probe?
A Pod is in CrashLoopBackOff — how do you tell an app crash from a failing liveness probe?
CrashLoopBackOff only means the container keeps dying and the kubelet backs off — not why. Check logs --previous and lastState.terminated: Reason Error, Exit 1 is an app crash. A liveness kill logs Liveness probe failed; exit 137 is a SIGKILL.
Common mistakes
- ✗Treating
CrashLoopBackOffas a root cause rather than a symptom - ✗Reading live logs instead of
logs --previousfor the dead container - ✗Assuming every restart is an app bug, ignoring a liveness-probe kill
Follow-up questions
- →What exit code would you expect from an OOM kill, and how does that differ here?
- →How does a too-aggressive
initialDelaySecondson liveness cause a crash loop?
MiddleDebuggingVery commonA Pod stays Pending with no node assigned — what do you check, in order?
A Pod stays Pending with no node assigned — what do you check, in order?
Pending means the scheduler placed the Pod on no node. The FailedScheduling Events line names why — here Insufficient memory: no node fits the memory requests. Other causes it names: a taint, no matching nodeSelector/affinity, or an unbound PVC.
Common mistakes
- ✗Confusing
Pending(unscheduled) with a not-yet-Ready running container - ✗Ignoring the
FailedSchedulingmessage that names the exact reason - ✗Assuming only capacity, missing taints, affinity, and PVC binding
Follow-up questions
- →How would the
FailedSchedulingmessage differ for an untolerated taint? - →Why can lowering a Pod's memory
requestlet it schedule but risk an OOM later?
JuniorDebuggingCommonA Pod is stuck in ImagePullBackOff — what are the usual causes and first checks?
A Pod is stuck in ImagePullBackOff — what are the usual causes and first checks?
ImagePullBackOff is the kubelet failing to pull the image and backing off. Read the describe Events — here latset is a typo, so the registry returns not-found; fix the tag. Other causes: a missing imagePullSecret, wrong host, or a rate limit.
Common mistakes
- ✗Confusing
ImagePullBackOffwith a crashing container instead of a failed pull - ✗Skipping the
describeEvents, which name the exact pull failure - ✗Forgetting that a private image needs an
imagePullSecret
Follow-up questions
- →How do you attach an
imagePullSecretso a Pod can pull from a private registry? - →Why is a mutable tag like
latestrisky for reproducible deploys?
JuniorDebuggingCommonA Pod is failing and prints no useful logs — which kubectl commands do you run, in order?
A Pod is failing and prints no useful logs — which kubectl commands do you run, in order?
Run kubectl describe pod first — it shows Events, Conditions, and each container's State/lastState with the exit code. If the process died, kubectl logs --previous reads its stdout; kubectl get events adds context. Guess only after describe.
Common mistakes
- ✗Treating a
Runningstatus as proof the container is healthy - ✗Forgetting
logs --previousfor a container that already restarted - ✗Guessing at fixes before reading the
describeEvents
Follow-up questions
- →What does a container's
lastState.terminatedblock tell you that live logs cannot? - →When would you reach for
kubectl debuginstead oflogsanddescribe?
MiddleDebuggingCommonAn Ingress returns 503 while the Service's Pods appear up — trace where traffic is lost.
An Ingress returns 503 while the Service's Pods appear up — trace where traffic is lost.
A 503 with kubectl get endpoints api empty means the Service has no ready backends, so kube-proxy cannot route. Two causes: the readiness probe never passes, so Pods stay 0/1 Running and out of the Service, or the Service selector misses the Pod labels.
Common mistakes
- ✗Blaming the Ingress controller before checking the Service endpoints
- ✗Forgetting a not-Ready Pod is removed from the Service endpoints
- ✗Overlooking a Service selector that does not match the Pod labels
Follow-up questions
- →How does a readiness probe differ from a liveness probe in this failure?
- →What command compares a Service's selector against a running Pod's labels?
MiddleDebuggingCommonA node is NotReady but its Pods still show Running and are unreachable — diagnose.
A node is NotReady but its Pods still show Running and are unreachable — diagnose.
The node stopped posting status, so it went NotReady; the Pods still read Running because that record predates the failure — the kubelet is silent. Check the node: dead kubelet, disk/memory pressure, or lost network. Past the timeout its Pods reschedule elsewhere.
Common mistakes
- ✗Trusting a stale
Runningstatus on a node that stopped reporting - ✗Assuming the app is down when it is the kubelet heartbeat that failed
- ✗Not knowing a NotReady node's Pods get evicted after the timeout
Follow-up questions
- →What is the default
node.kubernetes.io/unreachabletoleration timeout before eviction? - →How would you confirm from the node whether kubelet or networking is the failure?
MiddleDebuggingCommonA container shows Reason OOMKilled and Exit Code 137 — explain it and fix it.
A container shows Reason OOMKilled and Exit Code 137 — explain it and fix it.
Exit 137 is 128 + 9 (SIGKILL), and Reason OOMKilled means the cgroup killed the container for exceeding its memory limit (128Mi). Raise the limit to the real working set, or fix the leak. A too-low CPU limit only throttles — it never kills, so no exit 137.
Common mistakes
- ✗Reading exit 137 as anything other than a SIGKILL (128 + 9)
- ✗Confusing a memory-limit OOM kill with CPU throttling
- ✗Raising the limit blindly instead of also checking for a leak
Follow-up questions
- →How do
requestsandlimitsfor memory drive a Pod's QoS class and eviction order? - →Why does a memory limit hard-kill while a CPU limit only throttles?
MiddleDebuggingOccasionalkubectl exec fails because the image has no shell — how do you get a shell to debug it?
kubectl exec fails because the image has no shell — how do you get a shell to debug it?
The image is distroless, so there is no sh to exec into. Attach an ephemeral debug container that shares the target's namespaces: kubectl debug -it api-5f6 --image=busybox --target=api. --target shares the process namespace, so you see its processes. No rebuild.
Common mistakes
- ✗Assuming a distroless Pod cannot be debugged without a rebuild
- ✗Forgetting
--targetso the debug container cannot see the app's processes - ✗Confusing
kubectl debugwith a plainexecinto the same container
Follow-up questions
- →Why is running a distroless image (no shell) a security benefit in production?
- →What does
--targetchange compared with omitting it onkubectl debug?
MiddleDebuggingOccasionalA Pod is Evicted for low ephemeral-storage on the node — find the cause and fix.
A Pod is Evicted for low ephemeral-storage on the node — find the cause and fix.
This is kubelet node-pressure eviction: the node hit DiskPressure and evicted the Pod, which used 4Gi ephemeral-storage with no request. It is node disk for logs, emptyDir, and the writable layer. Fix with ephemeral-storage requests/limits or a PersistentVolume.
Common mistakes
- ✗Reading
Evictedas an OOM kill rather than node-pressure eviction - ✗Not knowing logs,
emptyDir, and the writable layer count as ephemeral-storage - ✗Leaving ephemeral-storage requests/limits unset on a disk-heavy Pod
Follow-up questions
- →How do a Pod's QoS class and priority change which Pod the kubelet evicts first?
- →Why does a Pod with no ephemeral-storage
requestget evicted before one that set it?
SeniorDebuggingOccasionalEvery Pod is CreateContainerConfigError after a config change — find the root cause.
Every Pod is CreateContainerConfigError after a config change — find the root cause.
CreateContainerConfigError is a creation error, not a crash: the kubelet cannot build the container because a referenced ConfigMap/Secret or key is missing. Here an env var maps DB_PASSWORD from a Secret, but the change dropped that key — restore it or fix the reference.
Common mistakes
- ✗Reading
CreateContainerConfigErroras an app crash with logs to read - ✗Missing that a dropped Secret/ConfigMap key blocks container creation
- ✗Not treating it as a Waiting state like
ImagePullBackOff
Follow-up questions
- →How does marking an env
valueFromasoptional: truechange this failure? - →Why do a Secret rename and a dropped key produce the same Pod-level symptom?
SeniorDebuggingOccasionalIntermittent 5xx come only from Pods on one node — isolate node-local from app cause.
Intermittent 5xx come only from Pods on one node — isolate node-local from app cause.
Same image and config, but errors only on node-7, so isolate to the node, not the app. Group errors by spec.nodeName: if one node stands out, the app is cleared. Node-local suspects: memory/disk pressure, a bad CoreDNS or kube-proxy, or full conntrack. Cordon and drain it.
Common mistakes
- ✗Blaming the app before grouping the errors by
spec.nodeName - ✗Overlooking node memory/disk pressure driving OOM and eviction
- ✗Forgetting per-node CoreDNS, kube-proxy, and conntrack as suspects
Follow-up questions
- →Why does a full conntrack table on one node cause intermittent, not constant, 5xx?
- →What does
kubectl cordondo thatdrainadds on top of it?
SeniorDebuggingOccasionalA Deployment rollout is degraded with ProgressDeadlineExceeded — diagnose the stall.
A Deployment rollout is degraded with ProgressDeadlineExceeded — diagnose the stall.
ProgressDeadlineExceeded means the new ReplicaSet never reached enough Ready Pods within progressDeadlineSeconds; the rollout stalls, it does not auto-roll-back, and maxUnavailable keeps old Pods up. Describe a new Pod — usually readiness, ImagePull, capacity, or a PDB.
Common mistakes
- ✗Expecting
ProgressDeadlineExceededto auto-roll-back rather than stall - ✗Not describing a new Pod to find why it never goes Ready
- ✗Ignoring readiness, ImagePull, capacity, and PDB as the usual roots
Follow-up questions
- →How do
maxUnavailableandmaxSurgeshape what stays available during the stall? - →When is
kubectl rollout undothe right call versus fixing the new Pods forward?
