Intermittent 5xx come only from Pods on one node — isolate node-local from app cause.
Users see intermittent 5xx. The error rate, grouped by the Pod's node, is near zero everywhere except one node. The Pods run the same image and config. Isolate whether this is a node-local problem or an application bug, and act.
# 5xx rate by node (from request logs, joined to spec.nodeName)
api-1 1/1 Running node-1 0.0% 5xx
api-2 1/1 Running node-3 0.1% 5xx
api-9 1/1 Running node-7 9.4% 5xx <-- errors concentrate here
$ kubectl describe node node-7 | grep -E 'Pressure|Ready'
MemoryPressure True
Ready TrueDiagnose and fix.
Same image and config, but errors only on node-7, so isolate to the node, not the app. Group errors by spec.nodeName: if one node stands out, the app is cleared. Node-local suspects: memory/disk pressure, a bad CoreDNS or kube-proxy, or full conntrack. Cordon and drain it.
- ✗Blaming the app before grouping the errors by
spec.nodeName - ✗Overlooking node memory/disk pressure driving OOM and eviction
- ✗Forgetting per-node CoreDNS, kube-proxy, and conntrack as suspects
- →Why does a full conntrack table on one node cause intermittent, not constant, 5xx?
- →What does
kubectl cordondo thatdrainadds on top of it?
Solution
1. Isolation: node or app
The same image/config on every Pod, yet 5xx concentrate on node-7. The logic: if the app were at fault, errors would spread across nodes. So it is a node-local cause. Grouping errors by spec.nodeName is the key step.
2. Node-local suspects
$ kubectl describe node node-7 | grep Pressure # MemoryPressure True
$ kubectl get pods -n kube-system -o wide | grep node-7 # coredns/kube-proxy healthy here?
# on the node: conntrack -S | grep drop ; dmesg | grep -i 'oom\|conntrack'- memory/disk pressure (as here) → OOM/eviction of Pods;
- CoreDNS/kube-proxy on the node failing → timeouts/resolution errors → 5xx;
- conntrack table full → a fraction of new connections dropped (hence intermittent).
3. Action
kubectl cordon node-7 (block new Pods) → kubectl drain node-7 (evacuate existing). Traffic moves to healthy nodes and the bleeding stops; then repair (memory, restart CoreDNS/kube-proxy, conntrack limits) or replace the node.
