A node is NotReady but its Pods still show Running and are unreachable — diagnose.
A node flipped to NotReady, yet its Pods still list Running and are unreachable. Explain why the Pods still read Running, what you check on the node, and what happens to those Pods next.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node-2 NotReady <none> 40d v1.29.3
$ kubectl get pods -o wide --field-selector spec.nodeName=node-2
NAME READY STATUS RESTARTS AGE NODE
web-a1 1/1 Running 0 6h node-2
$ kubectl describe node node-2 | sed -n '/Conditions/,/Events/p'
Ready Unknown KubeletNotReady kubelet stopped posting node status
MemoryPressure UnknownDiagnose and fix.
The node stopped posting status, so it went NotReady; the Pods still read Running because that record predates the failure — the kubelet is silent. Check the node: dead kubelet, disk/memory pressure, or lost network. Past the timeout its Pods reschedule elsewhere.
- ✗Trusting a stale
Runningstatus on a node that stopped reporting - ✗Assuming the app is down when it is the kubelet heartbeat that failed
- ✗Not knowing a NotReady node's Pods get evicted after the timeout
- →What is the default
node.kubernetes.io/unreachabletoleration timeout before eviction? - →How would you confirm from the node whether kubelet or networking is the failure?
Solution
1. Why Running while NotReady
A Pod's status in etcd is updated by the node's kubelet. If the kubelet stops sending heartbeats, the node becomes NotReady after node-monitor-grace-period (~40s), and the Pods' state freezes at the last report — hence Running, even though they are unreachable.
2. What to check on the node
$ kubectl describe node node-2 | grep -A1 Ready # KubeletNotReady?
# on the node (if reachable):
$ systemctl status kubelet # is the process alive?
$ df -h ; free -m # DiskPressure / MemoryPressure
$ journalctl -u kubelet -e # why it dropped outCauses: kubelet crashed/hung, disk/memory pressure, or lost network to the API server.
3. What happens to the Pods next
After the node.kubernetes.io/unreachable toleration (default 300s), the Pods are marked for deletion and the controller creates replacements on healthy nodes. Fix by restoring kubelet/networking, or drain and replace the node.
