A Pod is Evicted for low ephemeral-storage on the node — find the cause and fix.
A Pod shows Evicted. The message blames node ephemeral-storage. Explain what ephemeral-storage is, why the kubelet evicted this Pod, and the durable fix.
$ kubectl get pod log-shipper-3
NAME READY STATUS RESTARTS AGE
log-shipper-3 0/1 Evicted 0 12m
$ kubectl describe pod log-shipper-3
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage.
Container app was using 4Gi, request is 0.
$ kubectl describe node node-5 | grep Pressure
DiskPressure TrueDiagnose and fix.
This is kubelet node-pressure eviction: the node hit DiskPressure and evicted the Pod, which used 4Gi ephemeral-storage with no request. It is node disk for logs, emptyDir, and the writable layer. Fix with ephemeral-storage requests/limits or a PersistentVolume.
- ✗Reading
Evictedas an OOM kill rather than node-pressure eviction - ✗Not knowing logs,
emptyDir, and the writable layer count as ephemeral-storage - ✗Leaving ephemeral-storage requests/limits unset on a disk-heavy Pod
- →How do a Pod's QoS class and priority change which Pod the kubelet evicts first?
- →Why does a Pod with no ephemeral-storage
requestget evicted before one that set it?
Solution
1. What happened
Message: The node was low on resource: ephemeral-storage. Container app using 4Gi, request 0.
DiskPressure TrueThis is kubelet node-pressure eviction: the node crossed the DiskPressure threshold, so the kubelet started evicting Pods to reclaim disk. Pods over their request go first (here request 0 against 4Gi used).
2. What ephemeral-storage is
The node's local disk shared by: container logs (stdout), emptyDir volumes, and the container writable layer. Not a PersistentVolume — it is temporary node space.
3. Durable fix
predictable eviction order;
- set
resources.requests/limitsforephemeral-storageso the Pod is bounded and gets a - rotate / cap logs;
- write bulk data to a PersistentVolume, not the node disk.
