A Pod is stuck Pending — read the scheduler events and give the diagnosis.
A newly rolled Deployment has one Pod that never schedules. The cluster has 5 nodes: 2 general-purpose and 3 GPU nodes tainted dedicated=gpu:NoSchedule. The Pod's resources and the scheduler events:
resources:
requests:
cpu: "4"
memory: 8Gi$ kubectl describe pod web-7d9f
Status: Pending
Events:
Type Reason From Message
---- ------ ---- -------
Warning FailedScheduling default-scheduler 0/5 nodes are available:
2 Insufficient cpu, 3 node(s) had untolerated taint {dedicated: gpu}.
preemption: 0/5 nodes: 2 No preemption victims, 3 Preemption not helpful.Is this insufficient resources, an untolerated taint, an unsatisfiable affinity, or no matching node? Give the diagnosis and the fix.
No node fits, for two reasons. The 2 general nodes fail on Insufficient cpu (the Pod asks 4 CPU). The 3 GPU nodes carry an untolerated taint dedicated=gpu; preemption won't help — it frees no CPU and lifts no taint. Fix: shrink the request or add nodes; for GPU nodes add a toleration and nodeAffinity.
- ✗Blaming only CPU and ignoring the untolerated-taint half of the message
- ✗Expecting preemption to place a Pod that no node can physically fit
- ✗Thinking a toleration alone makes the Pod prefer the GPU nodes
- →What single change would let this Pod schedule on a general node?
- →Why does
Preemption not helpfulappear for the GPU nodes here?
The scheduler message holds two independent causes, not one:
Pod because it carries no matching toleration.
2 Insufficient cpu— the 2 general nodes cannot fit the requested 4 CPU.3 node(s) had untolerated taint {dedicated: gpu}— the 3 GPU nodes repel the
The Preemption not helpful line means evicting other Pods would achieve nothing: it adds no CPU to the general nodes and removes no taint from the GPU nodes.
The fix. To land on a general node, shrink requests.cpu or add capacity. To use the GPU nodes, add BOTH a toleration (to lift the bar) and a nodeAffinity (to attract the Pod there), because a toleration alone does not attract:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: accelerator
operator: In
values: ["gpu"]
