Introduction

A pod stuck in Pending state means the Kubernetes scheduler cannot find a suitable node to place it on. The pod definition is valid, but something about the cluster's current state—resource availability, scheduling constraints, or storage requirements—prevents assignment. Unlike CrashLoopBackOff or Error states, a Pending pod has not even started running yet.

The root cause is rarely the container image or application code. Instead, look at cluster capacity, node labels, taints and tolerations, affinity rules, PersistentVolumeClaim bindings, and namespace quotas.

Symptoms

  • kubectl get pods shows one or more pods with STATUS: Pending for extended periods
  • kubectl describe pod shows scheduling-related events but no container startup logs
  • New deployments scale to zero running replicas despite sufficient cluster nodes
  • Some pods schedule successfully while others from the same deployment remain pending
  • Events mention insufficient resources, node selectors, taints, or unbound PVCs

Common Causes

  • **Insufficient cluster resources**: No node has enough allocatable CPU or memory for the pod's requests
  • **Node selector mismatch**: The pod specifies a node label that no node possesses
  • **Taints without tolerations**: Nodes are tainted and the pod lacks the required toleration
  • **Affinity rules too restrictive**: Pod affinity or anti-affinity rules eliminate all candidate nodes
  • **PersistentVolumeClaim not bound**: The pod references a PVC that cannot be provisioned or bound
  • **Namespace resource quota exceeded**: The quota limit prevents new pod scheduling
  • **PriorityClass preemption blocked**: Lower-priority pods cannot be preempted to make room
  • **DaemonSet pressure**: Critical DaemonSets consume most node resources, leaving no room for new pods

Step-by-Step Fix

### 1. Check the pod events and scheduling status

bash kubectl describe pod <pod-name> -n <namespace>

Focus on the Events section at the bottom. Common messages:

| Event Message | Likely Cause | |--------------|--------------| | 0/5 nodes are available: 5 Insufficient cpu | No node has enough CPU | | 0/5 nodes are available: 5 node(s) didn't match node selector | Node label missing | | 0/5 nodes are available: 5 node(s) had taints that the pod didn't tolerate | Missing toleration | | 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims | PVC not bound | | exceeded quota: requests.cpu | Namespace quota hit |

### 2. Verify cluster resource availability

Check if nodes have enough allocatable resources:

bash kubectl top nodes kubectl describe node <node-name> | grep -A 5 "Allocated resources" kubectl describe node <node-name> | grep "Allocatable:" -A 5

Compare Allocatable vs. Requested resources. If all nodes show 90%+ utilization, the cluster is at capacity.

**Fix options:** - Scale down unnecessary workloads: kubectl scale deployment <name> --replicas=0 - Add more nodes to the cluster (autoscaler or manual) - Reduce resource requests in the pod spec if over-provisioned

### 3. Check node selector and labels

If the pod uses nodeSelector, verify matching labels exist:

```bash # Get the nodeSelector from pod spec kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeSelector}'

# List all node labels kubectl get nodes --show-labels ```

**Fix:** Add the missing label to a node:

bash kubectl label node <node-name> <label-key>=<label-value>

### 4. Check taints and tolerations

Nodes may have taints that repel pods without matching tolerations:

```bash # Check node taints kubectl describe node <node-name> | grep "Taints:"

# Check if pod has tolerations kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.tolerations}' ```

**Fix:** Add tolerations to the pod spec:

yaml spec: tolerations: - key: "node-role.kubernetes.io/master" operator: "Equal" value: "" effect: "NoSchedule"

### 5. Review affinity rules

Pod affinity/anti-affinity can inadvertently exclude all nodes:

bash kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.affinity}'

Common issues: - podAffinity requires pods that don't exist - podAntiAffinity is too strict (e.g., requiredDuringSchedulingIgnoredDuringExecution) - nodeAffinity rules conflict with available node labels

**Fix:** Relax affinity rules or use preferredDuringSchedulingIgnoredDuringExecution instead of required.

### 6. Check PersistentVolumeClaim status

If the pod references a PVC, verify it's bound:

bash kubectl get pvc -n <namespace> kubectl describe pvc <pvc-name> -n <namespace>

Status Pending means: - No StorageClass matches the PVC request - No PersistentVolume is available for static provisioning - Cloud provider quota limits volume creation - Access mode conflicts (e.g., ReadWriteOnce already mounted elsewhere)

**Fix options:** - Create a matching PV for static provisioning - Verify StorageClass exists: kubectl get sc - Check cloud provider volume quota - Change access mode or use a different PVC

### 7. Check namespace resource quota

bash kubectl describe quota -n <namespace> kubectl describe limitrange -n <namespace>

If quota is exceeded, either: - Request quota increase from cluster admin - Delete unnecessary workloads in the namespace - Reduce resource requests in pod specs

### 8. Use scheduler logs for complex cases

For persistent issues, check the scheduler itself:

bash # For managed clusters (EKS, GKE, AKS), check cloud console # For self-hosted clusters: kubectl logs -n kube-system -l component=kube-scheduler --tail=100

### 9. Enable scheduler debugging with kubectl

Use kubectl debug for advanced scheduling analysis:

bash kubectl debug -it <pod-name> --image=busybox --target=<container-name>

Or simulate scheduling with kube-scheduler dry-run (Kubernetes 1.23+):

bash kubectl get pod <pod-name> -o yaml | \ kubectl replace --dry-run=server -f -

Prevention Checklist

  • [ ] Set reasonable resource requests (not just limits)
  • [ ] Use kubectl top nodes regularly to monitor cluster capacity
  • [ ] Document required node labels in deployment manifests
  • [ ] Test taints/tolerations in staging before production
  • [ ] Use preferredDuringScheduling instead of required when possible
  • [ ] Monitor PVC provisioning latency and failures
  • [ ] Set up alerts for pending pods older than 5 minutes
  • [ ] Review namespace quotas during capacity planning
  • [Fix Kubernetes Pod CrashLoopBackOff](/articles/fix-kubernetes-pod-crashloopbackoff)
  • [Fix Kubernetes ImagePullBackOff](/articles/fix-kubernetes-imagepullbackoff)
  • [Fix Kubernetes PersistentVolumeClaim Pending](/articles/fix-kubernetes-pvc-pending)
  • [Fix Kubernetes Node NotReady](/articles/fix-kubernetes-node-notready)