Your pods aren't being scheduled on the nodes you expect, and node affinity might be the cause. Node affinity constrains which nodes pods can run on based on node labels. It's more flexible than nodeSelector but can still cause scheduling issues when labels don't match or rules are too restrictive.
Understanding Node Affinity
Node affinity allows you to specify rules for which nodes a pod can schedule on, using node labels. There are two types: required (hard) and preferred (soft). Required affinity must be satisfied for scheduling; preferred affinity is a hint the scheduler tries to satisfy.
Node affinity uses matchExpressions with operators like In, NotIn, Exists, DoesNotExist, Gt, Lt to create flexible matching rules.
Diagnosis Commands
Check pod node affinity:
```bash # Check pod node affinity rules kubectl get pod pod-name -n namespace -o yaml | grep -A 30 affinity
# Get specific affinity rules kubectl get pod pod-name -n namespace -o jsonpath='{.spec.affinity.nodeAffinity}' ```
Check node labels:
```bash # Check all node labels kubectl get nodes --show-labels
# Check specific node labels kubectl describe node node-name | grep -A 20 Labels
# Get specific label values kubectl get nodes -o custom-columns='NAME:.metadata.name,LABEL-VALUE:.metadata.labels.label-key' ```
Check scheduling events:
```bash # Describe pending pod for affinity errors kubectl describe pod pending-pod -n namespace | grep -A 10 "Events:" kubectl get events -n namespace | grep -i "affinity|node(s) didn't match"
# Check scheduler logs kubectl logs -n kube-system kube-scheduler-master | grep -i affinity ```
Common Solutions
Solution 1: Fix Node Label Mismatch
Pod affinity expects labels that don't exist:
```bash # Check what labels pod expects kubectl get pod pod-name -n namespace -o yaml | grep -A 10 matchExpressions
# Check actual node labels kubectl get nodes --show-labels ```
Fix by adding labels to nodes:
```bash # Add missing label to node kubectl label node node-name key=value
# Example: Add zone label kubectl label node node-1 zone=us-west-1a
# Verify label added kubectl describe node node-1 | grep zone ```
Fix by updating affinity rules:
```yaml # Pod expecting wrong label affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-east-1a # Wrong zone
# Fix: Update to match actual labels affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - us-west-1a # Correct zone ```
Solution 2: Fix Required Affinity Too Strict
Required affinity that no node satisfies blocks scheduling:
# Overly strict required affinity
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: zone
operator: In
values:
- zone-x # No nodes have this zone
- key: disktype
operator: In
values:
- ssd-nvme # AND must have this too - very strictRelax required affinity:
```yaml # Use OR logic with multiple nodeSelectorTerms affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: # Term 1 - zone A - key: zone operator: In values: - us-west-1a - matchExpressions: # Term 2 - zone B (OR) - key: zone operator: In values: - us-west-1b
# Or switch to preferred affinity affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: disktype operator: In values: - ssd ```
Solution 3: Fix Operator Issues
Different operators have different behaviors:
```yaml # In operator - value must be in the list matchExpressions: - key: zone operator: In values: - us-west-1a - us-west-1b
# NotIn operator - value must NOT be in the list matchExpressions: - key: zone operator: NotIn values: - us-east-1a # Don't schedule in east
# Exists operator - label must exist (any value) matchExpressions: - key: disktype operator: Exists # Node must have disktype label
# DoesNotExist operator - label must not exist matchExpressions: - key: temporary operator: DoesNotExist
# Gt/Lt operators - numeric comparison matchExpressions: - key: memory-size operator: Gt values: - "32" # memory-size > 32 ```
Solution 4: Fix Multiple Selector Terms
Multiple nodeSelectorTerms are OR-ed together:
```yaml # Each nodeSelectorTerm is an OR condition affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: # Term 1: zone=a AND disk=ssd - key: zone operator: In values: - zone-a - key: disktype operator: In values: - ssd - matchExpressions: # Term 2: zone=b (OR with Term 1) - key: zone operator: In values: - zone-b
# This means: (zone=a AND disk=ssd) OR zone=b ```
Within a term, matchExpressions are AND-ed:
# Single term with multiple expressions: all must match (AND)
nodeSelectorTerms:
- matchExpressions:
- key: zone
operator: In
values:
- zone-a
- key: disktype
operator: In
values:
- ssd
# Both must match: zone=a AND disktype=ssdSolution 5: Add Preferred Affinity
Preferred affinity gives flexibility:
affinity:
nodeAffinity:
# Required: must have zone label
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: zone
operator: Exists
# Preferred: prefer SSD nodes
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80 # Higher weight = more preference
preference:
matchExpressions:
- key: disktype
operator: In
values:
- ssd
- weight: 20 # Lower weight preference
preference:
matchExpressions:
- key: zone
operator: In
values:
- us-west-1aSolution 6: Fix Anti-Affinity Conflicts
Pod anti-affinity might conflict with node affinity:
```yaml # Pod wants specific zone but anti-affinity prevents it affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: zone operator: In values: - zone-a # Only one node in zone-a podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: myapp topologyKey: kubernetes.io/hostname # Can't be on same node
# Problem: 2 replicas, 1 node in zone-a -> can't schedule both ```
Fix by relaxing constraints:
# Relax anti-affinity
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: myapp
topologyKey: kubernetes.io/hostname
# Or add more nodes to zone-aSolution 7: Remove Node Affinity
Remove affinity to allow scheduling anywhere:
```yaml # Pod with restrictive affinity spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: special-hardware operator: Exists
# Remove or disable affinity spec: # Remove affinity block entirely, or: affinity: nodeAffinity: null ```
Solution 8: Check TopologyKey Issues
For pod affinity/anti-affinity, topologyKey must be valid:
```yaml # Valid topology keys topologyKey: kubernetes.io/hostname # Node level topologyKey: topology.kubernetes.io/zone # Zone level topologyKey: topology.kubernetes.io/region # Region level
# Custom topology key (requires consistent node labeling) topologyKey: custom-zone ```
Ensure nodes have the topology label:
```bash # Check if nodes have topology labels kubectl get nodes -o custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'
# Add missing topology labels kubectl label node node-name topology.kubernetes.io/zone=zone-a ```
Solution 9: Combine with Tolerations
Node affinity and taints work together:
```yaml # Node affinity selects nodes by labels # Taints repel pods without tolerations # Both must be satisfied
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: dedicated operator: In values: - gpu tolerations: - key: "dedicated" operator: "Equal" value: "gpu" effect: "NoSchedule" ```
Solution 10: Debug with Describe
Use describe to see scheduling constraints:
```bash # Full pod description kubectl describe pod pod-name -n namespace
# Check events section kubectl describe pod pod-name -n namespace | tail -20
# Look for affinity-related messages: # "0/3 nodes available: 3 node(s) didn't match node affinity" ```
Verification
After fixing node affinity issues:
```bash # Verify node labels kubectl get nodes --show-labels | grep key
# Verify pod affinity rules kubectl get pod pod-name -n namespace -o yaml | grep -A 30 affinity
# Check pod scheduled on expected node kubectl get pod pod-name -n namespace -o wide
# Verify pod is running kubectl get pods -n namespace ```
Node Affinity Operators Reference
| Operator | Behavior | Example |
|---|---|---|
| In | Label value in values list | zone In [a,b] |
| NotIn | Label value NOT in list | zone NotIn [c] |
| Exists | Label exists (any value) | disktype Exists |
| DoesNotExist | Label doesn't exist | temp DoesNotExist |
| Gt | Label value > number | memory Gt 32 |
| Lt | Label value < number | memory Lt 64 |
Node Affinity Issues Summary
| Issue | Check | Solution |
|---|---|---|
| No matching nodes | kubectl get nodes --show-labels | Add labels or fix affinity |
| Too strict required | kubectl describe pod | Relax to preferred or add terms |
| Wrong operator | kubectl get pod -o yaml | Use correct operator |
| AND vs OR confusion | Multiple terms vs expressions | Review logic structure |
| Missing topology labels | kubectl describe nodes | Add topology labels |
| Anti-affinity conflict | kubectl describe pod | Relax one constraint |
| Taint blocking | kubectl describe node | Add tolerations |
Prevention Best Practices
Label nodes consistently with meaningful labels (zone, region, disktype, etc.). Use preferred affinity when possible for flexibility. Test affinity rules before production. Monitor pending pods for affinity issues. Document affinity rules and their purpose. Keep required affinity minimal. Combine with tolerations for dedicated nodes.
Node affinity issues usually come down to labels not matching what the pod expects. The kubectl describe pod events tell you exactly why scheduling failed, and kubectl get nodes --show-labels shows what labels are actually available.