Fix Kubernetes PriorityClass Issues

Your pods aren't being scheduled correctly despite having PriorityClass set, or critical pods are being preempted unexpectedly. PriorityClass controls pod scheduling order and preemption behavior, but misconfiguration can cause pods to not get the priority you expect or critical pods to be evicted.

Understanding PriorityClass

PriorityClass defines the relative importance of pods. Higher priority pods are scheduled before lower priority pods and can preempt (evict) lower priority pods when resources are scarce. Every pod has a priority - either explicitly set via priorityClassName or using the default priority class.

PriorityClass values range from -2147483648 to 1000000000, with higher values meaning higher priority. Values above 1000000000 are reserved for system critical pods.

Diagnosis Commands

Check PriorityClass configuration:

```bash # List PriorityClasses kubectl get priorityclasses kubectl get pc # Short name

# Describe specific PriorityClass kubectl describe priorityclass priorityclass-name

# Check default PriorityClass kubectl get priorityclasses -o jsonpath='{.items[?(@.spec.globalDefault==true)].metadata.name}' ```

Check pod priority:

```bash # Check pod priorityClassName kubectl get pod pod-name -n namespace -o jsonpath='{.spec.priorityClassName}'

# Check actual priority value kubectl get pod pod-name -n namespace -o jsonpath='{.spec.priority}'

# List pods with priorities kubectl get pods -n namespace -o custom-columns='NAME:.metadata.name,PRIORITY:.spec.priority,PRIORITY-CLASS:.spec.priorityClassName' ```

Check scheduling/preemption events:

```bash # Check events for scheduling issues kubectl get events -n namespace --sort-by='.lastTimestamp' | grep -i "priority|preemption"

# Look for preemption events kubectl get events -A | grep -i Preempt ```

Common Solutions

Solution 1: Create Missing PriorityClass

Pod can't use PriorityClass that doesn't exist:

```bash # Check if PriorityClass exists kubectl get priorityclass high-priority

# If missing, create it kubectl apply -f priorityclass.yaml ```

Create PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority for critical workloads"
preemptionPolicy: PreemptLowerPriority  # Default

Create system critical PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: system-critical
value: 1000000001  # Above system threshold
globalDefault: false
description: "System critical pods - highest priority"
preemptionPolicy: PreemptLowerPriority

Solution 2: Fix PriorityClassName Reference

Pod might reference wrong PriorityClass:

```bash # Check pod's PriorityClass reference kubectl get pod pod-name -n namespace -o yaml | grep priorityClassName

# Check available PriorityClasses kubectl get priorityclasses ```

Fix reference:

```yaml # Pod with incorrect PriorityClass name spec: priorityClassName: wrong-priority-class # Doesn't exist

# Fix: Use correct name spec: priorityClassName: high-priority ```

Solution 3: Fix GlobalDefault Setting

Only one PriorityClass can have globalDefault=true:

```bash # Check current global default kubectl get priorityclasses -o custom-columns='NAME:.metadata.name,GLOBAL-DEFAULT:.spec.globalDefault'

# If multiple have globalDefault=true, fix this ```

Set single global default:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: default-priority
value: 0
globalDefault: true  # Only one can be true
description: "Default priority for pods without priorityClassName"

Remove globalDefault from others:

# Patch to remove globalDefault
kubectl patch priorityclass other-pc -p '{"spec":{"globalDefault":false}}'

Solution 4: Fix PreemptionPolicy

PreemptionPolicy controls how pods preempt others:

```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000000 preemptionPolicy: PreemptLowerPriority # Can preempt lower priority pods

# Other options: # Never - Pod won preempt others even with higher priority ```

Use Never for pods that shouldn't preempt:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: non-preempting-high
value: 1000000
preemptionPolicy: Never  # High priority but won't evict other pods
description: "High priority without preemption"

Solution 5: Fix Priority Value Order

Priority values must be in correct order:

# Check PriorityClass values
kubectl get priorityclasses -o custom-columns='NAME:.metadata.name,VALUE:.spec.value' | sort -k2 -n

Typical hierarchy:

# Low priority (batch jobs)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 100
---
# Normal priority
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: normal-priority
value: 1000
---
# High priority (critical services)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 100000
---
# System critical (system components)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: system-critical
value: 1000000001

Solution 6: Check Pod Preemption

Higher priority pods preempt lower priority pods:

```bash # Check preemption events kubectl get events -n namespace | grep -i preempt

# Check which pods were preempted kubectl describe pod preempted-pod -n namespace | grep -A 5 "Preempt" ```

Preemption information in events:

```bash # Look for preemption reason kubectl describe pod pod-name -n namespace | grep -B 5 -A 5 "PreemptBy"

# Check scheduler logs for preemption decisions kubectl logs -n kube-system kube-scheduler-master | grep -i preempt ```

Solution 7: Prevent Critical Pod Preemption

Critical pods shouldn't be preempted unexpectedly:

# Ensure critical pods use high PriorityClass
apiVersion: v1
kind: Pod
metadata:
  name: critical-pod
spec:
  priorityClassName: system-critical  # Very high priority
  # This pod won't be preempted by normal workloads

Create non-preemptible PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-non-preemptible
value: 1000000001
preemptionPolicy: Never
# Won't preempt others, but won't be preempted either

Solution 8: Fix Scheduling Issues Despite Priority

High priority pods might still not schedule:

```bash # Check pod status kubectl describe pod pending-pod -n namespace

# Even with high priority, pods need resources, nodes, etc. # Priority only affects order, not capacity ```

Check scheduling constraints:

# Check if pod has other constraints blocking scheduling
kubectl describe pod pod-name -n namespace | grep -A 20 "Events:"
kubectl describe pod pod-name -n namespace | grep -A 5 "nodeSelector\|affinity\|tolerations"

Solution 9: Check Priority Admission Controller

Priority admission controller handles pod priorities:

```bash # Check admission controllers kubectl get pods -n kube-system -l component=kube-apiserver

# Check API server logs kubectl logs -n kube-system kube-apiserver-master | grep -i priority ```

Verify Priority admission controller is enabled:

# kube-apiserver configuration
--enable-admission-plugins=Priority,...

Solution 10: Check Pod Disruption Budget Interaction

PDBs can prevent preemption:

```bash # Check PDB settings kubectl get pdb -n namespace

# Even high priority pods may not preempt if PDB blocks it ```

Verification

After fixing PriorityClass issues:

```bash # Verify PriorityClass exists kubectl get priorityclass priorityclass-name

# Check pod has correct priority kubectl get pod pod-name -n namespace -o jsonpath='{.spec.priority}' kubectl get pod pod-name -n namespace -o jsonpath='{.spec.priorityClassName}'

# Verify pod scheduling kubectl get pods -n namespace -o wide

# Check no unexpected preemption kubectl get events -n namespace | grep -i preempt ```

PriorityClass Quick Reference

PriorityClass Issues Summary

Prevention Best Practices

Create PriorityClasses for different workload tiers before deploying. Use consistent naming (low, normal, high, critical). Set only one globalDefault. Document priority hierarchy for your cluster. Use preemptionPolicy: Never for pods that shouldn't evict others. Monitor preemption events for unexpected behavior. Combine PriorityClass with resource management for true criticality.

PriorityClass issues typically come down to missing classes, wrong references, or misunderstanding that priority only affects scheduling order - it doesn't create resources or bypass other scheduling constraints.