Fix Kubernetes Deployment Not Rolling Out

What's Actually Happening

Kubernetes deployment rollout stalls when new pods fail to start or become ready. The deployment controller cannot replace old pods with new ones, leaving the application in an inconsistent state.

The Error You'll See

Rollout status:

```bash $ kubectl rollout status deployment/my-app

Waiting for deployment "my-app" rollout to finish: 1 out of 3 new replicas have been updated... # Hangs indefinitely ```

Deployment status:

```bash $ kubectl describe deployment my-app

Replicas: 3 desired | 1 updated | 2 total | 1 available | 0 unavailable Conditions: Type Status Reason ---- ------ ------ Progressing False ProgressDeadlineExceeded Available True MinimumReplicasAvailable ```

Pod status:

```bash $ kubectl get pods -l app=my-app

NAME READY STATUS RESTARTS AGE my-app-6b7f8c9d4-abc12 0/1 ImagePullBackOff 0 5m my-app-5a6e7b8c9-xyz34 1/1 Running 0 1h ```

Why This Happens

1. 1.Image pull failure - Container image not found or access denied
2. 2.Readiness probe failing - Pod never becomes ready
3. 3.Resource constraints - Insufficient CPU/memory
4. 4.CrashLoopBackOff - Application crashing on startup
5. 5.No template change - Deployment spec unchanged
6. 6.Paused rollout - Rollout explicitly paused

Step 1: Check Deployment Status

```bash # Get deployment status kubectl get deployment my-app -o wide

# Describe deployment for details kubectl describe deployment my-app

# Check deployment conditions kubectl get deployment my-app -o jsonpath='{.status.conditions}'

# View replica sets kubectl get rs -l app=my-app

# Check which replicaset is active kubectl describe rs my-app-6b7f8c9d4

# View rollout history kubectl rollout history deployment/my-app

# Check specific revision kubectl rollout history deployment/my-app --revision=2 ```

Step 2: Check Pod Status

```bash # List all pods for deployment kubectl get pods -l app=my-app

# Check pod details kubectl describe pod my-app-6b7f8c9d4-abc12

# Check events for pod kubectl get events --field-selector involvedObject.name=my-app-6b7f8c9d4-abc12

# Check pod logs kubectl logs my-app-6b7f8c9d4-abc12

# Check previous container logs if crashed kubectl logs my-app-6b7f8c9d4-abc12 --previous

# Check container status kubectl get pod my-app-6b7f8c9d4-abc12 -o jsonpath='{.status.containerStatuses}' ```

Step 3: Fix Image Pull Issues

```bash # Check if image exists kubectl describe pod my-app-xxx | grep -A 5 "Events:"

# Common image pull errors: # ImagePullBackOff - cannot pull image # ErrImagePull - temporary pull failure

# Verify image name is correct kubectl get deployment my-app -o jsonpath='{.spec.template.spec.containers[0].image}'

# Test image pull locally docker pull my-registry/my-app:v2

# For private registries, create secret: kubectl create secret docker-registry regcred \ --docker-server=my-registry.com \ --docker-username=user \ --docker-password=pass

# Reference secret in deployment: spec: template: spec: imagePullSecrets: - name: regcred containers: - image: my-registry/my-app:v2

# Apply update kubectl apply -f deployment.yaml ```

Step 4: Fix Readiness Probe

```bash # Check readiness probe configuration kubectl get deployment my-app -o yaml | grep -A 10 readinessProbe

# If probe failing, check probe endpoint kubectl exec my-app-xxx -- curl -s localhost:8080/health

# Temporarily remove probe to debug: kubectl patch deployment my-app --type json -p='[ {"op": "remove", "path": "/spec/template/spec/containers/0/readinessProbe"} ]'

# Or increase initial delay: readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 30 # Increase periodSeconds: 10 failureThreshold: 3

# Check if app responding on probe port kubectl exec my-app-xxx -- netstat -tlnp | grep 8080 ```

Step 5: Check Resource Constraints

```bash # Check resource requests/limits kubectl get deployment my-app -o yaml | grep -A 10 resources

# Check node resources kubectl describe nodes | grep -A 5 "Allocated resources"

# Check if pods pending due to resources kubectl get pods -l app=my-app | grep Pending

# Describe pending pod kubectl describe pod my-app-xxx | grep -A 10 Events

# Reduce resource requests if too high: resources: requests: cpu: "100m" # Reduce from 500m memory: "128Mi" # Reduce from 512Mi limits: cpu: "500m" memory: "256Mi"

# Apply and check kubectl apply -f deployment.yaml kubectl get pods -w ```

Step 6: Fix CrashLoopBackOff

```bash # Check why pod is crashing kubectl logs my-app-xxx kubectl logs my-app-xxx --previous

# Common causes: # - Missing environment variables # - Missing config/secrets # - Application error on startup # - Missing dependencies

# Check pod events kubectl describe pod my-app-xxx

# Check exit code kubectl get pod my-app-xxx -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

# Run interactive shell to debug kubectl run debug --image=my-app:v2 -it --rm -- /bin/sh

# Check configmaps and secrets kubectl get configmap -l app=my-app kubectl get secret -l app=my-app

# Verify environment variables kubectl exec my-app-xxx -- env | sort ```

Step 7: Verify Template Changed

```bash # If deployment not updating, template may not have changed

# Check current template hash kubectl get deployment my-app -o jsonpath='{.spec.template.metadata.annotations}' | jq

# Force new rollout by changing annotation: kubectl patch deployment my-app -p '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'"$(date +%s)"'"}}}}}'

# Or update image (even with same tag): kubectl set image deployment/my-app app=my-app:v2 --record

# Check rollout started kubectl rollout status deployment/my-app

# View rollout history kubectl rollout history deployment/my-app ```

Step 8: Check if Rollout Paused

```bash # Check if deployment is paused kubectl get deployment my-app -o jsonpath='{.spec.paused}' # true = paused

# Resume rollout kubectl rollout resume deployment/my-app

# Pause rollout if needed kubectl rollout pause deployment/my-app

# When paused, new changes are not applied # Resume to continue

# Check deployment annotation for pause kubectl describe deployment my-app | grep -i pause ```

Step 9: Rollback Failed Deployment

```bash # If rollout stuck, rollback to previous version

# View rollout history kubectl rollout history deployment/my-app

# Rollback to previous revision kubectl rollout undo deployment/my-app

# Rollback to specific revision kubectl rollout undo deployment/my-app --to-revision=2

# Check rollback status kubectl rollout status deployment/my-app

# Verify pods running kubectl get pods -l app=my-app

# Check deployment is healthy kubectl describe deployment my-app ```

Step 10: Adjust Deployment Strategy

```yaml # Modify deployment strategy for smoother rollout

apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # Max pods over desired count maxUnavailable: 0 # Max pods that can be unavailable # maxUnavailable: 0 + maxSurge: 1 = blue-green style # maxUnavailable: 1 + maxSurge: 0 = one at a time

# Increase progress deadline if slow to start progressDeadlineSeconds: 600 # Default 600s

# Increase revision history revisionHistoryLimit: 10

# Minimum ready seconds before considered available minReadySeconds: 5

# Apply changes kubectl apply -f deployment.yaml ```

Deployment Rollout Troubleshooting Checklist

Verify the Fix

```bash # After fixing rollout issue

# 1. Check rollout status kubectl rollout status deployment/my-app # Should complete: "deployment "my-app" successfully rolled out"

# 2. Verify pods running kubectl get pods -l app=my-app # All should be Running 1/1

# 3. Check deployment conditions kubectl get deployment my-app -o jsonpath='{.status.conditions[?(@.type=="Progressing")]}' # status should be "True"

# 4. Verify new pods are from new replicaset kubectl get rs -l app=my-app # Desired replicas should match new RS

# 5. Test application kubectl port-forward deployment/my-app 8080:8080 curl localhost:8080/health

# 6. Check logs kubectl logs -l app=my-app --tail=50 ```

Related Issues

• [Fix Kubernetes Pod CrashLoopBackOff](/articles/fix-kubernetes-pod-crashloopbackoff)
• [Fix Kubernetes ImagePullBackOff](/articles/fix-kubernetes-imagepullbackoff)
• [Fix Kubernetes Pod Stuck in Pending](/articles/fix-kubernetes-pod-stuck-pending)