Introduction

ImagePullBackOff means Kubernetes cannot pull the container image specified in the pod spec. The kubelet tries to pull the image, fails, and then backs off exponentially before retrying. This is different from CrashLoopBackOff (container crashes after starting) and Pending (pod cannot be scheduled). With ImagePullBackOff, the pod never starts because the image itself cannot be retrieved.

Common causes include typos in image names, missing registry credentials for private images, network connectivity issues, or rate limiting from container registries.

Symptoms

  • kubectl get pods shows STATUS: ImagePullBackOff or ErrImagePull
  • kubectl describe pod shows Failed to pull image errors with specific reasons
  • Events mention 401 Unauthorized, 404 Not Found, 403 Forbidden, or timeout errors
  • Pod remains in ContainerCreating state before transitioning to ImagePullBackOff
  • Other pods on different nodes may pull the same image successfully (network or node-specific issue)

Common Causes

  • **Image name typo**: Repository name, tag, or digest is incorrect
  • **Private image without credentials**: Image requires authentication but no imagePullSecrets specified
  • **Registry rate limiting**: Anonymous pulls exceed Docker Hub or registry rate limits
  • **Network connectivity**: Node cannot reach the registry due to firewall, proxy, or DNS issues
  • **Image does not exist**: Tag was deleted, repository is private, or image was never pushed
  • **Registry certificate issues**: Self-signed or expired SSL certificates on private registries
  • **Service account missing pull secret**: Default service account lacks imagePullSecrets reference

Step-by-Step Fix

### 1. Check the exact error message

bash kubectl describe pod <pod-name> -n <namespace>

Look at the Events section for the specific pull failure reason:

| Error Message | Cause | |--------------|-------| | pull access denied, repository does not exist or may require authorization | 401/403 - Auth or missing repo | | manifest for <image>:<tag> not found | 404 - Image/tag doesn't exist | | dial tcp: lookup registry.example.com: no such host | DNS resolution failure | | timeout waiting for response from registry | Network connectivity issue | | certificate signed by unknown authority | Self-signed or invalid SSL cert |

### 2. Verify image name and tag

Check the pod spec for the exact image reference:

bash kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

Common issues: - Typo in repository name: gcr.io/my-projct/app (missing 'e') - Tag doesn't exist: nginx:1.21.999 (never released) - Missing tag defaults to latest, which may not exist

**Fix:** Verify the image exists by pulling it locally:

bash docker pull <image-name> # Or with crane for faster verification crane pull <image-name>

### 3. Check imagePullSecrets for private registries

Private images require authentication credentials:

```bash # Check if pod has imagePullSecrets kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

# List available secrets in namespace kubectl get secret -n <namespace> | grep docker-registry

# Verify secret contains valid credentials kubectl get secret <secret-name> -n <namespace> -o jsonpath='{.data}' | base64 -d ```

**Fix:** Create or update the pull secret:

```bash # Create new secret kubectl create secret docker-registry regcred \ --docker-server=<registry-url> \ --docker-username=<username> \ --docker-password=<password> \ --docker-email=<email> \ -n <namespace>

# Add secret to service account kubectl patch serviceaccount default \ -p '{"imagePullSecrets": [{"name": "regcred"}]}' \ -n <namespace> ```

Or add directly to pod spec:

yaml spec: imagePullSecrets: - name: regcred containers: - name: app image: registry.example.com/private/app:v1.0

### 4. Check for rate limiting (Docker Hub)

Docker Hub limits anonymous pulls: - Anonymous: 100 pulls per 6 hours - Authenticated free: 200 pulls per 6 hours - Pro: Unlimited

If rate limited, you'll see 429 Too Many Requests errors.

**Fix:** - Create a Docker Hub secret and use imagePullSecrets - Mirror images to a different registry (GHCR, ECR, GCR) - Use a pull-through cache or registry mirror

### 5. Test node connectivity to registry

SSH into a node and test registry access:

```bash # Test DNS resolution nslookup registry.example.com

# Test HTTPS connectivity curl -I https://registry.example.com/v2/

# Test authentication docker login registry.example.com docker pull registry.example.com/image:tag ```

Common node issues: - Firewall blocks outbound HTTPS (port 443) - Corporate proxy required but not configured - DNS server unreachable from node network

**Fix for proxy environments:**

Configure kubelet with proxy settings:

bash # /etc/systemd/system/kubelet.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://proxy.example.com:8080" Environment="HTTPS_PROXY=http://proxy.example.com:8080" Environment="NO_PROXY=localhost,127.0.0.1,.cluster.local"

### 6. Check certificate issues (private registries)

For self-signed certificates, nodes need to trust the CA:

bash # On each node, copy CA cert to docker certs directory sudo mkdir -p /etc/docker/certs.d/registry.example.com/ sudo cp ca.crt /etc/docker/certs.d/registry.example.com/ sudo systemctl restart containerd # Or for Docker sudo systemctl restart docker

For Kubernetes, configure insecure registries (not recommended for production):

bash # /etc/containerd/config.toml [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.example.com"] endpoint = ["http://registry.example.com"]

### 7. Check if image was deleted or moved

Images can disappear if: - Repository was made private after pod spec was written - Tag was deleted to free space - Image was moved to a different repository

**Fix:** Verify image still exists:

```bash # Check Docker Hub curl -s https://hub.docker.com/v2/repositories/<user>/<repo>/tags/<tag>

# Check GCR gcloud container images describe gcr.io/<project>/<image>:<tag>

# Check ECR aws ecr describe-images --repository-name <repo> --image-ids imageTag=<tag> ```

### 8. Check ServiceAccount imagePullSecrets

Even if pod spec has imagePullSecrets, the ServiceAccount might override:

bash kubectl get serviceaccount default -n <namespace> -o yaml

If imagePullSecrets is missing but required, patch it:

bash kubectl patch serviceaccount default \ -n <namespace> \ -p '{"imagePullSecrets": [{"name": "regcred"}]}'

### 9. Verify container runtime is working

Sometimes the container runtime itself is broken:

```bash # Check containerd status systemctl status containerd crictl info

# Check Docker status (if using dockershim) systemctl status docker

# Check kubelet logs journalctl -u kubelet | grep -i "imagepull" ```

Quick Debugging Commands

```bash # 1. Get pod status and events kubectl describe pod <pod-name> -n <namespace>

# 2. Check image reference kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

# 3. Check imagePullSecrets kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

# 4. List available secrets kubectl get secret -n <namespace>

# 5. Test pull locally docker pull <image-name>

# 6. Check node conditions kubectl describe node <node-name> | grep -A 5 "Conditions" ```

Prevention Checklist

  • [ ] Use specific image tags instead of latest
  • [ ] Store images in a reliable registry with high availability
  • [ ] Configure imagePullSecrets at ServiceAccount level for consistency
  • [ ] Use private registry mirrors to avoid rate limiting
  • [ ] Set up image pull health monitoring and alerts
  • [ ] Document required secrets for new deployments
  • [ ] Use image digest (sha256:...) for immutable deployments
  • [ ] Test image pulls from clean nodes before production deployment
  • [Fix Kubernetes Pod Stuck in Pending](/articles/fix-kubernetes-pod-stuck-pending)
  • [Fix Kubernetes Pod CrashLoopBackOff](/articles/fix-kubernetes-pod-crashloopbackoff)
  • [Fix Kubernetes Container Creating Forever](/articles/fix-kubernetes-container-creating-forever)
  • [Fix Kubernetes Node NotReady](/articles/fix-kubernetes-node-notready)