What's Actually Happening

ImagePullBackOff occurs when Kubernetes cannot download the container image from the registry. After multiple pull failures, Kubernetes backs off (increases retry delay) but continues trying. The pod stays in pending state with this error.

The Error You'll See

```bash $ kubectl get pods NAME READY STATUS RESTARTS AGE my-app-abc123d-x1y2z 0/1 ImagePullBackOff 0 5m

$ kubectl describe pod my-app-abc123d-x1y2z ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 5m kubelet Pulling image "my-registry/app:v1" Warning Failed 5m kubelet Failed to pull image "my-registry/app:v1": rpc error: code = Unknown desc = Error response from daemon: pull access denied Warning BackOff 4m kubelet Back-off pulling image "my-registry/app:v1" ```

Why This Happens

  1. 1.Image doesn't exist - Wrong image name or tag
  2. 2.Private registry without credentials - Missing or wrong secret
  3. 3.Registry unreachable - Network issues, wrong URL
  4. 4.Rate limits - Docker Hub pull limits exceeded
  5. 5.Image too large - Pull times out
  6. 6.Wrong registry URL - Typos in image name

Step 1: Check the Image Name

bash
kubectl get pod my-app-abc123d-x1y2z -o yaml | grep image:

Verify the image name is correct:

yaml
image: my-registry.io/my-app:v1.0  # Check this matches registry

Test image name locally:

bash
docker pull my-registry.io/my-app:v1.0

If this fails locally, image name is wrong.

Step 2: Verify Image Exists in Registry

For Docker Hub:

```bash # Check image exists curl -s "https://hub.docker.com/v2/repositories/library/nginx/" | grep name

# Or search docker search nginx ```

For private registry:

```bash # List images in registry curl -s -u user:pass https://my-registry.io/v2/_catalog

# Check specific image tags curl -s -u user:pass https://my-registry.io/v2/my-app/tags/list ```

Step 3: Check Registry Credentials

For private registries, Kubernetes needs a secret:

```bash # Check if secret exists kubectl get secrets

# Check secret is referenced in deployment kubectl get deployment my-app -o yaml | grep -A5 "imagePullSecrets" ```

Should show:

yaml
imagePullSecrets:
- name: registry-credentials

Create secret if missing:

bash
kubectl create secret docker-registry registry-credentials \
  --docker-server=my-registry.io \
  --docker-username=myuser \
  --docker-password=mypass \
  --docker-email=my@email.com

Add to deployment:

yaml
spec:
  template:
    spec:
      imagePullSecrets:
      - name: registry-credentials

Step 4: Check Docker Hub Rate Limits

Docker Hub has pull limits (100 pulls/6 hours for free):

bash
kubectl describe pod my-app-abc123d-x1y2z | grep -i "rate limit"

If rate limited:

bash
Warning  Failed  ...  toomanyrequests: You have reached your pull rate limit

Solutions: - Use authenticated Docker Hub account (higher limits) - Mirror images to private registry - Use alternative registry (quay.io, ghcr.io)

Create Docker Hub secret:

bash
kubectl create secret docker-registry docker-hub-secret \
  --docker-server=docker.io \
  --docker-username=dockerhubuser \
  --docker-password=dockerhubtoken

Step 5: Check Network Connectivity

```bash # From a node, test registry access curl -I https://my-registry.io/v2/

# Check DNS resolves nslookup my-registry.io

# Test with specific node kubectl debug node/node-name --image=busybox -- curl -I https://my-registry.io/v2/ ```

Step 6: Check Image Tag Issues

If using latest tag:

yaml
image: my-app:latest  # Problematic - tag might not update

Use specific tags:

yaml
image: my-app:v1.2.3  # Specific, reproducible

If tag doesn't exist:

bash
# List available tags
curl -s https://my-registry.io/v2/my-app/tags/list

Step 7: Check Node Image Cache

Sometimes nodes have cached corrupt images:

bash
# SSH to node and clean image
ssh node-1
docker rmi my-registry.io/my-app:v1
docker pull my-registry.io/my-app:v1

Step 8: Verify Secret Namespace

Secret must be in same namespace as pod:

```bash kubectl get secrets -n my-namespace

# Create secret in correct namespace kubectl create secret docker-registry registry-credentials \ --namespace=my-namespace \ --docker-server=my-registry.io \ ... ```

Step 9: Check for Proxy Configuration

If cluster uses HTTP proxy:

bash
# Check node proxy config
kubectl debug node/node-name --image=busybox -- env | grep -i proxy

Add NO_PROXY for registry if needed.

Verify the Fix

After fixing:

```bash kubectl rollout restart deployment/my-app kubectl get pods -w

# Should see: # Pending -> ContainerCreating -> Running # No ImagePullBackOff ```

Prevention Tips

When deploying images:

```bash # Use specific tags, not latest image: my-app:v1.2.3

# Create registry secret before deployment kubectl create secret docker-registry registry-credentials ...

# Test image pull locally first docker pull my-registry.io/my-app:v1

# Use imagePullPolicy wisely imagePullPolicy: IfNotPresent # Use cached if available ```