Your pod won't start. The status shows ImagePullBackOff and your application can't run. Kubernetes can't download the container image it needs.

ImagePullBackOff means Kubernetes tried to pull the container image, failed, and is now backing off before retrying. Let's diagnose why the image pull is failing.

Understanding ImagePullBackOff

When you create a pod, Kubernetes needs to download the container image from a registry. If the pull fails (wrong name, authentication issues, network problems), Kubernetes retries with exponential backoff. After multiple failures, the pod shows ImagePullBackOff.

Step 1: Check Pod Status

Identify pods with image pull issues:

bash
kubectl get pods -n <namespace>
bash
NAME                        READY   STATUS             RESTARTS   AGE
my-app-6b7c8d9f-x1k2m       0/1     ImagePullBackOff   0          5m

Step 2: Get Detailed Error Information

Describe the pod to see the specific error:

bash
kubectl describe pod my-app-6b7c8d9f-x1k2m -n <namespace>

Look at the Events section:

bash
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  5m                 default-scheduler  Successfully assigned default/my-app to node-1
  Normal   Pulling    5m                 kubelet            Pulling image "myregistry.io/my-app:v1"
  Warning  Failed     4m                 kubelet            Failed to pull image "myregistry.io/my-app:v1": rpc error: code = Unknown desc = Error response from daemon: pull access denied
  Warning  Failed     4m                 kubelet            Error: ErrImagePull
  Normal   BackOff    3m (x6 over 4m)    kubelet            Back-off pulling image "myregistry.io/my-app:v1"
  Warning  Failed     3m (x6 over 4m)     kubelet            Error: ImagePullBackOff

The error message reveals the root cause. Common messages: - pull access denied - Authentication issue - manifest unknown - Image or tag doesn't exist - connection refused - Registry unreachable - name unknown - Repository doesn't exist

Step 3: Verify Image Name and Tag

Check the exact image name in your deployment:

bash
kubectl get pod my-app-6b7c8d9f-x1k2m -n <namespace> -o jsonpath='{.spec.containers[*].image}'
bash
myregistry.io/my-app:v1

Common mistakes: - Typo in image name - Wrong tag name - Missing tag (defaults to :latest) - Wrong registry URL

Verify the image exists in your registry. For Docker Hub:

bash
docker search my-app

For private registries, check via registry UI or API:

bash
curl -s -u user:password https://myregistry.io/v2/my-app/tags/list | jq .

Check for typos in the tag:

```yaml # Wrong tag image: my-app:v1.0 # Should be v1.0.0

# Missing tag - uses latest image: my-app # Same as my-app:latest ```

Step 4: Check Image Pull Secrets

For private registries, Kubernetes needs credentials. Check if image pull secrets are configured:

bash
kubectl get pod my-app-6b7c8d9f-x1k2m -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

If empty, no secrets are configured. Create a secret for your registry:

bash
kubectl create secret docker-registry my-registry-secret \
  --docker-server=myregistry.io \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=myemail@example.com \
  -n <namespace>

Then add it to your deployment:

yaml
spec:
  imagePullSecrets:
    - name: my-registry-secret
  containers:
    - name: my-app
      image: myregistry.io/my-app:v1

Or patch an existing deployment:

bash
kubectl patch serviceaccount default -n <namespace> -p '{"imagePullSecrets": [{"name": "my-registry-secret"}]}'

Step 5: Verify Secret Content

If a secret exists but pulling still fails, verify the secret is valid:

bash
kubectl get secret my-registry-secret -n <namespace> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

Check that: - The registry URL is correct - Username and password are correct - The base64 encoding is valid

Test the credentials manually:

bash
docker login myregistry.io -u myuser -p mypassword
docker pull myregistry.io/my-app:v1

If this fails locally, the credentials are wrong. Recreate the secret:

bash
kubectl delete secret my-registry-secret -n <namespace>
kubectl create secret docker-registry my-registry-secret \
  --docker-server=myregistry.io \
  --docker-username=myuser \
  --docker-password=mypassword \
  -n <namespace>

Step 6: Check Network Connectivity

The node might not be able to reach the registry:

```bash # SSH into a node and test connectivity ssh node-1

# Test DNS resolution nslookup myregistry.io

# Test HTTPS connectivity curl -v https://myregistry.io/v2/ ```

If DNS fails:

bash
# Check node DNS configuration
cat /etc/resolv.conf

If HTTPS fails, check: - Firewall rules blocking egress - Proxy configuration - TLS certificate issues

For registries with self-signed certificates, configure nodes to trust the certificate:

bash
# On each node
sudo mkdir -p /etc/docker/certs.d/myregistry.io
sudo cp ca.crt /etc/docker/certs.d/myregistry.io/
sudo systemctl restart containerd

Step 7: Check Rate Limits

Docker Hub has rate limits for anonymous and authenticated pulls:

bash
# Check your rate limit status
curl -s -u user:password "https://hub.docker.com/v2/users/userid/" | jq .

If you're hitting rate limits: - Use authenticated pulls - Mirror images to your own registry - Use a higher-tier Docker Hub plan

Check for rate limit errors in events:

bash
kubectl describe pod my-app -n <namespace> | grep -i "rate limit\|too many"

Step 8: Check Node Disk Space

The node might not have space to store images:

bash
kubectl describe node node-1 | grep -A 5 "Conditions:"

Look for DiskPressure:

bash
# SSH into the node and check disk
df -h /var/lib/containerd
df -h /var/lib/docker

Clean up unused images:

```bash # For containerd crictl rmi --prune

# For Docker docker system prune -a ```

Step 9: Check for Architecture Mismatch

The image architecture might not match the node architecture:

bash
# Check node architecture
kubectl get node node-1 -o jsonpath='{.status.nodeInfo.architecture}'
bash
amd64

Check the image architecture:

bash
docker inspect myregistry.io/my-app:v1 --format '{{.Architecture}}'
bash
arm64

If they don't match, you need a multi-architecture image or an image built for your node's architecture.

Build a multi-arch image:

bash
docker buildx build --platform linux/amd64,linux/arm64 -t myregistry.io/my-app:v1 .

Common Error Patterns

Error: pull access denied, repository does not exist

bash
Error response from daemon: pull access denied for my-app, repository does not exist

Cause: The image doesn't exist publicly and no authentication is provided.

Solution: Either make the image public or add image pull secrets.

Error: manifest unknown

bash
Error response from daemon: manifest for myregistry.io/my-app:v2 not found

Cause: The tag v2 doesn't exist in the repository.

Solution: Verify the tag exists, or use a different tag:

bash
# List available tags
curl -s -u user:password https://myregistry.io/v2/my-app/tags/list | jq .tags

Error: unauthorized: authentication required

bash
Error response from daemon: unauthorized: authentication required

Cause: Credentials are missing or invalid.

Solution: Create or update the image pull secret.

Error: x509: certificate signed by unknown authority

bash
Error response from daemon: Get https://myregistry.io/v2/: x509: certificate signed by unknown authority

Cause: The registry uses a self-signed or internal CA certificate.

Solution: Add the CA certificate to nodes:

bash
sudo mkdir -p /etc/docker/certs.d/myregistry.io
sudo cp ca.crt /etc/docker/certs.d/myregistry.io/

Error: connection refused

bash
Error response from daemon: Get "https://myregistry.io/v2/": dial tcp: connection refused

Cause: The registry is down or unreachable.

Solution: Check registry availability and network connectivity.

Verification

After fixing the issue, delete the pod and let Kubernetes recreate it:

bash
kubectl delete pod my-app-6b7c8d9f-x1k2m -n <namespace>

Or trigger a rollout restart:

bash
kubectl rollout restart deployment/my-deployment -n <namespace>

Watch the new pod start:

bash
kubectl get pods -n <namespace> -w
bash
NAME                        READY   STATUS              RESTARTS   AGE
my-app-7c8d9e0f-y2l3n       0/1     ContainerCreating   0           5s
my-app-7c8d9e0f-y2l3n       1/1     Running             0           30s

Quick Diagnostic Script

```bash #!/bin/bash POD=$1 NAMESPACE=${2:-default}

echo "=== Pod Status ===" kubectl get pod $POD -n $NAMESPACE -o wide

echo -e "\n=== Image ===" kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[*].image}'

echo -e "\n=== Image Pull Secrets ===" kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.imagePullSecrets}'

echo -e "\n=== Events ===" kubectl describe pod $POD -n $NAMESPACE | grep -A 15 "Events:"

echo -e "\n=== Node Status ===" NODE=$(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.nodeName}') kubectl describe node $NODE | grep -A 5 "Conditions:" ```

Key Takeaways

  1. 1.Check kubectl describe pod to see the exact error message
  2. 2.Verify image name and tag exist in the registry
  3. 3.For private registries, ensure imagePullSecrets are configured with valid credentials
  4. 4.Check network connectivity from nodes to the registry
  5. 5.Watch for rate limits on Docker Hub
  6. 6.Ensure image architecture matches node architecture
  7. 7.Verify nodes have disk space for images

ImagePullBackOff always has a specific reason - the describe output will tell you exactly what's wrong.