Fix CoreDNS Resolution Failed in Kubernetes

What's Actually Happening

Kubernetes pods cannot resolve DNS names. DNS queries fail, preventing pods from connecting to services by name.

The Error You'll See

```bash $ kubectl exec -it pod -- nslookup kubernetes

Server: 10.96.0.10 Address: 10.96.0.10#53

** server can't find kubernetes: NXDOMAIN ```

DNS timeout:

```bash $ kubectl exec -it pod -- curl http://service

curl: (6) Could not resolve host: service ```

CoreDNS error:

```bash $ kubectl logs -n kube-system coredns-xxx

[ERROR] plugin/errors: 2 kubernetes.default.svc.cluster.local. A: unreachable backend ```

Why This Happens

1. 1.CoreDNS pods not running - CoreDNS deployment issues
2. 2.CoreDNS Service not available - Service or endpoints missing
3. 3.Configuration error - CoreDNS Corefile misconfigured
4. 4.Network policy blocking - Network policy preventing DNS traffic
5. 5.Node DNS issues - Node /etc/resolv.conf problems
6. 6.Resource limits - CoreDNS OOMKilled or CPU throttled

Step 1: Check CoreDNS Pods Status

```bash # Check CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns

# Should show Running status: # NAME READY STATUS RESTARTS # coredns-5644d7b6d9-abc12 1/1 Running 0 # coredns-5644d7b6d9-def34 1/1 Running 0

# Check pod details: kubectl describe pod -n kube-system -l k8s-app=kube-dns

# Check pod logs: kubectl logs -n kube-system -l k8s-app=kube-dns

# Check if pods are ready: kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

# Check resource usage: kubectl top pods -n kube-system -l k8s-app=kube-dns ```

Step 2: Check CoreDNS Service

```bash # Check CoreDNS service: kubectl get svc -n kube-system kube-dns

# Check service endpoints: kubectl get endpoints -n kube-system kube-dns

# Should show CoreDNS pod IPs: # NAME ENDPOINTS AGE # kube-dns 10.244.0.10:53,10.244.0.11:53 10d

# Check service details: kubectl describe svc -n kube-system kube-dns

# Check ClusterIP: kubectl get svc -n kube-system kube-dns -o jsonpath='{.spec.clusterIP}' # Typically: 10.96.0.10

# Check service selector: kubectl get svc -n kube-system kube-dns -o jsonpath='{.spec.selector}' # Should be: k8s-app:kube-dns ```

Step 3: Check CoreDNS Configuration

```bash # View CoreDNS ConfigMap: kubectl get configmap -n kube-system coredns -o yaml

# Check Corefile: kubectl get configmap -n kube-system coredns -o jsonpath='{.data.Corefile}'

# Default Corefile: # .:53 { # errors # health # ready # kubernetes cluster.local in-addr.arpa ip6.arpa { # pods insecure # fallthrough in-addr.arpa ip6.arpa # } # prometheus :9153 # forward . /etc/resolv.conf # cache 30 # loop # reload # loadbalance # }

# Edit ConfigMap: kubectl edit configmap -n kube-system coredns

# After editing, restart CoreDNS: kubectl rollout restart deployment -n kube-system coredns ```

Step 4: Test DNS Resolution

```bash # Test from CoreDNS pod: kubectl exec -n kube-system coredns-xxx -- nslookup kubernetes

# Test from application pod: kubectl exec -it pod -- nslookup kubernetes.default.svc.cluster.local

# Test external DNS: kubectl exec -it pod -- nslookup google.com

# Test with dig: kubectl exec -it pod -- dig @10.96.0.10 kubernetes.default.svc.cluster.local

# Test with FQDN: kubectl exec -it pod -- nslookup kubernetes.default.svc.cluster.local 10.96.0.10

# Test short name: kubectl exec -it pod -- nslookup kubernetes

# If short name fails, check search domains: kubectl exec -it pod -- cat /etc/resolv.conf ```

Step 5: Check Pod DNS Configuration

```bash # Check pod resolv.conf: kubectl exec -it pod -- cat /etc/resolv.conf

# Expected output: # nameserver 10.96.0.10 # search default.svc.cluster.local svc.cluster.local cluster.local # options ndots:5

# Check dnsPolicy: kubectl get pod pod -o jsonpath='{.spec.dnsPolicy}'

# dnsPolicy options: # ClusterFirst (default) - Use cluster DNS for cluster domains # Default - Use node's DNS configuration # None - Use custom dnsConfig

# Custom DNS configuration: # In pod spec: dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 searches: - default.svc.cluster.local options: - name: ndots value: "2" ```

Step 6: Check Network Policies

```bash # List network policies: kubectl get networkpolicies -A

# Check if policy blocks DNS: kubectl describe networkpolicy -n kube-system

# DNS uses: # - UDP port 53 # - TCP port 53 (for large responses)

# Allow DNS in network policy: # For namespace: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns namespace: default spec: podSelector: {} policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53

# Apply: kubectl apply -f dns-policy.yaml ```

Step 7: Check Node DNS Configuration

```bash # Check node resolv.conf: # SSH to node: cat /etc/resolv.conf

# CoreDNS uses node's DNS for external resolution: # forward . /etc/resolv.conf

# If node DNS wrong, fix: # Edit systemd-resolved or NetworkManager config

# Check DNS on node: nslookup google.com

# Test node can reach external DNS: dig @8.8.8.8 google.com

# Check node DNS servers: cat /etc/resolv.conf | grep nameserver ```

Step 8: Check Resource Limits

```bash # Check CoreDNS resource limits: kubectl get deployment -n kube-system coredns -o yaml | grep -A10 resources

# If OOMKilled, increase memory: kubectl set resources deployment -n kube-system coredns \ --limits=memory=200Mi \ --requests=memory=100Mi

# Check for OOMKilled: kubectl describe pod -n kube-system -l k8s-app=kube-dns | grep -i oom

# Check CPU throttling: kubectl top pods -n kube-system -l k8s-app=kube-dns

# Increase CPU if throttled: kubectl set resources deployment -n kube-system coredns \ --limits=cpu=200m,memory=200Mi \ --requests=cpu=100m,memory=100Mi ```

Step 9: Fix Common Issues

```bash # Issue 1: Forward loop # Error: "loop" detected

# Fix: Check upstream DNS # In Corefile, change: # forward . /etc/resolv.conf # To: # forward . 8.8.8.8 8.8.4.4

# Issue 2: External DNS not working # Fix: Check forward directive # Add explicit upstream DNS: # forward . 8.8.8.8 8.8.4.4 { # policy sequential # }

# Issue 3: Slow DNS # Fix: Increase cache size # cache 30 -> cache 100

# Issue 4: DNS timeout # Check: # - Network connectivity # - CoreDNS resources # - Upstream DNS servers

# Issue 5: Pod DNS fails, node DNS works # Fix: Check kube-proxy kubectl get pods -n kube-system -l k8s-app=kube-proxy ```

Step 10: CoreDNS Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-coredns.sh #!/bin/bash

echo "=== CoreDNS Pods ===" kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

echo "" echo "=== CoreDNS Service ===" kubectl get svc -n kube-system kube-dns

echo "" echo "=== CoreDNS Endpoints ===" kubectl get endpoints -n kube-system kube-dns

echo "" echo "=== CoreDNS Logs (last 20) ===" kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

echo "" echo "=== DNS Test from Pod ===" kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup kubernetes 2>/dev/null || echo "DNS test failed"

echo "" echo "=== External DNS Test ===" kubectl run dns-test-ext --image=busybox --rm -it --restart=Never -- nslookup google.com 2>/dev/null || echo "External DNS test failed"

echo "" echo "=== CoreDNS ConfigMap ===" kubectl get configmap -n kube-system coredns -o jsonpath='{.data.Corefile}' | head -20

echo "" echo "=== Resource Usage ===" kubectl top pods -n kube-system -l k8s-app=kube-dns 2>/dev/null || echo "Metrics not available" EOF

chmod +x /usr/local/bin/check-coredns.sh

# Run: /usr/local/bin/check-coredns.sh

# Monitor: watch -n 10 /usr/local/bin/check-coredns.sh ```

CoreDNS Checklist

Verify the Fix

```bash # After fixing CoreDNS

# 1. Check pods kubectl get pods -n kube-system -l k8s-app=kube-dns // All Running

# 2. Test internal DNS kubectl exec -it pod -- nslookup kubernetes // Resolves to ClusterIP

# 3. Test external DNS kubectl exec -it pod -- nslookup google.com // Resolves

# 4. Test service discovery kubectl exec -it pod -- curl http://service-name:port // Connection successful

# 5. Check logs kubectl logs -n kube-system -l k8s-app=kube-dns // No errors

# 6. Monitor watch kubectl get pods -n kube-system -l k8s-app=kube-dns // Stable ```

Related Issues

• [Fix Kubernetes Pod CrashLoopBackOff](/articles/fix-kubernetes-pod-crashloopbackoff)
• [Fix Kubernetes Service Not Found](/articles/fix-kubernetes-service-not-found)
• [Fix Kubernetes Node Not Ready](/articles/fix-kubernetes-node-not-ready)