Fix Kubernetes Ingress 502 Error

What's Actually Happening

Kubernetes Ingress 502 errors occur when the ingress controller cannot reach the backend service or receives an invalid response. The service may be down, misconfigured, or the ingress rules may be incorrect.

The Error You'll See

Browser error:

502 Bad Gateway
nginx

Ingress controller logs:

```bash $ kubectl logs -n ingress-nginx ingress-nginx-controller-xxx

[error] 12345#12345: *67890 connect() failed (111: Connection refused) while connecting to upstream upstream: "http://10.0.0.1:8080" ```

Ingress resource status:

```bash $ kubectl describe ingress my-ingress

Rules: Host Path Backends ---- ---- -------- example.com /api my-service:80 (10.0.0.1:8080) [error: no healthy upstream] ```

Why This Happens

1. 1.Backend service down - Pods not running or ready
2. 2.Wrong service port - Ingress points to wrong port
3. 3.Service selector mismatch - Service not selecting pods
4. 4.Ingress class missing - No ingress controller assigned
5. 5.TLS certificate issues - Invalid or missing certificates
6. 6.Resource limits - Backend pods OOMKilled

Step 1: Check Ingress Controller Logs

```bash # Get ingress controller pod kubectl get pods -n ingress-nginx

# View logs kubectl logs -n ingress-nginx ingress-nginx-controller-xxx

# Follow logs in real-time kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -f

# Check for errors kubectl logs -n ingress-nginx ingress-nginx-controller-xxx | grep -i error

# Check nginx configuration kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- cat /etc/nginx/nginx.conf | grep upstream -A 5 ```

Step 2: Verify Backend Service

```bash # Check service exists kubectl get svc my-service

# Check service endpoints kubectl get endpoints my-service # Should show pod IPs, not empty

# Describe service kubectl describe svc my-service

# Check service selector kubectl get svc my-service -o jsonpath='{.spec.selector}'

# Find pods matching selector kubectl get pods -l app=my-app

# Test service directly kubectl run test --image=busybox --rm -it -- wget -qO- my-service:80

# Check if service port is correct kubectl get svc my-service -o jsonpath='{.spec.ports[0].port}' ```

Step 3: Check Pod Health

```bash # Check pod status kubectl get pods -l app=my-app

# All pods should be Running and Ready (1/1)

# Check pod details kubectl describe pod my-app-xxx

# Check if readiness probe passing kubectl describe pod my-app-xxx | grep -A 5 Readiness

# Check pod logs kubectl logs my-app-xxx

# Check pod is accepting connections kubectl exec my-app-xxx -- curl -s localhost:8080/health

# Check resource usage kubectl top pods -l app=my-app ```

Step 4: Fix Service Port Mismatch

```yaml # Check ingress backend port apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress spec: rules: - host: example.com http: paths: - path: /api backend: service: name: my-service port: number: 80 # Must match service port

--- # Verify service port apiVersion: v1 kind: Service metadata: name: my-service spec: ports: - port: 80 # Must match ingress backend port targetPort: 8080 # Must match container port selector: app: my-app

--- # Fix mismatch kubectl patch ingress my-ingress --type=json -p='[ {"op": "replace", "path": "/spec/rules/0/http/paths/0/backend/service/port/number", "value: 8080} ]' ```

Step 5: Check Ingress Class

```bash # Check ingress class kubectl get ingressclass

# Check ingress has correct class kubectl get ingress my-ingress -o yaml | grep ingressClassName

# If missing, add ingress class kubectl patch ingress my-ingress --type=json -p='[ {"op": "add", "path": "/spec/ingressClassName", "value": "nginx"} ]'

# Verify ingress controller is running kubectl get pods -n ingress-nginx

# Check ingress controller service kubectl get svc -n ingress-nginx

# Default ingress class annotation # For older Kubernetes versions: metadata: annotations: kubernetes.io/ingress.class: nginx ```

Step 6: Check TLS Configuration

```bash # Check TLS secret exists kubectl get secret my-tls-secret

# Describe TLS secret kubectl describe secret my-tls-secret

# Verify secret has correct keys kubectl get secret my-tls-secret -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

# Check ingress TLS configuration kubectl get ingress my-ingress -o yaml | grep -A 10 tls

# Create TLS secret if missing kubectl create secret tls my-tls-secret \ --cert=path/to/cert.crt \ --key=path/to/cert.key

# Or use cert-manager kubectl apply -f - <<EOF apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: my-cert spec: secretName: my-tls-secret issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames: - example.com EOF ```

Step 7: Check Network Policies

```bash # Check if network policies blocking traffic kubectl get networkpolicy -n default

# Describe network policy kubectl describe networkpolicy allow-ingress

# Policy should allow ingress from ingress-nginx namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-ingress spec: podSelector: matchLabels: app: my-app policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: name: ingress-nginx ports: - port: 8080 protocol: TCP

# Check namespace labels kubectl get namespace ingress-nginx --show-labels

# Label namespace if needed kubectl label namespace ingress-nginx name=ingress-nginx ```

Step 8: Test Backend Connectivity

```bash # Test from ingress controller pod kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- curl -s http://my-service.default.svc.cluster.local:80

# Test backend pod directly kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- curl -s http://10.0.0.1:8080

# Check DNS resolution kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- nslookup my-service.default.svc.cluster.local

# Test with verbose output kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- curl -v http://my-service:80

# Check if backend returning errors kubectl exec -n ingress-nginx ingress-nginx-controller-xxx -- curl -s -o /dev/null -w "%{http_code}" http://my-service:80 ```

Step 9: Check Resource Limits

```bash # Check if pods are being OOMKilled kubectl describe pod my-app-xxx | grep -i oom

# Check resource usage kubectl top pods -l app=my-app

# Check resource limits kubectl get pod my-app-xxx -o yaml | grep -A 10 resources:

# Increase limits if needed kubectl set resources deployment/my-app \ --limits=cpu=500m,memory=512Mi \ --requests=cpu=250m,memory=256Mi

# Check pod restart count kubectl get pods -l app=my-app -o wide

# High restart count indicates resource issues ```

Step 10: Enable Access Logging

```yaml # Enable access logging in ingress controller # ConfigMap for nginx ingress controller apiVersion: v1 kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx data: access-log-path: /var/log/nginx/access.log error-log-path: /var/log/nginx/error.log log-level-upstream: info

--- # View access logs kubectl logs -n ingress-nginx ingress-nginx-controller-xxx | grep "GET /api"

# Check error logs for specific request kubectl logs -n ingress-nginx ingress-nginx-controller-xxx | grep "502" ```

Ingress 502 Troubleshooting Checklist

Verify the Fix

```bash # After fixing backend or ingress configuration

# 1. Check ingress status kubectl get ingress my-ingress # Should show ADDRESS

# 2. Test ingress URL curl -I https://example.com/api # Should return 200, not 502

# 3. Check ingress controller logs kubectl logs -n ingress-nginx ingress-nginx-controller-xxx --tail 20 # No 502 errors

# 4. Verify service endpoints kubectl get endpoints my-service # Shows healthy pod IPs

# 5. Test from inside cluster kubectl run test --image=busybox --rm -it -- wget -qO- https://example.com/api

# 6. Monitor for recurring issues kubectl logs -n ingress-nginx -f ingress-nginx-controller-xxx | grep -i error ```

Related Issues

• [Fix Kubernetes Service Endpoints Empty](/articles/fix-kubernetes-service-endpoints-empty)
• [Fix Kubernetes Pod Not Ready](/articles/fix-kubernetes-pod-not-ready)
• [Fix Nginx Ingress Controller Not Working](/articles/fix-nginx-ingress-controller-not-working)