Fix Istio Proxy Sidecar Not Ready

What's Actually Happening

Istio proxy sidecar containers fail to become ready, blocking pod startup and traffic flow. Applications cannot start or receive traffic.

The Error You'll See

Sidecar not ready:

```bash $ kubectl get pods

NAME READY STATUS RESTARTS myapp-abc123 1/2 Running 3 # Sidecar container not ready ```

Readiness probe failed:

```bash $ kubectl describe pod myapp-abc123

Warning Unhealthy 1m kubelet Readiness probe failed: Get http://10.0.0.1:15021/healthz/ready: dial tcp 10.0.0.1:15021: connection refused ```

Proxy container logs:

```bash $ kubectl logs myapp-abc123 -c istio-proxy

2024-01-01T00:00:00.000Z warning environment Failed to fetch certs from Istiod: connection refused ```

Why This Happens

1. 1.Istiod unavailable - Control plane not accessible
2. 2.Certificate failure - mTLS certificate provisioning failed
3. 3.Configuration error - Invalid proxy configuration
4. 4.Resource limits - Insufficient memory/CPU
5. 5.Network issues - Sidecar cannot reach Istiod
6. 6.Startup race - Application starts before sidecar ready

Step 1: Check Pod Status

```bash # Check pod status: kubectl get pods -n myapp

# Describe the pod: kubectl describe pod myapp-abc123

# Check container status: kubectl get pod myapp-abc123 -o jsonpath='{.status.containerStatuses[?(@.name=="istio-proxy")]}'

# Check sidecar logs: kubectl logs myapp-abc123 -c istio-proxy

# Check previous logs if restarted: kubectl logs myapp-abc123 -c istio-proxy --previous

# Check init container: kubectl logs myapp-abc123 -c istio-init ```

Step 2: Check Istiod Status

```bash # Check Istiod pods: kubectl get pods -n istio-system

# Check Istiod health: kubectl exec -n istio-system deployment/istiod -- pilot-agent request GET ready

# Check Istiod logs: kubectl logs -n istio-system deployment/istiod | grep -i error

# Check Istiod endpoints: kubectl get endpoints -n istio-system istiod

# Check Istiod service: kubectl get svc -n istio-system istiod

# Test Istiod connectivity from application namespace: kubectl run test --rm -it --image=curlimages/curl -- curl http://istiod.istio-system:15014/ready ```

Step 3: Check Certificate Provisioning

```bash # Check certificate: kubectl exec myapp-abc123 -c istio-proxy -- ls /etc/certs/

# Check certificate content: kubectl exec myapp-abc123 -c istio-proxy -- openssl x509 -in /etc/certs/cert-chain.pem -text -noout

# Check certificate expiry: kubectl exec myapp-abc123 -c istio-proxy -- openssl x509 -in /etc/certs/cert-chain.pem -noout -dates

# Check root CA: kubectl exec myapp-abc123 -c istio-proxy -- cat /etc/certs/root-cert.pem | openssl x509 -text -noout

# Force certificate rotation: kubectl delete pod myapp-abc123

# Check CSR requests: kubectl get csr

# Approve CSR if pending: kubectl certificate approve <csr-name> ```

Step 4: Check Proxy Status

```bash # Check proxy status: istioctl proxy-status

# Output shows: # CLUSTER CDS LDS EDS RDS # myapp SYNCED SYNCED SYNCED SYNCED

# If NOT SENT, proxy cannot connect to Istiod

# Check proxy config: istioctl proxy-config cluster myapp-abc123

# Check listener config: istioctl proxy-config listener myapp-abc123

# Check route config: istioctl proxy-config route myapp-abc123

# Check endpoint config: istioctl proxy-config endpoint myapp-abc123

# Get config dump: kubectl exec myapp-abc123 -c istio-proxy -- curl localhost:15000/config_dump ```

Step 5: Check Resource Limits

```bash # Check sidecar resource usage: kubectl top pods myapp-abc123 --containers

# Check resource limits: kubectl get pod myapp-abc123 -o jsonpath='{.spec.containers[?(@.name=="istio-proxy")].resources}'

# Increase resources in IstioOperator: apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: global: proxy: resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi

# Apply: istioctl apply -f istio-operator.yaml

# Check for OOMKilled: kubectl get pod myapp-abc123 -o jsonpath='{.status.containerStatuses[?(@.name=="istio-proxy")].lastState.terminated.reason}' ```

Step 6: Check Network Connectivity

```bash # Test connectivity from sidecar to Istiod: kubectl exec myapp-abc123 -c istio-proxy -- curl -v http://istiod.istio-system:15014/ready

# Check DNS: kubectl exec myapp-abc123 -c istio-proxy -- nslookup istiod.istio-system

# Check network policies: kubectl get networkpolicy -A

# Check if port 15012 is accessible: kubectl exec myapp-abc123 -c istio-proxy -- nc -zv istiod.istio-system 15012

# Check if port 15014 is accessible: kubectl exec myapp-abc123 -c istio-proxy -- nc -zv istiod.istio-system 15014

# Test from different namespace: kubectl run test -n myapp --rm -it --image=curlimages/curl -- curl http://istiod.istio-system:15014/ready ```

Step 7: Check Startup Configuration

```bash # Check proxy environment: kubectl exec myapp-abc123 -c istio-proxy -- env | grep ISTIO

# Key environment variables: # ISTIO_META_POD_NAME # ISTIO_META_NAMESPACE # ISTIO_META_CLUSTER_ID # POD_IP

# Check bootstrap config: kubectl exec myapp-abc123 -c istio-proxy -- cat /etc/istio/proxy/envoy-rev0.json

# Check proxy-config annotation: kubectl get pod myapp-abc123 -o jsonpath='{.metadata.annotations.proxy\.istio\.io/config}'

# Check for custom configuration: kubectl get pod myapp-abc123 -o yaml | grep -A 10 annotations

# Common annotations: # proxy.istio.io/config: | # holdApplicationUntilProxyStarts: true ```

Step 8: Fix Startup Race Condition

```yaml # Configure sidecar to start first:

# Option 1: Annotation on pod: apiVersion: v1 kind: Pod metadata: annotations: proxy.istio.io/config: | holdApplicationUntilProxyStarts: true

# Option 2: Global configuration in IstioOperator: apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: global: proxy: holdApplicationUntilProxyStarts: true

# Option 3: Use postStart hook in application container: lifecycle: postStart: exec: command: - /bin/sh - -c - "sleep 5" # Wait for sidecar

# Apply and restart pod: kubectl rollout restart deployment/myapp ```

Step 9: Check Istio Configuration

```bash # Check Istio configuration: kubectl get istiooperator -A

# Check mesh config: kubectl get configmap istio -n istio-system -o yaml

# Check global config: istioctl config list

# Check destination rules: kubectl get destinationrule -A

# Check virtual services: kubectl get virtualservice -A

# Check for conflicting configurations: istioctl analyze

# Common issues: # - Mismatched subset definitions # - Missing service entries # - Incorrect gateway bindings ```

Step 10: Monitor Sidecar Health

```bash # Create monitoring script: cat << 'EOF' > /usr/local/bin/monitor-istio-sidecar.sh #!/bin/bash

echo "=== Pods with Sidecar Issues ===" kubectl get pods -A -o json | jq '.items[] | select(.status.containerStatuses != null) | select(.status.containerStatuses[] | select(.name=="istio-proxy" and .ready==false)) | {name: .metadata.name, namespace: .metadata.namespace}'

echo "" echo "=== Istiod Status ===" kubectl get pods -n istio-system -l app=istiod

echo "" echo "=== Proxy Status ===" istioctl proxy-status

echo "" echo "=== Recent Sidecar Errors ===" kubectl logs -n istio-system deployment/istiod --since=1h | grep -i "push|error" | tail -10

echo "" echo "=== Certificate Expiry ===" for pod in $(kubectl get pods -A -l security.istio.io/tlsMode=istio -o jsonpath='{.items[*].metadata.name}'); do ns=$(kubectl get pod $pod -A -o jsonpath='{.items[0].metadata.namespace}') echo "Pod: $pod" kubectl exec $pod -n $ns -c istio-proxy -- openssl x509 -in /etc/certs/cert-chain.pem -noout -dates 2>/dev/null done EOF

chmod +x /usr/local/bin/monitor-istio-sidecar.sh

# Prometheus metrics: # envoy_server_live # envoy_cluster_upstream_cx_connect_fail # istio_proxy_workload_info

# Alerts: - alert: IstioSidecarNotReady expr: count(kube_pod_container_status_ready{container="istio-proxy"} == 0) > 0 for: 5m labels: severity: warning annotations: summary: "Istio sidecar containers not ready"

• alert: IstiodDown
• expr: up{job="istiod"} == 0
• for: 2m
• labels:
• severity: critical
• annotations:
• summary: "Istiod is not available"
• `

Istio Sidecar Not Ready Checklist

Verify the Fix

```bash # After fixing sidecar issue

# 1. Check pod ready kubectl get pods myapp-abc123 // 2/2 Running

# 2. Check proxy status istioctl proxy-status // All SYNCED

# 3. Test connectivity kubectl exec myapp-abc123 -c app -- curl http://localhost:15021/healthz/ready // 200 OK

# 4. Verify traffic flow kubectl exec myapp-abc123 -c app -- curl http://other-service/ // Response received

# 5. Check proxy config istioctl proxy-config cluster myapp-abc123 // Clusters configured

# 6. Monitor stability /usr/local/bin/monitor-istio-sidecar.sh // No issues ```

Related Issues

• [Fix Envoy Proxy Upstream Connection Refused](/articles/fix-envoy-proxy-upstream-connection-refused)
• [Fix Consul Connect Proxy Error](/articles/fix-consul-connect-proxy-error)
• [Fix Kubernetes Pod Not Ready](/articles/fix-kubernetes-pod-not-ready)