Fix Istio Service Mesh Traffic Not Routing

What's Actually Happening

Istio service mesh traffic is not routing correctly according to VirtualService and DestinationRule configurations. Requests go to wrong destinations or fail completely.

The Error You'll See

```bash $ kubectl exec -it client-pod -- curl http://service

curl: (7) Failed to connect to service port 80: Connection refused ```

Traffic going to wrong destination:

```bash $ kubectl logs client-pod

Request sent to service-v1 but response from service-v2 ```

Istio proxy errors:

```bash $ kubectl logs client-pod -c istio-proxy

[error][envoy] upstream connect error: 503 ```

Why This Happens

1. 1.Sidecar not injected - Pods running without Istio proxy
2. 2.VirtualService mismatch - Routing rules not matching requests
3. 3.DestinationRule issues - Subset definitions incorrect
4. 4.Service not in mesh - Service excluded from mesh
5. 5.Port mismatch - Service port configuration wrong
6. 6.Host header mismatch - VirtualService host not matching

Step 1: Check Istio Installation

```bash # Check Istio is installed: istioctl version

# Check Istiod status: kubectl get pods -n istio-system

# Check Istio components: kubectl get svc -n istio-system

# Check Istio config: kubectl get configmap -n istio-system istio -o yaml

# Check proxy status: istioctl proxy-status

# Check Istio operator: kubectl get iop -n istio-system ```

Step 2: Check Sidecar Injection

```bash # Check namespace has injection enabled: kubectl get namespace -L istio-injection

# Enable injection for namespace: kubectl label namespace myapp istio-injection=enabled

# Check pod has sidecar: kubectl get pods -n myapp -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

# Should show: app-container istio-proxy

# Check injection webhook: kubectl get deployment -n istio-system istiod

# Check pod annotation: kubectl get pod mypod -n myapp -o yaml | grep "sidecar.istio.io"

# Force injection on existing pod: istioctl kube-inject -f deployment.yaml | kubectl apply -f -

# Or manually inject: kubectl patch deployment myapp -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}' ```

Step 3: Check VirtualService Configuration

```bash # List VirtualServices: kubectl get virtualservice -n myapp

# Check VirtualService details: kubectl get virtualservice myservice -n myapp -o yaml

# Example: apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: myservice spec: hosts: - "myservice" - "myservice.myapp.svc.cluster.local" http: - route: - destination: host: myservicesubset: v1 weight: 100

# Check host matching: # Host in VS must match request host header

# Test request: kubectl exec -it client -- curl -H "Host: myservice" http://service/

# Check multiple hosts: kubectl get virtualservice myservice -o jsonpath='{.spec.hosts}'

# Check route rules: kubectl get virtualservice myservice -o jsonpath='{.spec.http[*].route}' ```

Step 4: Check DestinationRule Configuration

```bash # List DestinationRules: kubectl get destinationrule -n myapp

# Check DestinationRule details: kubectl get destinationrule myservice -n myapp -o yaml

# Example: apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: name: myservice spec: host: myservice subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2trafficPolicy: loadBalancer: simple: ROUND_ROBIN

# Check subset labels match pod labels: kubectl get pods -n myapp -L version

# DestinationRule subset labels must match pod labels!

# Check host in DestinationRule: kubectl get destinationrule myservice -o jsonpath='{.spec.host}' # Must match service name ```

Step 5: Check Service Configuration

```bash # Check Kubernetes service: kubectl get svc myservice -n myapp -o yaml

# Check service port name: # Must follow protocol[-suffix] format for Istio: # http-web, http, grpc, tcp

kubectl get svc myservice -o jsonpath='{.spec.ports[*].name}'

# Example correct port names: ports: - name: http port: 80 targetPort: 8080

# Wrong port name causes issues: # name: web # Not http prefixed

# Check service selector: kubectl get svc myservice -o jsonpath='{.spec.selector}' # Must match pod labels

# Check service endpoints: kubectl get endpoints myservice -n myapp

# Must have endpoints! ```

Step 6: Check Pod Labels

```bash # Check pod labels: kubectl get pods -n myapp -L app,version

# Labels needed: # app: myservice # version: v1 or v2 (for subsets)

# DestinationRule subsets reference these labels: subsets: - name: v1 labels: version: v1 # Must match pod label

# Check deployment labels: kubectl get deployment myservice-v1 -n myapp -o jsonpath='{.spec.template.metadata.labels}'

# Add missing labels: kubectl patch deployment myservice-v1 -p '{"spec":{"template":{"metadata":{"labels":{"version":"v1"}}}}}' ```

Step 7: Check Istio Proxy Logs

```bash # Check istio-proxy logs: kubectl logs mypod -n myapp -c istio-proxy

# Check envoy access logs: kubectl logs mypod -n myapp -c istio-proxy | grep "REQ"

# Enable access logging: # In IstioConfigMap: data: mesh: |- accessLogFile: /dev/stdout

# Check for routing errors: kubectl logs mypod -n myapp -c istio-proxy | grep -i error

# Check upstream errors: kubectl logs mypod -n myapp -c istio-proxy | grep " upstream"

# View envoy config: istioctl proxy-config cluster mypod -n myapp

# View routes: istioctl proxy-config route mypod -n myapp

# View listeners: istioctl proxy-config listener mypod -n myapp ```

Step 8: Debug Envoy Configuration

```bash # View all clusters: istioctl proxy-config cluster mypod -n myapp -o json

# View specific cluster: istioctl proxy-config cluster mypod -n myapp --cluster "outbound|80||myservice"

# View routes: istioctl proxy-config route mypod -n myapp --name "80"

# View listeners: istioctl proxy-config listener mypod -n myapp --port 80

# Check endpoint config: istioctl proxy-config endpoint mypod -n myapp

# Compare with expected: istioctl proxy-config cluster mypod -n myapp | grep myservice

# Dump full config: kubectl exec mypod -n myapp -c istio-proxy -- pilot-agent request GET config_dump

# Analyze config: istioctl analyze -n myapp ```

Step 9: Test Traffic Routing

```bash # Test direct service call (without Istio): kubectl exec -it client -- curl http://myservice:80

# Test with Istio routing: kubectl exec -it client-pod-with-sidecar -- curl http://myservice:80

# Check which version responds: kubectl exec -it client -- curl -s http://myservice:80/version

# Test with specific subset: kubectl exec -it client -- curl -H "Host: myservice" http://myservice/v1

# Generate load and check routing: for i in {1..100}; do kubectl exec -it client -- curl -s http://myservice:80/version done | sort | uniq -c

# Check traffic distribution: # Should match VirtualService weights

# Use Istio debug: istioctl dashboard envoy mypod -n myapp

# Check from outside mesh: kubectl port-forward svc/myservice 8080:80 curl http://localhost:8080 ```

Step 10: Check Istio Configuration Conflicts

```bash # Analyze Istio config: istioctl analyze -A

# Check for errors: istioctl analyze -n myapp --failure-level Error

# Check for warnings: istioctl analyze -n myapp --failure-level Warn

# Common issues: # [IST0101] VirtualService references nonexistent destination rule host # [IST0102] DestinationRule references nonexistent service host # [IST0103] Subsets not found # [IST0104] Conflicting VirtualService hosts # [IST0105] Port name not http prefixed

# Check conflicting VirtualServices: kubectl get virtualservice -A -o json | jq '.items[] | select(.spec.hosts[] | contains("myservice"))'

# Check conflicting DestinationRules: kubectl get destinationrule -A -o json | jq '.items[] | select(.spec.host == "myservice")'

# Remove conflicting configs: kubectl delete virtualservice conflicting-vs -n other-ns ```

Istio Routing Checklist

Verify the Fix

```bash # After fixing routing

# 1. Check sidecar kubectl get pods -n myapp -o jsonpath='{.items[*].spec.containers[*].name}' // istio-proxy present

# 2. Test routing kubectl exec -it client -- curl http://myservice // Correct response from v1

# 3. Check distribution for i in {1..100}; do curl -s http://myservice/version; done | sort | uniq -c // Matches weight configuration

# 4. Verify envoy config istioctl proxy-config route client-pod -n myapp // Shows correct routes

# 5. Check logs kubectl logs client-pod -c istio-proxy | tail -20 // No errors

# 6. Analyze config istioctl analyze -n myapp // No errors or warnings ```

Related Issues

• [Fix Kubernetes Service Not Found](/articles/fix-kubernetes-service-not-found)
• [Fix Istio Sidecar Injection Failed](/articles/fix-istio-sidecar-injection-failed)
• [Fix Istio mTLS Error](/articles/fix-istio-mtls-error)