Introduction

Consul Connect provides service mesh capabilities through proxy sidecars that handle mTLS and traffic routing. Proxy errors can result from certificate issues, upstream/downstream connectivity problems, intention configuration errors, or Envoy proxy failures. These errors prevent services from communicating through the mesh, causing connection failures or certificate validation errors.

Symptoms

Error messages in Consul and Envoy logs:

bash
Proxy: upstream connection failed: certificate verification failed
Connect proxy cannot reach upstream service
Sidecar proxy error: intention denied
No healthy upstream endpoints
x509: certificate signed by unknown authority

Observable indicators: - Services cannot communicate through Connect - Consul UI shows services unhealthy - Envoy sidecar logs show connection errors - Certificate validation failures - Intention blocking traffic - Upstream resolution failures

Common Causes

  1. 1.Certificate rotation issues - CA certificate not propagated
  2. 2.Intention configuration - Missing or deny intention blocking traffic
  3. 3.Upstream not healthy - Target service not registered or healthy
  4. 4.Sidecar not started - Proxy container not running
  5. 5.Gateway configuration - Ingress/terminating gateway misconfigured
  6. 6.Service registration - Service not properly registered with Connect
  7. 7.Envoy proxy crash - Sidecar Envoy process crashed

Step-by-Step Fix

Step 1: Check Consul Service Health

```bash # Check service health in Consul consul catalog services

# List service instances consul catalog services -nodes

# Check specific service health curl http://localhost:8500/v1/health/service/my-service

# Check Connect-enabled services consul catalog services | grep -i connect ```

Step 2: Check Proxy Status

```bash # Check proxy status via Consul consul connect proxy status

# List all proxies curl http://localhost:8500/v1/connect/ca/roots

# Check proxy configuration consul config write proxy-config.json

# View proxy state curl http://localhost:8500/v1/agent/proxies ```

Step 3: Check Sidecar Logs

```bash # Check Envoy sidecar logs (Docker) docker logs service-sidecar

# Check Consul agent logs consul logs -node=<node-name>

# View Envoy admin interface curl http://localhost:19000/clusters

# Check Envoy stats curl http://localhost:19000/stats | grep -i connect ```

Step 4: Check Certificate Status

```bash # Check CA roots curl http://localhost:8500/v1/connect/ca/roots

# Check leaf certificates curl http://localhost:8500/v1/connect/ca/leaf/<service-name>

# Verify certificate expiration consul connect cert show <service-name>

# Check certificate rotation curl http://localhost:8500/v1/connect/ca/configuration ```

Step 5: Check Intention Configuration

```bash # List intentions consul intention list

# Check specific intention consul intention check <source> <destination>

# Get intention details curl http://localhost:8500/v1/connect/intentions

# Check intention matches consul intention match -source <source-service> ```

```bash # Create allow intention consul intention create -allow source-service destination-service

# Or via API curl -X PUT http://localhost:8500/v1/connect/intentions \ -d '{"SourceType":"consul","SourceName":"source-service","DestinationName":"destination-service","Action":"allow"}' ```

Step 6: Check Service Registration

json
// Check service definition with Connect enabled
{
  "service": {
    "name": "my-service",
    "port": 8080,
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "upstream-service",
              "local_bind_port": 9090
            }
          ]
        }
      }
    }
  }
}

```bash # Register service with Connect consul services register service.json

# Check registration curl http://localhost:8500/v1/agent/services ```

Step 7: Fix Proxy Configuration

json
// Proxy configuration file
{
  "proxy": {
    "destination_service_name": "my-service",
    "destination_service_id": "my-service-1",
    "local_service_address": "127.0.0.1",
    "local_service_port": 8080,
    "upstreams": [
      {
        "destination_name": "upstream-service",
        "local_bind_port": 9090,
        "config": {
          "connect_timeout_ms": 5000,
          "health_check_config": {
            "interval_ms": 10000,
            "timeout_ms": 5000
          }
        }
      }
    ]
  }
}

```bash # Apply proxy configuration consul connect proxy -config-file=proxy-config.json

# Or configure via service definition consul services register -proxy-count=1 service-with-proxy.json ```

Step 8: Fix Gateway Configuration

json
// Ingress gateway configuration
{
  "service": {
    "name": "ingress-gateway",
    "kind": "ingress-gateway",
    "port": 8443,
    "connect": {
      "gateway": {
        "tls": {
          "enabled": true
        },
        "routes": [
          {
            "service": {
              "name": "backend-service"
            }
          }
        ]
      }
    }
  }
}

```bash # Check gateway status curl http://localhost:8500/v1/connect/gateways

# Verify gateway routing consul config write gateway-config.json ```

Step 9: Verify the Fix

```bash # Test upstream connectivity curl http://localhost:9090/ # Via proxy upstream port

# Check service-to-service communication consul connect proxy test -source source-service -dest destination-service

# Monitor proxy status consul connect proxy status --verbose

# Check intentions are allowing traffic consul intention check source-service destination-service ```

Advanced Diagnosis

Debug Envoy Configuration

```bash # Get Envoy config dump curl http://localhost:19000/config_dump

# Check specific cluster curl http://localhost:19000/config_dump | jq '.configs[] | select(.cluster!=null)'

# Check listeners curl http://localhost:19000/listeners

# Check upstream clusters curl http://localhost:19000/clusters | grep upstream ```

Certificate Issues

```bash # Force certificate rotation consul connect ca rotate

# Re-register to get new certificate consul services register service.json --force

# Check certificate chain openssl s_client -connect localhost:21000 -showcerts

# Verify certificate matches service consul connect cert show my-service | openssl x509 -text -noout | grep -i spiffe ```

Check Mesh Configuration

```bash # Get mesh configuration curl http://localhost:8500/v1/connect/config

# Update mesh config for troubleshooting curl -X PUT http://localhost:8500/v1/connect/config \ -d '{"MeshGatewayMode":"local"}'

# Check CA provider consul connect ca get-root ```

Namespace Issues

```bash # If using namespaces consul namespace list

# Check service in correct namespace consul catalog services -namespace=production

# Intention across namespaces consul intention create -namespace=source-ns -destination-namespace=dest-ns source-service dest-service ```

Common Pitfalls

  • Missing intention - Traffic blocked without allow intention
  • Certificate not propagated - CA rotation not reaching proxies
  • Upstream not healthy - Target service down or not registered
  • Wrong upstream port - Local bind port mismatch
  • Service not Connect-enabled - Missing sidecar_service in registration
  • Gateway routing missing - Ingress gateway not configured for service
  • TLS mismatch - Gateway TLS when service expects plaintext

Best Practices

json
// Complete service registration with Connect
{
  "service": {
    "name": "my-app",
    "port": 8080,
    "tags": ["v1"],
    "meta": {
      "version": "v1"
    },
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "db-service",
              "local_bind_port": 3306,
              "config": {
                "connect_timeout_ms": 5000
              }
            },
            {
              "destination_name": "cache-service",
              "local_bind_port": 6379
            }
          ],
          "config": {
            "bind_address": "0.0.0.0",
            "expose_paths": [
              {
                "path": "/health",
                "protocol": "http",
                "listener_port": 21500
              }
            ]
          }
        }
      }
    },
    "check": {
      "id": "my-app-health",
      "name": "HTTP Health Check",
      "http": "http://127.0.0.1:21500/health",
      "interval": "10s",
      "timeout": "5s"
    }
  }
}

```bash # Create comprehensive intentions consul intention create -allow web-api api-service consul intention create -allow api-service db-service consul intention create -allow api-service cache-service

# Deny unintended access consul intention create -deny unknown-service api-service ```

  • Istio Destination Rule Error
  • Envoy Cluster Unhealthy
  • Linkerd Traffic Split Error
  • HAProxy Backend Down