Fix Linkerd Service Mesh mTLS Error

What's Actually Happening

Linkerd service mesh mutual TLS (mTLS) fails between services. Connections are rejected or fall back to plaintext, exposing traffic.

The Error You'll See

```bash $ linkerd check

linkerd-identity --------------- × identity certificate is valid issuer certificate not found × trust anchors are valid no trust anchors found ```

Proxy error:

```bash $ kubectl logs pod -c linkerd-proxy

ERROR linkerd2_proxy::control::destination {service=service-a} TLS error: certificate verify failed ```

Connection error:

```bash $ kubectl exec -it client -- curl http://service-a

curl: (35) SSL connect error ```

Linkerd edges:

```bash $ linkerd edges -n myapp

SRC DST AUTHZ client service-a not_opted_in ```

Why This Happens

1. 1.Trust anchor missing - Root certificate not installed
2. 2.Identity issuer expired - Issuer certificate expired
3. 3.Proxy not injected - Pods missing Linkerd proxy
4. 4.Wrong trust domain - Identity domain mismatch
5. 5.Certificate rotation failed - Automatic rotation failed
6. 6.Network policy blocking - Policy preventing proxy communication

Step 1: Check Linkerd Installation

```bash # Check Linkerd status: linkerd check

# Check specific components: linkerd check --linkerd-identity

# Check proxy status: linkerd check --proxy

# Full diagnostics: linkerd check --all

# Check Linkerd version: linkerd version

# Check control plane pods: kubectl get pods -n linkerd

# Check control plane services: kubectl get svc -n linkerd

# Check Linkerd config: kubectl get configmap -n linkerd linkerd-config -o yaml ```

Step 2: Check Identity Certificates

```bash # Check identity certificate: linkerd check --linkerd-identity

# View identity issuer certificate: kubectl get secret -n linkerd linkerd-identity-issuer -o yaml

# Decode certificate: kubectl get secret -n linkerd linkerd-identity-issuer -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -text

# Check certificate dates: kubectl get secret -n linkerd linkerd-identity-issuer -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -dates

# Check trust anchors: kubectl get secret -n linkerd linkerd-trust-anchor -o yaml

# Decode trust anchor: kubectl get secret -n linkerd linkerd-trust-anchor -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -text ```

Step 3: Check Trust Anchors

```bash # Check trust anchors config: kubectl get configmap -n linkerd linkerd-config -o jsonpath='{.data.identity-trust-anchors}'

# Verify trust anchor certificate: linkerd check --linkerd-identity | grep trust

# Check trust anchor expiry: kubectl get secret -n linkerd linkerd-trust-anchor -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -dates

# If trust anchor expired: # Generate new trust anchor: step certificate create root.linkerd.cluster.local ca.crt ca.key --profile root-ca --no-password --insecure

# Update trust anchor: kubectl create secret generic linkerd-trust-anchor --from-file=ca.crt --namespace linkerd --dry-run=client -o yaml | kubectl apply -f -

# Restart identity service: kubectl rollout restart deployment/linkerd-identity -n linkerd ```

Step 4: Renew Identity Issuer Certificate

```bash # Check issuer certificate status: linkerd check --linkerd-identity

# If issuer expired or invalid:

# Generate new issuer certificate signed by trust anchor: step certificate create identity.linkerd.cluster.local issuer.crt issuer.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --no-password --insecure

# Create/update issuer secret: kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt --key=issuer.key --namespace linkerd --dry-run=client -o yaml | kubectl apply -f -

# Restart identity service: kubectl rollout restart deployment/linkerd-identity -n linkerd

# Check status: linkerd check --linkerd-identity

# Wait for rollout: kubectl rollout status deployment/linkerd-identity -n linkerd ```

Step 5: Check Proxy Injection

```bash # Check if pods have Linkerd proxy: kubectl get pods -n myapp -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

# Should show: app-container linkerd-proxy

# Check injection annotation: kubectl get pods -n myapp -o yaml | grep "linkerd.io/inject"

# Check namespace injection: kubectl get namespace myapp -o yaml | grep "linkerd.io/inject"

# Enable injection: kubectl annotate namespace myapp linkerd.io/inject=enabled

# Check proxy version: kubectl get pods -n myapp -o jsonpath='{.items[0].metadata.annotations.linkerd.io/proxy-version}'

# Force reinjection: kubectl get deployment -n myapp -o yaml | linkerd inject | kubectl apply -f -

# Check proxy status: linkerd check --proxy -n myapp ```

Step 6: Check Trust Domain

```bash # Check trust domain in config: kubectl get configmap -n linkerd linkerd-config -o jsonpath='{.data.config}' | grep trustDomain

# Default: cluster.local

# Check identity service trust domain: kubectl get deployment linkerd-identity -n linkerd -o yaml | grep -A5 identity-trust-domain

# Check proxy trust domain: kubectl get pods -n myapp -o jsonpath='{.items[0].spec.containers[?(@.name=="linkerd-proxy")].env[?(@.name=="LINKERD2_PROXY_IDENTITY_TRUST_DOMAIN")].value}'

# Trust domain must match! # If mismatched, reinstall Linkerd with correct trust domain:

linkerd install --identity-trust-domain=mydomain.com | kubectl apply -f -

# Or update config: kubectl get configmap linkerd-config -n linkerd -o yaml | sed 's/trustDomain: cluster.local/trustDomain: mydomain.com/' | kubectl apply -f -

# Restart proxies: kubectl rollout restart deployment -n myapp ```

Step 7: Check mTLS Status

```bash # Check mTLS for services: linkerd edges -n myapp

# Output shows AUTHZ column: # established = mTLS working # not_opted_in = no proxy (plaintext) # unknown = cannot determine

# Check tap for TLS: linkerd tap deploy -n myapp --to deploy/service-a

# Shows TLS status in output

# Check authorization policy: kubectl get authorizationpolicy -n myapp

# Check meshed services: linkerd stat -n myapp

# Check mTLS from pod: kubectl exec -it pod-with-proxy -- curl -v http://service-a:8080

# Look for TLS in response

# Check server side TLS: kubectl exec -it pod-with-proxy -- linkerd2-proxy get-identity ```

Step 8: Check Proxy Logs

```bash # Check proxy logs: kubectl logs pod -n myapp -c linkerd-proxy

# Filter for TLS errors: kubectl logs pod -n myapp -c linkerd-proxy | grep -i tls

# Filter for identity errors: kubectl logs pod -n myapp -c linkerd-proxy | grep -i identity

# Check identity service logs: kubectl logs -n linkerd deployment/linkerd-identity

# Check proxy injector logs: kubectl logs -n linkerd deployment/linkerd-proxy-injector

# Enable verbose logging: # In proxy config: kubectl annotate pod pod -n myapp linkerd.io/proxy-log-level=debug

# Restart pod: kubectl delete pod pod -n myapp # Pod recreated with debug logging ```

Step 9: Fix Certificate Rotation

```bash # Linkerd auto-rotates certificates # If rotation failed:

# Check identity service status: kubectl logs -n linkerd deployment/linkerd-identity

# Check for rotation errors: kubectl logs -n linkerd deployment/linkerd-identity | grep -i rotate

# Manual certificate rotation: # Use step CLI or cert-manager

# With cert-manager: # Install cert-manager first # Create issuer: cat << 'EOF' | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: linkerd-identity-issuer namespace: linkerd spec: ca: secretName: linkerd-trust-anchor EOF

# Create certificate: cat << 'EOF' | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: linkerd-identity-issuer namespace: linkerd spec: issuerRef: name: linkerd-identity-issuer secretName: linkerd-identity-issuer commonName: identity.linkerd.cluster.local duration: 24h renewBefore: 1h EOF

# Restart identity: kubectl rollout restart deployment/linkerd-identity -n linkerd ```

Step 10: Debug and Verify

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-linkerd-mtls.sh #!/bin/bash

echo "=== Linkerd Control Plane ===" kubectl get pods -n linkerd

echo "" echo "=== Linkerd Check ===" linkerd check --linkerd-identity

echo "" echo "=== Trust Anchor ===" kubectl get secret -n linkerd linkerd-trust-anchor -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -subject -dates

echo "" echo "=== Identity Issuer ===" kubectl get secret -n linkerd linkerd-identity-issuer -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -subject -dates

echo "" echo "=== Meshed Pods ===" kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.metadata.annotations.linkerd.io/inject}{"\n"}{end}' | grep enabled

echo "" echo "=== mTLS Edges ===" linkerd edges -A

echo "" echo "=== Test Connection ===" kubectl exec -it deployment/client -- curl -s http://service-a:8080 || echo "Connection failed" EOF

chmod +x /usr/local/bin/check-linkerd-mtls.sh

# Run verification: /usr/local/bin/check-linkerd-mtls.sh

# Full Linkerd check: linkerd check --all ```

Linkerd mTLS Checklist

Verify the Fix

```bash # After fixing mTLS

# 1. Check Linkerd linkerd check --linkerd-identity // All checks pass

# 2. Check certificates kubectl get secret -n linkerd linkerd-identity-issuer -o jsonpath='{.data.crt\.pem}' | base64 -d | openssl x509 -noout -dates // Valid dates

# 3. Check edges linkerd edges -n myapp // AUTHZ: established

# 4. Test connection kubectl exec -it client -- curl http://service-a // Successful with mTLS

# 5. Check proxy logs kubectl logs client -c linkerd-proxy | grep -i tls // No errors

# 6. Monitor status linkerd stat -n myapp // mTLS percentage high ```

Related Issues

• [Fix Istio Service Mesh Traffic Not Routing](/articles/fix-istio-service-mesh-traffic-not-routing)
• [Fix Istio mTLS Error](/articles/fix-istio-mtls-error)
• [Fix Kubernetes Service Not Found](/articles/fix-kubernetes-service-not-found)