Fix Kyverno Admission Controller Error

What's Actually Happening

Kyverno admission webhook is returning errors when Kubernetes tries to validate or mutate resources. All resource operations fail with webhook errors.

The Error You'll See

```bash $ kubectl apply -f deployment.yaml

Error from server (InternalError): Internal error occurred: failed calling webhook "kyverno-validating-webhook-cfg": Post "https://kyverno-svc.kyverno.svc:443/validate?timeout=10s": dial tcp 10.96.100.1:443: connect: connection refused ```

Webhook timeout:

Error: failed calling webhook "kyverno-mutating-webhook-cfg": context deadline exceeded

TLS error:

Error: x509: certificate signed by unknown authority

Service unavailable:

Error: service 'kyverno-svc' not found in namespace 'kyverno'

Why This Happens

1. 1.Kyverno pod not running - Admission controller crashed or not started
2. 2.Service endpoint missing - No endpoints for webhook service
3. 3.TLS certificate issue - Certificate expired or not trusted
4. 4.Port mismatch - Service port doesn't match webhook config
5. 5.Network policy blocking - NetworkPolicy prevents webhook traffic
6. 6.Resource limits - Kyverno pod OOMKilled or CPU throttled

Step 1: Check Kyverno Controller Status

```bash # Check Kyverno deployment: kubectl get deploy -n kyverno

# Check pod status: kubectl get pods -n kyverno

# Check pod details: kubectl describe pods -n kyverno

# Check pod logs: kubectl logs -n kyverno deploy/kyverno -c kyverno kubectl logs -n kyverno deploy/kyverno -c kyverno --previous

# Check all containers: kubectl logs -n kyverno deploy/kyverno --all-containers

# Check pod resource usage: kubectl top pods -n kyverno

# Check events: kubectl get events -n kyverno --sort-by='.lastTimestamp'

# Check for OOMKilled: kubectl describe pods -n kyverno | grep -i oom

# If pod crashed, check previous logs: kubectl logs -n kyverno deploy/kyverno --previous ```

Step 2: Check Webhook Service

```bash # Check service: kubectl get svc -n kyverno

# Check service details: kubectl describe svc kyverno-svc -n kyverno

# Check service port: kubectl get svc kyverno-svc -n kyverno -o jsonpath='{.spec.ports[0].port}' # Should be 443 for webhook

# Check target port: kubectl get svc kyverno-svc -n kyverno -o jsonpath='{.spec.ports[0].targetPort}' # Should match container port

# Check endpoints: kubectl get endpoints kyverno-svc -n kyverno

# Check endpoint addresses: kubectl get endpoints kyverno-svc -n kyverno -o yaml

# If no endpoints: # Pod not ready or selector mismatch

# Check pod labels match service selector: kubectl get svc kyverno-svc -n kyverno -o jsonpath='{.spec.selector}' kubectl get pods -n kyverno --show-labels

# Test service DNS: kubectl run test-dns --image=busybox --rm -it --restart=Never \ -- nslookup kyverno-svc.kyverno.svc.cluster.local

# Test connectivity: kubectl run test-curl --image=curlimages/curl --rm -it --restart=Never \ -- curl -k https://kyverno-svc.kyverno.svc:443/validate ```

Step 3: Verify TLS Configuration

```bash # Check webhook CA bundle: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg -o yaml

# Look for CABundle in clientConfig: # clientConfig: # caBundle: <base64-encoded-cert>

# Check certificate expiration: kubectl get secret -n kyverno

# Kyverno generates its own certs or uses provided ones # Check secret: kubectl get secret kyverno-svc.kyverno.svc.kyverno-tls-pair -n kyverno -o yaml

# Decode and check cert: kubectl get secret kyverno-svc.kyverno.svc.kyverno-tls-pair -n kyverno \ -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

# Check cert expiration: kubectl get secret kyverno-svc.kyverno.svc.kyverno-tls-pair -n kyverno \ -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

# Verify webhook uses correct CA: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg \ -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -text -noout

# Test TLS connection: kubectl run test-tls --image=curlimages/curl --rm -it --restart=Never \ -- curl -vk https://kyverno-svc.kyverno.svc:443/validate ```

Step 4: Check Webhook Configuration

```bash # Get webhook config: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg -o yaml

# Check service reference: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg \ -o jsonpath='{.webhooks[0].clientConfig.service}'

# Verify: # - name: kyverno-svc # - namespace: kyverno # - port: 443

# Check webhook path: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg \ -o jsonpath='{.webhooks[0].clientConfig.service.path}' # Should be /validate

# Check timeout: kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg \ -o jsonpath='{.webhooks[0].timeoutSeconds}' # Default 10s

# Check mutating webhook similarly: kubectl get mutatingwebhookconfiguration kyverno-mutating-webhook-cfg -o yaml

# If port or path mismatch, patch webhook: kubectl patch validatingwebhookconfiguration kyverno-validating-webhook-cfg \ --type='json' -p='[{"op": "replace", "path": "/webhooks/0/clientConfig/service/port", "value": 443}]' ```

Step 5: Check Network Policies

```bash # Check if network policies block webhook traffic: kubectl get networkpolicy -n kyverno

# Check policy details: kubectl describe networkpolicy -n kyverno

# Kyverno webhook traffic comes from kube-apiserver # Check if policy allows ingress from control plane

# If policy exists, verify it allows: # - Ingress from kube-system namespace # - Port 443

# Check namespace labels for policy matching: kubectl get namespace kube-system --show-labels

# Test by creating temporary allow policy: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-webhook namespace: kyverno spec: podSelector: {} policyTypes: - Ingress ingress: - from: - namespaceSelector: {} ports: - port: 443

# Or temporarily remove network policy: kubectl delete networkpolicy -n kyverno <policy-name> ```

Step 6: Check Resource Limits

```bash # Check Kyverno pod resources: kubectl describe deploy kyverno -n kyverno | grep -A10 "Containers:" | grep -A5 "Resources:"

# Check current resource usage: kubectl top pods -n kyverno

# Check limits: kubectl get deploy kyverno -n kyverno -o jsonpath='{.spec.template.spec.containers[0].resources}'

# If memory limit too low, increase: kubectl patch deploy kyverno -n kyverno --type='json' \ -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "512Mi"}]'

# Check for OOMKilled events: kubectl describe pods -n kyverno | grep -i "OOMKilled"

# Check pod restart count: kubectl get pods -n kyverno -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}'

# If high restart count, check logs: kubectl logs -n kyverno deploy/kyverno --previous ```

Step 7: Restart Kyverno

```bash # Restart Kyverno deployment: kubectl rollout restart deploy/kyverno -n kyverno

# Watch rollout: kubectl rollout status deploy/kyverno -n kyverno

# Check new pods: kubectl get pods -n kyverno -w

# Verify webhook recreated: kubectl get validatingwebhookconfiguration kubectl get mutatingwebhookconfiguration

# Test webhook: kubectl apply -f test-deployment.yaml --dry-run=server

# If still failing, delete and reinstall: kubectl delete deploy kyverno -n kyverno kubectl apply -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/install.yaml

# Or use Helm: helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --create-namespace ```

Step 8: Check API Server Connectivity

```bash # Check if API server can reach webhook service:

# Get API server pod (control plane): kubectl get pods -n kube-system | grep kube-apiserver

# Check API server logs: kubectl logs -n kube-system kube-apiserver-<node> | grep kyverno

# Test webhook from inside cluster: kubectl run test-webhook --image=curlimages/curl --rm -it --restart=Never \ -- curl -k -X POST https://kyverno-svc.kyverno.svc:443/validate \ -H "Content-Type: application/json" \ -d '{"kind":"Deployment","apiVersion":"apps/v1"}'

# Check if API server uses correct service CIDR: kubectl get svc -A | grep kyverno

# Verify webhook service IP: kubectl get svc kyverno-svc -n kyverno -o jsonpath='{.spec.clusterIP}' ```

Step 9: Debug with Verbose Logging

```bash # Enable verbose logging in Kyverno:

# Check current log level: kubectl get deploy kyverno -n kyverno -o yaml | grep -i log

# Set log level via environment variable: kubectl set env deploy/kyverno -n kyverno LOG_LEVEL=debug

# Or via Helm: helm upgrade kyverno kyverno/kyverno --set logLevel=debug

# Watch logs: kubectl logs -n kyverno deploy/kyverno -c kyverno -f

# Check webhook handler logs: kubectl logs -n kyverno deploy/kyverno | grep -i "webhook|validate|mutate"

# Look for errors: kubectl logs -n kyverno deploy/kyverno | grep -i "error|failed"

# Check admission requests: kubectl logs -n kyverno deploy/kyverno | grep -i "admissionReview" ```

Step 10: Kyverno Admission Controller Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-kyverno-webhook.sh #!/bin/bash

echo "=== Kyverno Pods ===" kubectl get pods -n kyverno -o wide

echo "" echo "=== Pod Status Details ===" kubectl describe pods -n kyverno | grep -A5 "Status:"

echo "" echo "=== Webhook Service ===" kubectl get svc kyverno-svc -n kyverno

echo "" echo "=== Service Endpoints ===" kubectl get endpoints kyverno-svc -n kyverno

echo "" echo "=== Validating Webhook ===" kubectl get validatingwebhookconfiguration kyverno-validating-webhook-cfg -o yaml | grep -A10 "clientConfig"

echo "" echo "=== TLS Certificate ===" kubectl get secret -n kyverno | grep tls

echo "" echo "=== Certificate Expiration ===" kubectl get secret kyverno-svc.kyverno.svc.kyverno-tls-pair -n kyverno \ -o jsonpath='{.data.tls\.crt}' 2>/dev/null | base64 -d 2>/dev/null | openssl x509 -enddate -noout 2>/dev/null || echo "Cert not found"

echo "" echo "=== Network Policies ===" kubectl get networkpolicy -n kyverno

echo "" echo "=== Resource Usage ===" kubectl top pods -n kyverno 2>/dev/null || echo "Metrics not available"

echo "" echo "=== Recent Events ===" kubectl get events -n kyverno --sort-by='.lastTimestamp' | tail -10

echo "" echo "=== Test Webhook ===" kubectl run test --image=nginx --dry-run=client -o yaml | kubectl apply --dry-run=server -f - 2>&1 | head -5

echo "" echo "=== Recommendations ===" echo "1. Ensure Kyverno pods are running and ready" echo "2. Verify webhook service has endpoints" echo "3. Check TLS certificate is valid and trusted" echo "4. Verify webhook service port is 443" echo "5. Check network policies allow webhook traffic" echo "6. Increase resource limits if OOMKilled" echo "7. Restart Kyverno if pods unhealthy" EOF

chmod +x /usr/local/bin/check-kyverno-webhook.sh

# Usage: /usr/local/bin/check-kyverno-webhook.sh ```

Kyverno Webhook Checklist

Verify the Fix

```bash # After fixing Kyverno admission controller

# 1. Check pods running kubectl get pods -n kyverno // All pods Running and Ready

# 2. Check service endpoints kubectl get endpoints kyverno-svc -n kyverno // Has endpoint addresses

# 3. Test webhook kubectl apply -f deployment.yaml --dry-run=server // No webhook error

# 4. Apply resource kubectl apply -f deployment.yaml // Created/Updated successfully

# 5. Check Kyverno logs kubectl logs -n kyverno deploy/kyverno --tail=10 // No errors

# 6. Verify policy applied kubectl describe deploy myapp // Kyverno annotations present ```

Related Issues

• [Fix Kubernetes Admission Webhook Failed](/articles/fix-kubernetes-admission-webhook-failed)
• [Fix Kyverno Policy Not Enforcing](/articles/fix-kyverno-policy-not-enforcing)
• [Fix Istio Sidecar Injection Disabled](/articles/fix-istio-sidecar-injection-disabled)