Fix Cilium Network Policy Not Enforcing

What's Actually Happening

Cilium network policies are created but not being enforced. Traffic flows between pods that should be blocked by policy rules.

The Error You'll See

$ kubectl exec -it pod1 -- curl pod2:8080
# Should be blocked but connection succeeds

Policy not applied:

```bash $ cilium policy get

# Policy exists but shows 0% enforcement ```

Endpoint status:

```bash $ cilium endpoint list

# Endpoints show "policy not enforced" ```

Why This Happens

1. 1.Policy syntax error - Invalid policy specification
2. 2.Selector mismatch - Labels don't match pods
3. 3.Cilium agent issues - Agent not running or misconfigured
4. 4.eBPF not loaded - BPF programs not attached
5. 5.Policy disabled - Policy enforcement disabled
6. 6.Rule priority conflict - Higher priority rule allowing traffic

Step 1: Check Cilium Status

```bash # Check Cilium pods: kubectl get pods -n kube-system -l k8s-app=cilium

# Check Cilium status: cilium status

# Detailed status: cilium status --verbose

# Check Cilium CLI: cilium version

# Check Cilium config: kubectl exec -n kube-system cilium-xxx -- cilium config

# Check Cilium operator: kubectl get pods -n kube-system -l io.cilium.app.operator

# Check for errors: kubectl logs -n kube-system -l k8s-app=cilium --tail=100 ```

Step 2: Verify Policy Enforcement Enabled

```bash # Check if policy enforcement enabled: kubectl exec -n kube-system cilium-xxx -- cilium config | grep -i policy

# Should show: # EnablePolicy: true

# Enable policy enforcement: kubectl exec -n kube-system cilium-xxx -- cilium config PolicyEnforcement=default

# Or via helm values: # --set policyEnforcementMode=default

# Check endpoint policy status: cilium endpoint list | grep -i policy

# View all config: kubectl get configmap -n kube-system cilium-config -o yaml ```

Step 3: Check Network Policy Syntax

```yaml # Correct CiliumNetworkPolicy syntax:

apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: deny-all namespace: default spec: endpointSelector: matchLabels: app: myapp ingress: - fromEndpoints: - matchLabels: app: frontend toPorts: - ports: - port: "8080" protocol: TCP

# Check policy created: kubectl get ciliumnetworkpolicies -A

# Check specific policy: kubectl describe ciliumnetworkpolicy deny-all

# Validate policy: kubectl apply -f policy.yaml --dry-run=client ```

Step 4: Check Label Selectors

```bash # Check pod labels: kubectl get pods --show-labels

# Check endpoint labels in Cilium: cilium endpoint list

# Verify selector matches pods: kubectl get pods -l app=myapp

# Check if endpoint selector matches: # In policy: endpointSelector: matchLabels: app: myapp

# Must match actual pod labels

# Check endpoint details: cilium endpoint get <endpoint-id>

# Look for "labels" and "identity" sections # Verify identity matches policy selectors ```

Step 5: Debug Policy Rules

```bash # Get policy rules: cilium policy get

# Check specific policy: cilium policy get -n default deny-all

# Check policy revision: cilium policy get | grep revision

# Check derived policy: kubectl exec -n kube-system cilium-xxx -- cilium policy get

# Check policy repository: cilium policy get -o json | jq .

# Test policy simulation: cilium policy trace --src-identity <id> --dst-identity <id> --dport 80/TCP

# View rule details: cilium bpf policy get ```

Step 6: Check Endpoint Status

```bash # List endpoints: cilium endpoint list

# Get specific endpoint: cilium endpoint get <id>

# Check endpoint state: # State: ready, waiting-to-regenerate, etc.

# Check endpoint policy: cilium endpoint get <id> | grep -A50 Policy

# Check endpoint labels: cilium endpoint get <id> | grep Labels

# Check identity: cilium endpoint get <id> | grep Identity

# Regenerate endpoint: cilium endpoint regenerate <id>

# Check endpoint connectivity: cilium bpf tunnel list cilium bpf endpoint list ```

Step 7: Check Identity Management

```bash # List identities: cilium identity list

# Get specific identity: cilium identity get <id>

# Check identity labels: cilium identity get <id> | grep Labels

# Check identity allocation: kubectl exec -n kube-system cilium-xxx -- cilium identity list

# Verify identity matches policy: # Policy endpointSelector must match identity labels

# Force identity resolution: kubectl exec -n kube-system cilium-xxx -- cilium policy resolve

# Check identity cache: cilium identity list | grep -E "ID|Labels" ```

Step 8: Check eBPF Program Status

```bash # List BPF programs: cilium bpf prog list

# Check BPF maps: cilium bpf map list

# Check policy map: cilium bpf policy get

# Verify BPF loaded: kubectl exec -n kube-system cilium-xxx -- ls /sys/fs/bpf/tc/

# Check BPF attachment: kubectl exec -n kube-system cilium-xxx -- tc filter show dev cilium_host ingress

# Reload BPF: kubectl exec -n kube-system cilium-xxx -- cilium bpf reload

# Check BPF statistics: cilium bpf metrics ```

Step 9: Check Connectivity and Tracing

```bash # Run connectivity test: cilium connectivity test

# Test specific connectivity: cilium connectivity test --test no-unexpected-packet-drops

# Enable Hubble for observability: cilium hubble enable

# Port forward Hubble UI: cilium hubble ui

# Check Hubble flows: hubble observe --namespace default

# Trace specific flow: hubble observe --from-pod default/pod1 --to-pod default/pod2

# Check flow logs: kubectl logs -n kube-system -l k8s-app=hubble

# Enable debug logging: kubectl exec -n kube-system cilium-xxx -- cilium debug enable ```

Step 10: Cilium Policy Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-cilium-policy.sh #!/bin/bash

echo "=== Cilium Status ===" cilium status --verbose

echo "" echo "=== Cilium Pods ===" kubectl get pods -n kube-system -l k8s-app=cilium

echo "" echo "=== Policy Enforcement Status ===" kubectl exec -n kube-system $(kubectl get pods -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}') -- cilium config | grep -i policy

echo "" echo "=== Network Policies ===" kubectl get ciliumnetworkpolicies -A

echo "" echo "=== Endpoint List ===" cilium endpoint list | head -20

echo "" echo "=== Identity List ===" cilium identity list | head -20

echo "" echo "=== BPF Programs ===" cilium bpf prog list | head -20

echo "" echo "=== Recent Cilium Logs ===" kubectl logs -n kube-system -l k8s-app=cilium --tail=20 | grep -i policy

echo "" echo "=== Recommendations ===" echo "1. Verify policy enforcement enabled" echo "2. Check label selectors match pods" echo "3. Validate policy syntax" echo "4. Ensure eBPF programs loaded" echo "5. Check endpoint identities" echo "6. Use Hubble for flow visualization" EOF

chmod +x /usr/local/bin/check-cilium-policy.sh

# Usage: /usr/local/bin/check-cilium-policy.sh ```

Cilium Policy Checklist

Verify the Fix

```bash # After fixing policy enforcement

# 1. Check policy is enforced cilium policy get // Shows policy with rules

# 2. Test blocked traffic kubectl exec pod1 -- curl pod2:8080 // Connection refused/timeout

# 3. Test allowed traffic kubectl exec frontend -- curl myapp:8080 // Connection succeeds

# 4. Check endpoint policy cilium endpoint get <id> // Policy visible

# 5. Monitor with Hubble hubble observe --namespace default // Flows show policy verdicts

# 6. Check BPF cilium bpf policy get // Policy rules in BPF ```

Related Issues

• [Fix Istio Service Mesh Traffic Not Routing](/articles/fix-istio-service-mesh-traffic-not-routing)
• [Fix Linkerd Service Mesh mTLS Error](/articles/fix-linkerd-service-mesh-mtls-error)
• [Fix Kubernetes Network Policy Not Working](/articles/fix-kubernetes-network-policy-not-working)