What's Actually Happening

A Kubernetes Service provides a stable endpoint to access pods. When the Service is not accessible, requests to the Service IP or DNS name fail. This happens when the Service has no endpoints (no matching pods), pods aren't ready, or network blocks traffic.

The Error You'll See

```bash $ kubectl get svc my-service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-service ClusterIP 10.96.100.100 <none> 80/TCP 5m

$ curl 10.96.100.100 curl: (7) Failed to connect to 10.96.100.100 port 80: Connection refused

# Or from inside cluster: $ kubectl run test --image=busybox --rm -it -- wget -qO- my-service:80 wget: can't connect to remote host (10.96.100.100): Connection refused ```

Why This Happens

  1. 1.No matching pods - Service selector doesn't match pod labels
  2. 2.Pods not ready - Readiness probe failing, endpoints not populated
  3. 3.Service port mismatch - Service port doesn't match container port
  4. 4.Network policy blocking - NetworkPolicy denying traffic
  5. 5.DNS issues - CoreDNS not resolving service name
  6. 6.kube-proxy issues - iptables/IPVS rules not set up

Step 1: Check Service Endpoints

bash
kubectl get endpoints my-service

Shows pods backing the service:

bash
NAME         ENDPOINTS                       AGE
my-service   10.244.1.5:8080,10.244.2.3:8080  5m

If ENDPOINTS is empty:

bash
NAME         ENDPOINTS   AGE
my-service   <none>      5m

This means no pods match the Service.

Step 2: Check Selector Match

bash
# Get service selector
kubectl get svc my-service -o yaml | grep -A5 selector

Shows:

yaml
selector:
  app: my-app
  version: v1

Now check pod labels:

bash
kubectl get pods --show-labels | grep my-app

Pods must have matching labels:

bash
my-app-abc123-x1y2  1/1  Running  app=my-app,version=v1,pod-template-hash=abc123

If labels don't match selector, fix either:

```bash # Fix pod labels (in deployment) metadata: labels: app: my-app version: v1

# Or fix service selector selector: app: my-app ```

Step 3: Check Pod Readiness

Pods must pass readiness probe to be in endpoints:

bash
kubectl describe pod my-app-abc123-x1y2 | grep -A10 "Readiness:"

If readiness probe fails:

bash
Readiness:      http-get http://:8080/health delay=5s timeout=1s period=10s #success=1 #failure=3
Events:
  Warning  Unhealthy  2m  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500

Pod won't be in endpoints until readiness passes.

Fix readiness probe:

yaml
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10  # Give app time to start
  periodSeconds: 5
  failureThreshold: 3

Step 4: Check Port Configuration

bash
kubectl get svc my-service -o yaml | grep -A10 "ports:"

Service ports must match container:

yaml
ports:
- port: 80        # Service listens on 80
  targetPort: 8080 # Sends to pod port 8080

Check pod container port:

bash
kubectl get pod my-app-abc123-x1y2 -o yaml | grep -A5 "ports:"

Should show:

yaml
ports:
- containerPort: 8080  # Must match targetPort

Step 5: Test Direct Pod Access

Bypass service and test pod directly:

```bash # Get pod IP kubectl get pod my-app-abc123-x1y2 -o wide

# Pod IP: 10.244.1.5 # Test direct kubectl run test --image=busybox --rm -it -- wget -qO- 10.244.1.5:8080 ```

If direct pod access works but service fails, service config is wrong. If direct pod access fails, app or pod has issues.

Step 6: Check Network Policies

bash
kubectl get networkpolicy

NetworkPolicy might block traffic:

yaml
# Example policy that blocks ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Ingress
  ingress: []  # Empty = no ingress allowed

Fix by allowing traffic:

yaml
ingress:
- from:
  - podSelector: {}  # Allow from any pod in namespace
  ports:
  - port: 8080

Step 7: Check DNS Resolution

bash
kubectl run test --image=busybox --rm -it -- nslookup my-service

Should resolve to ClusterIP:

``` Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: my-service Address 1: 10.96.100.100 my-service.default.svc.cluster.local ```

If DNS fails:

bash
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system coredns-xxx

Step 8: Check kube-proxy

bash
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system kube-proxy-xxx

kube-proxy sets up iptables/IPVS rules. If it's failing, services won't route.

Step 9: Test from Inside Cluster

Create test pod in same namespace:

bash
kubectl run test --image=busybox --rm -it --restart=Never -- sh
# Inside pod:
wget -qO- my-service:80
wget -qO- my-service.default.svc.cluster.local:80

Verify the Fix

After fixing:

```bash kubectl get endpoints my-service # Should show pod IPs

kubectl run test --image=busybox --rm -it -- wget -qO- my-service:80 # Should return app response ```

Prevention Tips

When creating services:

```bash # Always verify selector matches pod labels kubectl get pods --show-labels kubectl get svc my-service -o yaml | grep selector

# Use named ports for clarity ports: - name: http port: 80 targetPort: http # References container port name

# Set proper readiness probes readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 10 ```