What's Actually Happening

Kubernetes clusters created with kubeadm have certificates that expire after 365 days by default. When certificates expire, kubelet cannot authenticate with API server, kubectl commands fail, and the cluster becomes inaccessible.

The Error You'll See

```bash $ kubectl get nodes Unable to connect to the server: x509: certificate has expired or is not yet valid

# On kubelet logs: E0310 10:00:00.123456 1234 reflector.go:123] "Failed to watch" err="failed to list: Get https://10.96.0.1:443/api/v1/nodes: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA signature verification failure\" while trying to verify candidate authority certificate \"kubernetes\")" ```

Why This Happens

  1. 1.Certificates expired - Default 1-year expiration reached
  2. 2.Time skew - Node clocks wrong, certs appear invalid
  3. 3.Wrong CA - Client using wrong certificate authority
  4. 4.Renewal not automated - Manual renewal not performed

Step 1: Check Certificate Expiration

bash
# Check all kubeadm certs
kubeadm certs check-expiration

Output shows:

``` CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Apr 03, 2027 10:00 UTC 365d ca no apiserver Apr 03, 2027 10:00 UTC 365d ca no apiserver-etcd-client Apr 03, 2027 10:00 UTC 365d etcd-ca no apiserver-kubelet-client Apr 03, 2027 10:00 UTC 365d ca no controller-manager.conf Apr 03, 2027 10:00 UTC 365d ca no etcd-healthcheck-client Apr 03, 2027 10:00 UTC 365d etcd-ca no etcd-peer Apr 03, 2027 10:00 UTC 365d etcd-ca no etcd-server Apr 03, 2027 10:00 UTC 365d etcd-ca no front-proxy-client Apr 03, 2027 10:00 UTC 365d front-proxy-ca no kubelet.conf Apr 03, 2027 10:00 UTC 365d ca no scheduler.conf Apr 03, 2027 10:00 UTC 365d ca no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Apr 01, 2036 10:00 UTC 10y no etcd-ca Apr 01, 2036 10:00 UTC 10y no front-proxy-ca Apr 01, 2036 10:00 UTC 10y no ```

If any shows 0d or negative, cert is expired.

Step 2: Renew Certificates on Control Plane

bash
# Renew all certificates
kubeadm certs renew all

Output:

bash
[certs] Renewing "apiserver" certificate
[certs] Renewing "apiserver-kubelet-client" certificate
[certs] Renewing "front-proxy-client" certificate
[certs] Renewing "apiserver-etcd-client" certificate
[certs] Renewing "etcd-server" certificate
[certs] Renewing "etcd-peer" certificate
[certs] Renewing "etcd-healthcheck-client" certificate
[certs] Renewing "admin.conf" certificate
[certs] Renewing "kubelet.conf" certificate
[certs] Renewing "controller-manager.conf" certificate
[certs] Renewing "scheduler.conf" certificate

Step 3: Restart Control Plane Components

After renewal, restart components:

```bash # Restart kubelet systemctl restart kubelet

# Restart static pods (kubeadm manages these) # They will auto-restart when config changes ```

For manual restart of static pods:

bash
# Move manifests to trigger restart
mv /etc/kubernetes/manifests/*.yaml /tmp/
sleep 10
mv /tmp/*.yaml /etc/kubernetes/manifests/

Step 4: Update Admin kubeconfig

```bash # Generate new admin config kubeadm init phase kubeconfig admin --config=/etc/kubernetes/kubeadm-config.yaml

# Copy to your kubectl config cp /etc/kubernetes/admin.conf ~/.kube/config

# Or merge kubectl config set-cluster default-cluster --server=https://10.96.0.1:443 --certificate-authority=/etc/kubernetes/pki/ca.crt kubectl config set-credentials default-admin --client-certificate=/etc/kubernetes/pki/admin.crt --client-key=/etc/kubernetes/pki/admin.key ```

Step 5: Renew Worker Node Certificates

On each worker node:

```bash ssh node-2

# Generate new kubelet certs (must be done on control plane) # On control plane: kubeadm kubeconfig user --org system:nodes --client-name system:node:node-2 > node-2.conf

# Copy to worker scp node-2.conf node-2:/etc/kubernetes/kubelet.conf

# On worker, restart kubelet ssh node-2 systemctl restart kubelet ```

Or use automatic kubelet cert rotation (Kubernetes v1.19+):

bash
# Check if rotation is enabled
kubelet --version | grep -i rotation
# kubelet.conf should have rotateCertificates: true

Step 6: Verify Certificate Renewal

```bash kubeadm certs check-expiration

# Should show 365d for all kubectl get nodes # Should work now ```

Step 7: Enable Automatic Certificate Rotation

To prevent future expirations:

yaml
# In kubelet config
rotateCertificates: true

For kubeadm clusters, kubelet cert rotation is automatic. For API server certs:

bash
# Check for cert rotation controller
kubectl get pods -n kube-system | grep cert

Or set up a cron job for renewal:

bash
# Add to crontab on control plane
# Monthly check and renew if needed
0 0 1 * * kubeadm certs check-expiration | grep -q "0d" && kubeadm certs renew all && systemctl restart kubelet

Step 8: Handle Already Expired Cluster

If cluster is completely expired and inaccessible:

```bash # SSH to control plane ssh node-1

# Renew certs without API server access kubeadm certs renew all

# Restart kubelet systemctl restart kubelet

# Wait for API server to start kubectl get nodes --kubeconfig /etc/kubernetes/admin.conf ```

If this fails, may need to regenerate from scratch.

Step 9: Check Certificate Authority

If client certs are valid but still failing:

```bash # Verify CA cert openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver.crt

# Check client cert chain openssl x509 -in ~/.kube/config -text -noout | grep Issuer ```

Verify the Fix

```bash # Check cluster access kubectl get nodes kubectl get pods -A

# Verify cert expiration kubeadm certs check-expiration

# Test from worker node ssh node-2 kubectl get nodes --kubeconfig /etc/kubernetes/kubelet.conf ```

Prevention Tips

Set up monitoring for cert expiration:

```bash # Prometheus alert for cert expiration # Alert: KubernetesCertificateExpirySoon # Expression: (kubelet_certificate_expiration_timestamp_seconds - time()) / 86400 < 30

# Regular renewal before expiration # Run renewal when certs have < 90 days left kubeadm certs renew all ```