Introduction

A cluster migration can move an admission controller to the new service while the Kubernetes API server still posts admission requests to the old one. Deployments fail only on certain clusters, one webhook rule uses the new backend while another still calls the previous service, or resource creation breaks after the old controller is removed because ValidatingWebhookConfiguration, MutatingWebhookConfiguration, service references, and CA bundles often drift apart.

Treat this as an admission-routing problem instead of a generic Kubernetes outage. Start by checking which webhook endpoint the API server actually calls for an affected resource operation, because migrations often validate the new controller pod itself while cluster admission policy still references the older service or namespace.

Symptoms

  • A Kubernetes admission webhook still calls the old service after migration
  • Resource creation or updates fail only after the previous webhook backend is removed
  • One cluster, namespace, or webhook rule uses the new service while another still uses the old one
  • API server errors reference the retired service name, namespace, or certificate
  • The new admission controller is healthy, but admission requests never reach it
  • The issue started after moving admission controllers, namespaces, or cluster policy components

Common Causes

  • ValidatingWebhookConfiguration or MutatingWebhookConfiguration still references the old service name or namespace
  • CA bundles still trust the previous service certificate or issuer
  • Multiple webhook entries were updated inconsistently across clusters or environments
  • GitOps, Helm, or platform automation keeps reapplying the old webhook manifest
  • API server policy still includes the old URL-based webhook target
  • Validation confirmed the new controller pods were running but did not verify where live admission requests were actually posted

Step-by-Step Fix

  1. Capture one failed or affected admission request and record the webhook configuration, target service, namespace, and certificate path it actually uses, because the live admission target determines where the API server sends policy checks.
  2. Compare that active webhook path with the intended post-migration cluster design, because one stale webhook object can keep all create or update requests tied to the retired controller.
  3. Review ValidatingWebhookConfiguration, MutatingWebhookConfiguration, service definitions, endpoints, CA bundles, and GitOps manifests for references to the old backend, because Kubernetes admission depends on policy objects and trust configuration together.
  4. Check each cluster, namespace, and webhook rule separately if behavior differs, because migrations often fix one admission path while another still calls the previous service.
  5. Update the authoritative webhook configuration so affected admission requests target the intended service and certificate chain, because deploying the new controller alone does not retarget existing API server policy.
  6. Apply a controlled resource change and confirm the intended admission service receives and processes the request, because healthy pods do not prove the API server is calling the right backend.
  7. Verify the old webhook service no longer receives admission traffic from migrated clusters, because split control-plane paths can remain hidden while both services stay reachable.
  8. Review failure policies, CABundle contents, and service DNS if requests still fail, because the destination can be correct while TLS trust or service discovery still blocks admission.
  9. Document which team owns webhook manifests, CA rotation, and cluster validation so future controller migrations verify the actual admission endpoint before retiring the previous service.