Introduction

A secrets-platform migration can bring a new Vault cluster online while Vault Agent still authenticates and renders templates from the old one. Applications continue starting, but they receive stale secrets, one node renders from the new cluster while another still follows the previous API address, or failures begin only after the old cluster is restricted because auto-auth config, discovery settings, and local agent templates often drift separately.

Treat this as a secret-source problem instead of a generic Vault outage. Start by checking which Vault address an affected agent actually authenticates to and which secret path its rendered template uses, because migrations often validate the new cluster from the CLI while live workloads continue following older agent configuration.

Symptoms

  • Vault Agent still renders secrets from the old cluster after migration
  • Applications start, but they receive stale or unexpected secret values
  • One node uses the new Vault cluster while another still follows the previous API address
  • Failures begin only after the old cluster, auth method, or certificate is removed
  • The new cluster is healthy, but migrated workloads never render secrets from it
  • The issue started after moving Vault clusters, auth backends, or service-discovery endpoints

Common Causes

  • The agent config still points auto-auth or template rendering to the old Vault address
  • Service discovery, DNS, or sidecar bootstrap still resolves the retired cluster first
  • Template config still reads the previous secret path or namespace after the cluster move
  • One deployment, node pool, or sidecar injector path was updated while another still uses the earlier config
  • Token sink files or bootstrap artifacts were recreated from older configuration data
  • Validation confirmed the new cluster served secrets manually but did not verify which cluster live agents actually authenticated to

Step-by-Step Fix

  1. Capture one affected workload and record the agent config source, Vault address, auth method, rendered secret path, and sink location it actually uses, because the live secret path determines which cluster and namespace really supply application credentials.
  2. Compare that active rendering path with the intended post-migration secrets design, because one stale agent address or template path can keep workloads tied to the retired cluster.
  3. Review Vault Agent config, auto-auth stanzas, template blocks, sidecar injector settings, service-discovery records, and bootstrap artifacts for references to the old cluster, because secret rendering depends on both target selection and local template behavior.
  4. Check each workload class, node pool, and deployment path separately if behavior differs, because migrations often update one injection path while another still uses the previous cluster address.
  5. Update the authoritative Vault Agent target and template configuration so affected workloads authenticate to the intended cluster and render the right secret path, because bringing the new Vault cluster online alone does not retarget existing agents.
  6. Restart or roll the affected workload and confirm the intended cluster issues the token and renders the expected secret data, because a running application does not prove the right Vault source handled startup.
  7. Verify the old cluster no longer receives auto-auth or secret-read traffic from migrated workloads, because split secret sources can remain hidden while both clusters stay reachable.
  8. Review auth backend trust, certificate chains, namespace mapping, and template permissions if rendering still fails, because the destination can be correct while Vault policy or trust still blocks the new path.
  9. Document which team owns agent injection, service discovery, and migration validation so future Vault cutovers verify the actual runtime secret source before retiring the previous cluster.