Introduction

Vault Agent's auto-auth feature can use Kubernetes service account tokens to authenticate with Vault. When the service account token is missing from the expected path, has been rotated, or the service account itself was deleted, Vault Agent cannot authenticate. This blocks applications from receiving their Vault-managed secrets.

Symptoms

  • Vault Agent logs show service account token not found or authentication failed
  • Application sidecar cannot retrieve secrets from Vault
  • Vault audit logs show failed login attempts from the Kubernetes auth method
  • Pod starts but cannot access Vault-managed configuration
  • Error message: Error making API request: Post "https://vault:8200/v1/auth/kubernetes/login": token not found

Common Causes

  • Service account deleted or renamed in Kubernetes
  • Token file path changed in Kubernetes 1.24+ (no longer auto-mounted by default)
  • Vault Kubernetes auth role not configured for the service account name
  • Service account token expired and not refreshed (Kubernetes 1.21+ token expiration)
  • Vault Kubernetes auth method JWT reviewer not matching the current cluster

Step-by-Step Fix

  1. 1.Verify the service account exists and token is mounted: Check the pod's service account.
  2. 2.```bash
  3. 3.kubectl get sa my-app-sa -o jsonpath='{.secrets}'
  4. 4.# Check if token is mounted in the pod
  5. 5.kubectl exec my-app-pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
  6. 6.`
  7. 7.For Kubernetes 1.24+, create a token manually: Use TokenRequest API.
  8. 8.```yaml
  9. 9.# Update pod spec to request a token
  10. 10.apiVersion: v1
  11. 11.kind: Pod
  12. 12.metadata:
  13. 13.name: my-app
  14. 14.spec:
  15. 15.serviceAccountName: my-app-sa
  16. 16.containers:
  17. 17.- name: vault-agent
  18. 18.image: hashicorp/vault:latest
  19. 19.volumeMounts:
  20. 20.- name: vault-agent-config
  21. 21.mountPath: /etc/vault-agent
  22. 22.- name: token-volume
  23. 23.mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  24. 24.volumes:
  25. 25.- name: token-volume
  26. 26.projected:
  27. 27.sources:
  28. 28.- serviceAccountToken:
  29. 29.path: token
  30. 30.expirationSeconds: 3600
  31. 31.`
  32. 32.Verify Vault Kubernetes auth role configuration: Ensure the role matches the service account.
  33. 33.```bash
  34. 34.vault read auth/kubernetes/role/my-app
  35. 35.# Check bound_service_account_names and bound_service_account_namespaces
  36. 36.`
  37. 37.Update the Vault Agent auto-auth configuration: Point to the correct token path.
  38. 38.```hcl
  39. 39.auto_auth {
  40. 40.method "kubernetes" {
  41. 41.mount_path = "auth/kubernetes"
  42. 42.config = {
  43. 43.role = "my-app"
  44. 44.token_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  45. 45.}
  46. 46.}
  47. 47.sink "file" {
  48. 48.config = {
  49. 49.path = "/home/vault/.vault-token"
  50. 50.}
  51. 51.}
  52. 52.}
  53. 53.`
  54. 54.Restart the Vault Agent sidecar: Apply the configuration changes.
  55. 55.```bash
  56. 56.kubectl rollout restart deployment/my-app
  57. 57.kubectl logs -l app=my-app -c vault-agent -f
  58. 58.`

Prevention

  • Use Kubernetes projected service account tokens with configurable expiration for Vault Agent
  • Include Vault Agent authentication in pod readiness probes
  • Monitor Vault Kubernetes auth method failure rates and alert on sustained errors
  • Document the service account requirements for each application's Vault access
  • Test Vault Agent authentication after any Kubernetes version upgrade
  • Configure Vault Kubernetes auth with appropriate token_bound_cidrs for network-level security