Introduction

Vault leases control the lifetime of dynamically generated secrets. When a lease expires, Vault revokes the secret -- for example, deleting a database credential. If the application does not renew the lease before expiration, the credential is revoked and the application loses database access until it obtains a new credential.

Symptoms

  • Application database connections fail with authentication failed or password expired
  • Vault audit logs show lease expired and secret revoked events
  • Application logs show sudden loss of access to Vault-managed secrets
  • Database shows the Vault-managed user has been deleted
  • Error message: FATAL: password authentication failed for user "v-token-app"

Common Causes

  • Application not implementing lease renewal logic
  • Lease TTL set too short relative to the application's renewal check interval
  • Network partition preventing the application from reaching Vault before lease expiry
  • Application restart losing track of lease IDs that needed renewal
  • Vault server clock skew causing premature lease expiration

Step-by-Step Fix

  1. 1.Check the lease status in Vault: Identify expired leases.
  2. 2.```bash
  3. 3.vault lease lookup database/creds/my-role/abcd1234
  4. 4.# Check if the lease is still valid
  5. 5.`
  6. 6.Re-create the secret by requesting a new lease: Get a fresh credential.
  7. 7.```bash
  8. 8.vault read database/creds/my-role
  9. 9.# Output:
  10. 10.# Key Value
  11. 11.# --- -----
  12. 12.# lease_id database/creds/my-role/new-lease-id
  13. 13.# lease_duration 1h
  14. 14.# username v-token-app-xyz
  15. 15.# password auto-generated-password
  16. 16.`
  17. 17.Update the application with the new credentials: Rotate the application's secret reference.
  18. 18.```bash
  19. 19.# Update Kubernetes secret
  20. 20.kubectl create secret generic db-credentials \
  21. 21.--from-literal=username=v-token-app-xyz \
  22. 22.--from-literal=password=auto-generated-password \
  23. 23.--dry-run=client -o yaml | kubectl apply -f -
  24. 24.`
  25. 25.Implement automatic lease renewal in the application: Use the Vault agent or SDK.
  26. 26.```java
  27. 27.// Using Vault SDK with automatic renewal
  28. 28.VaultTemplate vaultTemplate = new VaultTemplate(endpoint);
  29. 29.Lease lease = vaultTemplate.read("database/creds/my-role");
  30. 30.// Schedule renewal before lease expiry (at 2/3 of lease duration)
  31. 31.long renewBefore = lease.getLeaseDuration() / 3;
  32. 32.scheduler.schedule(() -> vaultTemplate.renew(lease.getLeaseId()), renewBefore, TimeUnit.SECONDS);
  33. 33.`
  34. 34.Verify the application reconnects successfully: Test database connectivity.
  35. 35.```bash
  36. 36.# Check application logs for successful connection
  37. 37.kubectl logs -l app=my-app | grep -i "database.*connected"
  38. 38.`

Prevention

  • Deploy Vault Agent sidecar with auto_auth and sink for automatic secret renewal
  • Set lease TTLs long enough to tolerate temporary Vault unavailability (minimum 1 hour)
  • Implement lease renewal in application code at 2/3 of the lease duration
  • Use Vault Agent's template feature to automatically render secrets and trigger app reload
  • Monitor lease expiration rates and alert when leases expire without renewal
  • Configure connection pooling in applications to handle credential rotation gracefully