Introduction

Vault tokens have both a TTL (time-to-live) and a max_ttl (maximum total lifetime). While tokens can be renewed before their TTL expires, the total lifetime cannot exceed the max_ttl. When an application repeatedly renews a token and eventually hits the max_ttl, further renewal attempts are rejected, and the token becomes invalid.

Symptoms

  • Application receives 400 Bad Request when attempting to renew a token
  • Vault logs show token max TTL reached or cannot renew token
  • Application authentication fails as the token is no longer valid
  • Token renewal worked for days or weeks before suddenly failing
  • Error message: Error renewing token: max TTL exceeded

Common Causes

  • Token created with max_ttl that is too short for the application's expected lifetime
  • Long-running application continuously renewing a single token without rotation
  • Auth method configuration setting a low max_ttl for the token role
  • Token created with ttl=0 (no expiry) but max_ttl still enforced
  • Periodic tokens (with period parameter) not configured, causing max TTL accumulation

Step-by-Step Fix

  1. 1.Check the token's current TTL and max TTL: Inspect the token state.
  2. 2.```bash
  3. 3.vault token lookup
  4. 4.# Check:
  5. 5.# ttl: remaining seconds
  6. 6.# max_ttl: maximum total lifetime
  7. 7.# policies: attached policies
  8. 8.`
  9. 9.Authenticate with a fresh token: Obtain a new token from the auth method.
  10. 10.```bash
  11. 11.# For Kubernetes auth
  12. 12.vault write auth/kubernetes/login role=my-app jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  13. 13.# For AppRole auth
  14. 14.vault write auth/approle/login role_id=<role-id> secret_id=<secret-id>
  15. 15.`
  16. 16.Update the token role to allow longer max TTL: If the current max TTL is too restrictive.
  17. 17.```bash
  18. 18.vault write auth/kubernetes/role/my-app \
  19. 19.bound_service_account_names=my-app-sa \
  20. 20.bound_service_account_namespaces=default \
  21. 21.policies=my-app-policy \
  22. 22.max_ttl=768h # 32 days
  23. 23.`
  24. 24.Use periodic tokens for long-running services: Prevent max TTL accumulation.
  25. 25.```bash
  26. 26.vault token create -period=24h -policy=my-app-policy
  27. 27.# Periodic tokens renew indefinitely without hitting max TTL
  28. 28.`
  29. 29.Implement token rotation in the application: Automatically obtain new tokens before expiry.
  30. 30.```java
  31. 31.// Vault SDK: re-authenticate when token TTL is low
  32. 32.if (tokenInfo.getTtl() < 3600) { // Less than 1 hour remaining
  33. 33.VaultAuthenticationResponse newToken = vault.auth.login(authMethod);
  34. 34.vault.setToken(newToken.getClientToken());
  35. 35.}
  36. 36.`

Prevention

  • Use periodic tokens (period parameter) for long-running services that need continuous Vault access
  • Set max_ttl to at least 7 days for production applications
  • Implement automatic token rotation in applications before the token reaches 80% of its lifetime
  • Monitor token renewal failure rates and alert on max TTL rejections
  • Use Vault Agent with auto_auth for automatic token management and renewal
  • Configure appropriate max_ttl per auth method role based on the service's expected lifetime