Introduction

The Vault transit secrets engine handles encryption and decryption operations without storing the actual data. Each encryption key has multiple versions, and ciphertext includes the key version used. When keys are rotated and old versions are deleted or archived, data encrypted with those versions becomes undecryptable, causing application failures.

Symptoms

  • Decryption requests fail with unable to decrypt or key version not found
  • Application cannot read previously encrypted data after key rotation
  • Vault logs show no decryption key for version X
  • Some records decrypt successfully (recent ones) while older records fail
  • Error message: Error decrypting: ciphertext version not found in key versions

Common Causes

  • Key rotation configured with min_decryption_version set higher than the version used for old ciphertext
  • Key versions deleted manually or by automated cleanup policy
  • Data encrypted before the key existed in Vault (migrated from another encryption system)
  • Ciphertext tampered or corrupted, causing version prefix to be unreadable
  • Application using stale key version reference after rotation

Step-by-Step Fix

  1. 1.Check the current key versions and minimum decryption version: Inspect the key configuration.
  2. 2.```bash
  3. 3.vault read transit/keys/my-key
  4. 4.# Check:
  5. 5.# keys: map of versions to timestamps
  6. 6.# min_decryption_version: minimum allowed version
  7. 7.# min_encryption_version: minimum version for new encryption
  8. 8.`
  9. 9.Restore the missing key version: If the version was archived, restore it.
  10. 10.```bash
  11. 11.# If using Vault Enterprise with key version archival
  12. 12.vault write -f transit/keys/my-key/restore-version/version=3
  13. 13.`
  14. 14.Lower the min_decryption_version if the version exists: Allow older versions to decrypt.
  15. 15.```bash
  16. 16.vault write transit/keys/my-key/config min_decryption_version=1
  17. 17.`
  18. 18.Re-encrypt data with the current key version: Migrate old ciphertext to the current version.
  19. 19.```bash
  20. 20.# Rewrap old ciphertext to the latest key version
  21. 21.vault write transit/rewrap/my-key ciphertext="vault:v1:base64ciphertext"
  22. 22.`
  23. 23.Verify decryption works for both old and new data: Test end-to-end.
  24. 24.```bash
  25. 25.vault write transit/decrypt/my-key ciphertext="vault:v3:base64ciphertext"
  26. 26.`

Prevention

  • Never delete key versions that are still needed for decrypting existing data
  • Set min_decryption_version=1 unless there is a specific compliance requirement
  • Implement a data migration plan before rotating transit keys to rewrap existing ciphertext
  • Monitor transit engine decryption failure rates and correlate with key rotation events
  • Use convergent_encryption=true for searchable encryption with consistent ciphertext
  • Document key rotation procedures including rewrap steps for all dependent data stores