Introduction

Vault's KV version 2 secrets engine uses a different path structure than version 1. All read and write operations must include the /data/ prefix in the path (e.g., secret/data/my-app instead of secret/my-app). Requests using the v1 path format return 404 errors or incorrect data, causing applications to fail when retrieving secrets.

Symptoms

  • vault kv get secret/my-app/config returns No value found at secret/my-app/config
  • API requests to v1/secret/my-app return 404 Not Found
  • CLI works with vault kv get but direct API calls fail
  • Application secret retrieval fails after migrating from KV v1 to v2
  • Error message: Error reading secret/data/my-app/config: no handler for route

Common Causes

  • Application code using v1 API paths with a v2 KV mount
  • Terraform provider using incorrect path prefix for KV v2 secrets
  • Documentation or scripts referencing v1 paths after mount migration
  • Confusion between the CLI vault kv get (which auto-adds /data/) and raw API calls
  • Multiple KV mounts (v1 and v2) causing path ambiguity

Step-by-Step Fix

  1. 1.Verify the KV engine version on the mount: Check if it is v1 or v2.
  2. 2.```bash
  3. 3.vault secrets list -detailed | grep secret
  4. 4.# v2 mount shows: options map[version:2]
  5. 5.`
  6. 6.Use the correct path with /data/ prefix for API calls: Fix the API path.
  7. 7.```bash
  8. 8.# WRONG: v1 style path
  9. 9.curl -H "X-Vault-Token: $VAULT_TOKEN" http://vault:8200/v1/secret/my-app/config

# CORRECT: v2 style path curl -H "X-Vault-Token: $VAULT_TOKEN" http://vault:8200/v1/secret/data/my-app/config ```

  1. 1.For CLI, use vault kv commands which handle the prefix automatically: Use the correct CLI syntax.
  2. 2.```bash
  3. 3.vault kv get secret/my-app/config
  4. 4.vault kv put secret/my-app/config username=admin password=secret
  5. 5.`
  6. 6.Update application code to use v2 paths: Fix hardcoded paths.
  7. 7.```go
  8. 8.// WRONG: v1 path
  9. 9.secret, err := client.Logical().Read("secret/my-app/config")

// CORRECT: v2 path secret, err := client.Logical().Read("secret/data/my-app/config") // Data is nested under .Data.Data in the response password := secret.Data["data"].(map[string]interface{})["password"] ```

  1. 1.Verify secret access works with the corrected path: Test end-to-end.
  2. 2.```bash
  3. 3.vault kv get secret/my-app/config
  4. 4.# Should show the secret data
  5. 5.`

Prevention

  • Document KV engine version (v1 or v2) for each secrets mount in a central registry
  • Use vault kv CLI commands in scripts which automatically handle path prefixes
  • Add path prefix validation in CI/CD pipelines that deploy Vault-related code
  • When migrating from v1 to v2, update all application code paths before switching the mount
  • Use Vault provider in Terraform which abstracts the path prefix difference
  • Implement integration tests that verify secret retrieval using the actual API paths