Fix Vault Secrets Access Denied

What's Actually Happening

HashiCorp Vault denies access to secrets when applications or users attempt to read or write secrets. Access is rejected due to authentication or authorization issues.

The Error You'll See

```bash $ vault kv get secret/myapp

Error reading secret/myapp: Error making API request. URL: GET https://vault:8200/v1/secret/data/myapp Code: 403. Errors: * permission denied ```

Token invalid:

```bash $ vault read sys/health

Error: invalid token ```

Policy denied:

```bash $ vault kv put secret/myapp key=value

Error writing secret/myapp: permission denied ```

AppRole auth failed:

```bash $ vault write auth/approle/login role_id=xxx secret_id=yyy

Error: invalid role ID or secret ID ```

Why This Happens

1. 1.Token expired - Vault token has expired
2. 2.Wrong token - Using incorrect or invalid token
3. 3.Policy missing - No policy grants access to path
4. 4.Policy insufficient - Policy lacks required capabilities
5. 5.Auth method issues - Authentication configuration wrong
6. 6.Path mismatch - Secret path doesn't match policy path

Step 1: Check Vault Status

```bash # Check Vault status: vault status

# Output: # Sealed: false # Initialized: true

# Check Vault health: curl http://vault:8200/v1/sys/health

# Check Vault version: vault version

# Check leader: vault operator leader

# Check seal status (if sealed): vault operator seal-status

# Unseal Vault if sealed: vault operator unseal unseal-key-1 vault operator unseal unseal-key-2

# Check storage: vault operator raft list-peers ```

Step 2: Check Token Validity

```bash # Check current token: vault token lookup

# Output shows: # display_name # creation_time # expiration_time # policies

# Check if token expired: vault token lookup | grep expire_time

# Check token capabilities: vault token capabilities secret/myapp

# Output: # [read, list]

# If token invalid, get new token: # Root token: vault login root-token

# Or via auth method: vault login -method=userpass username=admin password=password

# Create new token: vault token create -policy=myapp-policy

# Renew token: vault token renew

# Revoke token: vault token revoke token-id ```

Step 3: Check Policy Configuration

```bash # List policies: vault policy list

# Read policy: vault policy read myapp-policy

# Policy should grant access: # Example policy: path "secret/data/myapp/*" { capabilities = ["create", "read", "update", "delete", "list"] }

path "secret/metadata/myapp/*" { capabilities = ["list", "read", "delete"] }

# Check policy for specific path: vault token capabilities secret/data/myapp/config

# Create policy: vault policy write myapp-policy - <<EOF path "secret/data/myapp/*" { capabilities = ["read", "list"] } EOF

# Update policy: vault policy write myapp-policy policy.hcl

# Delete policy: vault policy delete myapp-policy

# Check ACLs: vault read sys/policies/acl/myapp-policy ```

Step 4: Check Authentication Methods

```bash # List auth methods: vault auth list

# Check auth method status: vault auth enable approle

# Check AppRole configuration: vault read auth/approle/role/myapp

# Check role ID: vault read auth/approle/role/myapp/role-id

# Generate secret ID: vault write -f auth/approle/role/myapp/secret-id

# Login with AppRole: vault write auth/approle/login role_id=xxx secret_id=yyy

# Check userpass: vault read auth/userpass/users/admin

# Login with userpass: vault login -method=userpass username=admin

# Check token auth: vault read auth/token/tokens

# Check Kubernetes auth: vault read auth/kubernetes/config vault read auth/kubernetes/role/myapp ```

Step 5: Check Path Access

```bash # Verify secret exists: vault kv list secret/

# Check secret metadata: vault kv metadata get secret/myapp

# Read secret with correct path: # KV v2 requires /data/ in path: vault kv get secret/myapp # Internally: secret/data/myapp

# Manual API call: curl -H "X-Vault-Token: token" http://vault:8200/v1/secret/data/myapp

# Check mount configuration: vault read sys/mounts

# Check engine type: # KV v1: secret/ # KV v2: secret/ (with /data/ and /metadata/)

# Path matching in policy: # KV v2 paths: # secret/data/myapp/* - data # secret/metadata/myapp/* - metadata # secret/delete/myapp/* - delete

# For KV v1: # secret/myapp/* ```

Step 6: Create Missing Policy

```bash # Create policy for read access: vault policy write myapp-read - <<EOF path "secret/data/myapp/*" { capabilities = ["read", "list"] } EOF

# Create policy for write access: vault policy write myapp-write - <<EOF path "secret/data/myapp/*" { capabilities = ["create", "update", "read", "delete"] } path "secret/metadata/myapp/*" { capabilities = ["list", "read", "delete"] } EOF

# Assign policy to token: vault token create -policy=myapp-read

# Assign policy to role: vault write auth/approle/role/myapp policies=myapp-read

# Assign to user: vault write auth/userpass/users/admin policies=myapp-read

# Check token policies: vault token lookup | grep policies ```

Step 7: Fix AppRole Authentication

```bash # Check AppRole configuration: vault read auth/approle/role/myapp

# Check role_id: vault read auth/approle/role/myapp/role-id

# Output: role_id

# Generate new secret_id: vault write -f auth/approle/role/myapp/secret-id

# Output: secret_id

# Login with AppRole: vault write auth/approle/login role_id=ROLE_ID secret_id=SECRET_ID

# Get token from response: # client_token

# Configure role with proper policies: vault write auth/approle/role/myapp \ token_policies=myapp-policy \ token_ttl=1h \ token_max_ttl=4h

# Check secret_id TTL: vault write auth/approle/role/myapp \ secret_id_ttl=24h \ secret_id_num_uses=10

# Create wrapped secret_id: vault write -wrap-ttl=5m -f auth/approle/role/myapp/secret-id ```

Step 8: Fix Kubernetes Authentication

```bash # Check Kubernetes auth config: vault read auth/kubernetes/config

# Check role: vault read auth/kubernetes/role/myapp

# Configure Kubernetes auth: vault write auth/kubernetes/config \ kubernetes_host=https://kubernetes.api \ token_reviewer_jwt=SA_TOKEN \ kubernetes_ca_cert=@/path/to/ca.crt

# Create role: vault write auth/kubernetes/role/myapp \ bound_service_account_names=myapp-sa \ bound_service_account_namespaces=myapp \ policies=myapp-policy \ ttl=1h

# Login from Kubernetes: kubectl exec -it pod -- vault write auth/kubernetes/login role=myapp jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Check JWT validity: kubectl exec -it pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ```

Step 9: Check Audit Logs

```bash # Enable audit log: vault audit enable file file_path=/var/log/vault/audit.log

# Or syslog: vault audit enable syslog tag=vault

# Check audit log: tail -f /var/log/vault/audit.log

# Audit log shows: # Request: path, method, token # Response: error or success

# Look for denied requests: grep "403" /var/log/vault/audit.log grep "permission denied" /var/log/vault/audit.log

# Check which policy denied: # Audit log includes policy evaluation

# List audit devices: vault audit list

# Disable audit: vault audit disable file/ ```

Step 10: Vault Access Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-vault-access.sh #!/bin/bash

VAULT_ADDR=${VAULT_ADDR:-"http://localhost:8200"}

echo "=== Vault Status ===" vault status

echo "" echo "=== Current Token ===" vault token lookup 2>&1 | head -20

echo "" echo "=== Policies ===" vault policy list

echo "" echo "=== Auth Methods ===" vault auth list

echo "" echo "=== Secret Engines ===" vault read sys/mounts | grep -E "Path|Type|Accessor"

echo "" echo "=== Test Access ===" vault kv list secret/ 2>&1

echo "" echo "=== Token Capabilities ===" vault token capabilities secret/data/myapp 2>&1

echo "" echo "=== Audit Logs (last 10) ===" tail -10 /var/log/vault/audit.log 2>/dev/null | jq -r '.time + " " + .request.path + " " + .response.status_code' 2>/dev/null || echo "No audit logs"

echo "" echo "=== Health Check ===" curl -s $VAULT_ADDR/v1/sys/health | jq . EOF

chmod +x /usr/local/bin/check-vault-access.sh

# Run: /usr/local/bin/check-vault-access.sh

# Monitor: watch -n 5 vault status ```

Vault Access Checklist

Verify the Fix

```bash # After fixing access

# 1. Check token vault token lookup // Token valid with correct policies

# 2. Check capabilities vault token capabilities secret/data/myapp // [read, list, create]

# 3. Read secret vault kv get secret/myapp // Returns secret data

# 4. Write secret vault kv put secret/myapp key=value // Success

# 5. Check policy vault policy read myapp-policy // Correct path and capabilities

# 6. Test from application # Application can access secrets // No errors ```

Related Issues

• [Fix Vault Unsealed State](/articles/fix-vault-unsealed-state)
• [Fix Vault Token Expired](/articles/fix-vault-token-expired)
• [Fix Vault Seal Status](/articles/fix-vault-seal-status)