Fix Vault Seal Failed

What's Actually Happening

HashiCorp Vault fails to unseal. Vault remains sealed and cannot process requests.

The Error You'll See

```bash $ vault operator unseal

Error unsealing: Error making API request. URL: PUT http://vault:8200/v1/sys/unseal Code: 400. Errors: 'key' is not a valid unseal key for seal ```

Storage error:

Error: failed to get stored keys: failed to read key ring

HA error:

Error: Vault is sealed, cannot perform operation

Key threshold error:

Error: insufficient unseal keys to meet threshold

Why This Happens

1. 1.Wrong unseal key - Incorrect key provided
2. 2.Storage backend down - Cannot read from storage
3. 3.Key shares lost - Not enough keys to reach threshold
4. 4.Auto-unseal failure - Cloud KMS or Transit unavailable
5. 5.Corrupted storage - Data corruption in backend
6. 6.Network partition - HA node communication failure

Step 1: Check Vault Status

```bash # Check Vault status: vault status

# Output shows: # Sealed: true/false # Seal Type: shamir/awskms/... # Total Shares: 5 # Threshold: 3 # Unseal Progress: 1/3

# Check if initialized: vault status | grep "Initialized"

# Check seal type: vault status | grep "Seal Type"

# Check HA status: vault status | grep "HA"

# Via API: curl http://vault:8200/v1/sys/seal-status | jq .

# Check Vault logs: journalctl -u vault # Or: kubectl logs -n vault deploy/vault

# Check Vault process: ps aux | grep vault ```

Step 2: Verify Unseal Keys

```bash # Check if keys are available: # Keys are generated during vault operator init

# List unseal keys (if stored): # WARNING: This is sensitive data cat /etc/vault/unseal-keys.txt

# Verify key format: # Unseal keys are base64 encoded: # Example: hvs.xxxxxxxxxxxxx

# Check threshold needed: vault status | grep "Threshold"

# Need N keys to unseal where N >= threshold

# If keys are lost: # Cannot recover - vault data is lost # Must reinitialize vault

# Validate key format: echo "your-key-here" | base64 -d # Should produce binary data

# Key provided must match one of the generated keys exactly # Case-sensitive ```

Step 3: Unseal with Correct Keys

```bash # Unseal requires threshold keys:

# Method 1: CLI (run threshold times with different keys): vault operator unseal "key1" vault operator unseal "key2" vault operator unseal "key3"

# Method 2: API: curl -X PUT http://vault:8200/v1/sys/unseal \ -d '{"key": "your-unseal-key"}'

# Check progress after each key: vault status | grep "Unseal Progress"

# After threshold keys, Vault unseals: # Sealed: false

# Verify unsealed: vault status

# Reset unseal progress: vault operator unseal -reset

# Use unseal keys from file: for key in $(cat /etc/vault/unseal-keys.txt); do vault operator unseal "$key" done ```

Step 4: Check Storage Backend

```bash # Check storage configuration: cat /etc/vault/config.hcl | grep -A10 storage

# For Consul storage: consul kv get vault/core/seal

# For Raft storage: vault operator raft list-peers

# Check Raft logs: vault operator raft logs

# For S3 storage: aws s3 ls s3://vault-bucket/

# For file storage: ls -la /opt/vault/data/

# For etcd storage: etcdctl get /vault/ --prefix

# Check storage connectivity: # Verify network access to storage backend

# Test storage access: curl -I http://consul:8500/v1/kv/vault

# Check storage permissions: # Vault must have read/write access ```

Step 5: Fix Auto-Unseal Issues

```bash # Check auto-unseal configuration: cat /etc/vault/config.hcl | grep -A10 seal

# AWS KMS auto-unseal: seal "awskms" { region = "us-east-1" kms_key_id = "key-id" }

# Check AWS credentials: aws kms describe-key --key-id "key-id"

# Test KMS access: aws kms decrypt --ciphertext-blob fileb://encrypted

# Transit auto-unseal (from another Vault): seal "transit" { address = "https://vault-primary:8200" token = "s.xxxxx" disable_renewal = "false" key_name = "autounseal" }

# Verify transit key exists: vault read transit/keys/autounseal

# Check Vault token for transit: vault token lookup

# GCP KMS auto-unseal: seal "gcpckms" { credentials = "/path/to/creds.json" project = "my-project" region = "us-east1" key_ring = "vault" crypto_key = "unseal" }

# Azure Key Vault: seal "azurekeyvault" { tenant_id = "tenant-id" client_id = "client-id" client_secret = "client-secret" vault_name = "vault-name" key_name = "unseal-key" } ```

Step 6: Handle Raft Storage Issues

```bash # Check Raft cluster: vault operator raft list-peers

# Check Raft configuration: vault operator raft configuration

# Raft node stuck: # Remove and rejoin node:

# Remove node: vault operator raft remove-peer "node-2"

# Rejoin node: vault server -config=/etc/vault/config.hcl -join="http://node-1:8200"

# Force Raft recovery: # Only use if cluster broken beyond repair vault operator raft remove-peers

# Snapshot restore: vault operator raft snapshot save backup.snap vault operator raft snapshot restore backup.snap

# Check Raft leader: vault status | grep "Leader"

# Manual peer recovery: # Delete peer set file: rm /opt/vault/data/raft/peers.json # Restart vault with -retry-join ```

Step 7: Recover from Corruption

```bash # If storage corruption suspected:

# 1. Stop Vault: systemctl stop vault # Or: kubectl scale deploy vault --replicas=0

# 2. Check storage integrity: # For file storage: ls -la /opt/vault/data/core/

# For Raft: ls -la /opt/vault/data/raft/

# 3. Restore from backup: vault operator raft snapshot restore backup.snap

# 4. For non-Raft storage: # Restore storage backend from backup

# 5. Verify data after restore: vault kv list secret/

# 6. Restart Vault: systemctl start vault

# 7. Unseal after restart: vault operator unseal

# If no backup: # Data is lost, must reinitialize: vault operator init ```

Step 8: Handle HA Failover Issues

```bash # Check HA status: vault status | grep -A5 "HA"

# Check cluster nodes: vault operator cluster list

# Check node health: vault operator cluster health

# Force step-down (if stuck leader): vault operator step-down

# Check standby nodes: vault status -format=json | jq '.ha_cluster_info'

# Network partition recovery:

# 1. Check connectivity between nodes: ping vault-1 curl http://vault-1:8200/v1/sys/health

# 2. Verify leader is accessible: vault status | grep "Leader Address"

# 3. If partition, nodes may have different leaders # Resolve by ensuring network connectivity

# For Consul HA: # Check session: consul session list consul session info <session-id>

# Force leader change: consul session destroy <session-id> ```

Step 9: Reinitialize Vault

```bash # WARNING: Reinitializing destroys all data

# Use only if: # - No backup available # - All unseal keys lost # - Irrecoverable corruption

# 1. Stop Vault: systemctl stop vault

# 2. Clear storage: rm -rf /opt/vault/data/*

# Or for Consul: consul kv delete -recurse vault

# Or for S3: aws s3 rm s3://vault-bucket --recursive

# 3. Start Vault: systemctl start vault

# 4. Initialize: vault operator init -key-shares=5 -key-threshold=3

# 5. Save unseal keys and root token securely!

# 6. Unseal: vault operator unseal

# 7. Login: vault login <root-token>

# 8. Reconfigure: # Restore secrets from backup # Recreate policies, roles, etc. ```

Step 10: Vault Seal Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-vault-seal.sh #!/bin/bash

HOST=${1:-"localhost:8200"}

echo "=== Vault Status ===" vault status -address=http://$HOST 2>&1 || echo "Vault not responding"

echo "" echo "=== Seal Status ===" vault status -address=http://$HOST -format=json 2>/dev/null | jq '{sealed, total_shares, threshold, progress}' || echo "Cannot get seal status"

echo "" echo "=== Storage Config ===" cat /etc/vault/config.hcl 2>/dev/null | grep -A10 storage || echo "Config not found"

echo "" echo "=== Seal Config ===" cat /etc/vault/config.hcl 2>/dev/null | grep -A10 seal || echo "No seal config (Shamir)"

echo "" echo "=== Vault Process ===" ps aux | grep vault | grep -v grep || echo "Vault not running"

echo "" echo "=== Vault Logs (recent) ===" journalctl -u vault --no-pager -n 20 2>/dev/null || kubectl logs -n vault deploy/vault --tail=20 2>/dev/null || echo "Cannot get logs"

echo "" echo "=== Raft Peers ===" vault operator raft list-peers 2>/dev/null || echo "Not using Raft or sealed"

echo "" echo "=== Recommendations ===" echo "1. Verify unseal keys are correct" echo "2. Check storage backend connectivity" echo "3. Ensure auto-unseal KMS is accessible" echo "4. Verify network connectivity in HA cluster" echo "5. Check storage permissions" echo "6. Use vault operator unseal with threshold keys" echo "7. If keys lost, must reinitialize (data lost)" EOF

chmod +x /usr/local/bin/check-vault-seal.sh

# Usage: /usr/local/bin/check-vault-seal.sh localhost:8200 ```

Vault Seal Checklist

Verify the Fix

```bash # After fixing Vault seal issues

# 1. Check Vault status vault status // Sealed: false

# 2. Test Vault operation vault kv put secret/test key=value // Success

# 3. Read secret vault kv get secret/test // Returns value

# 4. Check health curl http://vault:8200/v1/sys/health // 200 OK

# 5. Verify HA (if applicable) vault status | grep HA // HA Enabled: true, HA Mode: active

# 6. Check unseal progress vault status | grep Progress // Progress: 0 (unsealed) ```

Related Issues

• [Fix Vault Unseal Failed](/articles/fix-vault-unseal-failed)
• [Fix Vault Secrets Access Denied](/articles/fix-vault-secrets-access-denied)
• [Fix Consul Service Not Registering](/articles/fix-consul-service-not-registering)