Introduction

Vault has a safety feature that blocks all requests when audit logging fails. If the audit log destination -- typically a file on disk -- runs out of space, Vault cannot write audit entries and refuses to process any API requests. This design ensures compliance but creates a total service outage that requires immediate disk space recovery.

Symptoms

  • All Vault API requests fail with 500 Internal Server Error
  • Vault logs show audit backend failed to write or disk full
  • vault status may still show healthy but all operations fail
  • Applications cannot authenticate or retrieve secrets
  • Error message: error writing audit log: write /var/log/vault/audit.log: no space left on device

Common Causes

  • Audit log file growing unbounded without log rotation
  • Disk partition for audit logs undersized for the audit log volume
  • Log rotation not configured or failing silently
  • Vault audit log in JSON format without compression, growing rapidly
  • Other processes consuming disk space on the same partition

Step-by-Step Fix

  1. 1.Confirm the audit log disk is full: Check disk usage.
  2. 2.```bash
  3. 3.df -h /var/log/vault
  4. 4.du -sh /var/log/vault/*
  5. 5.`
  6. 6.Free disk space by rotating or truncating old audit logs: Make space available immediately.
  7. 7.```bash
  8. 8.# Compress old audit logs
  9. 9.gzip /var/log/vault/audit.log.1
  10. 10.# Or truncate the current log (preserves the file)
  11. 11.truncate -s 0 /var/log/vault/audit.log
  12. 12.`
  13. 13.Configure log rotation for the audit log file: Prevent future disk exhaustion.
  14. 14.```bash
  15. 15.# /etc/logrotate.d/vault-audit
  16. 16./var/log/vault/audit.log {
  17. 17.daily
  18. 18.rotate 30
  19. 19.compress
  20. 20.delaycompress
  21. 21.missingok
  22. 22.notifempty
  23. 23.create 0640 vault vault
  24. 24.postrotate
  25. 25.systemctl reload vault
  26. 26.endscript
  27. 27.}
  28. 28.`
  29. 29.Verify Vault resumes operations: Check that requests succeed again.
  30. 30.```bash
  31. 31.vault status
  32. 32.vault kv get secret/my-app/config
  33. 33.`
  34. 34.Consider adding a secondary audit backend: Use syslog or socket for redundancy.
  35. 35.```bash
  36. 36.vault audit enable syslog tag=vault
  37. 37.# Or file-based with rotation
  38. 38.vault audit enable file file_path=/var/log/vault/audit.log
  39. 39.`

Prevention

  • Configure logrotate for all Vault audit log files with appropriate retention and compression
  • Mount audit logs on a dedicated partition with monitoring and alerts at 80% capacity
  • Use multiple audit backends (file + syslog) for redundancy
  • Monitor Vault audit log write errors and alert on any audit backend failures
  • Set up automated disk space monitoring for the audit log partition
  • Use Vault Enterprise's audit log forwarding to external systems (Splunk, Elastic) for long-term retention