Introduction

When a disk partition reaches 100% capacity, applications cannot write new data, create temporary files, or rotate logs. Web servers may stop accepting new connections, databases refuse writes, and the entire site becomes unavailable. Log files are the most common cause of unexpected disk exhaustion because they grow continuously and are often overlooked when the application is generating verbose output or error loops.

Symptoms

  • Website returns 500 Internal Server Error or connection refused
  • df -h shows a partition at 100% usage
  • Application logs show No space left on device
  • Database refuses writes with disk full error
  • SSH login works but commands fail with disk errors

Common Causes

  • Application error loop generating thousands of log lines per second
  • Log rotation not configured or not running (logrotate service disabled)
  • Debug logging accidentally left enabled in production
  • Log partition too small for the traffic volume
  • Compressed log archives not being cleaned up (rotated logs accumulating)

Step-by-Step Fix

  1. 1.Identify the full partition:
  2. 2.```bash
  3. 3.df -h
  4. 4.# Find the partition at 100%
  5. 5.`
  6. 6.Find the largest files consuming space:
  7. 7.```bash
  8. 8.sudo du -sh /var/log/*/ 2>/dev/null | sort -h | tail -10
  9. 9.sudo du -sh /var/log/* 2>/dev/null | sort -h | tail -10
  10. 10.# Find specific large files
  11. 11.sudo find /var/log -type f -size +100M -exec ls -lh {} \;
  12. 12.`
  13. 13.Emergency cleanup - truncate large log files:
  14. 14.```bash
  15. 15.# Do NOT delete the file (may not free space if process holds it open)
  16. 16.# Instead, truncate it:
  17. 17.sudo truncate -s 0 /var/log/nginx/access.log
  18. 18.sudo truncate -s 0 /var/log/nginx/error.log
  19. 19.sudo truncate -s 0 /var/log/syslog

# Or use the shell redirection method: sudo sh -c '> /var/log/nginx/access.log' ```

  1. 1.Clear old rotated logs:
  2. 2.```bash
  3. 3.# Remove compressed logs older than 30 days
  4. 4.sudo find /var/log -name "*.gz" -mtime +30 -delete
  5. 5.sudo find /var/log -name "*.log.[0-9]*" -mtime +7 -delete
  6. 6.`
  7. 7.Fix log rotation configuration:
  8. 8.```bash
  9. 9.sudo nano /etc/logrotate.d/nginx
  10. 10.`
  11. 11.Ensure:
  12. 12.`
  13. 13./var/log/nginx/*.log {
  14. 14.daily
  15. 15.rotate 14
  16. 16.compress
  17. 17.delaycompress
  18. 18.missingok
  19. 19.notifempty
  20. 20.create 0640 www-data adm
  21. 21.sharedscripts
  22. 22.postrotate
  23. 23.[ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
  24. 24.endscript
  25. 25.}
  26. 26.`
  27. 27.Ensure logrotate is running:
  28. 28.```bash
  29. 29.sudo systemctl status logrotate
  30. 30.sudo systemctl enable logrotate
  31. 31.# Force an immediate rotation
  32. 32.sudo logrotate -f /etc/logrotate.conf
  33. 33.`

Prevention

  • Set up disk usage monitoring with alerts at 70%, 80%, and 90% thresholds
  • Configure log rotation with appropriate retention periods
  • Use centralized log aggregation (ELK, Loki) to reduce local log storage needs
  • Set filesystem quotas on log partitions to prevent them from consuming the entire disk
  • Implement log level management - use INFO in production, DEBUG only in development