Introduction

A classic Linux administration nightmare: df shows a filesystem at 100% but du adds up to far less than the total. This discrepancy happens when files are deleted but still held open by running processes, so the kernel cannot free the underlying blocks. The inode and data blocks remain allocated until every file descriptor referencing the file is closed.

Symptoms

  • df -h reports filesystem at 95-100% usage
  • du -sh / or du -sh /var shows significantly lower usage than df
  • Applications fail with No space left on device despite apparent free space
  • df -i may show normal inode usage (not an inode exhaustion issue)

Common Causes

  • Log files rotated by logrotate but the writing process was not signaled to reopen
  • Large temporary files deleted while a process still writes to them
  • Docker container layers referencing deleted base image files
  • Database WAL or journal files unlinked but still in use
  • Apache/Nginx access logs deleted without sending SIGHUP to the daemon

Step-by-Step Fix

  1. 1.Confirm the discrepancy between df and du:
  2. 2.```bash
  3. 3.df -h /var
  4. 4.sudo du -sh /var
  5. 5.# If df shows 45G used but du shows only 12G, deleted files are the culprit
  6. 6.`
  7. 7.Find deleted files still held open:
  8. 8.```bash
  9. 9.sudo lsof +L1 2>/dev/null | awk '$7 > 0 {print $0}' | sort -k7 -n -r | head -20
  10. 10.`
  11. 11.The +L1 flag shows files with link count less than 1 (deleted). The 7th column shows the file size.
  12. 12.Identify the offending processes:
  13. 13.```bash
  14. 14.sudo lsof +L1 | grep -E "mysqld|nginx|java|docker"
  15. 15.`
  16. 16.Recover space by truncating the deleted file descriptors:
  17. 17.```bash
  18. 18.# Find the PID and file descriptor number from lsof output
  19. 19.# Truncate via /proc filesystem
  20. 20.PID=1234
  21. 21.FD=5
  22. 22.truncate -s 0 /proc/$PID/fd/$FD
  23. 23.`
  24. 24.Alternatively, restart the process holding the file:
  25. 25.```bash
  26. 26.sudo systemctl restart nginx
  27. 27.sudo systemctl restart mysql
  28. 28.`
  29. 29.Verify space is reclaimed:
  30. 30.```bash
  31. 31.df -h /var
  32. 32.`

Prevention

  • Configure logrotate with copytruncate for services that cannot be signaled, or use postrotate scripts with kill -HUP
  • Monitor disk usage with alerts using df rather than du for production monitoring
  • Use lsof +L1 as part of regular disk health checks
  • Implement filesystem quotas to prevent any single process from consuming all space