You're trying to save a file or install a package, but the system responds with "No space left on device." Even worse, sometimes this happens when df shows there should be free space. Let's fix this step by step.
Understanding the Problem
Disk full errors come in two varieties: running out of actual data blocks, or running out of inodes (file descriptors). Both produce the same error message but require different approaches.
Typical Error Messages
No space left on device
write error (disk full?)
cannot create temp file: No space left on device
mkdir: cannot create directory 'newdir': No space left on deviceYou might also see:
- Applications crashing or failing to start
- Unable to create new files even when df shows free space
- System logging stops working
- Package installation failures
Diagnosing the Issue
Check Disk Usage
```bash # Basic disk usage overview df -h
# Detailed output including filesystem type df -hT
# Check specific mount point df -h /var
# Human-readable sizes with file type df -h --output=source,size,used,avail,pcent,target ```
Check Inode Usage
Sometimes disk shows free space but inodes are exhausted:
```bash # Check inode usage (100% = out of inodes) df -i
# If a filesystem shows 100% inode usage, you have too many small files ```
Find the Largest Directories
```bash # Find top 10 largest directories du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20
# Focus on specific directories du -h --max-depth=2 /var 2>/dev/null | sort -hr | head -20 du -h --max-depth=2 /home 2>/dev/null | sort -hr | head -20 du -h --max-depth=2 /usr 2>/dev/null | sort -hr | head -20
# Find files larger than 100MB find / -type f -size +100M 2>/dev/null | xargs ls -lh 2>/dev/null | awk '{print $5, $9}' | sort -hr
# Find files modified in last 24 hours taking up space find /var -type f -mtime -1 -exec ls -lh {} \; 2>/dev/null | awk '{print $5, $9}' | sort -hr | head -20 ```
Solutions
1. Clear Package Manager Caches
```bash # For Debian/Ubuntu systems sudo apt-get clean # Remove all cached .deb files sudo apt-get autoclean # Remove obsolete package files sudo apt-get autoremove # Remove unused dependencies
# Check how much space apt cache uses du -sh /var/cache/apt/archives
# For RHEL/CentOS/Fedora sudo dnf clean all # DNF (Fedora/RHEL 8+) sudo yum clean all # YUM (older RHEL/CentOS)
# For Arch Linux sudo pacman -Sc # Remove uninstalled package cache sudo pacman -Scc # Remove all package cache (caution) ```
2. Remove Old Kernels
Old kernels accumulate and consume significant space in /boot:
```bash # Check current kernel uname -r
# List installed kernels (Debian/Ubuntu) dpkg --list 'linux-image-*' | grep ^ii
# Remove old kernels, keeping current and one backup (Ubuntu) sudo apt-get autoremove --purge
# Manual kernel removal (be careful!) # First, install byobu for safe kernel removal sudo apt-get install byobu sudo purge-old-kernels --keep 2
# For RHEL/CentOS, remove old kernels (package-cleanup needed) sudo yum install yum-utils sudo package-cleanup --oldkernels --count=2 ```
3. Clear System Logs
Logs can grow massive over time:
```bash # Check journal log size journalctl --disk-usage
# Limit journal to 100MB sudo journalctl --vacuum-size=100M
# Keep only last 7 days sudo journalctl --vacuum-time=7d
# Check log sizes du -sh /var/log/* du -sh /var/log/journal/
# Rotate logs manually sudo logrotate -f /etc/logrotate.conf
# Find and remove old compressed logs find /var/log -type f -name "*.gz" -mtime +30 -delete find /var/log -type f -name "*.1" -mtime +30 -delete
# Truncate large log files (don't delete, just empty) sudo truncate -s 0 /var/log/large-file.log ```
4. Clear Temporary Files
```bash # Check temp directory sizes du -sh /tmp /var/tmp
# Remove files older than 10 days sudo find /tmp -type f -atime +10 -delete sudo find /var/tmp -type f -atime +10 -delete
# Clear systemd temporary files sudo systemd-tmpfiles --clean ```
5. Docker Cleanup
Docker images and volumes often consume large amounts of space:
```bash # Check Docker disk usage docker system df
# Remove unused data docker system prune -a # Remove all unused images, containers, volumes docker system prune -a --volumes # Include volumes
# Remove specific items docker image prune -a # Remove unused images docker container prune # Remove stopped containers docker volume prune # Remove unused volumes
# Delete everything Docker-related (nuclear option) docker stop $(docker ps -aq) 2>/dev/null docker rm $(docker ps -aq) 2>/dev/null docker rmi $(docker images -q) 2>/dev/null docker volume rm $(docker volume ls -q) 2>/dev/null ```
6. Find and Remove Large Files
```bash # Interactive ncdu tool (highly recommended) sudo apt-get install ncdu ncdu /
# Without ncdu, use find find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | awk '{print $5, $9}' | sort -hr
# Find deleted files still held open by processes sudo lsof | grep deleted
# Check for large deleted files taking space sudo lsof +L1
# If processes hold large deleted files, restart them or kill sudo kill -HUP <pid> ```
7. Handle Inode Exhaustion
If df -i shows 100% usage:
```bash # Find directories with most files find / -type d -size +100k 2>/dev/null | head -20
# Count files in each directory for dir in /*; do count=$(find "$dir" -type f 2>/dev/null | wc -l) echo "$count $dir" done | sort -rn | head -20
# Common culprits: session files, small cached items # Clear PHP sessions sudo find /var/lib/php/sessions -type f -atime +7 -delete
# Clear old mail queue files sudo find /var/spool/postfix/maildrop -type f -delete ```
8. Expand Filesystem (LVM)
If you have LVM and space available:
```bash # Check LVM status sudo vgdisplay sudo lvdisplay
# Extend logical volume sudo lvextend -L +10G /dev/mapper/vg00-lv_root # Or use all available space sudo lvextend -l +100%FREE /dev/mapper/vg00-lv_root
# Resize filesystem sudo resize2fs /dev/mapper/vg00-lv_root # For ext4 sudo xfs_growfs / # For XFS ```
Verification
After cleaning up, verify your changes:
```bash # Check disk usage df -h
# Check inode usage df -i
# Verify specific applications work touch /tmp/testfile && rm /tmp/testfile && echo "File creation works"
# Check if system logging works logger "Test message after disk cleanup" journalctl -n 1 ```
Prevention Tips
- Set up log rotation for all applications (
/etc/logrotate.d/) - Monitor disk usage with alerts at 80% threshold
- Use
logwatchor similar tools for daily reports - Configure
logrotateto compress old logs - Set up automated cleanup cron jobs for temporary directories
- Consider using separate partitions for
/var,/tmp, and/hometo prevent system lockup - Use quotas for user home directories in multi-user systems