Introduction

Linux disk space full and "No space left on device" errors occur when filesystem storage is exhausted, preventing file writes, application operations, and system functions. This critical error can cause service outages, data loss, database corruption, and system instability. Disk exhaustion happens gradually through log accumulation, temporary file growth, application data expansion, or suddenly through large file creation, backup retention, or container image accumulation. Additionally, inode exhaustion can cause "No space left" errors even with free disk space, as inodes track file metadata and each file consumes one inode. Common causes include log files growing without rotation, database files expanding unchecked, temporary files not cleaned, Docker images and containers accumulating, backup files retained indefinitely, user home directories filling with data, package manager cache growing, core dump files consuming space, inode exhaustion from millions of small files, and deleted files still held open by processes. The fix requires understanding filesystem layout, disk usage analysis tools, inode management, log rotation, and safe cleanup procedures. This guide provides production-proven troubleshooting for disk space issues across RHEL/CentOS, Ubuntu/Debian, SUSE, and containerized environments.

Symptoms

  • df -h shows 100% disk usage
  • No space left on device error messages
  • Application write failures
  • Database cannot accept writes
  • SSH login fails (cannot create .Xauthority)
  • Log files stop growing
  • Service startup failures
  • No space left on device but df shows free space (inode exhaustion)
  • System becomes read-only
  • Emergency mode on boot

Common Causes

  • Log files without rotation
  • Database files growing unchecked
  • Temporary files not cleaned
  • Docker/container images accumulating
  • Backup files retained indefinitely
  • User data filling home directories
  • Package manager cache growing
  • Core dumps consuming space
  • Inode exhaustion from small files
  • Deleted files still held by processes

Step-by-Step Fix

### 1. Diagnose disk usage

Check overall disk usage:

```bash # Check filesystem usage df -h

# Output shows: # Filesystem Size Used Avail Use% Mounted on # /dev/sda1 100G 95G 0G 100% / # /dev/sdb1 500G 100G 375G 21% /data

# Key columns: # - Used: Space consumed # - Avail: Space available for non-root users # - Use%: Percentage used

# Include filesystem type df -hT

# Show inode usage (critical!) df -i

# Inode exhaustion causes "No space left" even with free space # Filesystem Inodes IUsed IFree IUse% Mounted on # /dev/sda1 6553600 6553500 100 100% / ```

Find space-consuming directories:

```bash # Check root directory sizes du -sh /* 2>/dev/null | sort -hr | head -20

# Or with human-readable output du -xh --max-depth=1 / | sort -hr

# Key directories to check: # - /var: Variable data (logs, databases, cache) # - /home: User home directories # - /opt: Optional software # - /usr: System software # - /tmp: Temporary files ```

Find large files:

```bash # Find files larger than 1GB find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Find top 20 largest files find / -type f -exec du -h {} \; 2>/dev/null | sort -hr | head -20

# Or using du du -ah / 2>/dev/null | sort -rh | head -20

# Find files not modified in 30 days (potential cleanup candidates) find / -type f -mtime +30 -size +100M 2>/dev/null ```

### 2. Identify files held by deleted processes

Check for deleted files still in use:

```bash # Find deleted files still held open lsof | grep deleted

# Or with more detail lsof +L1 # Files with link count < 1

# Output shows: # COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME # java 1234 app 1w REG 8,1 1073741824 5678 /var/log/app.log (deleted)

# The SIZE column shows space that can be reclaimed

# Calculate total space from deleted files lsof | grep deleted | awk '{sum += $7} END {print sum/1024/1024 " MB"}' ```

Reclaim space from deleted files:

```bash # Option 1: Restart the process holding the file PID=$(lsof | grep deleted | awk '{print $2}' | head -1) kill -HUP $PID # Or restart service systemctl restart service-name

# Option 2: Truncate the file via /proc PID=$(lsof | grep deleted | awk '{print $2}' | head -1) FD=$(lsof | grep deleted | awk '{print $4}' | head -1) > /proc/$PID/fd/$FD # Truncate file

# Option 3: If safe, kill the process kill $PID ```

### 3. Clean up log files

Find large log files:

```bash # Check /var/log sizes du -sh /var/log/*

# Find large log files find /var/log -type f -size +100M -exec ls -lh {} \;

# Check for uncompressed old logs find /var/log -name "*.gz" -o -name "*.bz2" -o -name "*.xz" | head -20 ```

Safely clean log files:

```bash # NEVER delete log files directly (breaks log rotation) # WRONG: rm /var/log/syslog

# CORRECT: Truncate log files sudo truncate -s 0 /var/log/syslog # Or sudo : > /var/log/syslog

# Truncate all logs older than 7 days find /var/log -type f -name "*.log" -mtime +7 -exec truncate -s 0 {} \;

# Clean old compressed logs sudo find /var/log -name "*.gz" -mtime +30 -delete sudo find /var/log -name "*.bz2" -mtime +30 -delete sudo find /var/log -name "*.xz" -mtime +30 -delete sudo find /var/log -name "*.old" -mtime +30 -delete ```

Configure log rotation:

```bash # /etc/logrotate.conf - Global settings weekly rotate 4 create dateext compress delaycompress

# /etc/logrotate.d/ - Application-specific # Example for custom application /var/log/myapp/*.log { daily rotate 7 compress delaycompress missingok notifempty create 0640 root adm postrotate systemctl reload myapp endscript }

# Test logrotate logrotate -d /etc/logrotate.conf # Dry run logrotate -f /etc/logrotate.conf # Force rotation ```

### 4. Clean up temporary files

Clean system temp directories:

```bash # Check temp directory sizes du -sh /tmp /var/tmp /var/cache

# Clean old temp files (safe to delete files older than 7 days) find /tmp -type f -atime +7 -delete find /var/tmp -type f -atime +7 -delete

# Clean package manager caches # Debian/Ubuntu apt-get clean apt-get autoclean apt-get autoremove

# RHEL/CentOS yum clean all dnf clean all

# Clear systemd journal logs journalctl --vacuum-time=7d journalctl --vacuum-size=500M

# Check journal size before/after journalctl --disk-usage ```

Clean user cache directories:

```bash # Check user cache sizes du -sh /home/*/.cache 2>/dev/null | sort -hr

# Clean browser caches (example for Chrome) find /home/*/.cache/google-chrome -type f -atime +30 -delete

# Clean npm cache npm cache clean --force

# Clean pip cache pip cache purge

# Or clean all user caches older than 30 days find /home/*/.cache -type f -atime +30 -delete 2>/dev/null ```

### 5. Clean up Docker containers and images

Docker disk usage:

```bash # Check Docker disk usage docker system df

# Output shows: # TYPE TOTAL ACTIVE SIZE RECLAIMABLE # Images 50 10 15GB 12GB (80%) # Containers 30 5 1GB 500MB # Local Volumes 20 10 5GB 2GB

# Check individual image sizes docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

# Check container sizes docker ps -s ```

Safe Docker cleanup:

```bash # Remove stopped containers docker container prune -f

# Remove dangling images (untagged) docker image prune -f

# Remove all unused images docker image prune -a -f

# Remove unused volumes (careful!) docker volume prune -f

# Remove build cache docker builder prune -f

# Complete cleanup (removes all unused) docker system prune -a --volumes -f

# Or set size limit docker system prune --filter "until=24h" -f # Older than 24 hours ```

Prevent Docker disk growth:

```bash # Configure log rotation for containers # /etc/docker/daemon.json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }

# Restart Docker after config change systemctl restart docker

# Or limit in docker-compose.yml # services: # app: # logging: # driver: "json-file" # options: # max-size: "10m" # max-file: "3" ```

### 6. Find and remove duplicate files

Find duplicate files:

```bash # Install fdupes apt install fdupes # Debian/Ubuntu yum install fdupes # RHEL/CentOS

# Find duplicates in directory fdupes -r /home

# Or with more details fdupes -r -S /home # Sort by size

# Find duplicates and delete interactively fdupes -r -d /home ```

Find duplicate large files:

```bash # Using rdfind (more efficient for large datasets) apt install rdfind rdfind -dryrun true /data # Dry run first

# Find duplicate files by hash find /data -type f -exec md5sum {} \; | sort | uniq -d -w32

# Clean package duplicates # Debian/Ubuntu deborphan # Find orphaned packages apt-get remove --purge $(deborphan)

# Remove old kernel versions (keep current and one previous) apt-get autoremove --purge ```

### 7. Clean up database files

MySQL/MariaDB cleanup:

```bash # Check database sizes mysql -e "SELECT table_schema, SUM(data_length+index_length)/1024/1024 AS 'Size (MB)' FROM information_schema.tables GROUP BY table_schema;"

# Check binary log size ls -lh /var/lib/mysql/mysql-bin.*

# Clean binary logs (keep last 7 days) mysql -e "PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 7 DAY);"

# Or set automatic expiration # /etc/mysql/mysql.conf.d/mysqld.cnf [mysqld] binlog_expire_logs_seconds = 604800 # 7 days expire_logs_days = 7

# Clean slow query log truncate -s 0 /var/log/mysql/mysql-slow.log

# Optimize tables (reclaim space) mysqlcheck -o --all-databases ```

PostgreSQL cleanup:

```bash # Check database sizes psql -c "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database ORDER BY pg_database_size(datname) DESC;"

# Check table sizes psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;"

# Vacuum database (reclaim space) vacuumdb --all --verbose

# Clean WAL archives # Configure in postgresql.conf # wal_keep_size = 1GB # max_wal_size = 4GB ```

### 8. Find and clean large backups

Find backup files:

```bash # Find backup files by extension find / -type f \( -name "*.bak" -o -name "*.backup" -o -name "*.tar.gz" -o -name "*.zip" \) -size +100M 2>/dev/null

# Check backup directories du -sh /backup/* /var/backups/* /home/*/backup/* 2>/dev/null

# Find old backup files find /backup -type f -mtime +30 -exec ls -lh {} \; ```

Safe backup cleanup:

```bash # Keep only last N backups # Example: Keep last 5 .tar.gz backups ls -t /backup/*.tar.gz | tail -n +6 | xargs rm -f

# Or by date (keep last 30 days) find /backup -type f -mtime +30 -delete

# Compress old backups find /backup -type f -name "*.tar" -mtime +7 -exec gzip {} \;

# Move old backups to archive storage find /backup -type f -mtime +7 -exec mv {} /archive/ \; ```

### 9. Monitor disk usage

Create disk monitoring script:

```bash #!/bin/bash # /usr/local/bin/disk-monitor.sh

THRESHOLD=85 ALERT_EMAIL="admin@example.com"

# Check disk usage df -h | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 " " $6 }' | while read output; do usage=$(echo $output | awk '{print $1}' | cut -d'%' -f1) partition=$(echo $output | awk '{print $2}') mountpoint=$(echo $output | awk '{print $3}')

if [ $usage -ge $THRESHOLD ]; then echo "Disk usage alert: $partition on $mountpoint is ${usage}%" echo "" echo "Top 10 directories:" du -ah $mountpoint 2>/dev/null | sort -rh | head -10

# Send alert echo "Disk usage at ${usage}%" | mail -s "Alert: Disk Space - $mountpoint" $ALERT_EMAIL fi done

# Check inode usage df -i | grep -vE '^Filesystem|tmpfs' | awk '{ print $5 " " $6 }' | while read output; do usage=$(echo $output | awk '{print $1}' | cut -d'%' -f1) mountpoint=$(echo $output | awk '{print $2}')

if [ $usage -ge $THRESHOLD ]; then echo "Inode usage alert: $mountpoint is ${usage}%" fi done ```

Configure automated monitoring:

```bash # Add to cron crontab -e

# Check disk every hour 0 * * * * /usr/local/bin/disk-monitor.sh

# Or use systemd timer # /etc/systemd/system/disk-monitor.timer [Unit] Description=Check disk usage hourly

[Timer] OnBootSec=10min OnUnitActiveSec=1h

[Install] WantedBy=timers.target

# Enable timer systemctl enable disk-monitor.timer systemctl start disk-monitor.timer ```

### 10. Emergency disk space recovery

Emergency cleanup commands:

```bash # Quick space recovery (safe commands)

# Clear journal logs journalctl --vacuum-time=1d

# Clear apt cache apt-get clean

# Clear systemd tmpfiles systemd-tmpfiles --clean

# Truncate core dumps find / -type f -name "core.*" -exec truncate -s 0 {} \;

# Clear /var/cache find /var/cache -type f -delete 2>/dev/null

# Emergency Docker cleanup docker system prune -a -f --volumes ```

Last resort options:

```bash # If system is completely full and unbootable:

# Boot into single-user mode or recovery mode # Mount filesystem read-write mount -o remount,rw /

# Remove large non-essential files rm -rf /var/log/*.gz rm -rf /var/cache/apt/archives/*.deb rm -rf /tmp/*

# Or extend filesystem (if LVM) lvextend -L +10G /dev/mapper/vg-root xfs_growfs / # For XFS # Or resize2fs /dev/mapper/vg-root # For ext4 ```

Prevention

  • Configure log rotation for all applications
  • Set up disk usage monitoring with alerts at 80%
  • Implement log aggregation (send logs to central server)
  • Use separate partitions for /var, /home, /data
  • Configure Docker log rotation limits
  • Automate backup cleanup and retention
  • Regular inode usage checks
  • Document disk space requirements per service
  • Implement quotas for user directories
  • **No space left on device**: Disk or inode exhaustion
  • **Disk quota exceeded**: User quota limit reached
  • **Read-only file system**: Filesystem error or mount option
  • **Write error**: Physical disk failure possible
  • **Filesystem full**: 100% disk usage