Introduction

When both physical RAM and swap space are fully consumed, Linux enters a thrashing state where the kernel spends all its time moving pages between RAM and disk. The system becomes extremely unresponsive, with I/O wait reaching 90-100%. This is one of the most severe performance conditions on Linux because even basic commands like ps or top may take minutes to complete.

Symptoms

  • vmstat 1 shows si (swap in) and so (swap out) both high and continuous
  • free -m shows swap at 100% usage
  • top shows %wa (I/O wait) above 80%
  • SSH connections take minutes to establish
  • dmesg shows Out of memory or page allocation failure
  • Load average extremely high despite low CPU utilization

Common Causes

  • Memory leak in application consuming all RAM then all swap
  • Insufficient swap space allocated for the workload
  • vm.swappiness set too high, causing aggressive swapping
  • Database buffer pool sized without accounting for OS needs
  • Fork bomb or runaway process spawning children consuming memory

Step-by-Step Fix

  1. 1.Assess current memory and swap situation:
  2. 2.```bash
  3. 3.free -m
  4. 4.vmstat 1 5
  5. 5.cat /proc/meminfo | grep -E "Swap|MemAvailable"
  6. 6.`
  7. 7.Identify top memory consumers:
  8. 8.```bash
  9. 9.ps -eo pid,ppid,comm,rss,vsz --sort=-rss | head -15
  10. 10.# Use smem for more accurate PSS accounting
  11. 11.sudo smem -t -k -r | head -20
  12. 12.`
  13. 13.Kill the memory-hogging process:
  14. 14.```bash
  15. 15.kill -TERM <pid>
  16. 16.# If unresponsive due to I/O wait:
  17. 17.kill -9 <pid>
  18. 18.`
  19. 19.Add emergency swap space:
  20. 20.```bash
  21. 21.# Create a 4GB swap file
  22. 22.sudo fallocate -l 4G /swapfile_emergency
  23. 23.sudo chmod 600 /swapfile_emergency
  24. 24.sudo mkswap /swapfile_emergency
  25. 25.sudo swapon /swapfile_emergency
  26. 26.# Verify
  27. 27.swapon --show
  28. 28.`
  29. 29.Tune swap behavior to reduce thrashing:
  30. 30.```bash
  31. 31.# Reduce swappiness temporarily
  32. 32.sudo sysctl -w vm.swappiness=10
  33. 33.# Make persistent
  34. 34.echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
  35. 35.`
  36. 36.Drop clean caches to free RAM:
  37. 37.```bash
  38. 38.sudo sysctl -w vm.drop_caches=3
  39. 39.`

Prevention

  • Allocate swap equal to RAM size for systems under 8GB RAM, or 2-4GB for larger systems
  • Set vm.swappiness=10 for database servers, vm.swappiness=60 (default) for general use
  • Configure vm.min_free_kbytes to reserve memory for kernel allocations
  • Use systemd MemoryMax= limits to prevent any single service from consuming all memory
  • Monitor swap usage with alerts at 50% and 80% thresholds