Introduction

When a Linux server runs out of memory, the kernel's OOM (Out of Memory) killer terminates one or more processes to free memory. The OOM killer often targets the largest memory consumer, which is frequently the web server (Nginx, Apache) or application process (Node.js, Java, Python). When the web server process is killed, the site becomes immediately unavailable until the service is restarted. This is a common cause of intermittent site outages on memory-constrained servers.

Symptoms

  • Site becomes suddenly unavailable with no prior warning
  • journalctl -k | grep -i "oom|killed" shows Out of memory: Killed process XXXX (nginx)
  • Web server service shows failed status: systemctl status nginx
  • dmesg shows oom_score and memory allocation failure details
  • Site works again after restarting the web server service

Common Causes

  • Memory leak in the application gradually consuming all available RAM
  • Traffic spike causing web server to spawn more worker processes than memory supports
  • No swap space configured as a buffer before OOM trigger
  • Other memory-hungry processes (database, caching) competing with the web server
  • Container memory limit too low for the workload

Step-by-Step Fix

  1. 1.Confirm OOM kill caused the outage:
  2. 2.```bash
  3. 3.dmesg -T | grep -i "killed process" | tail -5
  4. 4.journalctl -k --since "1 hour ago" | grep -i "oom"
  5. 5.`
  6. 6.Restart the terminated service:
  7. 7.```bash
  8. 8.sudo systemctl start nginx
  9. 9.sudo systemctl start myapp
  10. 10.`
  11. 11.Identify the memory consumption pattern:
  12. 12.```bash
  13. 13.free -m
  14. 14.ps -eo pid,comm,rss,vsz --sort=-rss | head -15
  15. 15.# RSS is in KB - divide by 1024 for MB
  16. 16.`
  17. 17.Protect the web server from OOM killing:
  18. 18.```bash
  19. 19.sudo systemctl edit nginx.service
  20. 20.`
  21. 21.Add:
  22. 22.```ini
  23. 23.[Service]
  24. 24.OOMScoreAdjust=-500
  25. 25.`
  26. 26.Then reload:
  27. 27.```bash
  28. 28.sudo systemctl daemon-reload
  29. 29.`
  30. 30.Set memory limits to prevent runaway consumption:
  31. 31.```bash
  32. 32.sudo systemctl edit myapp.service
  33. 33.`
  34. 34.Add:
  35. 35.```ini
  36. 36.[Service]
  37. 37.MemoryMax=2G
  38. 38.MemoryHigh=1500M
  39. 39.`
  40. 40.Add swap space as a buffer:
  41. 41.```bash
  42. 42.sudo fallocate -l 4G /swapfile
  43. 43.sudo chmod 600 /swapfile
  44. 44.sudo mkswap /swapfile
  45. 45.sudo swapon /swapfile
  46. 46.echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
  47. 47.`

Prevention

  • Monitor memory usage with alerts at 70% and 85% thresholds
  • Set appropriate OOMScoreAdjust values for critical services
  • Configure swap space as a buffer before OOM trigger
  • Use memory limits (cgroups) for each service to contain consumption
  • Implement application-level memory monitoring and graceful degradation
  • Size servers with at least 30% memory headroom above normal usage