Introduction

Apache's graceful restart (apachectl graceful) tells worker processes to finish serving current requests before exiting and loading the new configuration. When a worker process becomes stuck (e.g., waiting on an unresponsive database, hung external API call, or deadlocked thread), the graceful restart hangs indefinitely. The system shows:

bash
sudo apachectl graceful
# Command hangs, no output, Apache config not reloaded

The old workers continue running with the old configuration while the new master process waits for them to exit.

Symptoms

  • apachectl graceful command hangs indefinitely
  • systemctl reload apache2 times out after 90 seconds
  • Old worker processes remain running with previous configuration
  • New configuration changes are not applied
  • apachectl status shows workers stuck in "W" or "K" state for extended periods
  • Deployment scripts timeout waiting for Apache restart to complete

Common Causes

  • Worker process stuck waiting for response from an unresponsive backend (database, API, memcached)
  • PHP script with infinite loop or deadlock consuming a worker slot
  • Timeout directive set too high (default 300 seconds), delaying worker exit
  • External resource (NFS mount, DNS resolver) is unresponsive, causing workers to hang in I/O wait
  • mod_php with long-running scripts not respecting graceful shutdown signals

Step-by-Step Fix

  1. 1.Identify stuck worker processes:
  2. 2.```bash
  3. 3.ps aux | grep apache2 | grep -v grep
  4. 4.# Find workers with the oldest uptime
  5. 5.ps -eo pid,etime,cmd | grep apache2 | sort -k2 -r | head -10
  6. 6.`
  7. 7.Check what the stuck process is doing:
  8. 8.```bash
  9. 9.# Replace PID with the stuck worker PID
  10. 10.strace -p PID -c -T 2>&1 | head -20
  11. 11.# Or check open files
  12. 12.lsof -p PID | head -20
  13. 13.`
  14. 14.This reveals whether the process is waiting on a network connection, file I/O, or is truly deadlocked.
  15. 15.Reduce the Timeout directive to prevent workers from hanging too long:
  16. 16.```apache
  17. 17.Timeout 60
  18. 18.`
  19. 19.This limits how long a worker can spend on a single request before being killed.
  20. 20.Force kill stuck workers and then restart:
  21. 21.```bash
  22. 22.# Find and kill workers older than expected
  23. 23.sudo kill -TERM $(pgrep -o apache2)
  24. 24.# Wait 10 seconds
  25. 25.sleep 10
  26. 26.# If still running, force kill
  27. 27.sudo kill -9 $(pgrep -o apache2)
  28. 28.# Then restart
  29. 29.sudo systemctl start apache2
  30. 30.`
  31. 31.Configure a systemd timeout override to prevent indefinite hangs:
  32. 32.Create /etc/systemd/system/apache2.service.d/override.conf:
  33. 33.```ini
  34. 34.[Service]
  35. 35.TimeoutStopSec=30
  36. 36.`
  37. 37.Then run sudo systemctl daemon-reload. This ensures systemd kills Apache if it does not stop within 30 seconds.
  38. 38.Set RequestReadTimeout to catch slow requests:
  39. 39.```apache
  40. 40.<IfModule mod_reqtimeout.c>
  41. 41.RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
  42. 42.</IfModule>
  43. 43.`
  44. 44.This disconnects clients that send data too slowly, freeing worker slots.

Prevention

  • Set appropriate Timeout values (60-120 seconds) based on your application's longest expected request
  • Use mod_reqtimeout to prevent slow clients from holding worker slots indefinitely
  • Monitor worker process age and alert on workers running longer than expected
  • Implement application-level timeouts for all external service calls (database, APIs, cache)
  • Use apachectl graceful during deployments but have a systemctl restart apache2 fallback
  • Add a deployment script wrapper that checks for stuck workers before initiating a graceful restart:
  • ```bash
  • stuck=$(ps aux | grep apache2 | grep -c "W|K")
  • if [ "$stuck" -gt 5 ]; then
  • echo "$stuck workers stuck, performing hard restart"
  • sudo systemctl restart apache2
  • else
  • sudo apachectl graceful
  • fi
  • `