Fix Apache Graceful Restart Hanging With Stale Worker Processes

# Fix Apache Graceful Restart Hanging With Stale Worker Processes

You run sudo systemctl reload apache2 to apply a configuration change, but the command hangs indefinitely. Existing connections continue to work, but new connections are not served. Eventually you are forced to run sudo systemctl restart apache2, dropping all active connections.

Understanding Graceful Restart

1. 1.A graceful reload (apachectl graceful or systemctl reload apache2) tells Apache to:
2. 2.Read the new configuration
3. 3.Start new worker processes with the updated config
4. 4.Wait for old workers to finish their current requests
5. 5.Shut down old workers when they are idle

The hang occurs in step 3: old workers never become idle because they are stuck processing (or waiting for) requests that will never complete.

Diagnosing Stuck Workers

Check which workers are still active:

sudo apachectl status
# or
curl -s http://localhost/server-status?auto

Look at the scoreboard. Workers marked W (sending reply), R (reading request), or K (keepalive) are still active. Workers marked _ (waiting for connection) are idle.

Check individual process:

ps aux | grep apache2 | grep -v grep

Note the process IDs and their CPU/memory usage. Workers using 0% CPU and holding a connection for minutes are likely stuck.

Common Causes of Stuck Workers

1. Long-Running PHP Scripts

A PHP script with an infinite loop or blocking call:

# Check which PHP scripts are running
ps aux | grep php | grep -v grep

Kill the specific stuck process:

sudo kill -TERM <pid>

2. Slow Clients Holding Keepalive Connections

A client with a very slow network connection holds a keepalive connection open, preventing the worker from becoming idle. Fix with timeout tuning:

Timeout 60
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100

The KeepAliveTimeout 5 is aggressive enough to release connections from slow clients after 5 seconds of inactivity.

3. Backend Proxy Timeouts

When Apache proxies to a backend that never responds, the worker waits indefinitely:

ProxyTimeout 60

Without ProxyTimeout, Apache uses the global Timeout value, which may be 300 seconds (5 minutes). Set it to a reasonable value for your backend's expected response time.

Forcing Stuck Workers to Terminate

If a graceful reload is stuck, identify and kill the stubborn workers:

```bash # Find workers running longer than expected ps -eo pid,etime,args | grep apache2 | grep -v grep

# Kill specific stuck workers sudo kill -TERM <pid>

# If TERM does not work, force kill sudo kill -9 <pid> ```

Then retry the graceful reload:

sudo apachectl graceful

Preventing Future Hangs

Set Graceful Shutdown Timeout

In Apache 2.4.27+, use GracefulShutdownTimeout:

GracefulShutdownTimeout 30

This tells Apache to forcefully terminate old workers after 30 seconds of waiting during a graceful shutdown. Workers that have not finished their requests within 30 seconds are killed, preventing indefinite hangs.

Use mod_reqtimeout

<IfModule mod_reqtimeout.c>
    RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>

This module sets minimum data rates for incoming requests. If a client sends data slower than 500 bytes/second, Apache closes the connection. This prevents slow clients from holding workers indefinitely.

Automated Monitoring

```bash #!/bin/bash # Check for workers stuck for more than 120 seconds apache_pids=$(pgrep -f "apache2 -k start" | grep -v $(cat /var/run/apache2/apache2.pid 2>/dev/null))

for pid in $apache_pids; do elapsed=$(ps -p $pid -o etimes= 2>/dev/null | tr -d ' ') if [ -n "$elapsed" ] && [ "$elapsed" -gt 120 ]; then echo "WARNING: Worker $pid has been running for ${elapsed}s" cpu=$(ps -p $pid -o %cpu= | tr -d ' ') if [ "$(echo "$cpu < 1" | bc)" -eq 1 ]; then echo "Worker $pid has low CPU -- likely stuck. Killing." kill -TERM $pid fi fi done ```

Run this as a cron job every 5 minutes to automatically detect and kill stuck workers before they block graceful restarts.