What's Actually Happening
Nginx returns 504 Gateway Timeout when upstream server does not respond within the timeout period. Backend applications are slow or unresponsive.
The Error You'll See
504 Gateway Timeout:
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>Nginx error log:
```bash tail -f /var/log/nginx/error.log
upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.0.1, server: example.com, request: "GET /api/endpoint HTTP/1.1", upstream: "http://10.0.0.2:8080/api/endpoint" ```
Why This Happens
- 1.Backend slow - Upstream application processing slowly
- 2.Backend down - Upstream server not responding
- 3.Proxy timeout low - Timeout setting too short
- 4.Network latency - Slow network between Nginx and upstream
- 5.Backend overloaded - Too many requests to backend
- 6.Large response - Large response body taking long to transfer
- 7.Backend hung - Application process stuck
Step 1: Check Nginx Error Logs
```bash tail -f /var/log/nginx/error.log
grep -i "upstream timed out" /var/log/nginx/error.log
grep -i "504" /var/log/nginx/error.log | tail -20
grep -E "timed out|upstream" /var/log/nginx/error.log | tail -50
journalctl -u nginx -f | grep -i upstream ```
Step 2: Test Backend Directly
```bash # Test backend directly: curl -v http://backend-server:8080/api/endpoint
# Test with timing: time curl http://backend-server:8080/api/endpoint
# Test health endpoint: curl -v http://backend-server:8080/health
# Check backend process: ssh backend-server 'ps aux | grep application'
ssh backend-server 'systemctl status app-service'
# Check backend logs: ssh backend-server 'tail -f /var/log/app/error.log' ```
Step 3: Check Upstream Configuration
```bash # View upstream config: cat /etc/nginx/conf.d/upstream.conf
# Check nginx.conf: cat /etc/nginx/nginx.conf | grep -A 20 "upstream"
# Example upstream: upstream backend { server 10.0.0.2:8080; server 10.0.0.3:8080 backup; }
# Check server block: cat /etc/nginx/sites-enabled/default | grep -A 30 "location"
# Check proxy config: cat /etc/nginx/sites-enabled/default | grep proxy
# Test config: nginx -t ```
Step 4: Increase Proxy Timeout
```bash # Proxy timeout settings: # In location or server block:
location /api/ { proxy_pass http://backend; proxy_connect_timeout 60s; proxy_send_timeout 60s; proxy_read_timeout 60s; }
# Or globally: proxy_connect_timeout 60s; proxy_send_timeout 60s; proxy_read_timeout 60s;
# For slow backends, increase: proxy_read_timeout 300s;
# Apply changes: nginx -t systemctl reload nginx ```
Step 5: Check Backend Response Time
```bash # Check backend response time: time curl http://backend-server:8080/api/endpoint
# Check with verbose: curl -w "Time: %{time_total}s\n" http://backend-server:8080/api/endpoint
# Check backend metrics: # If backend has metrics endpoint: curl http://backend-server:8080/metrics
# Check backend process stats: ssh backend-server 'top -p $(pgrep -f application)'
# Check backend resource usage: ssh backend-server 'free -h' ssh backend-server 'df -h'
# Check backend connections: ssh backend-server 'ss -tnp | grep 8080' ```
Step 6: Check Upstream Keepalive
```bash # Keepalive connections to upstream: upstream backend { server 10.0.0.2:8080; keepalive 32; }
location /api/ { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; }
# Check current connections: ss -tnp | grep 8080
# Check keepalive stats: # In Nginx logs or metrics ```
Step 7: Check Backend Load
```bash # Check backend CPU: ssh backend-server 'top -b -n 1 | head -20'
# Check backend memory: ssh backend-server 'free -h'
# Check backend disk IO: ssh backend-server 'iostat -x 1 5'
# Check backend network: ssh backend-server 'iftop -i eth0'
# Check backend connections: ssh backend-server 'ss -s'
# Check backend process count: ssh backend-server 'ps aux | grep application | wc -l'
# Check backend request queue: ssh backend-server 'netstat -an | grep 8080 | grep SYN' ```
Step 8: Enable Upstream Health Checks
```bash # Passive health checks: upstream backend { server 10.0.0.2:8080 max_fails=3 fail_timeout=30s; server 10.0.0.3:8080 max_fails=3 fail_timeout=30s; }
# Active health checks (Nginx Plus): # Or use nginx-module-healthcheck
# Check upstream status: # If enabled, check: # http://nginx-server/upstream_status
# Monitor upstream: watch -n 5 'curl -s http://localhost/nginx_status 2>/dev/null || echo "Status not available"' ```
Step 9: Implement Retry Logic
```bash # Retry on failure: location /api/ { proxy_pass http://backend; proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 30s; }
# This retries on: # - error (connection error) # - timeout (upstream timeout) # - http_502, http_503, http_504 (bad responses)
# Reload nginx: nginx -t systemctl reload nginx ```
Step 10: Monitor Upstream Performance
```bash # Enable nginx status: # In nginx.conf: location /nginx_status { stub_status on; access_log off; allow 127.0.0.1; deny all; }
# Check status: curl http://localhost/nginx_status
# Nginx exporter for Prometheus: # Use nginx-prometheus-exporter
# Monitor upstream response time: # In log format: log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$upstream_response_time"';
# Check upstream response times: tail -f /var/log/nginx/access.log | grep -E "upstream_response_time" ```
Nginx Upstream Timeout Checklist
| Check | Command | Expected |
|---|---|---|
| Error logs | tail error.log | No timeout errors |
| Backend response | curl backend | Response received |
| Proxy timeout | nginx config | Sufficient time |
| Backend load | top/free | Resources available |
| Keepalive | upstream config | Configured |
| Health checks | upstream config | Enabled |
Verify the Fix
```bash # 1. Check no timeout errors tail -f /var/log/nginx/error.log | grep -i timeout // No new timeouts
# 2. Test endpoint curl -v http://example.com/api/endpoint // Response received
# 3. Check response time curl -w "%{time_total}s\n" -o /dev/null -s http://example.com/api/endpoint // Reasonable time
# 4. Test multiple requests for i in {1..10}; do curl -o /dev/null -s -w "%{http_code}\n" http://example.com/api/endpoint; done // All 200
# 5. Monitor upstream watch -n 5 'curl -s http://localhost/nginx_status' // Normal activity
# 6. Check nginx config nginx -t // Syntax ok ```
Related Issues
- [Fix Nginx 502 Bad Gateway](/articles/fix-nginx-502-bad-gateway)
- [Fix Nginx 503 Service Unavailable](/articles/fix-nginx-503-service-unavailable)
- [Fix Apache Backend Connection Failed](/articles/fix-apache-backend-connection-failed)