What's Actually Happening

Nginx reverse proxy returns 502 Bad Gateway error when forwarding requests to upstream servers. Backend services may be down or unreachable.

The Error You'll See

```bash $ curl http://localhost

<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html> ```

Nginx error log:

bash
upstream prematurely closed connection while reading response header from upstream

Connection refused:

bash
connect() failed (111: Connection refused) while connecting to upstream

Timeout error:

bash
upstream timed out (110: Connection timed out) while reading response header from upstream

Why This Happens

  1. 1.Upstream down - Backend server not running
  2. 2.Wrong upstream address - Incorrect host/port configuration
  3. 3.Connection refused - Backend not listening on expected port
  4. 4.Timeout - Backend takes too long to respond
  5. 5.Proxy buffer issues - Response headers too large
  6. 6.SELinux/AppArmor - Security blocking connections

Step 1: Check Nginx Error Log

```bash # Check error log: tail -f /var/log/nginx/error.log

# Grep for specific errors: grep -i "upstream|502" /var/log/nginx/error.log | tail -20

# Check access log: tail -f /var/log/nginx/access.log

# Enable debug logging: # In nginx.conf: error_log /var/log/nginx/error.log debug;

# Reload Nginx: nginx -s reload

# Check Nginx status: systemctl status nginx

# Check Nginx process: ps aux | grep nginx

# Check configuration: nginx -t

# Check upstream config: grep -A 10 "upstream" /etc/nginx/nginx.conf grep -A 10 "upstream" /etc/nginx/conf.d/*.conf ```

Step 2: Check Upstream Server

```bash # Identify upstream server: grep -E "proxy_pass|upstream" /etc/nginx/conf.d/*.conf

# Test upstream connectivity: # If upstream is localhost:3000 curl -v http://localhost:3000

# If upstream is remote: curl -v http://backend-server:8080

# Test with telnet: telnet backend-server 8080

# Test with netcat: nc -zv backend-server 8080

# Check if backend running: ssh backend-server "systemctl status myapp"

# Check backend process: ssh backend-server "ps aux | grep myapp"

# Check backend port: ssh backend-server "netstat -tlnp | grep 8080"

# Restart backend if needed: ssh backend-server "systemctl restart myapp"

# Check backend logs: ssh backend-server "journalctl -u myapp -f" ```

Step 3: Fix Upstream Configuration

```bash # Check upstream block: upstream backend { server 127.0.0.1:3000; }

# Or in proxy_pass: location / { proxy_pass http://127.0.0.1:3000; }

# Common configuration issues:

# 1. Wrong port: # Check actual port backend uses proxy_pass http://127.0.0.1:3001; # Wrong port?

# 2. Wrong protocol: proxy_pass http://backend; # Use http:// not https:// if backend is HTTP

# 3. Missing upstream block: # If using upstream name: upstream my_backend { server 127.0.0.1:3000; } server { location / { proxy_pass http://my_backend; } }

# 4. DNS resolution: # Use resolver for dynamic upstreams: resolver 8.8.8.8; set $upstream http://backend.example.com; proxy_pass $upstream;

# 5. Multiple backends: upstream backend { server 127.0.0.1:3000; server 127.0.0.1:3001; server 127.0.0.1:3002; }

# Test configuration: nginx -t

# Reload: nginx -s reload ```

Step 4: Fix Timeout Configuration

```bash # Increase timeout settings:

location / { proxy_pass http://backend;

# Connection timeout proxy_connect_timeout 60s;

# Send timeout proxy_send_timeout 60s;

# Read timeout proxy_read_timeout 60s; }

# Or in http block (global): http { proxy_connect_timeout 60s; proxy_send_timeout 60s; proxy_read_timeout 60s; }

# For slow backends: proxy_read_timeout 300s; # 5 minutes

# For WebSocket: proxy_read_timeout 3600s; # 1 hour

# Keepalive connections: upstream backend { server 127.0.0.1:3000; keepalive 32; # Number of keepalive connections }

# Use keepalive: proxy_http_version 1.1; proxy_set_header Connection "";

# Check current timeouts: nginx -T | grep timeout ```

Step 5: Fix Proxy Headers

```bash # Proper proxy headers:

location / { proxy_pass http://backend;

# Standard headers proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;

# For HTTPS proxy_set_header X-Forwarded-Proto https; }

# Missing Host header causes issues: proxy_set_header Host $host; # Or specific host: proxy_set_header Host backend.example.com;

# For WebSocket: proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";

# Check for header size issues: proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k;

# Large headers: fastcgi_buffer_size 32k; fastcgi_buffers 8 16k; ```

Step 6: Check SELinux/AppArmor

```bash # Check SELinux status: getenforce

# If Enforcing, may block connections

# Check SELinux boolean for HTTP proxy: getsebool -a | grep httpd

# Allow HTTP network connections: setsebool -P httpd_can_network_connect 1

# Allow connections to specific port: setsebool -P httpd_can_network_connect_db 1

# Check SELinux audit log: ausearch -m avc -ts recent | grep nginx

# Create custom policy: grep nginx /var/log/audit/audit.log | audit2allow -M mynginx semodule -i mynginx.pp

# Temporarily set permissive: setenforce 0

# Check AppArmor status: aa-status

# Check AppArmor profile: aa-status | grep nginx

# Set to complain mode: aa-complain /etc/apparmor.d/usr.sbin.nginx

# Or disable for nginx: aa-disable /etc/apparmor.d/usr.sbin.nginx ```

Step 7: Fix Buffer Configuration

```bash # Buffer configuration for large responses:

# In http or server block: proxy_buffer_size 4k; proxy_buffers 8 4k; proxy_busy_buffers_size 8k;

# For large responses: proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k;

# Disable buffering (for streaming): proxy_buffering off;

# Temp file handling: proxy_temp_file_write_size 64k; proxy_max_temp_file_size 1024m;

# For FastCGI: fastcgi_buffer_size 16k; fastcgi_buffers 16 16k;

# Check current buffer settings: nginx -T | grep buffer

# Reload after changes: nginx -s reload ```

Step 8: Check Upstream Health

```bash # Configure health checks:

upstream backend { server 127.0.0.1:3000 max_fails=3 fail_timeout=30s; server 127.0.0.1:3001 max_fails=3 fail_timeout=30s; }

# Active health checks (requires nginx plus): health_check;

# Passive health checks: upstream backend { server 127.0.0.1:3000; server 127.0.0.1:3001 backup; }

# Check upstream status page (nginx plus): location /upstream_status { upstream_status; }

# Manual health check script: for port in 3000 3001 3002; do if nc -zv localhost $port 2>&1 | grep succeeded; then echo "Port $port: OK" else echo "Port $port: DOWN" fi done

# Check upstream sockets: ss -tlnp | grep -E "3000|3001|3002"

# Mark server as down: upstream backend { server 127.0.0.1:3000; server 127.0.0.1:3001 down; } ```

Step 9: Handle Specific Scenarios

```bash # Unix socket upstream: upstream backend { server unix:/var/run/app.sock; }

# Check socket exists: ls -la /var/run/app.sock

# Fix socket permissions: chmod 666 /var/run/app.sock chown nginx:nginx /var/run/app.sock

# HTTPS upstream: upstream backend { server backend.example.com:443; }

location / { proxy_pass https://backend; proxy_ssl_verify off; # If self-signed proxy_ssl_server_name on; }

# Backend requires authentication: proxy_set_header Authorization "Basic base64encoded";

# Rate limiting causing 502: # Check rate limit config: limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; limit_req zone=one burst=20;

# Increase burst: limit_req zone=one burst=50 nodelay;

# Check if backend rejecting connections: # Check backend logs for errors # Check backend resource usage (CPU, memory) ```

Step 10: Nginx 502 Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-nginx-502.sh #!/bin/bash

echo "=== Nginx Status ===" systemctl status nginx 2>/dev/null || echo "Service not running"

echo "" echo "=== Nginx Process ===" ps aux | grep nginx | grep -v grep || echo "No nginx process"

echo "" echo "=== Configuration Test ===" nginx -t 2>&1

echo "" echo "=== Upstream Configuration ===" grep -E "upstream|proxy_pass" /etc/nginx/nginx.conf 2>/dev/null grep -E "upstream|proxy_pass" /etc/nginx/conf.d/*.conf 2>/dev/null | head -20

echo "" echo "=== Upstream Connectivity ===" # Extract upstream addresses for addr in $(grep -oE "server [^;]+" /etc/nginx/conf.d/*.conf 2>/dev/null | grep -v "server_name" | awk '{print $2}' | tr -d ';'); do if [[ $addr == *":"* ]]; then host=$(echo $addr | cut -d: -f1) port=$(echo $addr | cut -d: -f2) echo "Testing $addr:" nc -zv $host $port 2>&1 || echo " Connection failed" fi done

echo "" echo "=== Recent 502 Errors ===" grep "502|upstream" /var/log/nginx/error.log 2>/dev/null | tail -10 || echo "No 502 errors"

echo "" echo "=== Access Log (502s) ===" grep " 502 " /var/log/nginx/access.log 2>/dev/null | tail -5 || echo "No 502 responses"

echo "" echo "=== SELinux Status ===" getenforce 2>/dev/null || echo "SELinux not installed"

echo "" echo "=== Proxy Timeouts ===" nginx -T 2>/dev/null | grep -E "proxy_.*timeout" | head -10

echo "" echo "=== Current Connections ===" ss -tlnp | grep nginx || netstat -tlnp | grep nginx

echo "" echo "=== Recommendations ===" echo "1. Verify upstream server is running" echo "2. Check upstream address and port correct" echo "3. Increase proxy timeouts if backend slow" echo "4. Check SELinux/AppArmor policies" echo "5. Verify proxy headers configured" echo "6. Check buffer sizes for large responses" echo "7. Review upstream health check settings" EOF

chmod +x /usr/local/bin/check-nginx-502.sh

# Usage: /usr/local/bin/check-nginx-502.sh ```

Nginx 502 Bad Gateway Checklist

CheckExpected
Upstream runningBackend server up
Upstream reachablePort accessible
Proxy timeoutAdequate for backend
Proxy headersHost, X-Forwarded headers set
SELinuxNetwork connections allowed
Buffer sizeAdequate for response size
Upstream configCorrect address and port

Verify the Fix

```bash # After fixing Nginx 502 error

# 1. Check upstream reachable curl http://localhost:3000 // Backend responds

# 2. Check Nginx config nginx -t // Configuration valid

# 3. Reload Nginx nginx -s reload // Reloaded successfully

# 4. Test proxy curl http://localhost // Returns 200 OK

# 5. Check error log tail -f /var/log/nginx/error.log // No 502 errors

# 6. Check access log tail -f /var/log/nginx/access.log // Shows 200 responses ```

  • [Fix Nginx 504 Gateway Timeout](/articles/fix-nginx-504-gateway-timeout)
  • [Fix Nginx Upstream Timed Out](/articles/fix-nginx-upstream-timed-out)
  • [Fix HAProxy Backend Down](/articles/fix-haproxy-backend-down)