Fix Nginx Upstream Connection Refused

What's Actually Happening

Nginx upstream connection refused occurs when Nginx cannot connect to the backend server defined in upstream block. The backend service is not listening on the configured port, or is not reachable.

The Error You'll See

Nginx error log:

```bash $ tail /var/log/nginx/error.log

[error] connect() failed (111: Connection refused) while connecting to upstream client: 192.168.1.1, server: example.com upstream: "http://127.0.0.1:8080" host: "example.com", referrer: "http://example.com/" ```

Browser error:

502 Bad Gateway
nginx/1.24.0

curl response:

```bash $ curl -I http://example.com/

HTTP/1.1 502 Bad Gateway Server: nginx/1.24.0 ```

Nginx status:

upstream connect error or disconnect/reset before headers
reason: Connection refused

Why This Happens

1. 1.Backend not running - Application server stopped
2. 2.Wrong port - Backend listening on different port
3. 3.Wrong host - Incorrect upstream IP/hostname
4. 4.Backend crashed - Application crashed after startup
5. 5.Firewall blocking - Local firewall blocking connection
6. 6.Socket permissions - Unix socket permission issues

Step 1: Check Backend Service Status

```bash # Check if backend process is running ps aux | grep node # Node.js app ps aux | grep python # Python app ps aux | grep java # Java app

# Check systemd service status systemctl status my-app

# Check if backend port is listening netstat -tlnp | grep 8080 # Should show backend process listening

# Check port connectivity nc -zv 127.0.0.1 8080 # Should succeed if backend running

# Check backend logs tail -100 /var/log/my-app/app.log journalctl -u my-app -f

# Start backend if not running systemctl start my-app # Or node app.js & ```

Step 2: Verify Upstream Configuration

```nginx # Check Nginx configuration cat /etc/nginx/conf.d/upstream.conf

# Verify upstream block upstream backend { server 127.0.0.1:8080; # Check: Is this the correct host and port? }

server { location / { proxy_pass http://backend; # Check: Does this match upstream name? } }

# Common issues: # - Wrong IP (e.g., 127.0.0.1 vs actual backend IP) # - Wrong port (e.g., 8080 vs 3000) # - Backend hostname not resolving

# Test configuration nginx -t

# Reload Nginx nginx -s reload ```

Step 3: Test Backend Directly

```bash # Test backend without Nginx curl http://127.0.0.1:8080/ # Should return response from backend

# Test backend health endpoint curl http://127.0.0.1:8080/health # Should return healthy status

# Test with telnet telnet 127.0.0.1 8080 # Should connect

# Test from Nginx server curl -v http://127.0.0.1:8080/

# If direct test fails: # Backend is the problem, not Nginx # Focus on fixing backend service ```

Step 4: Fix Upstream Host and Port

```nginx # WRONG: Incorrect upstream configuration upstream backend { server 127.0.0.1:8080; # Backend not on this port }

# CORRECT: Match backend actual port # Check what port backend actually uses: # Node.js default: 3000 # Python Flask: 5000 # Django: 8000 # Go: 8080 or configured

upstream backend { server 127.0.0.1:3000; # Match actual backend port }

# For multiple backends upstream backend { server 127.0.0.1:3000; server 127.0.0.1:3001; server 127.0.0.1:3002; }

# For external backend upstream backend { server backend-server:8080; }

# Apply changes nginx -t && nginx -s reload ```

Step 5: Check Unix Socket Permissions

```nginx # If using Unix socket upstream upstream backend { server unix:/var/run/my-app.sock; }

# Check socket exists ls -la /var/run/my-app.sock

# Check socket permissions # Nginx user (www-data/nginx) must have read/write stat /var/run/my-app.sock

# Fix socket permissions chmod 777 /var/run/my-app.sock # Quick fix (less secure) # Or chown www-data:www-data /var/run/my-app.sock

# Check Nginx user ps aux | grep nginx | head -1 # Shows user running Nginx

# Create socket directory with proper permissions mkdir -p /var/run/my-app chown www-data:www-data /var/run/my-app chmod 755 /var/run/my-app ```

Step 6: Add Upstream Health Checks

```nginx # Enable passive health checks upstream backend { server 127.0.0.1:8080 max_fails=3 fail_timeout=30s; server 127.0.0.1:8081 max_fails=3 fail_timeout=30s; }

# max_fails: Number of failed attempts before marking down # fail_timeout: Time to wait before retry

# Active health checks (requires nginx-plus or third-party module) # Or use passive monitoring with proxy_next_upstream

upstream backend { server 127.0.0.1:8080; server 127.0.0.1:8081 backup; # Backup server }

server { location / { proxy_pass http://backend; proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 10s; } }

# This retries on next upstream server on error ```

Step 7: Configure Keepalive Connections

```nginx # Enable keepalive to reduce connection overhead upstream backend { server 127.0.0.1:8080; keepalive 32; # Keep 32 idle connections open }

server { location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; # Clear connection header for keepalive } }

# This prevents connection refused by reusing connections # Particularly helpful for high-traffic backends ```

Step 8: Check Firewall Between Nginx and Backend

```bash # Check iptables on backend server iptables -L -n -v

# Allow connections from Nginx iptables -I INPUT -s nginx-server-ip -p tcp --dport 8080 -j ACCEPT

# Or allow localhost connections iptables -I INPUT -s 127.0.0.1 -p tcp --dport 8080 -j ACCEPT

# Check firewalld firewall-cmd --list-all firewall-cmd --add-rich-rule='rule family="ipv4" source address="nginx-ip" port protocol="tcp" port="8080" accept' --permanent firewall-cmd --reload

# SELinux may block connections getenforce # If Enforcing, check SELinux audit ausearch -m avc -ts recent | grep nginx

# Allow Nginx to network connect setsebool -P httpd_can_network_connect 1 ```

Step 9: Monitor Backend Availability

```bash # Create backend monitoring script cat << 'EOF' > /usr/local/bin/check_backend.sh #!/bin/bash BACKEND="127.0.0.1:8080"

if ! nc -z -w 5 $BACKEND 2>/dev/null; then echo "ALERT: Backend $BACKEND not reachable"

# Try to restart backend systemctl restart my-app

# Alert if restart fails if ! nc -z -w 5 $BACKEND 2>/dev/null; then mail -s "Backend Down" admin@company.com <<< \ "Backend $BACKEND is down and restart failed" fi fi EOF

chmod +x /usr/local/bin/check_backend.sh

# Add to cron echo "*/1 * * * * root /usr/local/bin/check_backend.sh" > /etc/cron.d/backend-monitor

# Monitor Nginx upstream status watch -n 5 'tail -10 /var/log/nginx/error.log | grep upstream' ```

Step 10: Add Error Handling in Nginx

```nginx # Custom error page for upstream errors server { location / { proxy_pass http://backend;

# Custom error responses error_page 502 503 504 /50x.html;

location = /50x.html { root /usr/share/nginx/html; internal; } } }

# Proxy error handling server { location / { proxy_pass http://backend; proxy_connect_timeout 5s; proxy_send_timeout 10s; proxy_read_timeout 10s;

proxy_intercept_errors on; error_page 502 = @fallback; }

location @fallback { proxy_pass http://backup-backend; } }

# This provides graceful fallback on upstream failure ```

Upstream Connection Refused Checklist

Verify the Fix

```bash # After fixing upstream configuration or backend

# 1. Test backend directly curl http://127.0.0.1:8080/ # Should return backend response

# 2. Test through Nginx curl http://example.com/ # Should return backend response, not 502

# 3. Check Nginx error log tail /var/log/nginx/error.log # Should not show "Connection refused"

# 4. Verify upstream status nginx -t && nginx -s reload

# 5. Monitor for recurring errors tail -f /var/log/nginx/error.log | grep --line-buffered "Connection refused" # Should not appear

# 6. Check backend stays running systemctl status my-app # Should be active

# 7. Test from browser # Navigate to example.com # Should load without 502 error ```

Related Issues

• [Fix Nginx 502 Bad Gateway](/articles/fix-nginx-502-bad-gateway)
• [Fix Nginx Upstream Timeout](/articles/fix-nginx-upstream-timeout)
• [Fix Nginx Proxy Configuration Error](/articles/fix-nginx-proxy-configuration-error)