Introduction

A "connection refused" error from HAProxy to a backend server indicates that the TCP connection attempt was actively rejected. This is different from a timeout (no response) or a reset (connection established then dropped). The refusal typically means no service is listening on the target port, or a firewall is actively rejecting the connection with a TCP RST packet.

Symptoms

Error messages in HAProxy logs:

bash
Server backend/web1 is DOWN, reason: Layer4 connection problem
Connection refused
connect(10.0.0.1:8080) failed: Connection refused
backend web_servers has no server available

Observable indicators: - Health checks fail immediately (not after timeout) - check_status shows "L4TOUT" or "L4CON" - No latency before the error (instant refusal) - Backend server may be running but not on expected port

Common Causes

  1. 1.Backend service not running - Application crashed or not started
  2. 2.Wrong port configured - HAProxy pointing to incorrect port
  3. 3.Firewall blocking connections - iptables, security groups, or cloud firewall rules
  4. 4.Service binding to wrong interface - App listening on localhost instead of 0.0.0.0
  5. 5.Port already in use - Another process occupying the expected port
  6. 6.SELinux/AppArmor blocking - Security policies preventing connections

Step-by-Step Fix

Step 1: Verify Backend Service Status

```bash # Check if service is running ssh backend-server 'systemctl status myapp' ssh backend-server 'service myapp status'

# Check process is running ssh backend-server 'ps aux | grep myapp'

# Check what's listening on the port ssh backend-server 'netstat -tlnp | grep 8080' ssh backend-server 'ss -tlnp | grep 8080' ```

Step 2: Test Connectivity Directly

```bash # Test from HAProxy server telnet 10.0.0.1 8080 nc -zv 10.0.0.1 8080

# Test with curl curl -v http://10.0.0.1:8080/health

# Check network path traceroute 10.0.0.1 mtr 10.0.0.1 ```

Step 3: Check Firewall Rules

```bash # On backend server - check incoming rules iptables -L -n -v --line-numbers iptables -L INPUT -n -v nft list ruleset

# For cloud environments, check security groups # AWS aws ec2 describe-security-groups --group-ids sg-123456

# Check if port is allowed iptables -L -n | grep 8080 ```

Step 4: Verify Service Binding

```bash # Check what interface the service is bound to ssh backend-server 'ss -tlnp | grep 8080' # Output should show 0.0.0.0:8080 or :::8080 # Not 127.0.0.1:8080 (localhost only)

# Check application configuration ssh backend-server 'cat /etc/myapp/config.yml | grep -A5 bind' ```

Step 5: Check HAProxy Configuration

```haproxy backend web_servers balance roundrobin

# Verify port matches backend service server web1 10.0.0.1:8080 check server web2 10.0.0.2:8080 check

# Check if using correct IP (not 127.0.0.1 if different machine) server web1 10.0.0.1:8080 check ```

Step 6: Start or Restart Backend Service

```bash # Start the service ssh backend-server 'systemctl start myapp'

# Enable at boot ssh backend-server 'systemctl enable myapp'

# Check logs for startup issues ssh backend-server 'journalctl -u myapp -f' ```

Step 7: Verify the Fix

```bash # Check HAProxy stats echo "show stat" | socat stdio /var/run/haproxy.sock

# Monitor health check status watch -n 1 'echo "show stat" | socat stdio /var/run/haproxy.sock | grep web'

# Test through load balancer curl -v http://loadbalancer.example.com/ ```

Advanced Diagnosis

Check for Port Conflicts

```bash # Find what's using the port ssh backend-server 'lsof -i :8080' ssh backend-server 'fuser 8080/tcp'

# Kill conflicting process if needed ssh backend-server 'fuser -k 8080/tcp' ```

Debug SELinux Issues

```bash # Check SELinux status getenforce

# Check if SELinux is blocking ausearch -m avc -ts recent

# Allow HTTP network connections setsebool -P httpd_can_network_connect 1

# Allow specific port semanage port -a -t http_port_t -p tcp 8080 ```

Check Systemd Socket Activation

```bash # If using socket activation, check socket status systemctl status myapp.socket systemctl list-sockets

# Check socket configuration systemctl cat myapp.socket ```

Common Pitfalls

  • Container port mapping - Docker/Podman mapping wrong ports
  • Service mesh sidecar - Istio/Linkerd intercepting connections
  • IPv4 vs IPv6 - Service listening on one but HAProxy using the other
  • Kubernetes service ports - TargetPort vs Port confusion
  • Private IP change - Instance restarted with new IP
  • DNS resolution timing - Using hostname that resolves incorrectly

Best Practices

```haproxy # Robust backend configuration backend web_servers balance roundrobin option httpchk GET /health HTTP/1.1\r\nHost:\ example.com http-check expect status 200

# Health check settings default-server inter 5s fall 3 rise 2

# Use IP addresses for reliability server web1 10.0.0.1:8080 check

# Enable connection retries retries 3 option redispatch

# Connection settings timeout connect 10s timeout server 60s ```

bash
# Monitoring script for backend availability
#!/bin/bash
for server in web1 web2; do
    if ! nc -zv 10.0.0.$i 8080 2>/dev/null; then
        echo "ALERT: $server is not accepting connections"
    fi
done
  • HAProxy Backend Down
  • HAProxy Health Check Failing
  • HAProxy Maxconn Reached
  • Nginx Load Balancer Timeout