Introduction

When HAProxy reports "no healthy servers available," it means all backend servers in a pool have failed health checks or are administratively disabled. This is a critical failure that completely blocks traffic to that backend. The error typically appears as a 503 Service Unavailable response to clients while HAProxy itself remains operational.

Symptoms

Error messages you'll see in HAProxy logs:

bash
Server backend/web1 is DOWN, reason: Layer4 connection problem
Server backend/web2 is DOWN, reason: Layer7 wrong status
backend web_servers has no server available

Client-facing symptoms: - HTTP 503 Service Unavailable responses - HAProxy stats page shows all servers in MAINT or DOWN state - lastchg field shows recent state changes - Queue filling up on the backend

Common Causes

  1. 1.Network connectivity failure - Firewall rules, routing issues, or network segmentation
  2. 2.Application crash on all backends - Backend services stopped responding
  3. 3.Health check misconfiguration - Wrong endpoint, port, or expected response
  4. 4.Resource exhaustion - Backends overwhelmed, timing out health checks
  5. 5.DNS resolution failure - Server addresses not resolving
  6. 6.Certificate expiration - TLS handshake failures blocking health checks

Step-by-Step Fix

Step 1: Check HAProxy Stats

```bash # Via socket echo "show stat" | socat stdio /var/run/haproxy.sock | grep -E "backend|web"

# Via CLI haproxy -c -f /etc/haproxy/haproxy.cfg ```

Look for: - status column showing DOWN - lastchg showing when the state changed - check_status showing why health checks fail

Step 2: Verify Backend Connectivity

```bash # Test connectivity from HAProxy server to backend curl -v http://10.0.0.1:8080/health nc -zv 10.0.0.1 8080

# Check if backend process is running ssh backend-server 'systemctl status myapp' ssh backend-server 'netstat -tlnp | grep 8080' ```

Step 3: Check Health Check Configuration

haproxy
backend web_servers
    balance roundrobin
    option httpchk GET /health HTTP/1.1\r\nHost:\ example.com
    http-check expect status 200-399
    default-server inter 5s fall 3 rise 2
    server web1 10.0.0.1:8080 check
    server web2 10.0.0.2:8080 check

Verify the health check matches your application: - Endpoint path is correct - Port matches application listening port - Expected status code matches what app returns - Host header is set correctly if app requires it

Step 4: Check Firewall Rules

```bash # On HAProxy server - verify outbound rules iptables -L -n -v | grep 8080

# On backend server - verify inbound rules iptables -L -n -v | grep 8080

# Test from HAProxy to backend telnet 10.0.0.1 8080 ```

Step 5: Enable a Server Manually

```bash # Enable a specific server echo "enable server web_servers/web1" | socat stdio /var/run/haproxy.sock

# Or set to ready state echo "set server web_servers/web1 state ready" | socat stdio /var/run/haproxy.sock ```

Step 6: Verify the Fix

```bash # Check server status echo "show stat" | socat stdio /var/run/haproxy.sock

# Test through load balancer curl -v http://loadbalancer.example.com/

# Monitor for stability watch -n 1 'echo "show stat" | socat stdio /var/run/haproxy.sock' ```

Advanced Diagnosis

Debug Health Check Failures

```bash # Enable debug mode temporarily haproxy -d -f /etc/haproxy/haproxy.cfg

# Check specific backend details echo "show backend" | socat stdio /var/run/haproxy.sock

# View recent errors echo "show errors" | socat stdio /var/run/haproxy.sock ```

Check Agent Health

haproxy
# If using agent checks
backend web_servers
    server web1 10.0.0.1:8080 check agent-check agent-addr 10.0.0.1 agent-port 12345

Review Log Patterns

```bash # Find health check failures grep "Health check" /var/log/haproxy.log | tail -50

# Look for connection errors grep "connection refused|timeout|reset" /var/log/haproxy.log ```

Common Pitfalls

  • Health check too aggressive - inter 1s fall 1 marks servers down too quickly
  • Wrong health endpoint - App returns redirect instead of 200
  • Agent check conflicts - Agent and HTTP check disagreeing on health
  • Resolution dependency - Using hostnames that fail DNS during outages
  • Backend sticky sessions - Clients stuck to failed servers

Best Practices

```haproxy # Robust health check configuration backend web_servers balance roundrobin option httpchk GET /health HTTP/1.1\r\nHost:\ example.com http-check expect status 200-399

# Check every 5 seconds, need 3 failures to mark down # Need 2 successes to mark up again default-server inter 5s fall 3 rise 2

# Add grace period for slow starters server web1 10.0.0.1:8080 check on-error mark-down error-limit 5 observe layer7

# Enable backup server server backup 10.0.0.99:8080 backup check ```

  • HAProxy Health Check Failing
  • HAProxy Connection Refused to Backend
  • HAProxy SSL Handshake Failed
  • HAProxy Maxconn Reached