HAProxy 503 No Backend Available

Introduction

HAProxy returns 503 when all backend servers are down or queue is full. This guide provides step-by-step diagnosis and resolution with specific commands and configuration examples.

Symptoms

Typical symptoms and error messages when this issue occurs:

bash

HTTP/1.1 503 Service Unavailable
no server is available to handle this request
Queue limit reached, connection rejected

Observable indicators: - Load balancer returns 5xx errors to clients - Backend servers marked as unhealthy - Traffic not reaching expected backends

Common Causes

1.HAProxy issues are typically caused by:
2.Health check configuration mismatch with backend
3.Timeout values too aggressive for application
4.Maxconn limits preventing connection acceptance
5.ACL or routing rules incorrectly matching traffic

Step-by-Step Fix

Step 1: Check Current State

bash

haproxy -c -f /etc/haproxy/haproxy.cfg

Step 2: Identify Root Cause

bash

echo "show stat" | socat stdio /var/run/haproxy.sock

Step 3: Apply Primary Fix

bash

# Primary configuration fix
upstream backend {
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    keepalive 32;
}

Apply this configuration and reload the load balancer.

Step 4: Apply Alternative Fix (If Needed)

bash

# Alternative fix: adjust timeouts
proxy_connect_timeout 10s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;

Step 5: Verify the Fix

After applying the fix, verify with:

bash

echo "show stat" | socat stdio /var/run/haproxy.sock | grep -v DOWN

Expected output should show healthy backends and successful request routing.

Common Pitfalls

Health check path not matching application endpoint
Timeout values shorter than application response time
Maxconn set too low for traffic load
Sticky session breaking after backend restart

Best Practices

Use option httpchk for HTTP health checks
Set appropriate inter, fall, rise values
Monitor backend server metrics
Use stick-table for rate limiting with care