Introduction

HAProxy backend queue overflows when server response time exceeds timeout and queue limit. This guide provides step-by-step diagnosis and resolution with specific commands and configuration examples.

Symptoms

Typical symptoms and error messages when this issue occurs:

bash
Load balancer error: backend unavailable
Check health check configuration
Verify backend server status

Observable indicators: - Load balancer returns 5xx errors to clients - Backend servers marked as unhealthy - Traffic not reaching expected backends

Common Causes

  1. 1.HAProxy issues are typically caused by:
  2. 2.Health check configuration mismatch with backend
  3. 3.Timeout values too aggressive for application
  4. 4.Maxconn limits preventing connection acceptance
  5. 5.ACL or routing rules incorrectly matching traffic

Step-by-Step Fix

Step 1: Check Current State

bash
haproxy -c -f /etc/haproxy/haproxy.cfg

Step 2: Identify Root Cause

bash
echo "show stat" | socat stdio /var/run/haproxy.sock

Step 3: Apply Primary Fix

bash
# HAProxy backend health check configuration
backend web_servers
    balance roundrobin
    option httpchk GET /health HTTP/1.1\r\nHost:\ example.com
    http-check expect status 200
    default-server inter 5s fall 3 rise 2
    server web1 10.0.0.1:8080 check
    server web2 10.0.0.2:8080 check

Apply this configuration and reload the load balancer.

Step 4: Apply Alternative Fix (If Needed)

bash
# Alternative fix: adjust timeouts
proxy_connect_timeout 10s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;

Step 5: Verify the Fix

After applying the fix, verify with:

bash
echo "show stat" | socat stdio /var/run/haproxy.sock | grep -v DOWN

Expected output should show healthy backends and successful request routing.

Common Pitfalls

  • Health check path not matching application endpoint
  • Timeout values shorter than application response time
  • Maxconn set too low for traffic load
  • Sticky session breaking after backend restart

Best Practices

  • Use option httpchk for HTTP health checks
  • Set appropriate inter, fall, rise values
  • Monitor backend server metrics
  • Use stick-table for rate limiting with care
  • HAProxy Backend Server Down
  • HAProxy 503 No Backend Available
  • HAProxy SSL Termination Failed
  • HAProxy Sticky Session Not Working