Introduction

When HAProxy reaches its configured maxconn limit, new connections are queued and eventually rejected if the queue fills up. This manifests as connection timeouts, 503 errors, or connection reset errors to clients. The maxconn setting protects HAProxy from being overwhelmed but needs proper tuning based on expected traffic patterns and system resources.

Symptoms

Error messages in HAProxy logs:

bash
Queue maximum reached, dropping connection
Server web_servers/web1 is maxconn
backend web_servers has no server available
Connection limit reached, rejecting

Client-facing symptoms: - HTTP 503 Service Unavailable - Connection timeout errors - Intermittent failures during traffic spikes - HAProxy stats show qcur (queue current) > 0 and qlimit reached

Common Causes

  1. 1.Global maxconn too low - Not enough connection slots for traffic
  2. 2.Backend maxconn too restrictive - Individual server limits exceeded
  3. 3.Frontend maxconn bottleneck - Frontend connection limit reached
  4. 4.Slow backends - Connections held too long, queue builds up
  5. 5.No queue timeout - Connections stuck in queue indefinitely
  6. 6.Traffic spike - Sudden burst exceeding planned capacity
  7. 7.Resource limits - System file descriptor limits hit

Step-by-Step Fix

Step 1: Check Current Connection Stats

```bash # Check global connection stats echo "show info" | socat stdio /var/run/haproxy.sock | grep -i conn

# Check queue depth per backend echo "show stat" | socat stdio /var/run/haproxy.sock | grep -E "pxname|qcur|qlimit|scur|slim"

# Check per-server connections echo "show stat" | socat stdio /var/run/haproxy.sock | column -t ```

Step 2: Check System Limits

```bash # Check file descriptor limits ulimit -n cat /proc/$(pidof haproxy)/limits | grep "open files"

# Check current file descriptor usage ls /proc/$(pidof haproxy)/fd | wc -l

# Check kernel connection tracking (if applicable) cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_max ```

Step 3: Review Current Configuration

```haproxy # Global settings - current config global maxconn 10000 # May need to increase based on traffic

# Frontend settings frontend http_front bind *:80 maxconn 5000 # Frontend-specific limit

# Backend server settings backend web_servers server web1 10.0.0.1:8080 maxconn 100 # Per-server limit ```

Step 4: Increase Connection Limits

```haproxy global # Increase global maxconn (most important) maxconn 100000

# Increase spread checks to avoid thundering herd spread-checks 5

defaults # Connection timeout settings timeout connect 10s timeout client 60s timeout server 60s timeout queue 30s

# Enable connection reuse option http-server-close option http-keep-alive

frontend http_front bind *:80 # Increase frontend limit maxconn 20000

backend web_servers balance roundrobin

# Per-server connection limits server web1 10.0.0.1:8080 maxconn 500 server web2 10.0.0.2:8080 maxconn 500

# Add more servers to distribute load server web3 10.0.0.3:8080 maxconn 500 ```

Step 5: Increase System Limits

```bash # Temporary increase ulimit -n 200000

# Permanent increase - edit /etc/security/limits.conf cat >> /etc/security/limits.conf << EOF haproxy soft nofile 200000 haproxy hard nofile 200000 EOF

# System-wide limits sysctl -w fs.file-max=500000 echo "fs.file-max=500000" >> /etc/sysctl.conf ```

Step 6: Configure Queue Behavior

```haproxy backend web_servers balance roundrobin

# Queue timeout - how long to wait for connection timeout queue 30s

# Connection retries retries 3

# Redispatch to another server if connection fails option redispatch

# Use leastconn for better distribution with slow requests balance leastconn

server web1 10.0.0.1:8080 maxconn 500 weight 100 server web2 10.0.0.2:8080 maxconn 500 weight 100 ```

Step 7: Verify the Fix

```bash # Reload configuration haproxy -c -f /etc/haproxy/haproxy.cfg systemctl reload haproxy

# Monitor connection stats watch -n 1 'echo "show stat" | socat stdio /var/run/haproxy.sock'

# Check queue depth echo "show stat" | socat stdio /var/run/haproxy.sock | awk -F',' '{print $1,$3,$4,$19}'

# Monitor during traffic tail -f /var/log/haproxy.log | grep -i "queue|maxconn|reject" ```

Advanced Diagnosis

Monitor Real-time Connection Stats

bash
# Continuous monitoring script
#!/bin/bash
while true; do
    echo "=== $(date) ==="
    echo "show info" | socat stdio /var/run/haproxy.sock | grep -E "Conn:|Queued:"
    echo "show stat" | socat stdio /var/run/haproxy.sock | grep -v "^#"
    sleep 1
done

Check Stick Table Connection Counts

```haproxy # If using stick tables for connection tracking frontend http_front stick-table type ip size 100k expire 30m store conn_cur,conn_rate(10s) http-request track-sc0 src

# Reject if too many connections from single IP http-request deny deny_status 429 if { sc_conn_cur gt 50 } ```

bash
# Show stick table entries
echo "show table http_front" | socat stdio /var/run/haproxy.sock

Optimize Backend Performance

```haproxy backend web_servers # Enable keep-alive to reduce connection overhead option http-server-close timeout http-keep-alive 10s timeout http-request 30s

# Connection pooling option prefer-last-server

# Health checks to avoid sending to slow servers option httpchk GET /health server web1 10.0.0.1:8080 maxconn 500 check inter 5s fall 3 rise 2 ```

Common Pitfalls

  • Frontend maxconn < backend maxconn sum - Creates artificial bottleneck
  • No queue timeout - Connections wait forever, queue never drains
  • Forgetting to increase ulimit - HAProxy can't use configured maxconn
  • Only global maxconn increased - Frontend/backend limits still restrictive
  • Weight imbalance - One server getting more connections than it can handle
  • Slow health check recovery - rise value too high during recovery

Best Practices

```haproxy global # Set to 80% of ulimit maxconn 80000

# Tune for performance nbproc 4 nbthread 4 cpu-map auto:1/1-4 0-3 spread-checks 5

defaults timeout connect 10s timeout client 60s timeout server 60s timeout queue 30s timeout http-keep-alive 10s retries 3 option redispatch option http-server-close

frontend http_front bind *:80 bind *:443 ssl crt /etc/ssl/certs/server.pem

# Frontend limit should match expected peak + buffer maxconn 50000

backend web_servers balance leastconn option httpchk GET /health

# Per-server maxconn based on backend capacity # Total backend maxconn = sum of all server maxconn server web1 10.0.0.1:8080 maxconn 1000 check server web2 10.0.0.2:8080 maxconn 1000 check server web3 10.0.0.3:8080 maxconn 1000 check ```

  • HAProxy Backend Down
  • HAProxy Health Check Failing
  • Nginx Load Balancer Timeout
  • AWS ALB Target Unhealthy