Introduction When the load balancer's connection pool to backend servers is exhausted, new requests cannot be forwarded. This causes request queuing, increased latency, and eventual 503 errors.

Symptoms - Increased request latency - 503 errors during peak traffic - Backend connections at maximum - Error: "no live upstreams" in Nginx logs - Load balancer metrics show max connections reached

Common Causes - Connection pool size too small for traffic - Backend not closing connections (keepalive misuse) - Slow backend responses holding connections open - Connection pool not scaling with backend count - TCP connection leak between LB and backend

Step-by-Step Fix 1. **Check connection pool status': ```bash # Nginx curl http://localhost/nginx_status # HAProxy echo "show stat" | socat stdio /var/run/haproxy.sock | cut -d, -f1,2,5,6,8,33,34,36 ```

  1. 1.**Increase connection pool and keepalive':
  2. 2.```nginx
  3. 3.upstream backend {
  4. 4.server 10.0.1.10:8080;
  5. 5.server 10.0.1.11:8080;
  6. 6.keepalive 64;
  7. 7.}
  8. 8.server {
  9. 9.location / {
  10. 10.proxy_pass http://backend;
  11. 11.proxy_http_version 1.1;
  12. 12.proxy_set_header Connection "";
  13. 13.}
  14. 14.}
  15. 15.`

Prevention - Size connection pool based on expected concurrent connections - Enable keepalive between load balancer and backends - Monitor connection pool utilization - Set backend connection timeouts - Implement connection limits per backend server