Fix HAProxy Backend Health Check Failed

What's Actually Happening

HAProxy marks backend servers as down due to failed health checks. Traffic is not distributed to those servers even though they may be running.

The Error You'll See

```bash $ haproxy -vv

Backend servers showing DOWN status ```

HAProxy stats output:

# haproxy stats
backend mybackend
  server1 DOWN
  server2 DOWN

Log messages:

Health check for server server1/mybackend failed, reason: Layer4 connection problem
Server server1/mybackend is DOWN

Why This Happens

1. 1.Backend server not running - Application server stopped
2. 2.Wrong health check port - Check configured for wrong port
3. 3.Health check URI failing - Check endpoint returning errors
4. 4.Network connectivity - Firewall or network blocking checks
5. 5.Timeout too short - Health check timeout insufficient
6. 6.SSL certificate issues - HTTPS health check failing

Step 1: Check HAProxy Status

```bash # Check HAProxy service: systemctl status haproxy

# Check HAProxy stats: echo "show stat" | socat stdio /var/run/haproxy.sock

# Or via stats page: curl http://localhost:8404/stats

# Check logs: tail -f /var/log/haproxy.log

# Check configuration: haproxy -c -f /etc/haproxy/haproxy.cfg ```

Step 2: Check Backend Server Status

```bash # Check if backend server is running: curl -I http://backend-server:8080

# Check port: nc -zv backend-server 8080

# Check service on backend: ssh backend-server 'systemctl status nginx'

# Check backend logs: ssh backend-server 'tail -f /var/log/nginx/error.log'

# Test direct connection: curl http://backend-server:8080/health ```

Step 3: Check Health Check Configuration

```bash # View HAProxy config: cat /etc/haproxy/haproxy.cfg

# Check backend config: grep -A 20 "backend mybackend" /etc/haproxy/haproxy.cfg

# Example health check config: backend mybackend option httpchk GET /health http-check expect status 200 server server1 10.0.0.1:8080 check server server2 10.0.0.2:8080 check

# Check health check parameters: # check - enable health checking # inter - check interval (default 2s) # fall - number of fails to mark down (default 3) # rise - number of successes to mark up (default 2) ```

Step 4: Check Health Check Endpoint

```bash # Test health check endpoint: curl -v http://backend-server:8080/health

# Expected response: HTTP/1.1 200 OK

# Check for specific status: curl -s -o /dev/null -w "%{http_code}" http://backend-server:8080/health

# Test with HAProxy's check: curl -H "User-Agent: HAProxy" http://backend-server:8080/health

# Check if endpoint exists: curl http://backend-server:8080/status curl http://backend-server:8080/ping ```

Step 5: Check Network Connectivity

```bash # Test from HAProxy server: nc -zv backend-server 8080

# Check firewall: iptables -L -n | grep 8080

# Allow health check traffic: iptables -I INPUT -p tcp --dport 8080 -j ACCEPT

# Check routing: traceroute backend-server

# Check DNS: dig backend-server

# Ping test: ping -c 3 backend-server

# Check from HAProxy container: docker exec haproxy nc -zv backend-server 8080 ```

Step 6: Check Health Check Timeout

```bash # View timeout settings: grep -E "timeout|inter" /etc/haproxy/haproxy.cfg

# Common timeouts: timeout connect 5s timeout client 50s timeout server 50s timeout check 5s

# Server check parameters: server server1 10.0.0.1:8080 check inter 5s fall 3 rise 2

# Adjust if needed: # inter 10s - longer interval # fall 5 - require 5 failures # rise 3 - require 3 successes

# Edit config: vim /etc/haproxy/haproxy.cfg ```

Step 7: Fix HTTP vs TCP Health Check

```bash # HTTP health check (Layer 7): backend mybackend mode http option httpchk GET /health HTTP/1.1\r\nHost:\ myapp.com http-check expect status 200 server server1 10.0.0.1:8080 check

# TCP health check (Layer 4): backend mybackend mode tcp # No HTTP check, just TCP connect server server1 10.0.0.1:8080 check

# SSL health check: backend mybackend mode tcp option ssl-hello-chk server server1 10.0.0.1:443 check ssl verify none

# Or custom SSL check: backend mybackend option httpchk GET /health server server1 10.0.0.1:443 check ssl alpn h2,http/1.1 ```

Step 8: Check SSL Health Checks

```bash # For HTTPS backends: backend https_backend mode tcp option ssl-hello-chk server server1 10.0.0.1:443 check

# Or with certificate verification: backend https_backend server server1 10.0.0.1:443 check ssl verify required ca-file /etc/ssl/certs/ca.pem

# Test SSL connection: openssl s_client -connect backend-server:443

# Check certificate: echo | openssl s_client -connect backend-server:443 2>/dev/null | openssl x509 -noout -dates

# Skip verification for health check: backend https_backend server server1 10.0.0.1:443 check ssl verify none ```

Step 9: Force Backend UP for Testing

```bash # Enable backend server manually: echo "enable server mybackend/server1" | socat stdio /var/run/haproxy.sock

# Check status: echo "show stat" | socat stdio /var/run/haproxy.sock

# Disable server: echo "disable server mybackend/server1" | socat stdio /var/run/haproxy.sock

# Set weight: echo "set weight mybackend/server1 100" | socat stdio /var/run/haproxy.sock

# Reload HAProxy: systemctl reload haproxy

# Full restart: systemctl restart haproxy ```

Step 10: Monitor and Debug Health Checks

```bash # Enable debug logging: # In haproxy.cfg: global debug

# Or specific level: global log /dev/log local0 debug

# Watch logs: tail -f /var/log/haproxy.log | grep -i "health|check"

# Monitor stats: watch -n 1 'echo "show stat" | socat stdio /var/run/haproxy.sock'

# Check backend state: echo "show backend mybackend" | socat stdio /var/run/haproxy.sock

# Check server state: echo "show servers state" | socat stdio /var/run/haproxy.sock

# Create monitoring script: cat << 'EOF' > /usr/local/bin/check-haproxy.sh #!/bin/bash

echo "=== HAProxy Status ===" systemctl status haproxy | head -5

echo "" echo "=== Backend Status ===" echo "show stat" | socat stdio /var/run/haproxy.sock | grep -E "backend|server"

echo "" echo "=== Health Check Errors ===" tail -50 /var/log/haproxy.log | grep -i "health|check|DOWN"

echo "" echo "=== Backend Connectivity ===" for server in server1 server2; do nc -zv $server 8080 2>&1 done EOF

chmod +x /usr/local/bin/check-haproxy.sh ```

HAProxy Health Check Checklist

Verify the Fix

```bash # After fixing health check

# 1. Check HAProxy status systemctl status haproxy // Active: active (running)

# 2. Check backend servers echo "show stat" | socat stdio /var/run/haproxy.sock // Servers show UP

# 3. Test health endpoint curl backend-server:8080/health // Returns 200 OK

# 4. Check logs tail -f /var/log/haproxy.log // No DOWN messages

# 5. Test traffic distribution curl http://haproxy-server/ // Responses from backends

# 6. Monitor stats watch -n 1 'echo "show stat" | socat stdio /var/run/haproxy.sock' // All servers UP ```

Related Issues

• [Fix Nginx Upstream Timed Out](/articles/fix-nginx-upstream-timed-out)
• [Fix Load Balancer Backend Unhealthy](/articles/fix-load-balancer-backend-unhealthy)
• [Fix HAProxy Configuration Error](/articles/fix-haproxy-configuration-error)