Fix Fluentd Output Connection Refused

What's Actually Happening

Fluentd cannot connect to output destinations like Elasticsearch, Kafka, or S3. Logs fail to be delivered and may be lost or buffered indefinitely.

The Error You'll See

Connection refused:

[error]: #0 unexpected error error_class=Errno::ECONNREFUSED error="Connection refused"

Output error:

[error]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2024-01-01 00:00:00 +0000 chunk="5a8f3b..."
[error]: #0 error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Could not connect to Elasticsearch"

Timeout:

[warn]: #0 timed out to send data to output plugin
[error]: #0 send data failed error_class=Timeout::Error error="execution expired"

Why This Happens

1. 1.Destination down - Elasticsearch, Kafka, or other destination not running
2. 2.Wrong endpoint - Incorrect host or port in configuration
3. 3.Network blocked - Firewall or security group blocking connection
4. 4.DNS failure - Hostname not resolving correctly
5. 5.TLS issues - Certificate problems for HTTPS connections
6. 6.Service overload - Destination not accepting new connections

Step 1: Check Fluentd Status

```bash # Check Fluentd running: systemctl status td-agent # Or: systemctl status fluentd

# Check Fluentd logs: tail -f /var/log/td-agent/td-agent.log

# Check Fluentd configuration: cat /etc/td-agent/td-agent.conf

# Check Fluentd metrics: curl http://localhost:24220/api/plugins.json

# Check buffer status: ls -la /var/log/td-agent/buffer/ ```

Step 2: Check Destination Availability

```bash # Check Elasticsearch: curl -I http://elasticsearch:9200

# Check Kafka: nc -zv kafka 9092

# Check specific port: nc -zv destination-host 9200

# Test connectivity: telnet elasticsearch 9200

# Check DNS: nslookup elasticsearch dig elasticsearch

# Check if destination is running: # For Elasticsearch: systemctl status elasticsearch curl http://elasticsearch:9200/_cluster/health

# For Kafka: systemctl status kafka kafka-topics.sh --bootstrap-server kafka:9092 --list

# Check from Fluentd host: ping elasticsearch ```

Step 3: Check Output Configuration

```xml # Check td-agent.conf: <match *.**> @type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix fluentd

# Connection settings connect_timeout 10 request_timeout 60

# Buffer settings <buffer> @type file path /var/log/td-agent/buffer timekey 60 timekey_wait 30 flush_mode interval flush_interval 10s flush_thread_count 4 retry_type exponential_backoff retry_wait 1s retry_max_interval 60s retry_timeout 24h retry_forever true chunk_limit_size 16MB queue_limit_length 512 </buffer> </match> ```

Step 4: Check Network Connectivity

```bash # Check firewall: iptables -L -n | grep 9200 iptables -L -n | grep 9092

# Allow port: iptables -I INPUT -p tcp --dport 9200 -j ACCEPT

# For firewalld: firewall-cmd --list-ports firewall-cmd --add-port=9200/tcp --permanent firewall-cmd --reload

# Check SELinux: getenforce

# If enforcing, may block outbound connections: setsebool -P httpd_can_network_connect 1

# Check if port listening on destination: ss -tlnp | grep 9200

# Test from Fluentd server: curl -v http://elasticsearch:9200 ```

Step 5: Configure Retry Settings

```xml # Robust retry configuration: <match *.**> @type elasticsearch host elasticsearch port 9200

# Reconnect settings reconnect_on_error true reload_on_failure true reload_connections false

# Timeout settings connect_timeout 30 request_timeout 120 http_backend typhoeus

<buffer> @type file path /var/log/td-agent/buffer

# Time-based chunking timekey 60 timekey_wait 30

# Flush settings flush_mode interval flush_interval 10s flush_thread_count 8

# Retry settings retry_type exponential_backoff retry_wait 1s retry_max_interval 300s retry_timeout 72h retry_forever true

# Buffer limits chunk_limit_size 16MB total_limit_size 2GB queue_limit_length 1024

# Overflow action overflow_action block </buffer>

# TLS settings ssl_verify true ca_file /etc/ssl/certs/ca.crt </match> ```

Step 6: Check TLS Configuration

```xml # For HTTPS connections: <match secure.**> @type elasticsearch host elasticsearch-secure port 9243 scheme https

# TLS verification ssl_verify true ca_file /etc/td-agent/ca.crt client_cert /etc/td-agent/client.crt client_key /etc/td-agent/client.key client_key_passphrase your_passphrase </match>

# For testing (disable verification): <match secure.**> @type elasticsearch host elasticsearch-secure port 9243 scheme https ssl_verify false ssl_version TLSv1_2 </match>

# Check certificates: openssl s_client -connect elasticsearch:9243 -showcerts ```

Step 7: Check Authentication

```xml # Elasticsearch with authentication: <match *.**> @type elasticsearch host elasticsearch port 9200 user fluentd password your_password

# Or use API key: api_key your_api_key_base64 </match>

# Kafka with SASL: <match kafka.**> @type kafka2 brokers kafka1:9092,kafka2:9092 topic logs

<format> @type json </format>

sasl_over_ssl true username fluentd password your_password ssl_ca_cert /etc/td-agent/kafka-ca.crt </match>

# S3 with IAM: <match s3.**> @type s3 aws_key_id your_key aws_sec_key your_secret s3_bucket your-bucket s3_region us-east-1 path logs/ </match> ```

Step 8: Test Output Manually

```bash # Test Elasticsearch output: curl -X POST "http://elasticsearch:9200/fluentd/_doc" \ -H 'Content-Type: application/json' \ -d '{"message":"test"}'

# Test Kafka output: kafka-console-producer.sh --bootstrap-server kafka:9092 --topic logs # Type messages, should succeed

# Test with Fluentd debug: td-agent -c /etc/td-agent/td-agent.conf -vv

# Or with Fluentd command: fluentd -c /etc/td-agent/td-agent.conf -vv

# Send test message: curl -X POST http://localhost:8888/test.tag -d '{"message":"test"}' ```

Step 9: Monitor Buffer Status

```bash # Check buffer directory: ls -la /var/log/td-agent/buffer/

# Buffer files growing: du -sh /var/log/td-agent/buffer/ watch -n 5 'du -sh /var/log/td-agent/buffer/'

# Check buffer stats via API: curl http://localhost:24220/api/plugins.json | jq '.plugins[] | select(.type=="elasticsearch")'

# Monitor metrics: curl http://localhost:24220/api/metrics.json | jq

# Create monitoring script: cat << 'EOF' > /usr/local/bin/monitor-fluentd.sh #!/bin/bash

echo "=== Fluentd Status ===" systemctl status td-agent --no-pager | head -5

echo "" echo "=== Buffer Size ===" du -sh /var/log/td-agent/buffer/

echo "" echo "=== Output Errors ===" grep -c "error" /var/log/td-agent/td-agent.log

echo "" echo "=== Connection Status ===" curl -s http://localhost:24220/api/plugins.json | jq '.plugins[] | {type: .type, output: .output}'

echo "" echo "=== Recent Errors ===" tail -20 /var/log/td-agent/td-agent.log | grep error EOF

chmod +x /usr/local/bin/monitor-fluentd.sh ```

Step 10: Configure Fallback Output

```xml # Add fallback for resilience: <match **> @type copy <store> @type elasticsearch host elasticsearch-primary port 9200 <buffer> @type file path /var/log/td-agent/buffer-primary </buffer> </store> <store> @type file path /var/log/td-agent/fallback.log <buffer> @type file path /var/log/td-agent/buffer-fallback </buffer> </store> </match>

# Or use secondary output: <match **> @type elasticsearch host elasticsearch-primary port 9200

<buffer> @type file path /var/log/td-agent/buffer retry_forever true </buffer>

<secondary> @type file path /var/log/td-agent/failed-logs.log </secondary> </match> ```

Fluentd Output Connection Checklist

Verify the Fix

```bash # After fixing connection issue

# 1. Test connection nc -zv elasticsearch 9200 // Connected

# 2. Restart Fluentd systemctl restart td-agent

# 3. Send test log curl -X POST http://localhost:8888/test -d '{"message":"test"}' // Accepted

# 4. Check destination curl http://elasticsearch:9200/fluentd/_search?q=test // Document found

# 5. Check buffer draining du -sh /var/log/td-agent/buffer/ // Size decreasing

# 6. Monitor logs tail -f /var/log/td-agent/td-agent.log // No connection errors ```

Related Issues

• [Fix Fluentd Buffer Overflow](/articles/fix-fluentd-buffer-overflow)
• [Fix Elasticsearch Cluster Red Status](/articles/fix-elasticsearch-cluster-red-status)
• [Fix Kafka Producer Retries Exceeded](/articles/fix-kafka-producer-retries-exceeded)