Fix Fluentd Log Not Sending

What's Actually Happening

Fluentd is not forwarding logs to destinations. Logs are buffered but not delivered, queues are growing, or Fluentd process is stuck.

The Error You'll See

Buffer overflow:

```bash tail -f /var/log/td-agent/td-agent.log

buffer queue has full ```

Connection refused:

```bash failed to flush the buffer

Connection refused connecting to elasticsearch ```

No logs arriving:

# At destination:
# No logs received from Fluentd

Why This Happens

1. 1.Destination down - Elasticsearch or other destination unreachable
2. 2.Network issue - Network connectivity problem
3. 3.Buffer full - Buffer exceeded limits
4. 4.Configuration error - Wrong match or output config
5. 5.Permission denied - Cannot read log files
6. 6.Format mismatch - Log format not matching parser
7. 7.Resource limits - Fluentd out of memory

Step 1: Check Fluentd Status

```bash systemctl status td-agent

systemctl status fluentd

ps aux | grep fluentd

ss -tlnp | grep 24224

journalctl -u td-agent -f ```

Step 2: Check Fluentd Logs

```bash tail -f /var/log/td-agent/td-agent.log

tail -f /var/log/fluentd/fluentd.log

grep -i "error|warn|fail" /var/log/td-agent/td-agent.log | tail -20

# Enable debug: # In fluent.conf: <system> log_level debug </system> ```

Step 3: Check Configuration

```bash cat /etc/td-agent/td-agent.conf

cat /etc/fluentd/fluent.conf

# Validate config: fluentd --dry-run -c /etc/td-agent/td-agent.conf

td-agent --dry-run -c /etc/td-agent/td-agent.conf

# Check match patterns: grep -A 20 "<match" /etc/td-agent/td-agent.conf ```

Step 4: Check Buffer Status

```bash ls -la /var/log/td-agent/buffer/

ls -la /var/log/fluentd/buffer/

# Check buffer size: du -sh /var/log/td-agent/buffer/

# Check buffer config: grep -A 10 "<buffer" /etc/td-agent/td-agent.conf

# Clear stuck buffer (CAUTION): rm -rf /var/log/td-agent/buffer/*.buf ```

Step 5: Test Destination Connectivity

```bash # For Elasticsearch: curl -v http://elasticsearch:9200

nc -zv elasticsearch 9200

# For Kafka: nc -zv kafka 9092

# For S3: aws s3 ls s3://my-bucket

# Test from Fluentd host: ping -c 3 destination-host ```

Step 6: Check Input Plugins

```bash # Check if logs being collected: tail -f /var/log/nginx/access.log

# Test tail input: # In config: <source> @type tail path /var/log/nginx/access.log pos_file /var/log/td-agent/nginx.access.pos tag nginx.access <parse> @type nginx </parse> </source>

# Check pos file: cat /var/log/td-agent/nginx.access.pos

# Verify log format: head -5 /var/log/nginx/access.log ```

Step 7: Check Output Plugins

```bash # Common output issues:

# Elasticsearch: <match **> @type elasticsearch host elasticsearch port 9200 logstash_format true <buffer> @type file path /var/log/td-agent/buffer flush_interval 10s </buffer> </match>

# Kafka: <match **> @type kafka_buffered brokers kafka:9092 topic_key topic default_topic logs </match>

# File output for debugging: <match **> @type file path /var/log/fluentd/output </match> ```

Step 8: Check Permissions

```bash # Fluentd user: ps aux | grep fluentd | head -1

# Check log file permissions: ls -la /var/log/nginx/access.log

# Add td-agent to log group: usermod -a -G adm td-agent

# Check buffer directory: ls -la /var/log/td-agent/

chown -R td-agent:td-agent /var/log/td-agent/ ```

Step 9: Monitor Fluentd Metrics

```bash # Enable metrics: <system> enable_msgpack_time_support true </system>

# Prometheus metrics plugin: <match fluentd.**> @type prometheus </match>

<match fluentd.**> @type prometheus_output_monitor <labels> hostname ${hostname} </labels> </match>

# Check buffer metrics: curl http://localhost:24220/api/plugins.json ```

Step 10: Restart and Recover

```bash # Graceful restart: systemctl reload td-agent

# Full restart: systemctl restart td-agent

# Check after restart: systemctl status td-agent tail -f /var/log/td-agent/td-agent.log

# Verify logs flowing: watch -n 5 'ls -la /var/log/td-agent/buffer/' ```

Fluentd Log Not Sending Checklist

Verify the Fix

```bash systemctl status td-agent

tail -f /var/log/td-agent/td-agent.log

curl http://localhost:24220/api/plugins.json

ls -la /var/log/td-agent/buffer/

curl http://elasticsearch:9200/_cat/indices

tail -f /var/log/nginx/access.log ```

Related Issues

• [Fix Elasticsearch Cluster Red](/articles/fix-elasticsearch-cluster-red)
• [Fix Prometheus Scrape Error](/articles/fix-prometheus-scrape-error)
• [Fix Kafka Consumer Lag High](/articles/fix-kafka-consumer-lag-high)