Fix Kafka Consumer Lag High

What's Actually Happening

Kafka consumers are falling behind producers. Messages are not being processed in real-time, causing delays in data processing pipelines.

The Error You'll See

Consumer lag monitoring:

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG my-group my-topic 0 1000 50000 49000 ```

High lag alert:

Consumer lag for group my-group on topic my-topic exceeded threshold: 50000 > 1000

Why This Happens

1. 1.Consumer slow - Consumer processing rate below production rate
2. 2.Consumer down - Consumer instances stopped or crashed
3. 3.Partition imbalance - Uneven partition assignment
4. 4.Broker overloaded - Broker cannot serve consumers fast enough
5. 5.Network latency - Slow network between consumer and broker
6. 6.Message size large - Large messages slow processing
7. 7.Rebalancing frequent - Constant consumer rebalancing

Step 1: Check Consumer Lag

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

kafka-consumer-groups --bootstrap-server localhost:9092 --list

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --state

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --members --verbose

watch -n 5 kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group ```

Step 2: Check Consumer Status

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --members

ps aux | grep kafka-consumer

systemctl status kafka-consumer-service

journalctl -u kafka-consumer -f

grep -i error /var/log/kafka-consumer.log

systemctl restart kafka-consumer-service ```

Step 3: Check Producer Rate

```bash kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic my-topic --time -1

sleep 10 kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic my-topic --time -1

# Calculate rate: difference / 10 = messages per second ```

Step 4: Check Partition Distribution

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --members --verbose

kafka-topics --bootstrap-server localhost:9092 --describe --topic my-topic

kafka-topics --bootstrap-server localhost:9092 --alter --topic my-topic --partitions 20 ```

Step 5: Check Consumer Performance

```bash top -p $(pgrep -f kafka-consumer)

ps aux | grep kafka-consumer | awk {print $3}

ps aux | grep kafka-consumer | awk {print $4} ```

Step 6: Check Broker Performance

```bash top -p $(pgrep -f kafka.Kafka)

tail -f /var/log/kafka/server.log

grep -i error /var/log/kafka/server.log | tail -20

iostat -x 1

cat /etc/kafka/server.properties | grep threads ```

Step 7: Optimize Consumer Configuration

```bash fetch.min.bytes=1048576 fetch.max.bytes=52428800 max.poll.records=500 max.partition.fetch.bytes=1048576

# Increase for higher throughput: fetch.min.bytes=10485760 max.poll.records=1000

cat /etc/kafka/consumer.properties

systemctl restart kafka-consumer ```

Step 8: Add More Consumers

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --members

# Deploy additional consumer instances with same group.id

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --state

watch -n 5 kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group ```

Step 9: Check Network and Latency

ping -c 3 broker-host
nc -zv broker-host 9092
iperf3 -c broker-host
traceroute broker-host
tcpdump -i eth0 port 9092 -nn

Step 10: Monitor and Alert on Lag

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

watch -n 10 kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

# Use kafka_exporter for Prometheus metrics # Use Burrow for advanced lag monitoring ```

Kafka Consumer Lag Checklist

Verify the Fix

```bash kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group | awk NR>1 {sum+=$6} END {print sum

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --members | wc -l

watch -n 10 kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

journalctl -u kafka-consumer --since 10 min ago | grep -i error ```

Related Issues

• [Fix Kafka Broker Not Starting](/articles/fix-kafka-broker-not-starting)
• [Fix Kafka Producer Failed](/articles/fix-kafka-producer-failed)