Fix Kafka Consumer Group Rebalance Loop

What's Actually Happening

Kafka consumer group enters continuous rebalance cycles, causing consumer assignments to constantly change. Messages are reprocessed, consumers can't make progress, and throughput drops to near zero.

The Error You'll See

Consumer logs show rebalancing:

```bash # Consumer application logs: [Consumer clientId=consumer-1, groupId=my-group] Revoking previously assigned partitions [topic-0, topic-1]

[Consumer clientId=consumer-1, groupId=my-group] Joining group

[Consumer clientId=consumer-1, groupId=my-group] Successfully joined group with generation 123

[Consumer clientId=consumer-1, groupId=my-group] Setting newly assigned partitions [topic-2, topic-3]

# Repeats continuously every few seconds ```

Consumer group unstable:

```bash $ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG my-group topic 0 1000 15000 14000 # Partition assignments keep changing ```

High rebalance count:

```bash $ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group --state

GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE my-group broker-1 (1) range Rebalancing # Stuck in Rebalancing state ```

Why This Happens

1. 1.Session timeout too short - Consumers fail to send heartbeats in time
2. 2.Processing too slow - Message processing exceeds max.poll.interval.ms
3. 3.Heartbeat thread blocked - Consumer thread blocking heartbeat sender
4. 4.Too many consumers - Excessive consumers causing coordination overhead
5. 5.Network latency - High latency between consumers and brokers
6. 6.Broker overload - Group coordinator under load

Step 1: Identify Rebalance Pattern

```bash # Check consumer group status kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group --state

# Check consumer lag (lag growing = not making progress) kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group

# Monitor rebalance events kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group --members --verbose

# Check broker logs for rebalance grep -i "rebalance|join|sync" /var/log/kafka/server.log | tail -100

# Count rebalances in last hour grep "Completed load of group my-group" /var/log/kafka/server.log | \ grep "$(date -d '1 hour ago' '+%Y-%m-%d')" | wc -l ```

Step 2: Check Session Timeout Configuration

```bash # Check consumer configuration # Common settings:

# session.timeout.ms - Timeout for detecting consumer failures # Default: 10000 (10 seconds)

# heartbeat.interval.ms - Interval between heartbeats # Default: 3000 (3 seconds)

# max.poll.interval.ms - Max time between polls # Default: 300000 (5 minutes)

# If session.timeout.ms is too short, consumers timeout during processing

# Check current settings in consumer properties: session.timeout.ms=10000 heartbeat.interval.ms=3000 max.poll.interval.ms=300000

# Rule: session.timeout.ms should be at least 3x heartbeat.interval.ms # heartbeat.interval.ms should be < session.timeout.ms / 3 ```

Step 3: Increase Session Timeout

```properties # Increase session timeout to avoid false failures

# Original (may be too short): session.timeout.ms=10000 heartbeat.interval.ms=3000

# Recommended for longer processing: session.timeout.ms=30000 # 30 seconds heartbeat.interval.ms=10000 # 10 seconds max.poll.interval.ms=600000 # 10 minutes

# Rule of thumb: # - session.timeout.ms = 3x max expected GC pause + network latency # - heartbeat.interval.ms = session.timeout.ms / 3 # - max.poll.interval.ms = max processing time per batch

# In Java consumer: Properties props = new Properties(); props.put("session.timeout.ms", "30000"); props.put("heartbeat.interval.ms", "10000"); props.put("max.poll.interval.ms", "600000"); ```

Step 4: Reduce Poll Batch Size

```properties # If processing takes too long, reduce batch size

# max.poll.records - Max records per poll # Default: 500

# If processing 500 records takes longer than max.poll.interval.ms, # reduce batch size:

max.poll.records=100

# In Java consumer: props.put("max.poll.records", "100");

# This ensures consumer returns to poll() within max.poll.interval.ms

# Calculate: # If 1 record takes 100ms to process: # 500 records = 50 seconds # If max.poll.interval.ms = 60 seconds, this is fine # But if network hiccups occur, you might timeout # Better to use 100-200 records per poll ```

Step 5: Check Consumer Thread Architecture

```java // Wrong: Blocking processing in same thread as polling

while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord<String, String> record : records) { // Slow processing blocks heartbeat thread! processRecord(record); // Takes 5+ seconds per record } consumer.commitSync(); }

// Correct: Use separate thread for processing

ExecutorService executor = Executors.newFixedThreadPool(4);

while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); if (!records.isEmpty()) { // Submit to thread pool, poll returns quickly executor.submit(() -> { for (ConsumerRecord<String, String> record : records) { processRecord(record); } }); } // Heartbeat thread continues in background }

// Or use pause/resume pattern: // Pause partitions while processing, resume when done consumer.pause(consumer.assignment()); // Process records... consumer.resume(consumer.assignment()); ```

Step 6: Check Network Latency

```bash # Check network latency between consumers and brokers ping broker1:9092 ping broker2:9092

# Check for packet loss mtr broker1

# Check broker connection count netstat -an | grep 9092 | wc -l

# Check broker network throughput iftop -i eth0

# If latency > session.timeout.ms / 3, increase session.timeout.ms

# Check consumer to broker connectivity nc -zv broker1 9092

# Test from consumer machine: kafka-broker-api-versions --bootstrap-server broker1:9092 ```

Step 7: Monitor Broker Group Coordinator

```bash # Check which broker is group coordinator kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group --state

# Check broker metrics for group coordinator # JMX metric: kafka.controller:type=GroupCoordinator,name=ActiveGroupCount

# Check broker load kafka-broker-api-versions --bootstrap-server broker1:9092

# Check broker logs for group coordinator issues grep -i "group.*coordinator" /var/log/kafka/server.log

# If coordinator overloaded, consider: # 1. Add more brokers (spreads group coordination load) # 2. Increase num.io.threads on broker # 3. Increase group.min.session.timeout.ms

# Check broker resource usage top -p $(pgrep java) ```

Step 8: Check Consumer Count

```bash # Check number of consumers in group kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group --members

# Output: GROUP CONSUMER-ID HOST PARTITIONS my-group consumer-1-xxx /10.0.0.1 2 my-group consumer-2-xxx /10.0.0.2 2 my-group consumer-3-xxx /10.0.0.3 2

# Compare to partition count kafka-topics --bootstrap-server localhost:9092 \ --describe --topic my-topic

# Topic:my-topic PartitionCount:6

# Rule: Consumers <= Partitions # If more consumers than partitions, extra consumers sit idle and can cause rebalances

# Recommended: # - 1 consumer per partition (optimal) # - Or fewer consumers handling multiple partitions # - Avoid more consumers than partitions ```

Step 9: Tune Static Membership

```properties # Use static membership to reduce rebalances (Kafka 2.3+)

# Add to consumer config: group.instance.id=consumer-1-unique-id

# When consumer restarts, it rejoins as same instance # No rebalance needed if consumer returns within session.timeout.ms

# Configuration: group.instance.id=consumer-1 session.timeout.ms=300000 # 5 minutes to allow restart max.poll.interval.ms=600000

# In Java consumer: props.put("group.instance.id", "consumer-1-unique"); props.put("session.timeout.ms", "300000");

# With static membership: # - Consumer can restart without triggering rebalance # - Only rebalance if consumer doesn't return in time # - Reduces rebalance storms significantly ```

Step 10: Monitor Rebalance Metrics

```bash # Create monitoring script cat << 'EOF' > monitor_rebalance.sh #!/bin/bash GROUP=my-group BOOTSTRAP=localhost:9092

echo "=== Consumer Group State ===" kafka-consumer-groups --bootstrap-server $BOOTSTRAP \ --describe --group $GROUP --state

echo "" echo "=== Consumer Members ===" kafka-consumer-groups --bootstrap-server $BOOTSTRAP \ --describe --group $GROUP --members

echo "" echo "=== Consumer Lag ===" kafka-consumer-groups --bootstrap-server $BOOTSTRAP \ --describe --group $GROUP

echo "" echo "=== Recent Rebalances (Broker Log) ===" grep -i "rebalance|join group|sync group" /var/log/kafka/server.log | tail -10 EOF

chmod +x monitor_rebalance.sh

# Monitor with JMX: # kafka.consumer:type=consumer-coordinator-metrics,client-id=xxx # - rebalance-rate-per-sec # - rebalance-latency-avg # - join-rate-per-sec # - sync-rate-per-sec

# Prometheus metrics: # kafka_consumer_coordinator_rebalance_rate # kafka_consumer_coordinator_rebalance_latency_seconds ```

Kafka Consumer Rebalance Checklist

Verify the Fix

```bash # After adjusting consumer configuration

# 1. Check consumer group is stable kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group --state // STATE should be Stable

# 2. Monitor for rebalance events # Wait 5 minutes, check logs grep -i rebalance /var/log/kafka/server.log | tail -5 // No recent rebalances

# 3. Check consumer lag decreasing kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-group // LAG should be decreasing

# 4. Monitor consumer metrics # rebalance-rate-per-sec should be 0 or very low

# 5. Test consumer restart # Restart one consumer, verify no rebalance (with static membership) // Group remains stable ```

Related Issues

• [Fix Kafka Consumer Lag Growing](/articles/fix-kafka-consumer-lag-growing)
• [Fix Kafka Consumer Rebalance Failed](/articles/fix-kafka-consumer-rebalance-failed)
• [Fix Kafka No Leader For Partition](/articles/fix-kafka-no-leader-for-partition)