The Problem

Redis Streams consumer groups exhibit various failures: messages stuck in pending state, consumers not receiving messages, or group creation errors. You may see errors like:

bash
NOGROUP No such key 'mystream' or consumer group 'mygroup'
BUSYGROUP Consumer group name already exists

Or messages remain in the Pending Entries List (PEL) indefinitely, preventing proper message acknowledgment flow.

Understanding Redis Streams Consumer Groups

Consumer groups provide: - Multiple consumers - Group processes stream cooperatively - Message delivery tracking - Each message delivered to one consumer - Pending Entries List (PEL) - Messages delivered but not acknowledged - Message claiming - Stuck messages can be claimed by other consumers

Key commands: - XGROUP CREATE - Create consumer group - XREADGROUP - Read from group - XACK - Acknowledge message - XPENDING - View pending messages - XCLAIM - Claim pending message

Diagnosis Commands

Check Stream Exists

bash
redis-cli XINFO STREAM mystream

Output:

bash
length: 1000
entries-added: 5000
max-deleted-entry-id: 0-0
entries-added: 5000
first-entry: 1-0 value1
last-entry: 1000-0 value1000
groups: 2

Check Consumer Groups

bash
redis-cli XINFO GROUPS mystream

Output:

bash
name: mygroup
consumers: 3
pending: 50
entries-read: null
lag: 10
  • pending - Messages in PEL (delivered but not acknowledged)
  • lag - Messages not yet delivered to group

Check Consumers in Group

bash
redis-cli XINFO CONSUMERS mystream mygroup

Output:

bash
name: consumer1
pending: 20
idle: 5000  # milliseconds since last interaction

Check Pending Messages

bash
redis-cli XPENDING mystream mygroup

Output:

bash
pending messages count: 50
ID of first pending message: 100-0
ID of last pending message: 150-0
minimum idle time of pending messages: 1000
maximum idle time of pending messages: 300000

Detailed Pending Messages

bash
redis-cli XPENDING mystream mygroup - + 10

Output shows each pending message:

``` 1) 1) "100-0" 2) "consumer1" 3) "5000" 4) "1"

2) 1) "101-0" 2) "consumer2" 3) "10000" 4) "2" ```

  • Message ID
  • Consumer owning the message
  • Idle time (milliseconds)
  • Delivery count

Common Errors and Solutions

Error 1: NOGROUP - Group Doesn't Exist

bash
NOGROUP No such key 'mystream' or consumer group 'mygroup' in XREADGROUP command

Diagnosis:

```bash # Check stream exists redis-cli TYPE mystream

# Check group exists redis-cli XINFO GROUPS mystream ```

Solution:

Create the group:

```bash # Create group from beginning of stream redis-cli XGROUP CREATE mystream mygroup 0 MKSTREAM

# Create group from new messages only redis-cli XGROUP CREATE mystream mygroup $

# MKSTREAM creates stream if it doesn't exist ```

Error 2: BUSYGROUP - Group Already Exists

bash
BUSYGROUP Consumer group name already exists

Solution:

Either use existing group or delete and recreate:

```bash # Delete existing group redis-cli XGROUP DESTROY mystream mygroup

# Create new group redis-cli XGROUP CREATE mystream mygroup 0 ```

Error 3: Messages Stuck in PEL

Messages remain pending indefinitely when consumers crash or don't acknowledge.

Diagnosis:

```bash # Check pending count redis-cli XPENDING mystream mygroup

# Check consumer idle times redis-cli XINFO CONSUMERS mystream mygroup ```

Solution 1: Claim Stuck Messages

bash
# Claim messages idle for more than 60 seconds
redis-cli XCLAIM mystream mygroup newconsumer 60000 - + COUNT 10

Output:

bash
1) 1) "100-0"
   2) 1) "field1"
      2) "value1"
2) 1) "101-0"
   2) 1) "field2"
      2) "value2"

Solution 2: Delete Inactive Consumer

```bash # Find inactive consumer redis-cli XINFO CONSUMERS mystream mygroup

# Delete consumer (pending messages become claimable) redis-cli XGROUP DELCONSUMER mystream mygroup inactive_consumer ```

Error 4: High Pending Count Growing

Pending messages accumulate faster than being acknowledged.

Diagnosis:

```bash # Monitor pending count redis-cli XPENDING mystream mygroup

# Check consumer activity redis-cli XINFO CONSUMERS mystream mygroup ```

Solutions:

  1. 1.Add more consumers:
bash
# New consumer reads from group
redis-cli XREADGROUP GROUP mygroup newconsumer COUNT 10 BLOCK 5000 STREAMS mystream >
  1. 1.Process backlog with XCLAIM:
bash
# Claim oldest pending messages
redis-cli XCLAIM mystream mygroup activeconsumer 0 0-0 + COUNT 100
  1. 1.Acknowledge processed messages:
bash
redis-cli XACK mystream mygroup 100-0 101-0 102-0

Error 5: Consumer Not Receiving Messages

Consumer reads but gets no messages.

Diagnosis:

```bash # Check consumer's pending redis-cli XPENDING mystream mygroup - + 10 consumer1

# Check stream lag redis-cli XINFO GROUPS mystream ```

Causes:

  1. 1.Consumer has too many pending messages
  2. 2.Reading from wrong position
  3. 3.Stream has no new messages

Solution:

```bash # Use > to get new messages only redis-cli XREADGROUP GROUP mygroup consumer1 COUNT 10 STREAMS mystream >

# Or use pending to process backlog first redis-cli XREADGROUP GROUP mygroup consumer1 COUNT 10 STREAMS mystream 0 ```

Error 6: Delivery Count Exceeds Limit

Messages delivered many times without acknowledgment.

Diagnosis:

bash
# Check delivery counts
redis-cli XPENDING mystream mygroup - + 100

The 4th field shows delivery count.

Solution:

Set delivery limit with XCLAIM:

bash
# Claim messages with delivery count > 5 and reset count
redis-cli XCLAIM mystream mygroup newconsumer 60000 - + RETRYCOUNT 0 FORCE JUSTID

Or use XAUTOCLAIM (Redis 7.0+):

bash
redis-cli XAUTOCLAIM mystream mygroup newconsumer 60000 0 COUNT 10

Consumer Group Workflow Patterns

Reliable Consumer Pattern

```python import redis import time import json

r = redis.Redis() STREAM = 'mystream' GROUP = 'mygroup' CONSUMER = 'worker-1'

def process_message(message_id, data): """Process the message.""" print(f"Processing {message_id}: {data}") # Your processing logic here return True

def consume_stream(block_ms=5000, count=10, max_pending=100): """Consume messages from stream with acknowledgment."""

# Ensure group exists try: r.xgroup_create(STREAM, GROUP, id='0', mkstream=True) except redis.ResponseError as e: if 'BUSYGROUP' not in str(e): raise

while True: # Read new messages messages = r.xreadgroup( groupname=GROUP, consumername=CONSUMER, streams={STREAM: '>'}, count=count, block=block_ms )

if not messages: continue

for stream_name, stream_messages in messages: for message_id, message_data in stream_messages: try: if process_message(message_id, message_data): # Acknowledge on success r.xack(STREAM, GROUP, message_id) else: # On failure, leave in pending for retry pass except Exception as e: print(f"Error processing {message_id}: {e}") # Leave in pending for later retry

# Periodically claim stuck messages pending = r.xpending_range(STREAM, GROUP, '-', '+', count=1) if pending and pending[0]['idle'] > 60000: stuck = r.xclaim(STREAM, GROUP, CONSUMER, 60000, [pending[0]['message_id']]) for msg_id, msg_data in stuck: r.xack(STREAM, GROUP, msg_id) ```

Cleanup Stuck Messages Periodically

```python def claim_stuck_messages(stream, group, consumer, min_idle_ms=60000, count=100): """Claim messages idle for more than min_idle_ms.""" pending = r.xpending_range(stream, group, '-', '+', count=count)

for entry in pending: if entry['idle'] > min_idle_ms: claimed = r.xclaim(stream, group, consumer, min_idle_ms, [entry['message_id']], force=True) for msg_id, msg_data in claimed: try: process_message(msg_id, msg_data) r.xack(stream, group, msg_id) except Exception: pass # Will retry later ```

Configuration Best Practices

Stream Creation with Limits

```bash # Create stream with max length redis-cli XADD mystream MAXLEN ~ 10000 * field1 value1

# Or use TRIM to maintain size redis-cli XTRIM mystream MAXLEN ~ 10000 ```

Group Creation

```bash # Create from beginning redis-cli XGROUP CREATE mystream mygroup 0 MKSTREAM

# Create for new messages only redis-cli XGROUP CREATE mystream mygroup $ ```

Consumer Management

```bash # Create consumer explicitly redis-cli XGROUP CREATECONSUMER mystream mygroup consumer1

# Delete inactive consumers redis-cli XGROUP DELCONSUMER mystream mygroup inactive_consumer ```

Monitoring Consumer Groups

Health Check Script

```bash #!/bin/bash # Check consumer group health

STREAM="mystream" GROUP="mygroup"

echo "=== Stream Info ===" redis-cli XINFO STREAM $STREAM

echo "=== Group Info ===" redis-cli XINFO GROUPS $STREAM

echo "=== Consumers ===" redis-cli XINFO CONSUMERS $STREAM $GROUP

echo "=== Pending ===" PENDING=$(redis-cli XPENDING $STREAM $GROUP | head -1) echo "Pending messages: $PENDING"

if [ "$PENDING" -gt 100 ]; then echo "WARNING: High pending count" fi ```

Monitor Lag

bash
# Check group lag (messages not yet delivered)
redis-cli XINFO GROUPS mystream | grep -A5 "name: mygroup" | grep lag

Verification

After fixing issues:

```bash # Test group creation redis-cli XGROUP CREATE teststream testgroup 0 MKSTREAM

# Add test message redis-cli XADD teststream * testfield testvalue

# Read from group redis-cli XREADGROUP GROUP testgroup consumer1 STREAMS teststream >

# Acknowledge redis-cli XACK teststream testgroup <message_id>

# Verify pending is 0 redis-cli XPENDING teststream testgroup

# Cleanup redis-cli XGROUP DESTROY teststream testgroup redis-cli DEL teststream ```