The Problem

Redis clients get disconnected with errors about output buffer limits. Replicas fail to sync. Applications receive unexpected disconnections. The error message typically reads:

bash
Client id=12345 addr=10.0.0.5:54321 fd=10 name=app-client age=30 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=16384 oll=16384 omem=268435456 events=rw cmd=GET

The omem value shows the output buffer memory usage that triggered the disconnection.

Understanding Output Buffer Limits

Redis maintains output buffers for each client to queue responses. Three types of limits exist:

  1. 1.Normal clients - Regular application connections
  2. 2.Replica clients - Connections from replicas
  3. 3.Pub/Sub clients - Subscribers receiving messages

Each limit has two thresholds: - Hard limit - Immediate disconnection when exceeded - Soft limit - Disconnection after sustained period

Diagnosis Commands

Check Current Buffer Limits

bash
redis-cli CONFIG GET client-output-buffer-limit

Output format:

bash
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

Format: <class> <hard limit> <soft limit> <soft seconds>

Find Clients with High Buffer Usage

bash
redis-cli CLIENT LIST

Key fields in output:

bash
id=12345 addr=10.0.0.5:54321 ... omem=268435456 ...
  • omem - Output buffer memory in bytes
  • oll - Output list length (number of pending responses)

Find the Biggest Buffer Users

bash
redis-cli CLIENT LIST | sort -k20 -t= -n -r | head -20

Or for just omem values:

bash
redis-cli CLIENT LIST | grep -oP 'omem=\K[0-9]+' | sort -rn | head -10

Check for Blocked Clients

bash
redis-cli CLIENT LIST | grep -E 'flags=.*b'

The b flag indicates a blocked client (waiting on BLPOP, etc.).

Common Scenarios and Solutions

Scenario 1: Slow Consumer Replicas

Replicas can't keep up with the master's write rate.

Symptoms:

bash
redis-cli CLIENT LIST | grep flags=S

Shows replica with high omem value.

Diagnosis:

```bash # Check replication lag redis-cli INFO replication | grep -E "slave[0-9]:|lag"

# Check master's write rate redis-cli INFO stats | grep instantaneous_ops_per_sec ```

Solutions:

  1. 1.Increase replica buffer limits:
bash
redis-cli CONFIG SET client-output-buffer-limit "replica 512mb 128mb 120"
  1. 1.Optimize network between master and replica
  2. 2.Reduce write rate on master temporarily
  3. 3.Use diskless replication:
bash
redis-cli CONFIG SET repl-diskless-sync yes
redis-cli CONFIG SET repl-diskless-sync-delay 5

Scenario 2: Slow Pub/Sub Consumers

Subscribers can't process messages fast enough.

Symptoms:

bash
redis-cli CLIENT LIST | grep -E 'flags=.*P'

Shows pub/sub client with high output buffer.

Diagnosis:

```bash # Check pub/sub channels and patterns redis-cli PUBSUB CHANNELS redis-cli PUBSUB NUMPAT

# Check client subscription count redis-cli CLIENT LIST | grep -E 'sub=[1-9]|psub=[1-9]' ```

Solutions:

  1. 1.Adjust pub/sub buffer limits:
bash
redis-cli CONFIG SET client-output-buffer-limit "pubsub 64mb 16mb 60"
  1. 1.Implement message batching in publisher
  2. 2.Add backpressure mechanism in subscriber
  3. 3.Consider using Redis Streams instead of Pub/Sub for reliable delivery

Scenario 3: Application Client Disconnection

Normal clients exceed limits during large result sets.

Symptoms:

  • Application receives "Connection reset by peer"
  • Clients disconnected after large queries

Diagnosis:

```bash # Check client limits redis-cli CONFIG GET client-output-buffer-limit | grep normal

# If normal clients have limits (non-zero), they may be too restrictive ```

Solutions:

  1. 1.For normal clients, typically set to unlimited:
bash
redis-cli CONFIG SET client-output-buffer-limit "normal 0 0 0"
  1. 1.Implement pagination for large queries:

```bash # Instead of getting all keys at once redis-cli KEYS '*' # BAD - blocks and returns huge result

# Use SCAN redis-cli SCAN 0 COUNT 100 # GOOD - paginated ```

  1. 1.Use SELECT for large datasets or stream results

Scenario 4: Monitoring Tool Causing Issues

Monitoring tools querying large datasets can trigger limits.

Symptoms:

  • Disconnections correlate with monitoring queries
  • High omem for monitoring client names

Diagnosis:

bash
redis-cli CLIENT LIST | grep -E 'name=monitor|name=prometheus|name=grafana'

Solutions:

  1. 1.Give monitoring clients unlimited buffer:
bash
redis-cli CLIENT SETNAME monitor-client
# Then in redis.conf (requires restart):
# client-output-buffer-limit normal 0 0 0
  1. 1.Optimize monitoring queries to use SCAN

Configuration Best Practices

Recommended Buffer Settings

```conf # Normal clients - unlimited (handle in application) client-output-buffer-limit normal 0 0 0

# Replicas - generous limits for sync client-output-buffer-limit replica 512mb 128mb 120

# Pub/Sub - moderate limits with soft threshold client-output-buffer-limit pubsub 64mb 16mb 60 ```

Making Changes Persistent

Add to redis.conf:

conf
# In redis.conf
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 512mb 128mb 120
client-output-buffer-limit pubsub 64mb 16mb 60

Monitoring and Alerting

Script to Monitor Output Buffers

```bash #!/bin/bash THRESHOLD_MB=100

redis-cli CLIENT LIST | while read line; do omem=$(echo "$line" | grep -oP 'omem=\K[0-9]+') client=$(echo "$line" | grep -oP 'name=\K[^ ]+') addr=$(echo "$line" | grep -oP 'addr=\K[^ ]+')

if [ -n "$omem" ] && [ "$omem" -gt $((THRESHOLD_MB * 1024 * 1024)) ]; then echo "WARNING: Client $client ($addr) has ${omem} bytes in output buffer" fi done ```

Key Metrics to Track

```bash # Total output buffer memory redis-cli INFO memory | grep mem_clients_normal redis-cli INFO memory | grep mem_clients_slaves

# Client count redis-cli INFO clients | grep connected_clients ```

Emergency Response

If clients are being disconnected right now:

Immediate Fix

```bash # Temporarily increase limits redis-cli CONFIG SET client-output-buffer-limit "replica 1gb 256mb 180" redis-cli CONFIG SET client-output-buffer-limit "pubsub 128mb 32mb 120"

# Or disable limits entirely for emergency redis-cli CONFIG SET client-output-buffer-limit "normal 0 0 0" redis-cli CONFIG SET client-output-buffer-limit "replica 0 0 0" ```

Identify the Culprit

```bash # Find client with highest output buffer redis-cli CLIENT LIST | awk -F'[ =]' '{for(i=1;i<=NF;i++) if($i=="omem") print $(i+1), $0}' | sort -rn | head -5

# Kill specific problematic client redis-cli CLIENT KILL ID <client-id> ```

Full Resync for Replicas

If replicas keep disconnecting:

bash
# On replica
redis-cli REPLICAOF NO ONE
redis-cli DEBUG RELOAD NOSAVE
redis-cli REPLICAOF <master-ip> <master-port>

Verification

After making changes:

```bash # Check new limits redis-cli CONFIG GET client-output-buffer-limit

# Monitor client output buffers redis-cli CLIENT LIST | grep -oP 'omem=[0-9]+' | sort -t= -k2 -n | tail -10

# Check for disconnection errors in log grep -i "output buffer" /var/log/redis/redis-server.log ```