Introduction

RabbitMQ's prefetch count (QoS) controls how many unacknowledged messages a consumer can have in flight. When set too high, the broker buffers large numbers of messages in memory for each channel, waiting for acknowledgments. With many consumers and high prefetch values, this buffering can consume all available broker memory, triggering resource alarms and connection blocking.

Symptoms

  • RabbitMQ memory usage grows continuously despite consumers actively processing
  • Broker triggers memory alarm, blocking all publishing connections
  • rabbitmqctl status shows memory used exceeds the memory high watermark
  • Consumers show large numbers of unacknowledged messages
  • Error message: memory alarm - resource alarm: memory usage exceeds configured limit

Common Causes

  • Prefetch count set to 0 (unlimited) or very high value like 10000+
  • Many channels on the same connection each with high prefetch counts
  • Slow consumers not acknowledging messages, leaving them in the prefetch buffer
  • Prefetch set per-channel without accounting for total consumer count
  • Message payloads larger than expected, making each prefetched message consume more memory

Step-by-Step Fix

  1. 1.Check current memory usage and alarm state: Verify the broker is in alarm.
  2. 2.```bash
  3. 3.rabbitmqctl status | grep -A5 "memory"
  4. 4.rabbitmqctl list_queues name messages messages_unacknowledged memory
  5. 5.`
  6. 6.Reduce prefetch count to a reasonable value: Set prefetch based on consumer processing rate.
  7. 7.```java
  8. 8.// Set prefetch count to match consumer processing capacity
  9. 9.channel.basicQos(50); // Process 50 messages at a time
  10. 10.`
  11. 11.Identify channels with excessive unacknowledged messages: Find the worst offenders.
  12. 12.```bash
  13. 13.rabbitmqctl list_channels connection_pid prefetch_count messages_unacknowledged --format table
  14. 14.`
  15. 15.Cancel and re-consume with corrected prefetch: Apply the fix to existing consumers.
  16. 16.```java
  17. 17.channel.basicCancel(consumerTag);
  18. 18.channel.basicQos(50);
  19. 19.channel.basicConsume(queueName, false, deliverCallback, cancelCallback);
  20. 20.`
  21. 21.Configure memory thresholds and flow control: Set appropriate memory limits.
  22. 22.```properties
  23. 23.# rabbitmq.conf
  24. 24.vm_memory_high_watermark.relative = 0.6
  25. 25.vm_memory_high_watermark_paging_ratio = 0.5
  26. 26.`

Prevention

  • Set basicQos prefetch count to 10-100 based on average message processing time
  • Never use basicQos(0) (unlimited) in production environments
  • Monitor per-channel memory usage with rabbitmqctl list_channels
  • Implement consumer-side flow control that pauses fetching when processing backlog grows
  • Use consumer_prefetch policy to enforce maximum prefetch limits at the broker level
  • Size broker memory based on expected concurrent consumers multiplied by prefetch count and average message size