Introduction

RabbitMQ lazy queues store messages on disk rather than in RAM to handle large message backlogs. When lazy queues are not properly configured for durability, messages may remain in memory buffers and fail to persist to disk before a broker restart, resulting in message loss. This is particularly dangerous because lazy queues are often used precisely when message backlogs are large.

Symptoms

  • Messages in lazy queues are lost after broker restart
  • Queue shows messages before restart but zero messages after restart
  • Broker logs show lazy queue index file errors
  • Disk I/O is low despite large message backlog, indicating messages are not being flushed
  • Error message: Lazy queue index corrupted or Failed to write index segment

Common Causes

  • Queue declared as non-durable (durable=false), causing all messages to be lost on restart
  • Messages published as non-persistent (delivery_mode=1), bypassing disk writes
  • Lazy queue page file corruption due to unclean broker shutdown
  • Insufficient disk space preventing lazy queue index file writes
  • File system permissions preventing RabbitMQ from writing to the queue index directory

Step-by-Step Fix

  1. 1.Verify queue durability and message persistence settings: Check the queue configuration.
  2. 2.```bash
  3. 3.rabbitmqctl list_queues name durable policy
  4. 4.`
  5. 5.Declare queues as durable with lazy mode: Ensure the queue persists across restarts.
  6. 6.```java
  7. 7.Map<String, Object> args = new HashMap<>();
  8. 8.args.put("x-queue-mode", "lazy");
  9. 9.channel.queueDeclare("my-lazy-queue", true, false, false, args);
  10. 10.`
  11. 11.Publish messages with persistent delivery mode: Ensure messages are written to disk.
  12. 12.```java
  13. 13.AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
  14. 14..deliveryMode(2) // persistent
  15. 15..build();
  16. 16.channel.basicPublish("exchange", "routing-key", props, body);
  17. 17.`
  18. 18.Check disk space and file permissions: Verify the broker can write to disk.
  19. 19.```bash
  20. 20.df -h /var/lib/rabbitmq
  21. 21.ls -la /var/lib/rabbitmq/mnesia/
  22. 22.`
  23. 23.Enable quorum queues for stronger durability guarantees: Quorum queues replicate data across nodes.
  24. 24.```bash
  25. 25.rabbitmqctl set_policy quorum-all "^important-queue$" '{"queue-mode":"lazy","queue-type":"quorum"}' --apply-to queues
  26. 26.`

Prevention

  • Always declare lazy queues as durable=true and publish with delivery_mode=2
  • Use quorum queues instead of classic lazy queues for stronger durability guarantees
  • Monitor disk I/O patterns for lazy queues to confirm messages are being written
  • Set up monitoring for queue depth before and after broker restarts to detect message loss
  • Configure graceful broker shutdown procedures to flush lazy queue buffers
  • Test message persistence by restarting a staging broker and verifying queue contents