Introduction

When RabbitMQ messages expire due to a configured TTL (time-to-live), they are either routed to a dead letter exchange (DLX) or silently discarded. If no DLX is configured, expired messages are dropped without any trace, making it difficult to detect that messages are being lost and why.

Symptoms

  • Messages disappear from queues after the TTL period with no error or notification
  • Consumer message count is lower than producer message count with no apparent cause
  • Queue metrics show messages count decreasing but ack count does not match
  • No dead letter queue exists to inspect expired messages
  • Error message: Message expired and no dead letter exchange configured

Common Causes

  • Queue created with x-message-ttl but without x-dead-letter-exchange argument
  • DLX configured but the exchange does not exist or is not bound to a dead letter queue
  • Message TTL set too short for the consumer processing time
  • DLX routing key not matching any binding on the dead letter exchange
  • Per-message TTL overriding queue-level TTL with shorter values

Step-by-Step Fix

  1. 1.Check queue arguments for TTL and DLX configuration: Identify queues with TTL but no DLX.
  2. 2.```bash
  3. 3.rabbitmqctl list_queues name arguments --format json | jq '.[] | select(.arguments["x-message-ttl"] != null and .arguments["x-dead-letter-exchange"] == null)'
  4. 4.`
  5. 5.Create a dead letter exchange and queue: Set up the DLX infrastructure.
  6. 6.```bash
  7. 7.# Declare dead letter exchange
  8. 8.rabbitmqadmin declare exchange name=dlx type=direct
  9. 9.# Declare dead letter queue
  10. 10.rabbitmqadmin declare queue name=dead-letter-queue durable=true
  11. 11.# Bind DLX to dead letter queue
  12. 12.rabbitmqadmin declare binding source=dlx destination=dead-letter-queue routing_key=dlq
  13. 13.`
  14. 14.Update the queue with DLX argument: Add the dead letter exchange to the existing queue.
  15. 15.```bash
  16. 16.# Queues cannot have their arguments changed - recreate the queue
  17. 17.rabbitmqadmin delete queue name=my-queue
  18. 18.rabbitmqadmin declare queue name=my-queue durable=true \
  19. 19.arguments='{"x-message-ttl":60000,"x-dead-letter-exchange":"dlx","x-dead-letter-routing-key":"dlq"}'
  20. 20.`
  21. 21.Inspect dead letter queue for expired messages: Review what messages were expired.
  22. 22.```bash
  23. 23.rabbitmqctl list_queues dead-letter-queue messages
  24. 24.rabbitmqadmin get queue=dead-letter-queue count=10
  25. 25.`
  26. 26.Adjust TTL values based on actual consumer processing time: Set appropriate TTL.
  27. 27.```java
  28. 28.AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
  29. 29..expiration("300000") // 5 minutes
  30. 30..build();
  31. 31.channel.basicPublish("my-exchange", "routing-key", props, body);
  32. 32.`

Prevention

  • Always configure x-dead-letter-exchange when setting x-message-ttl on queues
  • Set TTL values based on p99 consumer processing latency plus a safety margin
  • Monitor dead letter queue depth as an indicator of messages expiring under load
  • Implement alerting on the difference between published and consumed message counts
  • Use per-message TTL judiciously -- prefer queue-level TTL for consistent behavior
  • Create a standard DLX template that all queue declarations include by default