Introduction

Kafka's idempotent producer prevents duplicate messages by tracking a per-partition sequence number on the broker. When a broker crashes and restarts, the in-memory sequence number state is lost. The producer detects this and resets its sequence numbers, which can result in duplicate messages being written to the partition during the recovery window.

Symptoms

  • Duplicate messages appear in the topic after broker restart despite idempotent producer enabled
  • Producer logs show ProducerId reset or OutOfOrderSequenceException
  • Broker logs show Producer epoch changed after restart
  • Consumer receives messages with duplicate keys and identical payloads
  • Error message: The broker received a sequence number out of order

Common Causes

  • Broker crash before producer idempotence state is persisted to the transaction log
  • Producer continues sending after broker restart without resetting its sequence number
  • enable.idempotence=true but acks not set to all, weakening idempotence guarantees
  • Multiple producer instances sharing the same transactional.id
  • Broker restart clears producer state from the __producer_state internal topic

Step-by-Step Fix

  1. 1.Verify idempotence configuration: Check producer settings.
  2. 2.```properties
  3. 3.enable.idempotence=true
  4. 4.acks=all
  5. 5.max.in.flight.requests.per.connection=5
  6. 6.retries=Integer.MAX_VALUE
  7. 7.`
  8. 8.Check for OutOfOrderSequenceException in producer logs: Confirm the idempotence break.
  9. 9.```bash
  10. 10.grep "OutOfOrderSequenceException" /var/log/kafka-producer.log | tail -10
  11. 11.`
  12. 12.Reset producer after broker recovery: Initialize the producer cleanly after the broker is stable.
  13. 13.```java
  14. 14.producer.close(Duration.ofSeconds(30));
  15. 15.// Create new producer instance to reset producer ID and sequence numbers
  16. 16.KafkaProducer<String, String> newProducer = new KafkaProducer<>(props);
  17. 17.`
  18. 18.Enable transactional producer for exactly-once semantics: Use transactions to guarantee no duplicates.
  19. 19.```java
  20. 20.props.put("transactional.id", "my-producer-" + UUID.randomUUID());
  21. 21.producer.initTransactions();
  22. 22.producer.beginTransaction();
  23. 23.producer.send(record);
  24. 24.producer.commitTransaction();
  25. 25.`
  26. 26.Deduplicate at the consumer side as a safety net: Implement idempotent consumption.
  27. 27.```java
  28. 28.// Use event ID for deduplication in downstream storage
  29. 29.database.upsert("INSERT INTO events (id, data) VALUES (?, ?) ON CONFLICT (id) DO NOTHING",
  30. 30.record.key(), record.value());
  31. 31.`

Prevention

  • Always pair enable.idempotence=true with acks=all for full idempotence guarantees
  • Use transactional producers with unique transactional.id per producer instance for exactly-once semantics
  • Implement consumer-side idempotency using unique message IDs regardless of producer idempotence
  • Monitor producer OutOfOrderSequenceException count as an indicator of idempotence breaks
  • Avoid sharing transactional.id across multiple producer instances
  • Configure transaction.timeout.ms appropriately to prevent premature transaction expiry