Introduction

Kafka message size limit configuration issues happen when one layer of the Kafka path allows a payload size that another layer still rejects. Producers, brokers, topic overrides, replicas, and consumers all have their own limits. If they are not aligned, you get large-message failures that are hard to diagnose because the error may appear on send, replication, or read.

Symptoms

  • Producers fail with message too large style errors
  • Brokers reject records larger than expected
  • Replication or fetch breaks when large messages are introduced
  • Consumers stall or fail when a topic begins carrying larger payloads

Common Causes

  • Producer max.request.size is lower than the intended payload
  • Broker or topic message.max.bytes still uses a smaller default
  • Replica or consumer fetch limits are lower than the broker accept limit
  • Large payload use cases grew without a coordinated size policy

Step-by-Step Fix

  1. 1.Check all size limits in the path
  2. 2.Review producer, broker, topic, replica, and consumer fetch settings together.
properties
max.request.size=5242880
message.max.bytes=5242880
replica.fetch.max.bytes=6291456
fetch.max.bytes=6291456
max.partition.fetch.bytes=6291456
  1. 1.Align the producer and broker limits first
  2. 2.The broker should not accept less than what the producer is configured to send.
bash
kafka-configs.sh --bootstrap-server broker1:9092 --entity-type topics --entity-name my-topic --describe
  1. 1.Update replica and consumer fetch settings
  2. 2.Large messages that replicate or fetch incorrectly often point to unaligned downstream limits.
bash
kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic my-topic
  1. 1.Retest with a controlled payload
  2. 2.Send a message near the intended limit and verify send, replication, and consumption all succeed.
bash
python produce-test-message.py --size-bytes 5000000

Prevention

  • Treat message-size policy as an end-to-end contract, not a single broker knob
  • Keep large-payload topics documented separately from normal topics
  • Test producer, replication, and consumer paths together after size changes
  • Prefer external object storage for oversized payloads when practical