Introduction

Kafka supports SASL/SCRAM authentication for client-broker communication. During credential rotation, there is a window where brokers may have updated credentials but clients still use old ones (or vice versa), causing authentication failures. This window can block all producer and consumer connections to the cluster if not managed carefully.

Symptoms

  • Clients fail to connect with SaslAuthenticationException: Authentication failed
  • Broker logs show Failed authentication with SCRAM-SHA-256 for rotating credentials
  • Producer and consumer connections are intermittently rejected during rotation window
  • Error message: javax.security.sasl.SaslException: Authentication failed: Invalid username or password
  • Metrics show authentication failure rate spiking during credential rotation

Common Causes

  • Credentials rotated on the broker before all clients receive the new credentials
  • ZooKeeper-based SCRAM credential storage not propagating updates to all brokers simultaneously
  • Client application caching old credentials without checking for rotation signals
  • Credential rotation script failing silently for some users but succeeding for others
  • SCRAM iteration count mismatch between broker and client configuration

Step-by-Step Fix

  1. 1.Identify which clients are failing authentication: Check broker authentication logs.
  2. 2.```bash
  3. 3.grep "SaslAuthenticationException|Failed authentication" /var/log/kafka/server.log | tail -20
  4. 4.`
  5. 5.Verify current SCRAM credentials in ZooKeeper: Check the stored credential state.
  6. 6.```bash
  7. 7.kafka-configs.sh --bootstrap-server localhost:9092 \
  8. 8.--entity-type users --entity-name my-client --describe
  9. 9.`
  10. 10.Update credentials on the broker using kafka-configs: Ensure the broker has the correct credentials.
  11. 11.```bash
  12. 12.kafka-configs.sh --bootstrap-server localhost:9092 \
  13. 13.--alter --add-config 'SCRAM-SHA-256=[password=newSecurePassword123],SCRAM-SHA-512=[password=newSecurePassword123]' \
  14. 14.--entity-type users --entity-name my-client
  15. 15.`
  16. 16.Update client JAAS configuration with new credentials: Deploy updated credentials to all clients.
  17. 17.```properties
  18. 18.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  19. 19.username="my-client" \
  20. 20.password="newSecurePassword123";
  21. 21.sasl.mechanism=SCRAM-SHA-256
  22. 22.security.protocol=SASL_SSL
  23. 23.`
  24. 24.Rolling restart clients with updated credentials: Update clients one at a time to maintain availability.
  25. 25.```bash
  26. 26.for client in client-1 client-2 client-3; do
  27. 27.kubectl rollout restart deployment/$client
  28. 28.kubectl rollout status deployment/$client --timeout=120s
  29. 29.done
  30. 30.`

Prevention

  • Implement a dual-credential rotation strategy: add new credentials before removing old ones
  • Use automated credential rotation with Kubernetes secrets and sidecar reload
  • Monitor SCRAM authentication failure rate and alert during rotation windows
  • Maintain a credential rotation runbook with pre- and post-rotation verification steps
  • Use short-lived credentials with automated renewal to reduce manual rotation frequency
  • Test credential rotation in staging to verify all client applications handle the rotation gracefully