Introduction

SSL certificate verification failures between message brokers and clients prevent secure communication, causing connection rejections and producer/consumer outages. This typically occurs when the certificate chain is incomplete, the certificate has expired, or the client truststore does not contain the issuing CA certificate.

Symptoms

  • Producer and consumer connections fail with SSLHandshakeException or certificate verify failed
  • Broker logs show handshake_failure or bad_certificate alert messages
  • Clients report PKIX path building failed: unable to find valid certification path
  • Secure port connectivity tests with openssl show certificate chain errors
  • Metrics show dropped connections on the SSL listener port

Common Causes

  • Broker certificate expired and automatic renewal did not execute
  • Intermediate CA certificate missing from the broker certificate chain
  • Client truststore not updated after CA certificate rotation
  • Hostname verification failure -- certificate SAN does not match the broker DNS name
  • TLS protocol version mismatch between broker and client (e.g., TLS 1.2 vs TLS 1.3)

Step-by-Step Fix

  1. 1.Verify the broker certificate chain: Use openssl to inspect the full certificate chain served by the broker.
  2. 2.```bash
  3. 3.openssl s_client -connect broker.example.com:9093 -showcerts < /dev/null 2>&1 | openssl x509 -noout -dates -subject -issuer
  4. 4.`
  5. 5.Check certificate expiration and SAN entries: Confirm the certificate is valid and covers the correct hostname.
  6. 6.```bash
  7. 7.openssl s_client -connect broker.example.com:9093 < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -A1 "Subject Alternative Name"
  8. 8.`
  9. 9.Update the client truststore with the full CA chain: Import the root and intermediate CA certificates.
  10. 10.```bash
  11. 11.keytool -importcert -alias root-ca -file root-ca.pem -keystore client.truststore -storepass changeit -noprompt
  12. 12.keytool -importcert -alias intermediate-ca -file intermediate-ca.pem -keystore client.truststore -storepass changeit -noprompt
  13. 13.`
  14. 14.Verify keystore and truststore configuration: Ensure the client SSL configuration points to the correct files.
  15. 15.```properties
  16. 16.ssl.truststore.location=/etc/kafka/client.truststore
  17. 17.ssl.truststore.password=changeit
  18. 18.ssl.keystore.location=/etc/kafka/client.keystore
  19. 19.ssl.keystore.password=changeit
  20. 20.ssl.endpoint.identification.algorithm=https
  21. 21.`
  22. 22.Test the SSL connection end-to-end: Use openssl to confirm a successful handshake.
  23. 23.```bash
  24. 24.openssl s_client -connect broker.example.com:9093 -CAfile root-ca.pem </dev/null 2>&1 | grep "Verify return code"
  25. 25.`

Prevention

  • Automate certificate renewal with cert-manager or similar tools and configure pre-expiry alerts at 30, 14, and 7 days
  • Include intermediate CA certificates in the broker certificate chain during initial setup
  • Maintain a certificate inventory with expiration dates for all brokers and clients
  • Use mutual TLS (mTLS) with short-lived certificates issued by an internal CA
  • Monitor SSL handshake failure rates as a leading indicator of certificate issues