Introduction

RabbitMQ federation links replicate messages between brokers across different clusters or data centers. When the upstream broker uses TLS, the federation link must verify the upstream certificate chain. Certificate verification failures -- caused by expired certificates, missing CA certificates, or hostname mismatches -- prevent the federation link from establishing, breaking cross-cluster message replication.

Symptoms

  • Federation status shows down with certificate verify failed error
  • Upstream connection attempts fail with TLS handshake errors
  • Messages are not replicated to the federated downstream queue
  • RabbitMQ logs show {ssl_error, {tls_alert, {bad_certificate, ...}}}
  • Error message: Federation link shutdown: upstream connection failed - certificate verify failed

Common Causes

  • Upstream broker TLS certificate expired and has not been renewed
  • Downstream broker truststore does not contain the upstream CA certificate
  • Upstream certificate SAN does not match the hostname used in the federation URI
  • Certificate chain incomplete -- intermediate CA certificate missing
  • TLS version mismatch between upstream and downstream brokers

Step-by-Step Fix

  1. 1.Check federation link status and error details: Identify the failing upstream connection.
  2. 2.```bash
  3. 3.rabbitmqctl federation_status
  4. 4.`
  5. 5.Verify the upstream broker certificate: Test the TLS connection from the downstream broker.
  6. 6.```bash
  7. 7.openssl s_client -connect upstream-broker:5671 -CAfile /etc/rabbitmq/ca.pem </dev/null 2>&1 | grep "Verify return code"
  8. 8.`
  9. 9.Update the truststore with the correct CA certificate: Ensure the downstream broker trusts the upstream CA.
  10. 10.```bash
  11. 11.cat upstream-ca.pem >> /etc/rabbitmq/ca-certificates.crt
  12. 12.# Restart RabbitMQ to reload the truststore
  13. 13.systemctl restart rabbitmq-server
  14. 14.`
  15. 15.Update the federation upstream URI with correct hostname: Ensure the URI matches the certificate SAN.
  16. 16.```bash
  17. 17.rabbitmqctl set_parameter federation-uri '{"uris":["amqps://upstream-broker.example.com:5671"]}'
  18. 18.`
  19. 19.Verify the federation link reconnects: Check that the link transitions to running state.
  20. 20.```bash
  21. 21.rabbitmqctl federation_status
  22. 22.# Should show: status: running
  23. 23.`

Prevention

  • Automate TLS certificate renewal for all brokers in the federation topology
  • Include all broker CA certificates in each broker's truststore
  • Use internal DNS names that match certificate SANs in federation URIs
  • Monitor federation link status and alert on any link in down or starting state
  • Implement certificate expiry monitoring with alerts at 30, 14, and 7 days before expiration
  • Test federation connectivity after any certificate rotation using automated health checks