Introduction

Systemd-journald includes a built-in rate limiter to prevent a single service from flooding the journal with log messages. When a service exceeds the configured burst limit within the rate interval, subsequent messages are dropped and replaced with a single Suppressed X messages entry. This can hide critical error information during incidents.

Symptoms

  • Journal shows Suppressed 3421 messages from myapp.service
  • Important error logs are missing during high-traffic periods
  • journalctl -u myapp has gaps where log entries should be
  • Intermittent issues are impossible to diagnose because logs are dropped

Common Causes

  • Application enters error loop generating thousands of messages per second
  • Default RateLimitBurst=10000 and RateLimitIntervalSec=30s too low for verbose services
  • Debug logging accidentally enabled in production
  • Multiple services sharing the same rate limit bucket under older systemd versions

Step-by-Step Fix

  1. 1.Check for suppressed messages in the journal:
  2. 2.```bash
  3. 3.journalctl -p warning --since "1 hour ago" | grep -i "suppressed"
  4. 4.journalctl --since "30 min ago" | grep -c "Suppressed"
  5. 5.`
  6. 6.Review current rate limit settings:
  7. 7.```bash
  8. 8.grep -E "RateLimit" /etc/systemd/journald.conf
  9. 9.# Default output: RateLimitIntervalSec=30s, RateLimitBurst=10000
  10. 10.`
  11. 11.Increase the rate limit in journald.conf:
  12. 12.```bash
  13. 13.sudo nano /etc/systemd/journald.conf
  14. 14.`
  15. 15.Add or modify:
  16. 16.```ini
  17. 17.[Journal]
  18. 18.RateLimitIntervalSec=5s
  19. 19.RateLimitBurst=50000
  20. 20.SystemMaxUse=2G
  21. 21.`
  22. 22.Alternatively, disable rate limiting for a specific service:
  23. 23.```bash
  24. 24.sudo systemctl edit myapp.service
  25. 25.`
  26. 26.Add:
  27. 27.```ini
  28. 28.[Service]
  29. 29.LogRateLimitIntervalSec=0
  30. 30.LogRateLimitBurst=0
  31. 31.`
  32. 32.Restart journald to apply changes:
  33. 33.```bash
  34. 34.sudo systemctl restart systemd-journald
  35. 35.# Verify the new settings
  36. 36.systemctl show systemd-journald | grep -i ratelimit
  37. 37.`
  38. 38.Fix the root cause - reduce application log verbosity:
  39. 39.```bash
  40. 40.# Check application log level
  41. 41.grep -r "LogLevel|log_level" /etc/myapp/
  42. 42.# Reduce from DEBUG to INFO or WARNING in production
  43. 43.`

Prevention

  • Set appropriate log levels (INFO or above) for production environments
  • Use structured logging with rate limiting at the application level
  • Monitor journal size with journalctl --disk-usage and set SystemMaxUse
  • Configure persistent journal storage: Storage=persistent in journald.conf
  • Use external log aggregation (rsyslog, fluentd) for high-volume services