Your logs have stopped working. Journalctl returns errors, or worse, shows nothing when you need to diagnose a critical issue. systemd-journald is the core logging service in modern Linux systems, and when it fails, you lose visibility into your system.

Understanding systemd-journald

systemd-journald collects and stores log data from the kernel, users, and services. Logs are stored in a structured, indexed binary format, typically in /var/log/journal/ for persistent storage or /run/log/journal/ for volatile storage.

Typical Error Messages

bash
Failed to get journal file descriptor: No such file or directory
Journal file /var/log/journal/.../system.journal is corrupted
Failed to open /var/log/journal/.../user-1000.journal: No space left on device
Failed to write entry to journal: No space left on device

Common symptoms: - journalctl commands return no output or errors - Logs aren't being written - Disk space issues in /var/log/journal/ - Systemd journal service failing to start

Diagnosing Journal Issues

Check journald Service Status

```bash # Check if journald is running systemctl status systemd-journald

# Check recent journald logs journalctl -u systemd-journald -n 50

# View journald statistics journalctl --disk-usage

# Check for journal corruption journalctl --verify ```

Check Storage Configuration

```bash # View current configuration cat /etc/systemd/journald.conf

# See effective configuration systemd-analyze cat-config systemd/journald.conf

# Check journal directories ls -la /var/log/journal/ ls -la /run/log/journal/

# Check journal file sizes du -sh /var/log/journal/* ```

Check Permissions

```bash # Verify directory permissions ls -la /var/log/journal/

# Should be: drwxr-sr-x root:systemd-journal # Files should be: -rw-r----- root:systemd-journal

# Check SELinux context (if applicable) ls -laZ /var/log/journal/ ```

Solutions

Solution 1: Free Up Journal Disk Space

When journal logs consume too much space:

```bash # Check current disk usage journalctl --disk-usage

# Vacuum to specific size sudo journalctl --vacuum-size=100M

# Keep only recent time period sudo journalctl --vacuum-time=7d

# Keep only N most recent files sudo journalctl --vacuum-files=10

# View what would be removed (dry run) sudo journalctl --vacuum-size=100M --dry-run ```

Configure automatic limits in /etc/systemd/journald.conf:

```bash # Edit configuration sudo nano /etc/systemd/journald.conf

# Set these values: [Journal] SystemMaxUse=500M # Maximum disk usage for system journal SystemKeepFree=1G # Always keep this much free space SystemMaxFileSize=100M # Maximum size per journal file RuntimeMaxUse=100M # Maximum for runtime (volatile) journal RuntimeKeepFree=200M MaxRetentionSec=1month # Maximum age of journal entries

# Apply changes sudo systemctl restart systemd-journald

# Verify journalctl --disk-usage ```

Solution 2: Fix Corrupted Journal Files

```bash # Verify journal integrity sudo journalctl --verify

# If corruption is found, try to recover sudo journalctl --verify --repair

# If that fails, backup and remove corrupted files sudo systemctl stop systemd-journald sudo mv /var/log/journal/$(cat /etc/machine-id)/system.journal /var/log/journal/$(cat /etc/machine-id)/system.journal.corrupted

# Restart journald (will create new journal) sudo systemctl start systemd-journald

# Verify it's working journalctl -n 10 ```

For persistent corruption issues:

```bash # Stop journald sudo systemctl stop systemd-journald

# Backup existing journals sudo tar -czf journal-backup-$(date +%Y%m%d).tar.gz /var/log/journal/

# Remove all journal files sudo rm -rf /var/log/journal/*

# Recreate directory with proper permissions sudo mkdir -p /var/log/journal/$(cat /etc/machine-id) sudo chmod 2755 /var/log/journal/$(cat /etc/machine-id) sudo chown root:systemd-journal /var/log/journal/$(cat /etc/machine-id)

# Start journald sudo systemctl start systemd-journald

# Verify journalctl --verify ```

Solution 3: Fix Permission Issues

```bash # Check if systemd-journal group exists getent group systemd-journal

# Create if missing sudo groupadd -r systemd-journal

# Set correct ownership sudo chown -R root:systemd-journal /var/log/journal/ sudo chown -R root:systemd-journal /run/log/journal/

# Set correct permissions sudo chmod 2755 /var/log/journal/ sudo chmod 2755 /var/log/journal/$(cat /etc/machine-id) sudo chmod 640 /var/log/journal/$(cat /etc/machine-id)/*.journal

# Add users to systemd-journal group to allow log access sudo usermod -a -G systemd-journal username

# Restart journald sudo systemctl restart systemd-journald ```

Solution 4: Fix Missing Journal Directory

```bash # Check if machine ID exists cat /etc/machine-id

# Generate if missing sudo systemd-machine-id-setup

# Create journal directory sudo mkdir -p /var/log/journal/$(cat /etc/machine-id) sudo chmod 2755 /var/log/journal/$(cat /etc/machine-id) sudo chown root:systemd-journal /var/log/journal/$(cat /etc/machine-id)

# Restart journald sudo systemctl restart systemd-journald ```

Solution 5: Switch Between Volatile and Persistent Storage

If you want logs to persist across reboots (persistent) or not (volatile):

```bash # Edit journald configuration sudo nano /etc/systemd/journald.conf

# For persistent storage (saved to disk) [Journal] Storage=persistent # Logs stored in /var/log/journal/

# For volatile storage (lost on reboot) [Journal] Storage=volatile # Logs stored in /run/log/journal/ RuntimeMaxUse=50M # Limit size for volatile storage

# For automatic (persistent if /var/log/journal exists) [Journal] Storage=auto # Default behavior

# Apply changes sudo systemctl restart systemd-journald

# Verify storage location ls -la /var/log/journal/ # For persistent ls -la /run/log/journal/ # For volatile ```

Solution 6: Configure Rate Limiting

Prevent log flooding:

```bash # Edit journald configuration sudo nano /etc/systemd/journald.conf

[Journal] RateLimitIntervalSec=30s # Time window for rate limiting RateLimitBurst=1000 # Max messages in the interval window

# Apply changes sudo systemctl restart systemd-journald ```

Solution 7: Fix Forwarding to Syslog

If you need logs forwarded to traditional syslog:

```bash # Ensure rsyslog is installed and running sudo systemctl status rsyslog

# Edit journald config to forward sudo nano /etc/systemd/journald.conf

[Journal] ForwardToSyslog=yes ForwardToKMsg=yes ForwardToConsole=no

# Restart both services sudo systemctl restart systemd-journald sudo systemctl restart rsyslog

# Verify forwarding tail -f /var/log/syslog ```

Solution 8: Access Logs for Specific Boot or Time

```bash # Current boot journalctl -b

# Previous boot journalctl -b -1

# Specific boot (list boots first) journalctl --list-boots journalctl -b 3

# Time range journalctl --since "2024-01-15 10:00:00" --until "2024-01-15 12:00:00" journalctl --since yesterday journalctl --since "2 hours ago"

# Real-time follow journalctl -f

# Filter by unit journalctl -u nginx.service journalctl -u nginx.service -u php-fpm.service

# Filter by priority journalctl -p err # Errors only journalctl -p warning # Warnings and above

# Filter by user journalctl _UID=1000

# Kernel messages only journalctl -k ```

Solution 9: Fix Split Journal Issues

User journals may have separate issues:

```bash # List all journals ls -la /var/log/journal/$(cat /etc/machine-id)/

# Check user journals ls -la /var/log/journal/$(cat /etc/machine-id)/user-*.journal

# View user-specific logs journalctl _UID=1000

# If user journal is corrupted sudo systemctl stop systemd-journald sudo mv /var/log/journal/$(cat /etc/machine-id)/user-1000.journal{,.corrupted} sudo systemctl start systemd-journald ```

Debugging journald Issues

Enable verbose debugging:

```bash # Enable debug logging for journald sudo mkdir -p /etc/systemd/system/systemd-journald.service.d/ cat << 'EOF' | sudo tee /etc/systemd/system/systemd-journald.service.d/debug.conf [Service] Environment=SYSTEMD_LOG_LEVEL=debug EOF

sudo systemctl daemon-reload sudo systemctl restart systemd-journald

# View debug output journalctl -u systemd-journald -f ```

Check for namespace issues:

```bash # Check for journal namespaces ls -la /var/log/journal/

# View all journal files find /var/log/journal -name "*.journal" -exec ls -lh {} \;

# Check machine IDs cat /etc/machine-id ls /var/log/journal/ ```

Verification

After applying fixes:

```bash # Verify journald is running systemctl status systemd-journald

# Check disk usage journalctl --disk-usage

# Verify journal integrity journalctl --verify

# Test log writing logger "Test message from troubleshooting" journalctl -n 1

# Verify you can read logs journalctl -n 20

# Check configuration is applied systemd-analyze cat-config systemd/journald.conf

# Verify storage mode journalctl -b 0 | head -1 ```

Best Practices

  • Set SystemMaxUse to limit disk usage (default is 10% of filesystem, max 4GB)
  • Set MaxRetentionSec to control log age (default is no limit)
  • Forward critical logs to remote syslog server for redundancy
  • Use journalctl -f for real-time monitoring
  • Combine with logrotate for traditional syslog files
  • Regularly check journalctl --disk-usage in monitoring
  • Back up journals before major system changes
bash
# Good default configuration
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month
RateLimitIntervalSec=30s
RateLimitBurst=1000