# Docker Logging Driver Error: Troubleshooting Container Logs

Docker containers generate logs, and Docker captures them through logging drivers. When the logging driver fails, you might see container startup errors, missing logs, or disk space being consumed by unbounded log files. Different drivers have different failure modes and solutions.

Common error messages:

bash
failed to initialize logging driver: log driver type "syslog" not supported
bash
Error response from daemon: failed to start logging driver: connection refused
bash
failed to register layer: Error processing tar file: write error: no space left on device
bash
error opening log file: permission denied

Understanding Logging Drivers

Docker supports multiple logging drivers:

DriverDescriptionUse Case
json-fileDefault, stores logs in JSON formatLocal development, simple setups
syslogSends to syslog daemonCentralized logging
journaldSends to systemd journalsystemd-based systems
fluentdSends to Fluentd collectorLog aggregation pipelines
awslogsSends to CloudWatchAWS environments
gcplogsSends to Google Cloud LoggingGCP environments
noneDisables loggingSecurity-sensitive containers

Check current driver:

bash
docker info | grep "Logging Driver"
docker inspect <container> --format '{{.HostConfig.LogConfig.Type}}'

Diagnosis Steps

Check Container Log Configuration

bash
docker inspect <container> --format '{{json .HostConfig.LogConfig}}' | jq

Check Log File Location

bash
docker inspect <container> --format '{{.LogPath}}'

Check Log File Size

bash
ls -lh $(docker inspect <container> --format '{{.LogPath}}')

Check Daemon Logs

bash
sudo journalctl -u docker.service --no-pager -n 50 | grep -i log

Common Causes and Fixes

Cause 1: Unbounded Log Growth

The json-file driver fills disk space when log rotation isn't configured.

Symptoms: `` write error: no space left on device Log file exceeds maximum size

Diagnosis: ```bash # Find large log files find /var/lib/docker/containers -name "*-json.log" -exec ls -lh {} \; | sort -k5 -h

# Check individual log size docker inspect <container> --format '{{.LogPath}}' ls -lh $(docker inspect <container> --format '{{.LogPath}}') ```

Fix 1: Configure log rotation globally

json
// /etc/docker/daemon.json
{
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
bash
sudo systemctl restart docker

Fix 2: Configure per container

bash
docker run --log-opt max-size=10m --log-opt max-file=3 <image>

In Docker Compose:

yaml
services:
  myapp:
    image: nginx
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Fix 3: Truncate existing large logs

bash
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Fix 4: Clean up and restart

bash
docker rm -f $(docker ps -aq)
sudo rm /var/lib/docker/containers/*/*-json.log
sudo systemctl restart docker

Cause 2: Syslog Driver Connection Refused

Syslog daemon isn't running or accessible.

Symptoms: `` failed to start logging driver: syslog: connection refused

Diagnosis: ```bash # Check syslog daemon sudo systemctl status rsyslog sudo systemctl status syslog-ng

# Check syslog is listening sudo netstat -tlnp | grep 514 ```

Fix 1: Start syslog daemon

bash
sudo systemctl start rsyslog
sudo systemctl enable rsyslog

Fix 2: Use correct syslog address

bash
docker run --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:514 <image>

For remote syslog:

bash
docker run --log-driver syslog \
  --log-opt syslog-address=tcp://192.168.1.100:514 \
  --log-opt syslog-tag="myapp" \
  <image>

Fix 3: Switch to default driver temporarily

bash
docker run --log-driver json-file <image>

Cause 3: Journald Driver Not Available

Journald requires systemd and proper permissions.

Symptoms: `` failed to initialize logging driver: log driver type "journald" not supported

Diagnosis: ```bash # Check systemd journal sudo systemctl status systemd-journald journalctl --version

# Check if systemd is running ps aux | grep systemd ```

Fix 1: Ensure systemd-journald running

bash
sudo systemctl start systemd-journald
sudo systemctl enable systemd-journald

Fix 2: Install journald driver plugin (older Docker)

bash
# On some systems, journald driver needs package
sudo apt install docker-logging-journald  # Ubuntu
sudo yum install docker-log-driver-journald  # RHEL

Fix 3: Use json-file instead

json
// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Cause 4: Fluentd Driver Connection Failed

Fluentd collector not reachable.

Symptoms: `` failed to start logging driver: fluentd: connection refused

Fix 1: Ensure Fluentd running

```bash # Check Fluentd sudo systemctl status td-agent sudo systemctl status fluentd

# Check listening port sudo netstat -tlnp | grep 24224 ```

Fix 2: Configure correct Fluentd address

bash
docker run --log-driver fluentd \
  --log-opt fluentd-address=127.0.0.1:24224 \
  --log-opt fluentd-tag=docker.myapp \
  <image>

Fix 3: Start Fluentd if needed

bash
sudo systemctl start td-agent
# Or
fluentd -c /etc/fluent/fluent.conf

Cause 5: Log File Permission Denied

Docker daemon can't write to log file location.

Symptoms: `` error opening log file: permission denied failed to create log file: permission denied

Fix: Check permissions

bash
ls -la /var/lib/docker/containers
sudo chown -R root:root /var/lib/docker/containers
sudo chmod -R 755 /var/lib/docker/containers
sudo systemctl restart docker

Cause 6: Invalid Logging Driver Options

Wrong configuration for the logging driver.

Symptoms: `` failed to initialize logging driver: invalid log option

Fix: Use valid options

```bash # Valid json-file options docker run --log-driver json-file \ --log-opt max-size=10m \ --log-opt max-file=3 \ --log-opt compress=true \ <image>

# Valid syslog options docker run --log-driver syslog \ --log-opt syslog-address=udp://127.0.0.1:514 \ --log-opt syslog-facility=daemon \ --log-opt syslog-tag="myapp" \ <image> ```

Cause 7: Container Logs Not Appearing

Logs aren visible but container is running.

Symptoms: `` docker logs <container> returns nothing

Diagnosis: ```bash # Check logging driver docker inspect <container> --format '{{.HostConfig.LogConfig.Type}}'

# If "none", logs disabled # If "syslog", logs go to syslog not docker logs ```

Fix 1: Use json-file driver for docker logs

bash
docker run --log-driver json-file <image>
docker logs <container>

Fix 2: Check syslog for logs

bash
sudo tail -f /var/log/syslog | grep <container_name>
journalctl CONTAINER_NAME=<container_name>

Fix 3: Check application writes to stdout/stderr

Docker only captures stdout and stderr:

bash
docker exec <container> ls -la /proc/1/fd/
# Check fd 1 (stdout) and fd 2 (stderr)

If application logs to files inside container, Docker won capture them:

bash
# Inside container, symlink logs to stdout
ln -sf /dev/stdout /var/log/app.log
ln -sf /dev/stderr /var/log/app-error.log

In Dockerfile:

dockerfile
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log

Cause 8: AWS CloudWatch Logs Error

awslogs driver can't send to CloudWatch.

Symptoms: `` failed to start logging driver: awslogs: AccessDenied

Fix: Configure AWS credentials

bash
docker run --log-driver awslogs \
  --log-opt awslogs-region=us-east-1 \
  --log-opt awslogs-group=my-log-group \
  --log-opt awslogs-stream=my-stream \
  -e AWS_ACCESS_KEY_ID=your-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret \
  <image>

Or configure daemon-wide:

json
// /etc/docker/daemon.json
{
  "log-driver": "awslogs",
  "log-opts": {
    "awslogs-region": "us-east-1",
    "awslogs-group": "docker-logs"
  }
}

Ensure IAM permissions allow logs:CreateLogStream and logs:PutLogEvents.

Log Rotation Configuration

Daemon-Level Configuration

json
// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5",
    "compress": "true"
  }
}

Apply:

bash
sudo systemctl restart docker

Note: This only affects new containers. Existing containers keep their configuration.

Container-Level Configuration

bash
docker run \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  --log-opt compress=true \
  <image>

Docker Compose Configuration

yaml
services:
  app:
    image: nginx
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
        compress: "true"

Disabling Logs for Specific Containers

For containers that don't need logs:

bash
docker run --log-driver none <image>

In Docker Compose:

yaml
services:
  one-time-job:
    image: alpine
    logging:
      driver: none
    command: echo "No logs captured"

Verification Steps

  1. 1.Check logging driver:
  2. 2.```bash
  3. 3.docker info | grep "Logging Driver"
  4. 4.docker inspect <container> --format '{{.HostConfig.LogConfig.Type}}'
  5. 5.`
  6. 6.Test logs appear:
  7. 7.```bash
  8. 8.docker logs <container>
  9. 9.docker logs --tail 100 <container>
  10. 10.`
  11. 11.Check log file size:
  12. 12.```bash
  13. 13.ls -lh $(docker inspect <container> --format '{{.LogPath}}')
  14. 14.`
  15. 15.Verify rotation works:
  16. 16.```bash
  17. 17.# Generate lots of logs
  18. 18.for i in {1..10000}; do docker exec <container> echo "Test log $i"; done

# Check log file size stayed bounded ls -lh $(docker inspect <container> --format '{{.LogPath}}') ```

  1. 1.Check external log collectors:
  2. 2.```bash
  3. 3.# For syslog
  4. 4.sudo tail /var/log/syslog | grep docker

# For journald journalctl CONTAINER_NAME=<container> -n 100

# For fluentd curl http://localhost:24224/api/logs ```

Logging driver errors usually stem from configuration issues or external service availability. Configure log rotation globally to prevent disk space exhaustion, and ensure the target log collector is running when using syslog, journald, or fluentd drivers.