Introduction

Self-hosted GitHub Actions runners that execute containerized jobs accumulate Docker image layers and build cache over time. Each workflow run that uses container: or builds Docker images adds layers to the local Docker storage. Without regular cleanup, the runner's disk fills up, causing workflow failures with no space left on device errors.

Symptoms

  • Workflow jobs fail with No space left on device during Docker operations
  • Runner logs show docker: failed to register layer: no space left on device
  • df -h on the runner shows disk usage at 100%
  • docker system df shows Docker consuming most of the disk
  • Error message: Error: Process completed with exit code 1: ENOSPC

Common Causes

  • Docker build cache not pruned between workflow runs
  • Multiple workflows building different images without cleanup
  • Runner shared across many repositories, each with their own Docker images
  • Docker --no-cache builds creating layers that are never reused
  • Dangling images from failed builds not automatically removed

Step-by-Step Fix

  1. 1.Check Docker disk usage: Identify what is consuming space.
  2. 2.```bash
  3. 3.docker system df
  4. 4.df -h /var/lib/docker
  5. 5.`
  6. 6.Prune unused Docker resources: Free disk space immediately.
  7. 7.```bash
  8. 8.# Remove all unused images, containers, volumes, and networks
  9. 9.docker system prune -af --volumes
  10. 10.# Remove dangling images only (safer)
  11. 11.docker image prune -f
  12. 12.`
  13. 13.Add disk cleanup as a workflow step: Prevent future accumulation.
  14. 14.```yaml
  15. 15.jobs:
  16. 16.build:
  17. 17.runs-on: self-hosted
  18. 18.steps:
  19. 19.- name: Cleanup Docker before build
  20. 20.run: |
  21. 21.docker system prune -f
  22. 22.docker image prune -f
  23. 23.- name: Run build
  24. 24.run: docker build -t myapp .
  25. 25.- name: Cleanup Docker after build
  26. 26.if: always()
  27. 27.run: |
  28. 28.docker system prune -f
  29. 29.`
  30. 30.Configure Docker storage driver and limits: Optimize Docker disk usage.
  31. 31.```json
  32. 32.# /etc/docker/daemon.json
  33. 33.{
  34. 34."storage-driver": "overlay2",
  35. 35."storage-opts": ["overlay2.override_kernel_check=true"],
  36. 36."log-driver": "json-file",
  37. 37."log-opts": {
  38. 38."max-size": "10m",
  39. 39."max-file": "3"
  40. 40.}
  41. 41.}
  42. 42.`
  43. 43.Set up automated cleanup on the runner: Schedule regular pruning.
  44. 44.```bash
  45. 45.# Cron job for Docker cleanup
  46. 46.crontab -e
  47. 47.0 */6 * * * docker system prune -af --volumes --filter "until=24h"
  48. 48.`

Prevention

  • Include Docker cleanup steps in all workflow templates for self-hosted runners
  • Monitor runner disk usage and alert when it exceeds 80% capacity
  • Use shared Docker registry to pull pre-built images instead of building on the runner
  • Configure Docker daemon with log rotation to prevent log file growth
  • Set up runner health checks that verify adequate disk space before accepting jobs
  • Consider using ephemeral runners that are recreated after each workflow run