Introduction

Jenkins archives build artifacts (test reports, binaries, logs) on the controller node by default. Over time, these artifacts accumulate, consuming disk space. When the controller disk fills up, new builds cannot archive artifacts, the Jenkins UI becomes unresponsive, and the entire CI/CD pipeline grinds to a halt.

Symptoms

  • Builds fail during the artifact archiving step with No space left on device
  • Jenkins UI loads slowly or returns 500 errors
  • df -h shows controller disk at 100% usage
  • New builds cannot start because Jenkins cannot write to the build record directory
  • Error message: java.io.IOException: No space left on device

Common Causes

  • Artifact archiving enabled for all builds without retention policy
  • Large build artifacts (Docker images, compiled binaries) archived on every build
  • Build history never cleaned up, accumulating artifacts from all past builds
  • Log files growing unbounded with verbose build output
  • No external artifact repository configured, storing everything on the controller

Step-by-Step Fix

  1. 1.Check disk usage by Jenkins directories: Identify the largest consumers of space.
  2. 2.```bash
  3. 3.du -sh /var/lib/jenkins/jobs/*/builds/* | sort -rh | head -20
  4. 4.du -sh /var/lib/jenkins/artifacts/ 2>/dev/null
  5. 5.`
  6. 6.Remove old build artifacts manually: Free disk space immediately.
  7. 7.```bash
  8. 8.# Remove builds older than 30 days
  9. 9.find /var/lib/jenkins/jobs -name "builds" -type d -exec find {} -maxdepth 1 -mtime +30 -type d \; -exec rm -rf {} \;
  10. 10.# Clean up workspace
  11. 11.find /var/lib/jenkins/workspace -type f -mtime +7 -delete
  12. 12.`
  13. 13.Configure build discarder policy: Automatically remove old builds.
  14. 14.```groovy
  15. 15.// In Jenkinsfile
  16. 16.options {
  17. 17.buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '5', daysToKeepStr: '30'))
  18. 18.}
  19. 19.`
  20. 20.Set up external artifact storage: Move artifacts off the controller.
  21. 21.```groovy
  22. 22.// Use Jenkins Artifact Manager on S3
  23. 23.// Configure in: Manage Jenkins > Configure System > Artifact Management
  24. 24.// Or in Jenkinsfile:
  25. 25.post {
  26. 26.success {
  27. 27.sh 'aws s3 sync ${WORKSPACE}/target/ s3://my-artifacts/${BUILD_NUMBER}/'
  28. 28.}
  29. 29.}
  30. 30.`
  31. 31.Configure log rotation: Prevent log files from consuming disk space.
  32. 32.```bash
  33. 33.# /etc/logrotate.d/jenkins
  34. 34./var/log/jenkins/*.log {
  35. 35.weekly
  36. 36.rotate 12
  37. 37.compress
  38. 38.delaycompress
  39. 39.missingok
  40. 40.}
  41. 41.`

Prevention

  • Configure build discarder policies on all jobs to limit retained builds and artifacts
  • Use external artifact storage (S3, Nexus, Artifactory) for large build artifacts
  • Monitor controller disk usage and alert at 70% and 85% capacity
  • Archive only essential artifacts -- exclude intermediate build outputs
  • Set up automated cleanup jobs that run weekly to remove orphaned files
  • Consider using Jenkins ephemeral controller patterns where the controller is rebuilt regularly