Introduction

GitLab Container Registry uses garbage collection to remove unused layers and reclaim disk space. During garbage collection, the registry enters read-only mode, blocking all push operations. If garbage collection runs for an extended period -- due to a large number of unused images -- push operations from CI/CD pipelines fail with errors, blocking deployments.

Symptoms

  • Docker push to GitLab Registry fails with blob upload unknown or repository read-only
  • GitLab registry returns HTTP 503 during garbage collection
  • CI/CD pipeline deployment step fails during image push
  • Registry admin page shows garbage collection in progress
  • Error message: Error pushing manifest: received unexpected HTTP status: 503 Service Unavailable

Common Causes

  • Garbage collection scheduled during active CI/CD pipeline hours
  • Large number of unused image layers causing extended GC duration
  • GC not configured with a maintenance window
  • Multiple projects pushing images simultaneously during GC
  • Registry storage driver (S3, filesystem) slow during GC operations

Step-by-Step Fix

  1. 1.Check if garbage collection is currently running: Verify GC status.
  2. 2.`
  3. 3.# GitLab Admin: Admin > Monitoring > Background Jobs
  4. 4.# Or via Rails console
  5. 5.gitlab-rails console
  6. 6.> Gitlab::BackgroundMigration::BackfillProjectStatistics.last
  7. 7.`
  8. 8.Wait for GC to complete before retrying the push: Simple but effective.
  9. 9.```bash
  10. 10.# Check GC status
  11. 11.curl --header "PRIVATE-TOKEN: $TOKEN" \
  12. 12."https://gitlab.example.com/api/v4/groups/$GROUP_ID/-/packages/registry/garbage_collection"
  13. 13.# Wait and retry
  14. 14.docker push registry.gitlab.example.com/group/project:latest
  15. 15.`
  16. 16.Schedule garbage collection during off-hours: Prevent conflicts with CI/CD.
  17. 17.```ruby
  18. 18.# GitLab Admin: configure GC schedule
  19. 19.# In gitlab.rb
  20. 20.registry['garbage_collection_enabled'] = true
  21. 21.registry['garbage_collection_schedule'] = '0 2 * * 0' # Sunday 2 AM
  22. 22.`
  23. 23.Enable online garbage collection (GitLab 15.0+): Allow pushes during GC.
  24. 24.```ruby
  25. 25.# In gitlab.rb for GitLab 15.0+
  26. 26.registry['garbage_collection_online_enabled'] = true
  27. 27.`
  28. 28.Implement retry logic in CI/CD pipelines: Handle temporary registry unavailability.
  29. 29.```yaml
  30. 30.# .gitlab-ci.yml
  31. 31.push:
  32. 32.stage: push
  33. 33.script:
  34. 34.- |
  35. 35.for i in 1 2 3 4 5; do
  36. 36.docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA && break
  37. 37.echo "Push failed, retrying in 30s..."
  38. 38.sleep 30
  39. 39.done
  40. 40.`

Prevention

  • Enable online garbage collection (GitLab 15.0+) to allow concurrent pushes
  • Schedule GC during maintenance windows when CI/CD activity is minimal
  • Monitor registry disk usage and GC duration to plan appropriate windows
  • Implement retry logic with exponential backoff in CI/CD image push steps
  • Set image cleanup policies to automatically remove old images before they accumulate
  • Use registry storage with adequate capacity to reduce GC frequency