Introduction

GitLab CI/CD jobs can specify tags to route them to specific runners. When no online runner has all the tags required by a job, the pipeline remains stuck in pending state indefinitely. This is common after runner decommissioning, tag changes, or when new jobs are added with tags that no existing runner supports.

Symptoms

  • Pipeline shows stuck or pending status with no jobs running
  • Job detail shows This job is stuck because no runner with these tags is online
  • All runners are online but none have the required tag combination
  • Pipeline timeout eventually kills the job after hours of waiting
  • Error message: This job is stuck because you don't have any active runners that can run this job

Common Causes

  • Runner with the required tag was decommissioned or went offline
  • Job tag was changed but no runner was updated with the new tag
  • Runner tags were modified without updating the pipeline configuration
  • Shared runners disabled at the project level and no specific runners available
  • New environment (e.g., staging) added to pipeline without corresponding runner

Step-by-Step Fix

  1. 1.Check the job's required tags: Identify what tags the job needs.
  2. 2.```yaml
  3. 3.# .gitlab-ci.yml
  4. 4.build:
  5. 5.stage: build
  6. 6.tags:
  7. 7.- docker
  8. 8.- kubernetes # Job requires both tags
  9. 9.`
  10. 10.Check available runners and their tags: Find matching runners.
  11. 11.```bash
  12. 12.# GitLab UI: Settings > CI/CD > Runners
  13. 13.# Check each runner's tags and online status
  14. 14.# Or via API
  15. 15.curl --header "PRIVATE-TOKEN: $TOKEN" \
  16. 16."https://gitlab.example.com/api/v4/runners" | jq '.[] | {id, description, tag_list, online}'
  17. 17.`
  18. 18.Register a new runner with the required tags: Add a matching runner.
  19. 19.```bash
  20. 20.gitlab-runner register \
  21. 21.--url https://gitlab.example.com/ \
  22. 22.--token $REGISTRATION_TOKEN \
  23. 23.--executor docker \
  24. 24.--docker-image docker:latest \
  25. 25.--tag-list "docker,kubernetes" \
  26. 26.--description "k8s-build-runner"
  27. 27.`
  28. 28.Or update the pipeline to use available runner tags: Match existing runners.
  29. 29.```yaml
  30. 30.# .gitlab-ci.yml - update to match available runners
  31. 31.build:
  32. 32.stage: build
  33. 33.tags:
  34. 34.- docker # Remove kubernetes tag if no runner has it
  35. 35.`
  36. 36.Enable shared runners as a fallback: Ensure jobs always have a runner.
  37. 37.`
  38. 38.# GitLab UI: Settings > CI/CD > Runners
  39. 39.# Enable "Enable shared runners for this project"
  40. 40.`

Prevention

  • Maintain a registry of runner tags and which jobs require them
  • Use a CI/CD lint tool to validate that all job tags have matching runners
  • Monitor pipeline pending duration and alert when jobs are stuck
  • Enable shared runners as a fallback for jobs without specific runner requirements
  • Document runner provisioning process when adding new tags to pipeline jobs
  • Implement runner health checks that verify tag coverage for all active pipelines