Introduction

Jenkins assigns builds to executors based on node labels defined in the job configuration. When no executor with the required label is online or available, the build remains in the queue indefinitely. This blocks the entire CI/CD pipeline and can delay releases, especially when the required nodes are offline or at capacity.

Symptoms

  • Build shows Stuck in queue with message Waiting for next available executor on nodes with label
  • Jenkins queue page shows the build waiting with no estimated start time
  • All nodes with the required label are offline or have all executors busy
  • Pipeline downstream jobs are blocked waiting for the stuck build
  • Error message: There are no nodes with the label 'docker-builder'

Common Causes

  • Agent node with the required label crashed or was terminated (e.g., spot instance)
  • All executors on matching nodes are occupied by other builds
  • Node label was changed or removed after the job was configured
  • Cloud agent plugin failing to provision new nodes due to quota limits
  • Jenkins controller lost connection to the agent due to network issues

Step-by-Step Fix

  1. 1.Check the queue item details: Identify the required label and waiting reason.
  2. 2.`
  3. 3.# Jenkins UI: Manage Jenkins > Nodes and Clouds > Build Queue
  4. 4.# Hover over the stuck build to see the label requirement
  5. 5.`
  6. 6.Check node status for the required label: Verify if matching nodes are online.
  7. 7.`
  8. 8.# Jenkins UI: Manage Jenkins > Nodes and Clouds
  9. 9.# Check nodes with the required label - are they online?
  10. 10.# Or via CLI
  11. 11.java -jar jenkins-cli.jar -s https://jenkins.example.com/ list-nodes
  12. 12.`
  13. 13.Restart or reconnect the offline agent: Bring the node back online.
  14. 14.```bash
  15. 15.# On the agent machine
  16. 16.java -jar agent.jar -jnlpUrl https://jenkins.example.com/computer/docker-builder/slave-agent.jnlp \
  17. 17.-secret <secret> -workDir "/home/jenkins"
  18. 18.`
  19. 19.If using cloud agents, check provisioning limits: Ensure the cloud can scale.
  20. 20.`
  21. 21.# Jenkins UI: Manage Jenkins > Clouds > configure
  22. 22.# Check instance cap and verify cloud provider quota
  23. 23.# For EC2:
  24. 24.aws ec2 describe-spot-instance-requests --filters "Name=state,Values=active"
  25. 25.`
  26. 26.Cancel and requeue the build with an alternative label: If the original nodes cannot be recovered.
  27. 27.`
  28. 28.# Jenkins UI: Build Queue > Cancel the stuck build
  29. 29.# Update the job label and re-trigger
  30. 30.# Or use pipeline with fallback labels:
  31. 31.agent { label 'docker-builder || linux-builder' }
  32. 32.`

Prevention

  • Configure multiple agent labels as fallbacks in pipeline agent definitions
  • Set up cloud-based agents with auto-scaling to handle queue buildup
  • Monitor executor availability and alert when all executors for a label are offline
  • Use Jenkins Shared Libraries to standardize label requirements across pipelines
  • Configure cloud agent instance caps above the expected maximum concurrent builds
  • Implement queue monitoring with plugins like "Queue Metrics" to detect stuck builds early