Fix Jenkins Pipeline Stuck in Queue

What's Actually Happening

Your Jenkins pipeline jobs are stuck in the build queue waiting for an executor but never starting. The queue grows and no builds are running despite having configured executors.

The Error You'll See

Build stuck in queue:

# Jenkins UI shows:
Build #123 (pending—Waiting for executor)

Queue growing:

```bash # Jenkins Script Console: Jenkins.instance.queue.items.each { println "${it.task.fullDisplayName}: ${it.why}" }

# Output: my-job #1: Waiting for executor my-job #2: Waiting for executor ```

All executors offline:

```bash # /queue API: curl -s http://localhost:8080/queue/api/json | jq '.items[].why'

# Output: "Waiting for executor" "Waiting for executor" ```

Why This Happens

1. 1.No executors available - All executors busy or offline
2. 2.Node offline - Agent node not connected
3. 3.Label mismatch - Job requires label no node provides
4. 4.Executor starvation - Long-running builds blocking
5. 5.Dead executors - Executors stuck in "in use" state
6. 6.Configuration error - Wrong number of executors set to 0
7. 7.Resource constraints - Not enough system resources
8. 8.Plugin issue - Plugin causing executor deadlock

Step 1: Check Executor Status

```bash # Via Script Console (http://localhost:8080/script):

// List all executors: Jenkins.instance.computers.each { computer -> println "${computer.name}: ${computer.countExecutors()} executors" println " Online: ${computer.isOnline()}" println " Offline cause: ${computer.offlineCause}" }

// Count total executors: def total = 0 def busy = 0 Jenkins.instance.computers.each { computer -> total += computer.countExecutors() computer.executors.each { executor -> if (executor.isBusy()) busy++ } } println "Total: $total, Busy: $busy, Available: ${total - busy}"

// Check master executors: println "Master executors: ${Jenkins.instance.numExecutors}" ```

Step 2: Check Node Status

```bash # Via Script Console:

// List all nodes and status: Jenkins.instance.nodes.each { node -> def computer = node.toComputer() println "Node: ${node.name}" println " Online: ${computer?.isOnline()}" println " Offline cause: ${computer?.offlineCause}" println " Labels: ${node.labelString}" println " Executors: ${node.numExecutors}" }

// Check specific node: def node = Jenkins.instance.getNode('my-agent') def computer = node?.toComputer() println "Node: ${node?.name}, Online: ${computer?.isOnline()}"

// Get offline nodes: Jenkins.instance.computers.each { computer -> if (!computer.isOnline()) { println "${computer.name}: ${computer.offlineCause}" } } ```

Step 3: Bring Nodes Online

```bash # Via Script Console:

// Bring node online: def node = Jenkins.instance.getNode('my-agent') def computer = node.toComputer() computer.connect()

// Force online all nodes: Jenkins.instance.computers.each { computer -> if (!computer.isOnline()) { computer.connect() println "Connecting ${computer.name}" } }

// Check node connection log: def node = Jenkins.instance.getNode('my-agent') def computer = node.toComputer() computer.log?.each { line -> println line }

# Via CLI: java -jar jenkins-cli.jar -s http://localhost:8080/ online-node my-agent

# Via SSH: ssh -p 22 my-agent "systemctl status jenkins-agent" ```

Step 4: Check Label Configuration

```bash # Via Script Console:

// Check job label requirements: def job = Jenkins.instance.getItemByFullName('my-job') if (job instanceof hudson.model.FreeStyleProject) { println "Label: ${job.assignedLabel}" }

// Find nodes with specific label: def label = 'linux' Jenkins.instance.nodes.each { node -> if (node.labelString.contains(label)) { println "${node.name}: ${node.labelString}" } }

// Or via API: Jenkins.instance.labelAtoms.find { it.name == 'linux' }.nodes.each { node -> println node.name }

// Fix job label: def job = Jenkins.instance.getItemByFullName('my-job') job.setAssignedLabel(null) // Remove label constraint ```

Step 5: Handle Executor Starvation

```bash # Via Script Console:

// List running builds: Jenkins.instance.computers.each { computer -> computer.executors.each { executor -> if (executor.isBusy()) { def workUnit = executor.currentWorkUnit println "${computer.name}: ${workUnit?.work?.context?.get(hudson.model.Run)}" } } }

// Find long-running builds: Jenkins.instance.computers.each { computer -> computer.executors.each { executor -> if (executor.isBusy()) { def build = executor.currentExecutable def duration = System.currentTimeMillis() - build.startTimeInMillis if (duration > 3600000) { // > 1 hour println "${build.fullDisplayName}: ${duration/60000} minutes" } } } }

// Abort stuck build: def build = Jenkins.instance.getItemByFullName('my-job').getBuildByNumber(123) build.doKill()

// Abort all builds on a node: def computer = Jenkins.instance.getComputer('my-agent') computer.interrupt() ```

Step 6: Fix Dead Executors

```bash # Via Script Console:

// Find dead executors: Jenkins.instance.computers.each { computer -> computer.executors.each { executor -> if (executor.isBusy() && executor.currentExecutable == null) { println "Dead executor: ${computer.name}-${executor.number}" } } }

// Clear dead executors: Jenkins.instance.computers.each { computer -> computer.executors.each { executor -> if (executor.isBusy() && executor.currentExecutable == null) { executor.interrupt() } } }

// Restart agent from script: def node = Jenkins.instance.getNode('my-agent') def computer = node.toComputer() computer.disconnect() Thread.sleep(5000) computer.connect() ```

Step 7: Clear Build Queue

```bash # Via Script Console:

// View queue: Jenkins.instance.queue.items.each { item -> println "${item.task.fullDisplayName}: ${item.why}" }

// Cancel all queued builds: Jenkins.instance.queue.clear()

// Cancel specific job: Jenkins.instance.queue.items.findAll { it.task.name == 'my-job' }.each { Jenkins.instance.queue.cancel(it.task) }

// Cancel builds older than X minutes: Jenkins.instance.queue.items.each { item -> def age = (System.currentTimeMillis() - item.inQueueSince) / 60000 if (age > 30) { println "Cancelling ${item.task.fullDisplayName} (age: ${age}m)" Jenkins.instance.queue.cancel(item.task) } } ```

Step 8: Add More Executors

```bash # Via Script Console:

// Increase master executors: Jenkins.instance.setNumExecutors(4) Jenkins.instance.save()

// Add executors to node: def node = Jenkins.instance.getNode('my-agent') node.setNumExecutors(4) Jenkins.instance.save()

# Via Groovy file in JENKINS_HOME/init.groovy.d/: import jenkins.model.* import hudson.model.*

Jenkins.instance.setNumExecutors(4) Jenkins.instance.save()

# Via configuration: # Edit $JENKINS_HOME/config.xml: # <numExecutors>4</numExecutors>

# Restart Jenkins: systemctl restart jenkins ```

Step 9: Monitor Queue Health

```bash # Create monitoring script: cat << 'EOF' > /usr/local/bin/jenkins-queue-monitor.sh #!/bin/bash

JENKINS_URL="http://localhost:8080" CRUMB=$(curl -s "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)")

# Get queue length QUEUE_LENGTH=$(curl -s "$JENKINS_URL/queue/api/json" | jq '.items | length')

# Get busy executors BUSY=$(curl -s "$JENKINS_URL/computer/api/json" | jq '[.computer[] | select(.busy == true)] | length')

# Get offline executors OFFLINE=$(curl -s "$JENKINS_URL/computer/api/json" | jq '[.computer[] | select(.offline == true)] | length')

echo "Queue: $QUEUE_LENGTH" echo "Busy executors: $BUSY" echo "Offline executors: $OFFLINE"

if [ "$QUEUE_LENGTH" -gt 10 ]; then echo "WARNING: Queue length > 10" fi

if [ "$OFFLINE" -gt 0 ]; then echo "WARNING: $OFFLINE executors offline" fi EOF

chmod +x /usr/local/bin/jenkins-queue-monitor.sh

# Prometheus metrics via Script Console: import jenkins.model.* import hudson.model.*

def metrics = [ queue_length: Jenkins.instance.queue.items.size(), total_executors: 0, busy_executors: 0, offline_nodes: 0 ]

Jenkins.instance.computers.each { computer -> metrics.total_executors += computer.countExecutors() if (!computer.isOnline()) metrics.offline_nodes++ computer.executors.each { executor -> if (executor.isBusy()) metrics.busy_executors++ } }

metrics.each { k, v -> println "${k} ${v}" } ```

Step 10: Production Best Practices

```groovy // Jenkinsfile with proper resource management:

pipeline { agent { label 'linux' }

options { timeout(time: 30, unit: 'MINUTES') buildDiscarder(logRotator(numToKeepStr: '10')) disableConcurrentBuilds() }

stages { stage('Build') { steps { script { // Check available executors def available = jenkins.model.Jenkins.instance.computers.collect { it.countExecutors() - it.executors.count { e -> e.isBusy() } }.sum()

echo "Available executors: ${available}" }

sh 'make build' } } }

post { always { // Clean workspace cleanWs() }

failure { // Notify team emailext subject: "Build failed: ${env.JOB_NAME}", body: "Build ${env.BUILD_NUMBER} failed", to: 'team@example.com' } } } ```

# Kubernetes Jenkins agent configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-agent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jenkins-agent
  template:
    metadata:
      labels:
        app: jenkins-agent
    spec:
      containers:
      - name: jenkins-agent
        image: jenkins/inbound-agent:latest
        env:
        - name: JENKINS_URL
          value: "http://jenkins:8080"
        - name: JENKINS_SECRET
          valueFrom:
            secretKeyRef:
              name: jenkins-agent-secret
              key: secret
        - name: JENKINS_AGENT_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          limits:
            cpu: "2"
            memory: "4Gi"
          requests:
            cpu: "1"
            memory: "2Gi"

Jenkins Queue Troubleshooting Checklist

Verify the Fix

```bash # After fixing queue issues:

# 1. Check executors online curl -s http://localhost:8080/computer/api/json | jq '.computer[] | select(.offline==true)' # Output: Empty

# 2. Check queue draining curl -s http://localhost:8080/queue/api/json | jq '.items | length' # Output: Decreasing

# 3. Verify builds running curl -s http://localhost:8080/computer/api/json | jq '[.computer[] | select(.busy==true)] | length' # Output: Builds processing

# 4. Test new build curl -X POST http://localhost:8080/job/my-job/build # Check if starts immediately

# Compare before/after: # Before: 50 in queue, 0 executors, all offline # After: 0 in queue, 4 executors, builds running ```

Related Issues

• [Fix Jenkins Agent Not Connecting](/articles/fix-jenkins-agent-not-connecting)
• [Fix Jenkins Build Failing](/articles/fix-jenkins-build-failing)
• [Fix Jenkins Plugin Conflict](/articles/fix-jenkins-plugin-conflict)