Fix GitLab Runner Not Picking Jobs

What's Actually Happening

GitLab Runner is registered but not picking up jobs from the queue. CI/CD pipelines remain pending, jobs never start executing.

The Error You'll See

```bash $ gitlab-runner list

Configured runners: my-runner Token=xxx URL=https://gitlab.com ```

But jobs stuck in pending:

Job #123 pending
Runner: waiting for runner to pick job...

Runner offline:

Runner status: offline
Last contact: never

Job never starts:

# Pipeline stuck at:
stage: build
status: pending
# No runner assigned

Why This Happens

1. 1.Runner offline - Runner not connected to GitLab
2. 2.Tag mismatch - Job tags don't match runner tags
3. 3.Runner paused - Runner administratively paused
4. 4.Max jobs reached - Runner at concurrent job limit
5. 5.Registration issue - Runner token invalid or expired
6. 6.Network issue - Runner cannot reach GitLab

Step 1: Check Runner Status

```bash # List registered runners: gitlab-runner list

# Check runner status locally: gitlab-runner status

# Verify runner config: cat /etc/gitlab-runner/config.toml

# Check runner is running: systemctl status gitlab-runner

# Check runner logs: journalctl -u gitlab-runner -f

# Or: gitlab-runner --debug run

# Check GitLab UI for runner status: # GitLab -> Settings -> CI/CD -> Runners ```

Step 2: Verify Runner Registration

```bash # Check registration in GitLab: # Settings -> CI/CD -> Runners -> Expand

# Runner should show: # - Status: online (green circle) # - Version: current version # - Last contact: recent timestamp

# Verify runner token: grep "token" /etc/gitlab-runner/config.toml

# Re-register runner if needed: gitlab-runner register \ --non-interactive \ --url "https://gitlab.com" \ --registration-token "REGISTRATION_TOKEN" \ --description "my-runner" \ --tag-list "docker,linux" \ --executor "docker" \ --docker-image "alpine:latest"

# Verify registration: gitlab-runner verify

# Check runner can reach GitLab: curl -I https://gitlab.com ```

Step 3: Check Runner Tags

```bash # View runner tags in config: cat /etc/gitlab-runner/config.toml | grep -A5 "runners"

# Runner config example: [[runners]] name = "my-runner" url = "https://gitlab.com" token = "xxx" executor = "docker" [runners.docker] image = "alpine:latest" [runners.tags] tags = ["docker", "linux", "production"]

# Check job tags in .gitlab-ci.yml: job_name: tags: - docker - production script: - echo "Hello"

# Tags must match: # - Job requires specific tags # - Runner must have those tags # - If job has no tags, runner must have "Run untagged jobs" enabled

# Enable runner to pick untagged jobs: # GitLab UI -> Runner settings -> "Run untagged jobs" checkbox ```

Step 4: Check Runner Configuration

```bash # View full runner config: cat /etc/gitlab-runner/config.toml

# Check executor type: # shell, docker, docker-ssh, ssh, kubernetes, parallels, virtualbox

# Example docker executor config: [[runners]] name = "docker-runner" url = "https://gitlab.com" token = "xxx" executor = "docker" [runners.docker] image = "alpine:latest" privileged = false volumes = ["/cache"] [runners.cache] Type = "s3" ServerAddress = "s3.amazonaws.com"

# Check concurrent jobs limit: concurrent = 4

# Check request concurrency: [[runners]] request_concurrency = 1

# Edit config: vim /etc/gitlab-runner/config.toml

# Restart after edit: systemctl restart gitlab-runner ```

Step 5: Check Job Queue

```bash # In GitLab UI: # Project -> CI/CD -> Pipelines

# Check pending jobs: # - Job details should show required tags # - Check "Run on" section for runner requirements

# Via API: curl --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/projects/$PROJECT_ID/jobs?status=pending"

# Check runners that can pick job: curl --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/projects/$PROJECT_ID/runners"

# Check runner details: curl --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/runners/$RUNNER_ID"

# Check runner jobs: curl --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/runners/$RUNNER_ID/jobs" ```

Step 6: Fix Runner Communication

```bash # Test GitLab connectivity: curl -v https://gitlab.com/api/v4/version

# Check firewall: # Runner needs to reach GitLab on HTTPS (443)

# Test from runner host: ping gitlab.com telnet gitlab.com 443

# Check SSL/TLS: openssl s_client -connect gitlab.com:443

# If using self-hosted GitLab: # Check certificate validity curl --cacert /path/to/cert.pem https://gitlab.example.com/api/v4/version

# Check DNS: dig gitlab.com nslookup gitlab.com

# If runner behind proxy: # In config.toml: [[runners]] [runners.http] http_proxy = "http://proxy:8080" https_proxy = "http://proxy:8080" no_proxy = "localhost,internal.example.com" ```

Step 7: Check Runner Executor

```bash # For docker executor: # Check docker is running: systemctl status docker

# Check docker access: docker ps docker run alpine:latest echo test

# Check docker images: docker images

# For kubernetes executor: # Check kubernetes access: kubectl get pods

# Check service account: kubectl auth can-i create pods --as=system:serviceaccount:default:gitlab-runner

# For shell executor: # Check shell access: bash -c "echo test"

# Check user permissions: id

# Test executor manually: gitlab-runner exec docker job_name ```

Step 8: Update Runner

```bash # Check runner version: gitlab-runner --version

# Update runner: # For package installation: sudo apt update sudo apt install gitlab-runner

# For manual installation: curl -L https://packages.gitlab-runner.com/binaries/gitlab-runner/latest/linux-amd64 -o gitlab-runner chmod +x gitlab-runner mv gitlab-runner /usr/local/bin/

# Restart after update: systemctl restart gitlab-runner

# Verify version matches GitLab: # Runner version should be compatible with GitLab version ```

Step 9: Unpause Runner

```bash # In GitLab UI: # Settings -> CI/CD -> Runners -> Find runner -> Edit # Uncheck "Paused" checkbox

# Via API: curl --request PUT --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/runners/$RUNNER_ID" \ --data "paused=false"

# Check runner is active: curl --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/runners/$RUNNER_ID" | jq .paused

# Start runner locally: gitlab-runner run

# Or as service: systemctl start gitlab-runner ```

Step 10: GitLab Runner Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-gitlab-runner.sh #!/bin/bash

echo "=== Runner Status ===" gitlab-runner status

echo "" echo "=== Runner Version ===" gitlab-runner --version

echo "" echo "=== Runner List ===" gitlab-runner list

echo "" echo "=== Runner Config ===" cat /etc/gitlab-runner/config.toml

echo "" echo "=== Runner Service ===" systemctl status gitlab-runner --no-pager

echo "" echo "=== GitLab Connectivity ===" curl -s -o /dev/null -w "%{http_code}" https://gitlab.com/api/v4/version echo ""

echo "" echo "=== Docker Status (if docker executor) ===" systemctl status docker --no-pager 2>/dev/null || echo "Docker not used" docker ps 2>/dev/null | head -5 || echo "Docker not available"

echo "" echo "=== Recent Runner Logs ===" journalctl -u gitlab-runner --no-pager -n 20

echo "" echo "=== Test Job Execution ===" echo "Run manually: gitlab-runner exec docker <job_name>" EOF

chmod +x /usr/local/bin/check-gitlab-runner.sh

# Usage: /usr/local/bin/check-gitlab-runner.sh

# Monitor: watch -n 30 /usr/local/bin/check-gitlab-runner.sh ```

GitLab Runner Checklist

Verify the Fix

```bash # After fixing runner

# 1. Check runner status gitlab-runner status // Running

# 2. Verify GitLab shows runner online # GitLab UI -> Settings -> CI/CD -> Runners // Green circle, recent contact

# 3. Trigger a pipeline # Push commit or run pipeline manually // Job starts running

# 4. Check job assignment # In job details // Runner assigned, job executes

# 5. Monitor logs journalctl -u gitlab-runner -f // No errors, jobs processed

# 6. Test multiple jobs # Run several jobs simultaneously // All picked up within concurrent limit ```

Related Issues

• [Fix GitLab CI Pipeline Stuck](/articles/fix-gitlab-ci-pipeline-stuck)
• [Fix Jenkins Build Stuck](/articles/fix-jenkins-build-stuck)
• [Fix GitHub Actions Workflow Failed](/articles/fix-github-actions-workflow-failed)