Fix GitLab CI Pipeline Stuck in Pending

What's Actually Happening

GitLab CI pipeline jobs remain stuck in "pending" state, never being picked up by runners. Pipelines queue indefinitely, blocking deployments and causing development delays.

The Error You'll See

Pipeline stuck pending:

```bash # In GitLab UI: Pipeline #123 - pending - Created 1 hour ago

Job 1: build - pending (stuck) Job 2: test - pending (waiting for build) Job 3: deploy - pending (waiting for test) ```

Job details:

This job is stuck, because the project doesn't have any runners online assigned to it.

Runner status:

```bash $ gitlab-runner list

Configured runners: my-runner Token=xxx URL=https://gitlab.com Status=offline ```

Why This Happens

1. 1.No runners available - All runners offline or paused
2. 2.Runner capacity full - concurrent limit reached
3. 3.Tag mismatch - Job tags don't match runner tags
4. 4.Runner paused - Runner explicitly paused
5. 5.Project not shared - Runner not assigned to project
6. 6.Job limit exceeded - Project job limit reached

Step 1: Check Runner Status

```bash # List runners in GitLab UI # Admin Area > CI/CD > Runners

# Or via API: curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/runners"

# Check specific runner curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/runners/<runner-id>"

# Check runner status locally gitlab-runner status

# List configured runners gitlab-runner list

# Verify runner is running systemctl status gitlab-runner docker ps | grep gitlab-runner ```

Step 2: Verify Runner is Online

```bash # Check runner process ps aux | grep gitlab-runner

# Check runner logs journalctl -u gitlab-runner -f docker logs gitlab-runner -f

# Common issues: # - Runner crashed # - Network connectivity to GitLab # - Token expired # - Config file missing

# Restart runner systemctl restart gitlab-runner docker restart gitlab-runner

# Verify runner connects gitlab-runner verify

# Output should show: # Runner my-runner is alive ```

Step 3: Check Runner Tags

```bash # Check job tags in .gitlab-ci.yml cat .gitlab-ci.yml

build: stage: build tags: - docker - production

# Check runner tags # In GitLab UI: Admin > Runners > <runner> > Tags # Or API: curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/runners/<id>" | jq '.tag_list'

# Job will only run on runner with matching tags # If job needs "docker" tag, runner must have "docker" tag

# Add tags to runner via API: curl --request PUT --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/runners/<id>" \ --data "tag_list[]=docker&tag_list[]=production"

# Or in config.toml: [[runners]] name = "my-runner" tags = ["docker", "production"] ```

Step 4: Check Runner Concurrency

```bash # Check runner concurrent limit cat /etc/gitlab-runner/config.toml

concurrent = 4 # Max jobs this runner can run

# If concurrent = 4 and 4 jobs running, new jobs stay pending

# Increase concurrent limit concurrent = 10

# Restart runner systemctl restart gitlab-runner

# Check current running jobs curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/runners/<id>/jobs" | jq '.[] | select(.status=="running")'

# Check runner capacity # In GitLab UI: Admin > Runners > shows "Active jobs" ```

Step 5: Check Project Runner Assignment

```bash # Check project runners curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<project-id>/runners"

# If runner is "shared", it's available to all projects # If runner is "specific", it must be explicitly assigned

# Assign runner to project curl --request POST --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<project-id>/runners" \ --data "runner_id=<runner-id>"

# Enable shared runners for project # In GitLab UI: Project > Settings > CI/CD > Runners > Enable shared runners

# Check if project has paused runners curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<project-id>/runners" | jq '.[] | select(.paused==true)' ```

Step 6: Unpause Runner

```bash # Check if runner is paused curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/runners/<id>" | jq '.paused'

# Unpause runner via API curl --request PUT --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/runners/<id>" \ --data "paused=false"

# Or in GitLab UI: Admin > Runners > <runner> > Unpause

# Verify runner status curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/runners/<id>" | jq '.status' # Should be "online" ```

Step 7: Check Job Queue Limit

```bash # Check project CI/CD settings # In GitLab UI: Project > Settings > CI/CD > General pipelines

# Check "Maximum jobs per pipeline" limit # If exceeded, jobs stay pending

# Check "CI/CD minutes quota" for group # Group > Settings > CI/CD > Quota

# If minutes quota exceeded, jobs cannot run

# Check via API: curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<project-id>" | jq '.ci_cd_settings'

# For group quota: curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/groups/<group-id>" | jq '.shared_runners_minutes_limit' ```

Step 8: Register New Runner

```bash # If no runners available, register new one

# Get registration token # In GitLab UI: Project > Settings > CI/CD > Runners > New project runner # Or: Admin > Runners > New instance runner

# Register runner gitlab-runner register \ --non-interactive \ --url https://gitlab.com \ --registration-token <token> \ --name "my-runner" \ --tag-list "docker,production" \ --executor "docker" \ --docker-image "alpine:latest"

# Verify runner registered gitlab-runner list

# Or use docker: docker run -d --name gitlab-runner --restart always \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest register \ --non-interactive \ --url https://gitlab.com \ --registration-token <token> \ --executor docker \ --docker-image alpine:latest ```

Step 9: Check Runner Executor Issues

```bash # Runner executor may have issues

# Check executor type cat /etc/gitlab-runner/config.toml | grep executor

# Common executor types: shell, docker, docker+machine, kubernetes

# For docker executor, check docker daemon: docker ps docker info

# For kubernetes executor, check cluster: kubectl get pods -n gitlab-runner

# For shell executor, check shell access: ssh user@runner-host

# Check runner can pull docker images docker pull alpine:latest

# Check runner disk space df -h

# Check runner memory free -h ```

Step 10: Monitor Runner and Pipeline

```bash # Create monitoring script cat << 'EOF' > monitor_gitlab_ci.sh #!/bin/bash TOKEN="<private-token>" PROJECT="<project-id>" GITLAB="https://gitlab.com"

echo "=== Project Runners ===" curl -s --header "PRIVATE-TOKEN: $TOKEN" \ "$GITLAB/api/v4/projects/$PROJECT/runners" | jq '.[] | {name, status, active, paused}'

echo "" echo "=== Pending Jobs ===" curl -s --header "PRIVATE-TOKEN: $TOKEN" \ "$GITLAB/api/v4/projects/$PROJECT/pipelines?status=pending" | jq '.[] | {id, status, created_at}'

echo "" echo "=== Runner Status (local) ===" gitlab-runner status

echo "" echo "=== Runner Process ===" ps aux | grep gitlab-runner EOF

chmod +x monitor_gitlab_ci.sh

# Monitor in loop watch -n 30 ./monitor_gitlab_ci.sh

# Enable GitLab CI/CD monitoring # Admin > Settings > Metrics and profiling > Prometheus ```

GitLab CI Runner Checklist

Verify the Fix

```bash # After fixing runner issues

# 1. Check runner is online curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/runners/<id>" | jq '.status' // Should be "online"

# 2. Verify job starts # Check pipeline in GitLab UI // Jobs should change from "pending" to "running"

# 3. Monitor job execution gitlab-runner logs // Shows job being processed

# 4. Check pipeline completes curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.com/api/v4/projects/<project-id>/pipelines/<pipeline-id>" | jq '.status' // Should be "success" or "failed" (not "pending")

# 5. Verify runner picks future jobs # Trigger new pipeline // Should be picked up immediately ```

Related Issues

• [Fix GitLab CI Runner Not Picking Jobs](/articles/fix-gitlab-ci-runner-not-picking-jobs)
• [Fix GitLab CI Pipeline Stuck](/articles/fix-gitlab-ci-pipeline-stuck)
• [Fix GitLab Runner Timeout](/articles/fix-gitlab-runner-timeout)