GitLab CI Pipeline Stuck: Complete Troubleshooting Guide

# GitLab CI Pipeline Stuck: Complete Troubleshooting Guide

Your GitLab pipeline shows "pending" or "stuck" and jobs aren't starting. The little spinner keeps spinning, but nothing happens. This is one of the most frustrating CI/CD situations because it's not always clear what's blocking your jobs.

Let me walk through every reason a GitLab CI pipeline gets stuck and how to fix each one.

Understanding Pipeline States

First, understand what "stuck" actually means:

If jobs show "Pending" for a long time, something is preventing runners from picking them up.

Fix 1: No Available Runners

The most common cause—no runners are available or online.

Diagnosis:

Check runner status:

1. 1.Go to Settings → CI/CD → Runners
2. 2.Look for green circles (online) vs gray/red (offline)
3. 3.Check if runners are shared or specific to your project

Or via API:

curl --header "PRIVATE-TOKEN: your_token" \
  "https://gitlab.com/api/v4/projects/PROJECT_ID/runners"

Solution:

If no runners are online:

1. 1.For shared runners: Check if shared runners are enabled for your project:
2. 2.- Go to Settings → CI/CD → Runners
3. 3.- Enable "Enable shared runners"
4. 4.For specific runners: Start the runner:

```bash # On the runner machine sudo gitlab-runner start

# Check status sudo gitlab-runner status

# Verify registration sudo gitlab-runner verify ```

If the runner won't start:

```bash # Check runner logs sudo journalctl -u gitlab-runner -f

# Common fix: re-register the runner sudo gitlab-runner unregister --all-runners sudo gitlab-runner register --url https://gitlab.com --registration-token YOUR_TOKEN ```

Fix 2: Runner Tags Mismatch

Your job requires tags that no runner has:

job_name:
  tags:
    - docker
    - production
  script:
    - echo "Deploying"

If no runner has both docker AND production tags, the job stays stuck.

Diagnosis:

# List all runners and their tags
curl --header "PRIVATE-TOKEN: your_token" \
  "https://gitlab.com/api/v4/projects/PROJECT_ID/runners" | jq '.[] | {id, tag_list}'

Solution:

Either add tags to an existing runner:

```bash # On runner machine, edit config sudo nano /etc/gitlab-runner/config.toml

# Add tags [[runners]] name = "my-runner" tags = "docker,production,linux"

# Restart runner sudo gitlab-runner restart ```

Or remove/modify tags in your .gitlab-ci.yml:

job_name:
  tags:
    - docker  # Remove the 'production' tag if not needed
  script:
    - echo "Deploying"

Fix 3: Runner Quota Exceeded

Shared runners have minute limits on free tiers. When exceeded, jobs queue indefinitely.

Diagnosis:

1. 1.Check your CI/CD minutes quota:
2. 2.- Go to Settings → CI/CD → CI/CD Minutes
3. 3.- View remaining quota
4. 4.Look at pipeline details for "This job is queued and will be delayed" message

Solution:

If quota is exceeded:

1. 1.Purchase additional CI minutes (GitLab.com)
2. 2.Set up your own runners (no minute limits)
3. 3.Reduce CI usage:
4. 4.- Only run pipelines on relevant changes
5. 5.- Use only/except rules
6. 6.- Cancel unnecessary pipelines

# Only run on main branch changes
job_name:
  only:
    - main
  script:
    - echo "Building"

Fix 4: Resource Group Locking

GitLab has a feature called resource groups that locks jobs when another job holds the resource:

deploy_production:
  resource_group: production
  script:
    - ./deploy.sh

If a previous deployment is stuck or running, new jobs wait for the lock.

Diagnosis:

1. 1.Go to CI/CD → Pipelines
2. 2.Click on stuck job
3. 3.Look for "This job is waiting for a resource group lock"

Solution:

Cancel or finish the job holding the lock:

1. 1.Go to the running/previous pipeline
2. 2.Cancel the job holding the resource group
3. 3.Your queued job will start

Or remove the resource group constraint:

deploy_production:
  # resource_group: production  # Remove or comment out
  script:
    - ./deploy.sh

Fix 5: Runner Concurrency Limits

The runner has reached its concurrent job limit.

Diagnosis:

```bash # On runner machine, check current jobs sudo gitlab-runner list

# Check config for limit cat /etc/gitlab-runner/config.toml | grep concurrent ```

Solution:

Increase concurrency limit:

```bash # Edit runner config sudo nano /etc/gitlab-runner/config.toml

# Increase concurrent jobs concurrent = 10 # Default is often 1-4 ```

Then restart:

sudo gitlab-runner restart

Fix 6: Docker Image Pull Issues

Jobs stuck at "Preparing environment" often mean Docker image pull failures:

job_name:
  image: private-registry.com/my-image:latest
  script:
    - echo "Running"

Diagnosis:

Check job logs for: - ERROR: Job failed (system failure): failed to pull image - image pull policy: if-not-present - Network timeouts

Solution:

For private registries, add credentials:

job_name:
  image: private-registry.com/my-image:latest
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD private-registry.com
  script:
    - echo "Running"

Or use GitLab's built-in registry:

job_name:
  image: $CI_REGISTRY_IMAGE:latest
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - echo "Running"

Fix 7: Cache/Artifact Expiration

Jobs waiting for artifacts from previous stages might get stuck if artifacts expired:

Downloading artifacts for job_name (id=123)...
ERROR: Downloading artifacts from coordinator... 404

Diagnosis:

Check artifact settings in .gitlab-ci.yml:

artifacts:
  expire_in: 1 week  # Artifacts expire after 1 week

Solution:

Increase artifact expiration:

artifacts:
  expire_in: 1 month
  # Or never expire
  # expire_in: never

For stuck jobs, manually re-run the job that produces the artifacts.

Fix 8: Environment Protection Rules

Deployments to protected environments require approvals:

deploy_production:
  environment:
    name: production
    url: https://production.example.com
  script:
    - ./deploy.sh

Diagnosis:

1. 1.Go to Settings → CI/CD → Protected environments
2. 2.Check if environment requires approval
3. 3.Look for "Waiting for approval" in job status

Solution:

Approve the deployment:

1. 1.Go to CI/CD → Pipelines → [Pipeline] → [Environment]
2. 2.Click "Approve" or add required approvers

Or modify protection rules:

1. 1.Go to Settings → CI/CD → Protected environments
2. 2.Adjust "Allowed to deploy" settings

Fix 9: Long-Running Scripts Without Output

Scripts that produce no output for too long can timeout or appear stuck:

job_name:
  script:
    - ./long-process.sh  # No output for 5+ minutes

Solution:

Add output or increase timeout:

job_name:
  timeout: 2h  # Increase timeout
  script:
    - echo "Starting long process..."
    - ./long-process.sh
    - echo "Process complete"

Or modify script to output progress:

#!/bin/bash
# long-process.sh
while true; do
  echo "Still working..."
  sleep 60
done

Fix 10: Git LFS Issues

If your repository uses Git LFS, jobs can get stuck during checkout:

Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/project/.git/

Diagnosis:

Check if LFS files are causing issues:

variables:
  GIT_LFS_SKIP_SMUDGE: 1  # Skip LFS during checkout

Solution:

Fetch LFS files explicitly:

```yaml variables: GIT_LFS_SKIP_SMUDGE: 1

before_script: - git lfs pull # Fetch LFS files after checkout ```

Or ensure runners have Git LFS installed:

# On runner machine
sudo apt-get install git-lfs
git lfs install

Quick Reference: Stuck Pipeline Checklist

Debugging Commands

Run these on the GitLab runner machine:

```bash # Check runner status sudo gitlab-runner status

# View runner logs sudo journalctl -u gitlab-runner -f

# List registered runners sudo gitlab-runner list

# Verify runner connection sudo gitlab-runner verify

# Check runner config cat /etc/gitlab-runner/config.toml

# Run a test job sudo gitlab-runner run --debug ```