GitLab CI Pipeline Stuck

# GitLab CI Pipeline Stuck

Common Error Patterns

GitLab CI stuck pipelines show:

Job is pending and waiting for a runner

Runner is offline

Job stuck due to resource constraints

Pipeline blocked by manual action

The job exceeded the maximum execution time

Root Causes and Solutions

1. No Runner Available

No runner is available to pick up the job.

Solution:

Check runner status:

```bash # Via GitLab API curl -s --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/runners" | jq

# Via GitLab UI # Navigate to Settings > CI/CD > Runners ```

For shared runners (GitLab.com):

# Specify runner tags
job:
  tags:
    - docker
    - linux
  script: ./build.sh

For self-hosted runners:

```bash # Check runner service sudo gitlab-runner status

# Restart runner sudo gitlab-runner restart

# Verify runner is registered sudo gitlab-runner list ```

Register new runner:

sudo gitlab-runner register \
  --non-interactive \
  --url https://gitlab.com \
  --registration-token $REGISTRATION_TOKEN \
  --executor docker \
  --docker-image alpine:latest \
  --description "my-runner" \
  --tag-list "docker,linux"

2. Runner Offline

The runner is registered but offline.

Solution:

Check runner connectivity:

```bash # Test GitLab API access curl -s https://gitlab.com/api/v4/version

# Check runner logs sudo journalctl -u gitlab-runner -f

# Verify runner configuration cat /etc/gitlab-runner/config.toml ```

Common issues: - Runner service stopped - Network connectivity issues - GitLab server unreachable - Runner token expired

```bash # Renew runner token sudo gitlab-runner verify

# If verification fails, re-register sudo gitlab-runner unregister --all-runners sudo gitlab-runner register ```

3. Job Dependency Blocking

Job waiting for dependent job to complete.

Solution:

Check job dependencies:

```yaml # .gitlab-ci.yml stages: - build - test - deploy

build: stage: build script: make build

test: stage: test needs: [build] # Explicit dependency script: make test

deploy: stage: deploy needs: [test] # Wait for test script: make deploy ```

View pipeline graph:

1. 1.Navigate to CI/CD > Pipelines
2. 2.Click pipeline ID
3. 3.View dependency graph

4. Manual Job Blocking

Pipeline waiting for manual action.

Solution:

Check for manual jobs:

deploy_production:
  stage: deploy
  when: manual  # Requires manual trigger
  script: ./deploy.sh production

Trigger manual job:

```bash # Via GitLab CLI (glab) glab api projects/:id/pipelines/:pipeline_id/jobs \ --paginate | jq '.[] | select(.status=="manual") | .id'

# Trigger manual job glab api projects/:id/jobs/:job_id/play -X POST ```

1. 1.Or in GitLab UI:
2. 2.Navigate to pipeline
3. 3.Find manual job (has play button)
4. 4.Click to trigger

5. Resource Constraints

Job waiting for resources (concurrency limit).

Solution:

Check runner concurrency:

```bash # View runner config cat /etc/gitlab-runner/config.toml

# Check concurrent setting grep concurrent /etc/gitlab-runner/config.toml ```

Increase concurrency:

```toml # /etc/gitlab-runner/config.toml concurrent = 10 # Increase from default

[[runners]] limit = 5 # Limit per runner ```

Restart runner after changes:

sudo gitlab-runner restart

6. Job Timeout

Job exceeded maximum execution time.

Solution:

Set appropriate timeout:

job:
  timeout: 2h  # Job-specific timeout
  script: ./long-build.sh

Or globally:

# .gitlab-ci.yml
default:
  timeout: 1h

For runner timeout:

# /etc/gitlab-runner/config.toml
[[runners]]
  timeout = 3600  # Seconds

7. Artifact Blocking

Job waiting for artifacts from previous job.

Solution:

Configure artifact passing:

```yaml build: stage: build script: make build artifacts: paths: - build/ expire_in: 1 week

test: stage: test needs: - job: build artifacts: true # Explicitly request artifacts script: make test ```

Or disable artifact dependency:

test:
  needs:
    - job: build
      artifacts: false  # Don't wait for artifacts

8. Environment Lock

Job waiting for protected environment lock.

Solution:

Check environment configuration:

1. 1.Navigate to Settings > CI/CD > Protected environments
2. 2.View lock status

deploy_production:
  stage: deploy
  environment:
    name: production
    url: https://production.example.com
  script: ./deploy.sh

Unlock environment:

# Via API
curl -X DELETE --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/projects/:id/environments/:environment_id/stop"

Debugging Commands

```bash # List stuck pipelines glab api projects/:id/pipelines?status=pending | jq

# Check specific pipeline glab api projects/:id/pipelines/:pipeline_id | jq

# List jobs in pipeline glab api projects/:id/pipelines/:pipeline_id/jobs | jq

# Get job log glab api projects/:id/jobs/:job_id/trace

# Cancel stuck job glab api projects/:id/jobs/:job_id/cancel -X POST ```

Runner Management

```bash # Check runner status sudo gitlab-runner verify

# View runner details sudo gitlab-runner list

# Clear runner cache sudo gitlab-runner clear-cache

# Prune old runners sudo gitlab-runner prune

# Check runner health sudo gitlab-runner health-check ```

Pipeline Troubleshooting

Cancel Stuck Pipeline

```bash # Via API curl -X POST --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/projects/:id/pipelines/:pipeline_id/cancel"

# Via glab glab pipeline cancel <pipeline-id> ```

Retry Failed Job

```bash # Via API curl -X POST --header "PRIVATE-TOKEN: $TOKEN" \ "https://gitlab.com/api/v4/projects/:id/jobs/:job_id/retry"

# Via glab glab job retry <job-id> ```

View Job Details

glab api projects/:id/jobs/:job_id | jq '{status,stage,name,runner}'

Quick Reference Table

Prevention Tips

1. 1.Monitor runner health with alerts
2. 2.Set appropriate timeouts for each job
3. 3.Use auto-cancel for redundant pipelines
4. 4.Configure proper concurrency limits
5. 5.Use DAG pipelines with explicit needs
6. 6.Set up runner autoscaling for peak demand

Related Articles

• [GitHub Actions Workflow Failed](#)
• [CircleCI Job Timeout](#)
• [Docker Build Failed in CI](#)