Introduction

During a GitLab upgrade, database migrations are run automatically to update the PostgreSQL schema. If a migration fails -- due to incompatible data, insufficient disk space, conflicting extensions, or a bug in the migration script -- GitLab will not start, leaving the instance in a broken state. This is a critical failure that requires careful investigation and manual intervention.

Symptoms

  • GitLab fails to start after package upgrade
  • gitlab-ctl reconfigure output shows database migration error
  • GitLab Rails console shows pending migrations
  • PostgreSQL logs show constraint violation or data type mismatch
  • Error message: StandardError: An error has occurred, this and all later migrations canceled

Common Causes

  • Database contains data that violates a new constraint added by the migration
  • Insufficient disk space for migration operations (creating new columns, indexes)
  • PostgreSQL extension version incompatible with the new GitLab version
  • Skipping multiple GitLab versions, missing intermediate migrations
  • Custom database modifications conflicting with standard migrations

Step-by-Step Fix

  1. 1.Check the migration error details: Identify the failing migration.
  2. 2.```bash
  3. 3.# Check reconfigure output
  4. 4.gitlab-ctl reconfigure 2>&1 | grep -i "migration|error"
  5. 5.# Check Rails logs
  6. 6.gitlab-ctl tail rails
  7. 7.# Check PostgreSQL logs
  8. 8.gitlab-ctl tail postgresql
  9. 9.`
  10. 10.Check database disk space: Ensure there is enough space for the migration.
  11. 11.```bash
  12. 12.df -h /var/opt/gitlab/postgresql
  13. 13.# Should have at least 20% free space
  14. 14.`
  15. 15.Run the migration manually with verbose output: Get detailed error information.
  16. 16.```bash
  17. 17.gitlab-rake db:migrate:status
  18. 18.# Find the migration that failed (shows "down" status)
  19. 19.gitlab-rake db:migrate VERSION=20240101000000 # Run up to the failing migration
  20. 20.`
  21. 21.Fix the underlying data issue: Resolve data conflicts before re-running.
  22. 22.```ruby
  23. 23.# In Rails console
  24. 24.gitlab-rails console
  25. 25.# Fix data that violates the new constraint
  26. 26.# Example: set null values for a new NOT NULL column
  27. 27.ActiveRecord::Base.connection.execute("UPDATE table_name SET column_name = '' WHERE column_name IS NULL")
  28. 28.`
  29. 29.Re-run the migration and restart GitLab: Complete the upgrade.
  30. 30.```bash
  31. 31.gitlab-rake db:migrate
  32. 32.gitlab-ctl reconfigure
  33. 33.gitlab-ctl restart
  34. 34.# Verify GitLab is running
  35. 35.gitlab-rake gitlab:check
  36. 36.`

Prevention

  • Always review GitLab upgrade documentation for database migration notes before upgrading
  • Back up the database before any upgrade: gitlab-backup create
  • Do not skip major GitLab versions -- follow the recommended upgrade path
  • Test upgrades in a staging environment that mirrors production data volume
  • Ensure adequate disk space (at least 2x current database size) before upgrading
  • Monitor PostgreSQL logs during upgrade for early warning of migration issues