Introduction

Upgrading Grafana introduces new features and security fixes, but the process can fail due to database migration errors, plugin incompatibilities, configuration changes, or breaking changes in major version upgrades. Understanding how to diagnose and fix upgrade failures ensures you can safely move to newer Grafana versions while preserving your dashboards and data.

Symptoms

  • Grafana fails to start after upgrade with database migration error
  • Error: "Migration failed" or "Database schema version mismatch"
  • Dashboards or plugins are missing after upgrade
  • Error: "Plugin incompatible with Grafana version"
  • Grafana UI shows errors or blank pages after upgrade
  • Login fails with "Invalid session" or authentication errors
  • Configuration settings are reset or missing

Common Causes

  • Database migration scripts fail due to existing data conflicts
  • Installed plugins are incompatible with new Grafana version
  • Configuration file format changed between versions
  • Breaking changes in major version upgrades (e.g., 8 to 9)
  • Insufficient disk space for database migration
  • Permissions changed on Grafana directories
  • SQLite database locked during migration

Step-by-Step Fix

Pre-Upgrade Preparation

  1. 1.Always backup before upgrading:
  2. 2.```bash
  3. 3.# Backup SQLite database
  4. 4.cp /var/lib/grafana/grafana.db /backup/grafana.db.backup

# Backup configuration cp /etc/grafana/grafana.ini /backup/grafana.ini.backup

# Backup dashboards tar -czf /backup/dashboards.tar.gz /var/lib/grafana/dashboards

# For PostgreSQL/MySQL, backup database pg_dump grafana > /backup/grafana.sql mysqldump grafana > /backup/grafana.sql ```

  1. 1.Check current Grafana version:
  2. 2.```bash
  3. 3.grafana-cli --version
  4. 4.`
  5. 5.List installed plugins:
  6. 6.```bash
  7. 7.grafana-cli plugins ls
  8. 8.`
  9. 9.Check plugin compatibility:
  10. 10.- Visit https://grafana.com/plugins for version requirements
  11. 11.- Note plugins that may need updating

Database Migration Issues

  1. 1.Check migration log for errors:
  2. 2.```bash
  3. 3.journalctl -u grafana-server | grep -i "migration|schema"
  4. 4.# Or check migration log table
  5. 5.sqlite3 /var/lib/grafana/grafana.db "SELECT * FROM migration_log ORDER BY id DESC LIMIT 20;"
  6. 6.`
  7. 7.For migration failure, check error details:
  8. 8.```bash
  9. 9.# Common migration errors
  10. 10.# - "Duplicate column name"
  11. 11.# - "Foreign key constraint"
  12. 12.# - "Table already exists"
  13. 13.`
  14. 14.If migration is incomplete, restore backup and retry:
  15. 15.```bash
  16. 16.systemctl stop grafana-server
  17. 17.cp /backup/grafana.db.backup /var/lib/grafana/grafana.db
  18. 18.chown grafana:grafana /var/lib/grafana/grafana.db
  19. 19.systemctl start grafana-server
  20. 20.`
  21. 21.For manual migration fix:
  22. 22.```bash
  23. 23.grafana-cli admin data-migration
  24. 24.`
  25. 25.Check database schema version:
  26. 26.```bash
  27. 27.sqlite3 /var/lib/grafana/grafana.db "SELECT * FROM migration_log WHERE success = 1 ORDER BY id DESC LIMIT 1;"
  28. 28.`

Plugin Compatibility Issues

  1. 1.Update plugins before Grafana upgrade:
  2. 2.```bash
  3. 3.grafana-cli plugins update-all
  4. 4.`
  5. 5.Check for incompatible plugins after upgrade:
  6. 6.```bash
  7. 7.grafana-cli plugins ls | grep -i "error|incompatible"
  8. 8.journalctl -u grafana-server | grep -i "plugin.*error"
  9. 9.`
  10. 10.Remove incompatible plugins:
  11. 11.```bash
  12. 12.grafana-cli plugins uninstall incompatible-plugin-name
  13. 13.`
  14. 14.Install compatible plugin version:
  15. 15.```bash
  16. 16.# Check available versions
  17. 17.grafana-cli plugins list-versions plugin-name

# Install specific compatible version grafana-cli plugins install plugin-name compatible-version ```

  1. 1.For unsigned plugin issues:
  2. 2.```ini
  3. 3.# In grafana.ini
  4. 4.[plugins]
  5. 5.allow_loading_unsigned_plugins = plugin-id
  6. 6.`

Configuration Changes

  1. 1.Check for deprecated configuration options:
  2. 2.```bash
  3. 3.# Compare old and new default config
  4. 4.diff /backup/grafana.ini.backup /etc/grafana/grafana.ini
  5. 5.`
  6. 6.Review breaking changes for major versions:

Grafana 8 to 9 changes: - Unified alerting replaces legacy alerting - API key → Service accounts - New provisioning format

Grafana 9 to 10 changes: - Azure AD OAuth configuration changes - Prometheus query editor updates

  1. 1.Update deprecated settings:
  2. 2.```ini
  3. 3.# Old (Grafana 7)
  4. 4.[alerting]
  5. 5.enabled = true

# New (Grafana 9+) [unified_alerting] enabled = true ```

  1. 1.Check for new required settings:
  2. 2.```ini
  3. 3.# Grafana 9+ service accounts
  4. 4.[service_accounts]
  5. 5.enabled = true
  6. 6.`

Session and Authentication Issues

  1. 1.Clear old sessions after upgrade:
  2. 2.```bash
  3. 3.sqlite3 /var/lib/grafana/grafana.db "DELETE FROM user_session;"
  4. 4.`
  5. 5.For PostgreSQL:
  6. 6.```sql
  7. 7.TRUNCATE TABLE user_session;
  8. 8.`
  9. 9.Reset auth tokens if login fails:
  10. 10.```bash
  11. 11.grafana-cli admin reset-admin-password newpassword
  12. 12.`

Rollback Procedure

  1. 1.If upgrade fails completely, rollback to previous version:
  2. 2.```bash
  3. 3.# Stop Grafana
  4. 4.systemctl stop grafana-server

# Restore database cp /backup/grafana.db.backup /var/lib/grafana/grafana.db chown grafana:grafana /var/lib/grafana/grafana.db

# Install previous version apt-get install grafana=previous-version # Or for Docker docker run grafana/grafana:previous-version

# Start Grafana systemctl start grafana-server ```

  1. 1.For Docker rollback:
  2. 2.```bash
  3. 3.docker stop grafana
  4. 4.docker rm grafana
  5. 5.docker run -d --name grafana \
  6. 6.-v grafana-storage:/var/lib/grafana \
  7. 7.grafana/grafana:previous-version
  8. 8.`

Post-Upgrade Verification

  1. 1.Verify Grafana health:
  2. 2.```bash
  3. 3.curl -s http://localhost:3000/api/health | jq .
  4. 4.grafana-cli --version
  5. 5.`
  6. 6.Verify dashboards intact:
  7. 7.```bash
  8. 8.curl -s -u admin:password http://localhost:3000/api/search?type=dash-db | jq 'length'
  9. 9.`
  10. 10.Verify datasources:
  11. 11.```bash
  12. 12.curl -s -u admin:password http://localhost:3000/api/datasources | jq '.[] | .name'
  13. 13.`
  14. 14.Verify alert rules:
  15. 15.```bash
  16. 16.curl -s -u admin:password http://localhost:3000/api/v1/rules | jq '.[] | .name'
  17. 17.`

Upgrade Best Practices

  1. 1.Test upgrade in staging first:
  2. 2.- Clone production database
  3. 3.- Run upgrade on staging environment
  4. 4.- Verify all functionality
  5. 5.- Document any required changes
  6. 6.Plan upgrade path for major versions:
  7. 7.```bash
  8. 8.# For major version jumps, upgrade incrementally
  9. 9.# 7.x -> 8.x -> 9.x -> 10.x
  10. 10.`
  11. 11.Enable upgrade logging:
  12. 12.```ini
  13. 13.[log]
  14. 14.level = debug

[log.filters] migration = debug ```

Verification

  • Verify Grafana starts successfully
  • Test login functionality
  • Open critical dashboards and verify data displays
  • Test datasource connectivity
  • Verify alert rules are functional
  • Check plugins are loaded correctly
  • Run health check API