How to Fix Grafana Provisioning Error

Introduction

Grafana provisioning allows you to configure dashboards, datasources, and alert rules using YAML or JSON files instead of the UI. This enables GitOps workflows and consistent configuration across environments. When provisioning fails, you may see errors in logs like "failed to provision dashboard" or "invalid configuration" without obvious UI indicators of what went wrong.

Symptoms

• Dashboards, datasources, or alerts defined in provisioning files don't appear in Grafana
• Error in logs: "provisioning failed" or "invalid provisioning configuration"
• Dashboards are loaded but datasource references are broken
• Changes to provisioning files are not reflected after restart
• Duplicate resources appear from conflicting provisioning files
• Alert rules show "provisioning error" status

Common Causes

• YAML/JSON syntax errors in provisioning files
• Incorrect file paths or directory structure
• Datasource UID mismatch between provisioned dashboards and datasources
• File permissions prevent Grafana from reading configuration
• Invalid or missing required fields in provisioning YAML
• Conflicting definitions between multiple provisioning files
• Provisioned dashboards reference deleted datasources

Step-by-Step Fix

Verify Provisioning Directory Structure

1. 1.Check provisioning directory exists and has correct structure:
2. 2.```bash
3. 3.ls -la /etc/grafana/provisioning/
4. 4.ls -la /etc/grafana/provisioning/dashboards/
5. 5.ls -la /etc/grafana/provisioning/datasources/
6. 6.ls -la /etc/grafana/provisioning/alerting/
7. 7.`
8. 8.Verify Grafana provisioning configuration:
9. 9.```ini
10. 10.# In grafana.ini
11. 11.[paths]
12. 12.provisioning = /etc/grafana/provisioning
13. 13.`
14. 14.Check file permissions:
15. 15.```bash
16. 16.chown -R grafana:grafana /etc/grafana/provisioning
17. 17.chmod -R 755 /etc/grafana/provisioning
18. 18.chmod 644 /etc/grafana/provisioning/datasources/*.yaml
19. 19.`

Datasource Provisioning Errors

1. 1.Validate datasource YAML syntax:
2. 2.```yaml
3. 3.# Example valid datasource provisioning
4. 4.apiVersion: 1

datasources: - name: Prometheus type: prometheus access: proxy url: http://prometheus:9090 uid: prometheus isDefault: true editable: false

deleteDatasources: - name: Old-Prometheus ```

1. 1.Test datasource configuration:
2. 2.```bash
3. 3.# Use Python to validate YAML
4. 4.python3 -c "import yaml; yaml.safe_load(open('/etc/grafana/provisioning/datasources/datasources.yaml'))"
5. 5.`
6. 6.Check for required fields in datasource definitions:
7. 7.- name - unique datasource name
8. 8.- type - valid datasource type (prometheus, mysql, etc.)
9. 9.- access - proxy or direct
10. 10.- url - valid endpoint URL
11. 11.- uid - unique identifier (required for dashboard references)

Dashboard Provisioning Errors

1. 1.Validate dashboard provisioning configuration:
2. 2.```yaml
3. 3.# Example valid dashboard provider
4. 4.apiVersion: 1

providers: - name: 'default' orgId: 1 folder: '' folderUid: '' type: file disableDeletion: false updateIntervalSeconds: 30 allowUiUpdates: true options: path: /var/lib/grafana/dashboards foldersFromFilesStructure: true ```

1. 1.Check dashboard JSON files are valid:
2. 2.```bash
3. 3.python3 -c "import json; json.load(open('/var/lib/grafana/dashboards/dashboard.json'))"
4. 4.`
5. 5.Verify dashboard datasource references:
6. 6.```bash
7. 7.# Check datasource UID in dashboard JSON matches provisioned datasource
8. 8.grep -r "datasource" /var/lib/grafana/dashboards/*.json | grep -i "uid"
9. 9.`
10. 10.Fix mismatched datasource UIDs:
11. 11.```json
12. 12.// In dashboard JSON, ensure datasource references match provisioned UID
13. 13."datasource": {
14. 14."type": "prometheus",
15. 15."uid": "prometheus" // Must match datasource provisioning UID
16. 16.}
17. 17.`

Alert Rule Provisioning Errors

1. 1.Validate alert rule YAML syntax:
2. 2.```yaml
3. 3.# Example valid alert rule provisioning
4. 4.apiVersion: 1

groups: - orgId: 1 name: system-alerts folder: alerts interval: 1m rules: - uid: cpu-high title: High CPU Usage condition: B data: - refId: A relativeTimeRange: from: 600 to: 0 datasourceUid: prometheus model: expr: rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8 refId: A - refId: B datasourceUid: prometheus model: type: threshold conditions: - evaluator: params: [0.8] type: gt operator: type: and query: params: [A] type: query noDataState: NoData executionErrorState: Alerting ```

1. 1.Check for required fields in alert rules:
2. 2.- uid - unique identifier for the rule
3. 3.- title - alert title
4. 4.- condition - refId of the condition expression
5. 5.- data - query and condition definitions
6. 6.- datasourceUid - must match provisioned datasource

Provisioning Conflict Resolution

1. 1.Check for duplicate definitions:
2. 2.```bash
3. 3.grep -r "name:" /etc/grafana/provisioning/datasources/*.yaml
4. 4.grep -r "uid:" /etc/grafana/provisioning/datasources/*.yaml
5. 5.`
6. 6.Remove conflicting definitions or use deleteDatasources:
7. 7.```yaml
8. 8.apiVersion: 1

deleteDatasources: - name: Old-Datasource orgId: 1 ```

Debug Provisioning Issues

1. 1.Enable provisioning debug logging:
2. 2.```ini
3. 3.# In grafana.ini
4. 4.[log]
5. 5.level = debug

[log.filters] provisioning = debug ```

1. 1.Check provisioning logs:
2. 2.```bash
3. 3.journalctl -u grafana-server | grep -i "provision|dashboard|datasource"
4. 4.`
5. 5.Force reload provisioning by restarting:
6. 6.```bash
7. 7.systemctl restart grafana-server
8. 8.`

Verification

1. 1.Verify datasources are loaded:
2. 2.```bash
3. 3.curl -s -u admin:password http://localhost:3000/api/datasources | jq '.[] | {name: .name, type: .type, uid: .uid}'
4. 4.`
5. 5.Verify dashboards are loaded:
6. 6.```bash
7. 7.curl -s -u admin:password http://localhost:3000/api/search?type=dash-db | jq '.[] | {title: .title, uid: .uid}'
8. 8.`
9. 9.Verify alert rules are loaded:
10. 10.```bash
11. 11.curl -s -u admin:password http://localhost:3000/api/v1/rules | jq '.[] | {name: .name}'
12. 12.`
13. 13.In Grafana UI:
14. 14.- Navigate to Configuration > Data sources - verify all provisioned datasources
15. 15.- Navigate to Dashboards - verify all provisioned dashboards
16. 16.- Navigate to Alerting > Alert rules - verify all provisioned alert rules