Introduction

Grafana's annotation API allows programmatic creation, update, and deletion of annotations on dashboards. API keys in Grafana have scoped permissions (Viewer, Editor, Admin). When an API key with insufficient scope is used to create or modify annotations, the API returns a 403 Forbidden error, blocking automated deployment pipelines and incident response tools that rely on annotations.

Symptoms

  • curl or API client receives HTTP/1.1 403 Forbidden when posting annotations
  • Grafana logs show Forbidden: insufficient permissions for annotation API calls
  • Deployment pipeline fails at the step that adds deployment annotations
  • API key works for reading dashboards but fails for writing annotations
  • Error message: {"message":"Forbidden","traceID":"00000000000000000000000000000000"}

Common Causes

  • API key created with Viewer or Editor scope instead of Admin
  • Grafana RBAC (Role-Based Access Control) restricting annotation write permissions
  • API key expired or revoked after rotation
  • Organization mismatch: API key belongs to a different Grafana organization
  • Grafana configuration disable_api_keys enabled in newer versions

Step-by-Step Fix

  1. 1.Verify the API key scope: Check the permissions of the current API key.
  2. 2.```bash
  3. 3.# Test the API key by reading dashboards (should work with any scope)
  4. 4.curl -s -H "Authorization: Bearer eyJrIjoi..."" http://grafana:3000/api/dashboards | jq '.[].title'
  5. 5.`
  6. 6.Create a new API key with Admin scope: Generate a key with sufficient permissions.
  7. 7.```bash
  8. 8.# Via Grafana UI: Configuration > API Keys > Add API Key
  9. 9.# Name: annotation-writer
  10. 10.# Role: Admin

# Or via API with an existing Admin key curl -X POST http://grafana:3000/api/auth/keys \ -H "Authorization: Bearer existing-admin-key" \ -H "Content-Type: application/json" \ -d '{"name": "annotation-writer", "role": "Admin", "secondsToLive": 0}' ```

  1. 1.Test the annotation API with the new key: Verify write access works.
  2. 2.```bash
  3. 3.curl -X POST http://grafana:3000/api/annotations \
  4. 4.-H "Authorization: Bearer new-admin-key" \
  5. 5.-H "Content-Type: application/json" \
  6. 6.-d '{"dashboardId": 1, "panelId": 1, "text": "Test annotation"}'
  7. 7.# Should return: {"id": 123, "message": "Annotation added"}
  8. 8.`
  9. 9.Update the deployment pipeline with the new API key: Replace the old key.
  10. 10.```bash
  11. 11.# In your CI/CD pipeline secret store
  12. 12.export GRAFANA_API_KEY="glsa_new_admin_key_here"
  13. 13.# Test deployment annotation
  14. 14.curl -X POST http://grafana:3000/api/annotations \
  15. 15.-H "Authorization: Bearer $GRAFANA_API_KEY" \
  16. 16.-H "Content-Type: application/json" \
  17. 17.-d '{"text": "Deployment v2.3.1", "tags": ["deployment"]}'
  18. 18.`
  19. 19.Verify RBAC permissions if using Grafana Enterprise: Check role assignments.
  20. 20.```bash
  21. 21.# Check if the API key's role has annotation:write permission
  22. 22.curl -s http://grafana:3000/api/access-control/user/permissions \
  23. 23.-H "Authorization: Bearer $GRAFANA_API_KEY"
  24. 24.`

Prevention

  • Document the required API key scope for each Grafana API endpoint in a runbook
  • Use the minimum required scope -- Admin for annotation writes, Editor for dashboard edits
  • Rotate API keys regularly and test all dependent integrations after rotation
  • Monitor Grafana API 403 response rates and alert on authentication failures
  • Store API keys in a secrets manager with rotation automation
  • Test annotation API access as part of Grafana deployment health checks