Introduction

Grafana API keys and service account tokens enable programmatic access to Grafana for automation, CI/CD integration, and external tooling. When API authentication fails, requests return 401 Unauthorized or 403 Forbidden errors. These issues typically stem from expired keys, insufficient permissions, or incorrect authentication header format. In Grafana 9+, service accounts replace the older API key system.

Symptoms

  • API request returns 401 Unauthorized
  • Error: "Invalid API key" or "Token expired"
  • API request returns 403 Forbidden despite correct key
  • Error: "Permission denied" for specific API endpoints
  • Service account cannot perform expected operations
  • API key creation fails with "Permission denied"
  • Key works for some endpoints but fails for others

Common Causes

  • API key has expired or was revoked
  • API key role doesn't have required permissions
  • Authentication header format is incorrect
  • Service account is disabled or locked
  • API key belongs to different organization
  • API endpoint requires Admin role but key has Viewer role
  • Rate limiting is blocking requests

Step-by-Step Fix

Check API Key Status

  1. 1.List existing API keys:
  2. 2.```bash
  3. 3.curl -s -u admin:password http://localhost:3000/api/auth/keys | jq .
  4. 4.`
  5. 5.For Grafana 9+ service accounts:
  6. 6.```bash
  7. 7.curl -s -u admin:password http://localhost:3000/api/serviceaccounts | jq .
  8. 8.curl -s -u admin:password http://localhost:3000/api/serviceaccounts/1/tokens | jq .
  9. 9.`
  10. 10.Check key expiration:
  11. 11.- API keys can have expiration dates
  12. 12.- Check if key has expired in the UI
  13. 13.- Navigate to Configuration > API Keys

Fix Authentication Header

  1. 1.Verify correct authentication header format:
  2. 2.```bash
  3. 3.# For API key (Bearer token)
  4. 4.curl -H "Authorization: Bearer eyJrIjoi..." http://localhost:3000/api/dashboards/home

# For Basic auth with API key curl -u api_key:eyJrIjoi... http://localhost:3000/api/dashboards/home

# Wrong format - missing "Bearer" curl -H "Authorization: eyJrIjoi..." http://localhost:3000/api/dashboards/home ```

  1. 1.Test API key authentication:
  2. 2.```bash
  3. 3.# Verify key works
  4. 4.curl -s -H "Authorization: Bearer YOUR_API_KEY" \
  5. 5.http://localhost:3000/api/user | jq .
  6. 6.`

Create New API Key

  1. 1.Create new API key via UI:
  2. 2.- Navigate to Configuration > API Keys
  3. 3.- Click "Add API key"
  4. 4.- Set name, role, and expiration
  5. 5.- Copy key immediately (shown only once)
  6. 6.Create API key via API:
  7. 7.```bash
  8. 8.curl -X POST -u admin:password \
  9. 9.-H "Content-Type: application/json" \
  10. 10.-d '{"name": "automation-key", "role": "Admin", "secondsToLive": 86400}' \
  11. 11.http://localhost:3000/api/auth/keys
  12. 12.`
  13. 13.Create service account token (Grafana 9+):
  14. 14.```bash
  15. 15.# Create service account
  16. 16.curl -X POST -u admin:password \
  17. 17.-H "Content-Type: application/json" \
  18. 18.-d '{"name": "automation", "role": "Admin"}' \
  19. 19.http://localhost:3000/api/serviceaccounts

# Create token for service account curl -X POST -u admin:password \ -H "Content-Type: application/json" \ -d '{"name": "automation-token"}' \ http://localhost:3000/api/serviceaccounts/1/tokens ```

Fix Permission Issues

  1. 1.Verify API key role:
  2. 2.```bash
  3. 3.curl -s -u admin:password http://localhost:3000/api/auth/keys | jq '.[] | {name: .name, role: .role}'
  4. 4.`
  5. 5.API key roles and their permissions:
  6. 6.- Viewer - Read access to dashboards and data sources
  7. 7.- Editor - Read/write access to dashboards (cannot delete)
  8. 8.- Admin - Full access including user/team management
  9. 9.Check if endpoint requires higher role:
  10. 10.```bash
  11. 11.# Admin-only endpoints:
  12. 12.# /api/admin/settings
  13. 13.# /api/users
  14. 14.# /api/orgs
  15. 15.# /api/datasources (POST/DELETE)

# Test with Admin key curl -s -H "Authorization: Bearer ADMIN_KEY" \ http://localhost:3000/api/admin/stats ```

  1. 1.Create key with appropriate role:
  2. 2.```bash
  3. 3.curl -X POST -u admin:password \
  4. 4.-H "Content-Type: application/json" \
  5. 5.-d '{"name": "admin-key", "role": "Admin"}' \
  6. 6.http://localhost:3000/api/auth/keys
  7. 7.`

Organization Issues

  1. 1.Check API key organization:
  2. 2.```bash
  3. 3.# API keys are scoped to organization
  4. 4.curl -s -u admin:password http://localhost:3000/api/auth/keys | jq '.[] | {orgId: .orgId}'
  5. 5.`
  6. 6.For multi-org access, use service accounts:
  7. 7.```bash
  8. 8.# Service accounts can be granted org access
  9. 9.curl -X POST -u admin:password \
  10. 10.-H "Content-Type: application/json" \
  11. 11.-d '{"orgId": 2}' \
  12. 12.http://localhost:3000/api/serviceaccounts/1/orgs
  13. 13.`
  14. 14.Verify request includes correct org header:
  15. 15.```bash
  16. 16.# Add X-Grafana-Org-Id header
  17. 17.curl -H "Authorization: Bearer TOKEN" \
  18. 18.-H "X-Grafana-Org-Id: 2" \
  19. 19.http://localhost:3000/api/dashboards/home
  20. 20.`

Rate Limiting Issues

  1. 1.Check for rate limiting:
  2. 2.```bash
  3. 3.# Look for 429 Too Many Requests
  4. 4.curl -v -H "Authorization: Bearer TOKEN" \
  5. 5.http://localhost:3000/api/dashboards/home
  6. 6.`
  7. 7.Configure rate limits:
  8. 8.```ini
  9. 9.# In grafana.ini
  10. 10.[security]
  11. 11.api_key_max_seconds_to_live = -1 # No expiration
  12. 12.auth_token_max_seconds_to_live = -1

[rate_limiting] enabled = true per_second = 100 burst = 200 ```

Token Revocation

  1. 1.Delete/revoke problematic key:
  2. 2.```bash
  3. 3.curl -X DELETE -u admin:password \
  4. 4.http://localhost:3000/api/auth/keys/1
  5. 5.`
  6. 6.Revoke service account token:
  7. 7.```bash
  8. 8.curl -X DELETE -u admin:password \
  9. 9.http://localhost:3000/api/serviceaccounts/1/tokens/1
  10. 10.`
  11. 11.Disable service account:
  12. 12.```bash
  13. 13.curl -X PATCH -u admin:password \
  14. 14.-H "Content-Type: application/json" \
  15. 15.-d '{"isDisabled": true}' \
  16. 16.http://localhost:3000/api/serviceaccounts/1
  17. 17.`

Debug API Issues

  1. 1.Enable API debug logging:
  2. 2.```ini
  3. 3.[log]
  4. 4.level = debug

[log.filters] api = debug auth = debug ```

  1. 1.Check authentication logs:
  2. 2.```bash
  3. 3.journalctl -u grafana-server | grep -i "auth|api.*key"
  4. 4.`

Verification

  1. 1.Test API key authentication:
  2. 2.```bash
  3. 3.curl -s -H "Authorization: Bearer YOUR_NEW_KEY" \
  4. 4.http://localhost:3000/api/user | jq .
  5. 5.`
  6. 6.Verify key has correct permissions:
  7. 7.```bash
  8. 8.# Test read operation
  9. 9.curl -s -H "Authorization: Bearer YOUR_KEY" \
  10. 10.http://localhost:3000/api/dashboards/home

# Test write operation (requires Editor/Admin) curl -X POST -H "Authorization: Bearer YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{"dashboard": {}, "overwrite": false}' \ http://localhost:3000/api/dashboards/db ```

  1. 1.Verify service account works:
  2. 2.```bash
  3. 3.curl -s -H "Authorization: Bearer SERVICE_ACCOUNT_TOKEN" \
  4. 4.http://localhost:3000/api/serviceaccounts/1
  5. 5.`