Introduction

Annotations in Grafana mark significant events on dashboards, such as deployments, incidents, or custom notes. They help correlate metrics with events for better troubleshooting. Annotation errors can prevent events from displaying on graphs, cause API failures when creating annotations, or result in incorrect time ranges for event markers.

Symptoms

  • Annotations don't appear on dashboard panels despite being created
  • Error: "Failed to query annotations" in dashboard
  • Annotation API returns 400 or 500 error when creating events
  • Annotations show wrong timestamps or time zone issues
  • Annotations created via API don't sync with dashboard
  • Query-based annotations fail to load from datasource
  • Annotation colors and tags don't display correctly

Common Causes

  • Annotation query returns data in incorrect format
  • Annotation datasource is unreachable or misconfigured
  • API annotation creation has invalid payload format
  • Time zone mismatch between annotation and dashboard
  • Annotation query time range doesn't include events
  • Annotation tags are not configured correctly
  • Dashboard annotations are disabled or filtered out

Step-by-Step Fix

Annotation Query Issues

  1. 1.Test annotation query in datasource:
  2. 2.```bash
  3. 3.# For Prometheus-based annotations
  4. 4.curl -s "http://prometheus:9090/api/v1/query?query=deploy_events"

# For Elasticsearch-based annotations curl -s "http://elasticsearch:9200/events/_search?q=type:deploy" ```

  1. 1.Check annotation configuration in dashboard:
  2. 2.- Navigate to Dashboard settings > Annotations
  3. 3.- Verify annotation query settings
  4. 4.- Test annotation query manually
  5. 5.Fix query format for annotation datasource:
  6. 6.```yaml
  7. 7.# Annotation query must return:
  8. 8.# - time (timestamp in milliseconds or ISO format)
  9. 9.# - title (annotation text)
  10. 10.# - tags (optional, comma-separated)
  11. 11.# - text (optional, description)
  12. 12.`
  13. 13.For Prometheus annotations:
  14. 14.```promql
  15. 15.# Query must return a metric with timestamp
  16. 16.timestamp(deploy_time{event="deployment"})
  17. 17.`
  18. 18.For Elasticsearch annotations:
  19. 19.```json
  20. 20.{
  21. 21."query": {
  22. 22."match": { "type": "deployment" }
  23. 23.}
  24. 24.}
  25. 25.`

Annotation API Issues

  1. 1.Test annotation API creation:
  2. 2.```bash
  3. 3.curl -X POST -u admin:password \
  4. 4.-H "Content-Type: application/json" \
  5. 5.-d '{
  6. 6."time": 1609459200000,
  7. 7."title": "Deployment",
  8. 8."tags": ["deploy", "production"],
  9. 9."text": "Version 1.2.3 deployed"
  10. 10.}' \
  11. 11.http://localhost:3000/api/annotations
  12. 12.`
  13. 13.Check required fields in annotation payload:
  14. 14.- time - Unix timestamp in milliseconds (required)
  15. 15.- title - Annotation title (required)
  16. 16.- tags - Array of tag strings (optional)
  17. 17.- text - Description text (optional)
  18. 18.- dashboardId - Target dashboard ID (optional)
  19. 19.- panelId - Target panel ID (optional)
  20. 20.Fix time format issues:
  21. 21.```bash
  22. 22.# Correct: milliseconds since epoch
  23. 23.time: 1609459200000

# Convert ISO time to milliseconds date -d "2021-01-01T00:00:00" +%s | awk '{print $1*1000}' ```

  1. 1.Create annotation for specific dashboard:
  2. 2.```bash
  3. 3.# Get dashboard ID
  4. 4.curl -s -u admin:password http://localhost:3000/api/dashboards/uid/my-dashboard | jq '.dashboard.id'

# Create annotation for dashboard curl -X POST -u admin:password \ -H "Content-Type: application/json" \ -d '{ "dashboardId": 1, "time": 1609459200000, "title": "Event", "tags": ["alert"] }' \ http://localhost:3000/api/annotations ```

Annotation Display Issues

  1. 1.Check if annotations are enabled for dashboard:
  2. 2.- Navigate to Dashboard settings > Annotations
  3. 3.- Verify annotation queries are checked/enabled
  4. 4.- Check "Built-in annotations" are enabled
  5. 5.Verify annotation query time range:
  6. 6.- Annotation queries use dashboard time range
  7. 7.- If event occurred outside time range, it won't show
  8. 8.- Expand dashboard time range to include annotation time
  9. 9.Check annotation tags filter:
  10. 10.- Click annotation icon on panel
  11. 11.- Check if tags are being filtered
  12. 12.- Enable all tags to see all annotations
  13. 13.Fix annotation color settings:
  14. 14.```yaml
  15. 15.# In annotation query settings
  16. 16.Specify color: #FF0000 # or use tag-based colors
  17. 17.`

Time Zone Issues

  1. 1.Check Grafana time zone settings:
  2. 2.```ini
  3. 3.# In grafana.ini
  4. 4.[date_formats]
  5. 5.default_timezone = browser

# Or set to UTC default_timezone = utc ```

  1. 1.Verify annotation timestamp interpretation:
  2. 2.- Annotations use UTC timestamps
  3. 3.- Dashboard may display in local time
  4. 4.- Check time zone offset for correct display
  5. 5.Fix time zone in annotation creation:
  6. 6.```bash
  7. 7.# Use UTC timestamp
  8. 8.TZ=UTC date -d "2021-01-01T00:00:00" +%s000
  9. 9.`

Annotation Synchronization

  1. 1.Check annotation provisioning:
  2. 2.```yaml
  3. 3.# Annotations in dashboard JSON
  4. 4."annotations": {
  5. 5."list": [
  6. 6.{
  7. 7."datasource": "Prometheus",
  8. 8."enable": true,
  9. 9."expr": "deploy_events",
  10. 10."iconColor": "blue",
  11. 11."name": "Deployments"
  12. 12.}
  13. 13.]
  14. 14.}
  15. 15.`
  16. 16.Verify annotation datasource UID:
  17. 17.```bash
  18. 18.# Check datasource UID matches annotation config
  19. 19.jq '.annotations.list[].datasource' dashboard.json
  20. 20.curl -s -u admin:password http://localhost:3000/api/datasources | jq '.[] | {name: .name, uid: .uid}'
  21. 21.`
  22. 22.Delete problematic annotations:
  23. 23.```bash
  24. 24.# List annotations
  25. 25.curl -s -u admin:password "http://localhost:3000/api/annotations?from=now-24h&to=now"

# Delete specific annotation curl -X DELETE -u admin:password http://localhost:3000/api/annotations/123 ```

Annotation Query Errors

  1. 1.Enable annotation debug logging:
  2. 2.```ini
  3. 3.# In grafana.ini
  4. 4.[log]
  5. 5.level = debug

[log.filters] annotation = debug ```

  1. 1.Check annotation query errors in logs:
  2. 2.```bash
  3. 3.journalctl -u grafana-server | grep -i annotation
  4. 4.`
  5. 5.Test annotation query directly:
  6. 6.```bash
  7. 7.curl -s -u admin:password \
  8. 8."http://localhost:3000/api/annotations?dashboardId=1&from=now-24h&to=now"
  9. 9.`

Verification

  1. 1.Verify annotations appear on dashboard:
  2. 2.- Open dashboard with annotations
  3. 3.- Check annotation markers on time series panels
  4. 4.- Hover over annotations to see title and text
  5. 5.Verify annotation query returns data:
  6. 6.```bash
  7. 7.curl -s -u admin:password \
  8. 8."http://localhost:3000/api/annotations?from=$(date -d '1 day ago' +%s)000&to=$(date +%s)000"
  9. 9.`
  10. 10.Test annotation creation via API:
  11. 11.- Create test annotation
  12. 12.- Verify annotation appears on dashboard
  13. 13.- Check annotation details in UI
  14. 14.Verify annotation filtering:
  15. 15.- Click annotation filter icon
  16. 16.- Filter by specific tags
  17. 17.- Confirm only matching annotations display