Introduction

Grafana panels execute queries against configured datasources (Prometheus, Elasticsearch, PostgreSQL, etc.). When a datasource query takes longer than the configured timeout, Grafana aborts the request and displays an error on the panel. This is common with complex PromQL queries, large time ranges, or datasources under heavy load.

Symptoms

  • Panel displays Panel data not available or Query timed out error
  • Grafana server logs show context deadline exceeded for datasource queries
  • Specific panels fail while others on the same dashboard load successfully
  • Query inspector shows the query duration exceeding the timeout threshold
  • Error message: Error: request timed out: Get "http://prometheus:9090/api/v1/query_range": context deadline exceeded

Common Causes

  • Query time range too broad (e.g., 30 days of high-cardity metrics)
  • Datasource under heavy load from concurrent queries
  • Complex PromQL with multiple subqueries and aggregations
  • Grafana datasource timeout set too low for the query complexity
  • Network latency between Grafana and the datasource server

Step-by-Step Fix

  1. 1.Identify the slow query using the Query Inspector: Check the exact query and its duration.
  2. 2.`
  3. 3.# In Grafana UI: Click panel title > Inspect > Query
  4. 4.# Look at the "Query" tab for execution time
  5. 5.`
  6. 6.Increase the datasource query timeout: Allow more time for complex queries.
  7. 7.`
  8. 8.# Grafana UI: Configuration > Data Sources > select datasource
  9. 9.# Set Timeout to 60s (default is 30s)
  10. 10.`
  11. 11.Optimize the PromQL query: Reduce query complexity and data volume.
  12. 12.```promql
  13. 13.# BEFORE: Querying all series over 30 days
  14. 14.rate(http_requests_total[5m])

# AFTER: Aggregate by job and limit range sum by (job) (rate(http_requests_total{status=~"2.."}[5m])) ```

  1. 1.Reduce the default dashboard time range: Limit the data queried by default.
  2. 2.`
  3. 3.# Set dashboard time range to last 6 hours instead of last 30 days
  4. 4.# Use relative time overrides per panel for heavy panels
  5. 5.`
  6. 6.Configure query caching for repeated queries: Reduce load on the datasource.
  7. 7.```ini
  8. 8.# grafana.ini
  9. 9.[caching]
  10. 10.enabled = true
  11. 11.TTL = 60
  12. 12.`

Prevention

  • Set appropriate datasource timeouts based on the p99 query duration for each datasource type
  • Use dashboard variables with limited cardinality to reduce query scope
  • Implement query optimization guidelines for teams creating Grafana dashboards
  • Monitor datasource query latency and alert when p99 exceeds the timeout threshold
  • Use Grafana's built-in query caching for dashboards that do not require real-time data
  • Split heavy dashboards into multiple focused dashboards to reduce per-query complexity