Introduction

Grafana template variables allow dashboard users to filter panels by selecting values from dropdown menus. When the variable query -- typically a PromQL label_values() or a database query -- returns no results, the dropdown appears empty. This breaks dashboard interactivity and can cause all panels that depend on the variable to show no data.

Symptoms

  • Dashboard template variable dropdown shows No options found or is completely empty
  • Panels using the variable display no data because the variable evaluates to empty
  • Variable query preview in the variable editor returns zero results
  • label_values() query returns empty when the metric or label does not exist
  • Error message: Template variable query returned no results

Common Causes

  • Metric name or label in the variable query changed after a service update
  • Datasource for the variable query is down or unreachable
  • Variable query uses incorrect PromQL syntax, e.g., label_values(metric) instead of label_values(metric, label)
  • Time range of the dashboard does not include data for the queried metric
  • Permission issue: the datasource API key does not have read access to the queried metrics

Step-by-Step Fix

  1. 1.Preview the variable query in the variable editor: Identify why it returns no results.
  2. 2.`
  3. 3.# In Grafana UI: Dashboard Settings > Variables > select variable > Preview of values
  4. 4.# Check the query being executed
  5. 5.`
  6. 6.Verify the metric and label exist in Prometheus: Test the query directly.
  7. 7.```bash
  8. 8.curl -s 'http://prometheus:9090/api/v1/label/job/values' | jq '.data'
  9. 9.# Or test label_values
  10. 10.curl -s 'http://prometheus:9090/api/v1/series?match[]={__name__=~".*"}' | jq '.data | length'
  11. 11.`
  12. 12.Fix the variable query syntax: Correct the PromQL expression.
  13. 13.```promql
  14. 14.# WRONG: missing label parameter
  15. 15.label_values(http_requests_total)

# CORRECT: specify metric and label label_values(http_requests_total, job)

# OR: get all values for a specific label across all metrics label_values({__name__=~".+"}, instance) ```

  1. 1.Set a fallback default value: Ensure the dashboard works even when the variable is empty.
  2. 2.`
  3. 3.# In variable settings:
  4. 4.# - Selection Options > Include All option: enabled
  5. 5.# - Custom all value: .*
  6. 6.# - Multi-value: enabled
  7. 7.`
  8. 8.Verify the datasource connection: Ensure the variable can query the datasource.
  9. 9.`
  10. 10.# In Grafana UI: Configuration > Data Sources > verify datasource is healthy
  11. 11.# Check datasource permissions if using API keys
  12. 12.`

Prevention

  • Use label_values() with both metric name and label parameters for reliable queries
  • Include an "All" option in template variables with a regex fallback (e.g., .*)
  • Test dashboard variables after any metric name or label changes in monitored services
  • Monitor datasource health and alert when variables start returning empty results
  • Document variable query requirements in dashboard creation guidelines
  • Use Grafana dashboard as code (JSON provisioning) to version control variable definitions