What's Actually Happening
Grafana dashboard variables do not populate with values, or variable selection does not update panels.
The Error You'll See
Variable dropdown empty:
Variable: server
No options foundQuery error:
Variable Query Error
Failed to evaluate queryPanel not updating:
# Selecting variable value doesn't update panel dataTemplate error:
Templating
Failed to upgrade legacy queriesWhy This Happens
- 1.Data source error - Data source query fails
- 2.Query syntax wrong - Variable query incorrect
- 3.Variable name mismatch - Wrong variable name in panel
- 4.Permission issue - User lacks query permission
- 5.Data source not configured - Missing or wrong data source
- 6.Variable dependency - Dependent variable not resolving
Step 1: Check Variable Configuration
```bash # View variable settings: # Dashboard -> Settings -> Variables
# Check variable definition: # Name: server # Type: Query # Data source: Prometheus # Query: label_values(up, instance)
# Test variable query: # In variable edit, click "Preview values"
# Check variable types: # Query - Values from data source query # Custom - Manually defined values # Constant - Fixed value # Text box - User input # Interval - Time intervals # Data source - Select data source
# Test query in Explore: # Grafana -> Explore -> Run same query label_values(up, instance) ```
Step 2: Fix Prometheus Variable Queries
```bash # Common Prometheus variable queries:
# Get all label values: label_values(metric_name, label_name)
# Get all instances: label_values(up, instance)
# Get all job names: label_values(up, job)
# Get all metric names: metrics()
# Get metric names with prefix: metrics(http_.*)
# Query with filter: label_values({job="nginx"}, instance)
# Use query result: query_result(count by (instance) (up))
# Regex extraction from values: label_values(up, instance) -> Regex: /([^:]+):.*/ ```
Step 3: Fix InfluxDB Variable Queries
```bash # InfluxDB variable queries:
# Show measurements: SHOW MEASUREMENTS
# Show tag keys: SHOW TAG KEYS FROM "measurement"
# Show tag values: SHOW TAG VALUES FROM "measurement" WITH KEY = "tag_key"
# Show field keys: SHOW FIELD KEYS
# With regex: SHOW TAG VALUES FROM "measurement" WITH KEY = "host" WHERE "region" =~ /^$region$/
# Flux query (InfluxDB 2.x): import "influxdata/influxdb/schema" schema.tagValues(bucket: "bucket", tag: "host") ```
Step 4: Fix Elasticsearch Variable Queries
```bash # Elasticsearch variable queries:
# Find unique values: { "find": "terms", "field": "host.keyword" }
# With query filter: { "find": "terms", "field": "host.keyword", "query": "status:200" }
# Buckets query: { "bucketAggs": [ { "type": "terms", "field": "host.keyword", "settings": { "size": "100" } } ], "metrics": [{"type": "count", "id": "1"}] } ```
Step 5: Fix Variable Dependencies
```bash # Chained variables - one depends on another:
# Variable 1: region # Query: label_values(aws_ec2_instances, region)
# Variable 2: instance (depends on region) # Query: label_values(aws_ec2_instances{region="$region"}, instance)
# Note the $region syntax
# Check dependency order: # Variables are evaluated in order # Dependent variables must come after
# Common issues: # - Wrong variable name ($region vs $Region) # - Variable not defined yet # - Circular dependency
# Test each variable individually: # Disable dependency, test parent variable ```
Step 6: Check Variable Syntax in Panels
```bash # Using variables in panel queries:
# Prometheus: up{instance="$server"} up{instance=~"$server"} # Regex match (multi-select) up{instance!~"$server"} # Negative regex
# InfluxDB: SELECT value FROM cpu WHERE host = '$server' SELECT value FROM cpu WHERE host =~ /^$server$/
# Elasticsearch: host: "$server" host.keyword: "$server"
# Variable formats: # $var - Simple reference # ${var} - With braces (recommended) # [[var]] - Old format (deprecated)
# Multi-select variables: # $var = value1|value2|value3 # Use regex match: instance=~"$var"
# All option: # Enable "Include All option" # $var = .* for all ```
Step 7: Check Data Source Configuration
```bash # Check data source: # Configuration -> Data Sources -> Select data source
# Test connection: # Click "Save & Test"
# Check data source permissions: # User must have access to data source
# Check data source variables: # Type: Data source variable # Query: prometheus, influxdb, etc.
# Check default data source: # Variables use default if not specified
# Verify data source in query: # Panel -> Query -> Data source dropdown ```
Step 8: Debug Variable Queries
```bash # Enable query inspector: # Panel -> Query Inspector
# Check query sent to data source: # Shows expanded variable values
# Check variable resolution: # Dashboard -> View variable values: # Click variable dropdown -> Show all
# Use Grafana API to debug: curl -u admin:password http://grafana:3000/api/dashboards/uid/dashboard-uid | jq .dashboard.templating
# Check server logs: docker logs grafana journalctl -u grafana-server
# Enable debug logging: # In grafana.ini: [log] level = debug ```
Step 9: Fix Common Variable Issues
```bash # Issue 1: Variable shows "No options"
# Fix: Check query returns data # Test in Explore view first
# Issue 2: Variable shows IDs instead of names
# Fix: Add label transformation label_values(metric{label!=""}, label)
# Issue 3: Multi-select not working
# Fix: Use regex match in query instance=~"$var" # Not instance="$var"
# Issue 4: Refresh not updating values
# Fix: Set refresh option # Variable -> Refresh -> On Dashboard Load / On Time Range Change
# Issue 5: Sorting wrong
# Fix: Set sort option # Variable -> Sort -> Alphabetical (asc/desc)
# Issue 6: Custom values not working
# Fix: Use correct format # Values separated by comma: value1,value2,value3 # Or with display: text:value, text2:value2 ```
Step 10: Grafana Variable Verification Script
```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-grafana-vars.sh #!/bin/bash
GRAFANA_URL=${1:-"http://localhost:3000"} DASHBOARD_UID=${2:-""} USER=${3:-"admin"} PASS=${4:-"admin"}
echo "=== Grafana Status ===" curl -s -u $USER:$PASS $GRAFANA_URL/api/health | jq .
if [ -n "$DASHBOARD_UID" ]; then echo "" echo "=== Dashboard Variables ===" curl -s -u $USER:$PASS $GRAFANA_URL/api/dashboards/uid/$DASHBOARD_UID | \ jq '.dashboard.templating.list[] | {name: .name, type: .type, query: .query, current: .current}'
echo "" echo "=== Variable Values ===" VARS=$(curl -s -u $USER:$PASS $GRAFANA_URL/api/dashboards/uid/$DASHBOARD_UID | \ jq -r '.dashboard.templating.list[].name')
for var in $VARS; do echo "Variable: $var" curl -s -u $USER:$PASS "$GRAFANA_URL/api/dashboards/uid/$DASHBOARD_UID/variables" | \ jq --arg v "$var" '.[] | select(.name == $v) | .current' done fi
echo "" echo "=== Data Sources ===" curl -s -u $USER:$PASS $GRAFANA_URL/api/datasources | jq '.[] | {name: .name, type: .type, isDefault: .isDefault}'
echo "" echo "=== Recent Dashboard Errors ===" docker logs grafana 2>&1 | grep -i "variable|template" | tail -10 EOF
chmod +x /usr/local/bin/check-grafana-vars.sh
# Usage: /usr/local/bin/check-grafana-vars.sh http://grafana:3000 dashboard-uid admin password
# Quick variable test: alias grafana-vars='curl -s -u admin:password http://localhost:3000/api/dashboards/uid/UID | jq .dashboard.templating' ```
Grafana Variable Checklist
| Check | Command | Expected |
|---|---|---|
| Variable defined | Dashboard Settings | Variable listed |
| Query returns data | Explore view | Values shown |
| Data source works | Save & Test | Data source OK |
| Syntax correct | Panel query | Variable resolves |
| Dependencies | Order check | No circular deps |
| Permissions | User access | Can query |
Verify the Fix
```bash # After fixing variable issue
# 1. Check variable dropdown # Open dashboard, click variable // Options populated
# 2. Select value # Choose a value // Panels update
# 3. Test multi-select # Select multiple values // All included in query
# 4. Test "All" option # Select "All" // All values included
# 5. Check query inspector # Panel -> Query Inspector // Variable expanded correctly
# 6. Refresh dashboard # Reload dashboard // Variables refresh ```
Related Issues
- [Fix Grafana Dashboard Not Loading](/articles/fix-grafana-dashboard-not-loading)
- [Fix Prometheus Scrape Error](/articles/fix-prometheus-scrape-error)
- [Fix Grafana Data Source Connection Failed](/articles/fix-grafana-data-source-connection-failed)