Introduction

Prometheus relabeling rules use regex matching to include, exclude, or modify metrics during scraping. A misconfigured keep or drop rule with an overly broad regex can inadvertently filter out all metrics from a target. Because relabeling happens silently, the target appears healthy in the targets page but reports zero ingested samples.

Symptoms

  • Prometheus target shows status UP but scrape_samples_scraped is 0
  • No metrics from the target appear in query results
  • Prometheus config shows target as healthy but queries return no data
  • Recent relabel config change correlates with metric disappearance
  • Error message: target dropped by relabeling (visible in debug logs)

Common Causes

  • action: keep with regex that does not match any metric name or label value
  • action: drop with regex .+ or .* matching everything
  • Incorrect source_labels referencing labels that do not exist on the target
  • Regex anchoring error -- ^http_ vs ^http_requests.* -- causing unintended matches
  • Relabel config applied to wrong scrape job due to YAML indentation error

Step-by-Step Fix

  1. 1.Validate the current relabel configuration: Check for problematic rules.
  2. 2.```bash
  3. 3.curl -s http://localhost:9090/api/v1/status/config | grep -A20 "relabel_configs"
  4. 4.`
  5. 5.Enable debug logging to see relabeling decisions: Identify which rule is dropping metrics.
  6. 6.```yaml
  7. 7.# prometheus.yml - add debug level
  8. 8.global:
  9. 9.scrape_interval: 15s
  10. 10.log_level: debug
  11. 11.`
  12. 12.Fix the relabel regex to match the intended metrics: Correct the keep/drop rule.
  13. 13.```yaml
  14. 14.# WRONG: drops all metrics
  15. 15.relabel_configs:
  16. 16.- source_labels: [__name__]
  17. 17.regex: ".*"
  18. 18.action: drop

# CORRECT: drops only debug metrics relabel_configs: - source_labels: [__name__] regex: "go_.*|process_.*" action: keep ```

  1. 1.Test relabel config using promtool: Validate before applying.
  2. 2.```bash
  3. 3.promtool check config /etc/prometheus/prometheus.yml
  4. 4.# Test relabeling against sample targets
  5. 5.promtool tsdb analyze /var/lib/prometheus/metrics2
  6. 6.`
  7. 7.Reload Prometheus configuration: Apply the fix.
  8. 8.```bash
  9. 9.curl -X POST http://localhost:9090/-/reload
  10. 10.# Verify metrics are flowing
  11. 11.curl -s 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result[].metric.job'
  12. 12.`

Prevention

  • Always test relabel configuration changes in a staging Prometheus instance first
  • Use promtool check config to validate relabel rules before deployment
  • Start with action: drop rules that are narrowly scoped to specific metric patterns
  • Monitor scrape_samples_scraped per target and alert when it drops to zero for an UP target
  • Document all relabel rules with comments explaining the intent of each rule
  • Use replacement action to rename labels instead of complex keep/drop chains