Introduction

Prometheus remote_read problems usually show up when Prometheus can query local data but fails to fetch older or external data from a remote backend such as Thanos, Cortex, Mimir, or VictoriaMetrics. The failure may be transport, auth, timeout, or simply that the remote backend returns no data for the labels and time range you asked for.

Symptoms

  • Queries for historical data fail while recent data still works
  • Logs show remote read timeout, 500 responses, or no data returned
  • Dashboards become incomplete or gap-filled for older time ranges
  • The issue is worse on broad, high-cardinality queries

Common Causes

  • The remote storage endpoint is slow or unreachable
  • Auth, TLS, or custom headers for the remote backend are wrong
  • Query range or cardinality is too large for the current timeout settings
  • External labels or matcher expectations no longer align with stored data

Step-by-Step Fix

  1. 1.Check Prometheus logs and remote read metrics
  2. 2.Confirm whether the failure is timeout, auth, TLS, or empty data.
bash
kubectl logs deploy/prometheus
curl -s http://prometheus:9090/metrics | grep remote_read
  1. 1.Test the remote backend directly
  2. 2.Query the remote store outside Prometheus to see whether it is healthy and returning data.
bash
curl -vk https://remote-store.example.com
  1. 1.Review timeout and query scope
  2. 2.Broad queries can overwhelm remote backends even when local Prometheus remains healthy.
yaml
remote_read:
  - url: https://remote-store.example.com/api/v1/read
    read_recent: false
  1. 1.Retest with a narrow known-good query
  2. 2.Validate one low-cardinality query first before broadening time range and label scope again.
promql
up{job="node-exporter"}

Prevention

  • Monitor remote read latency and failure rate alongside dashboard health
  • Keep remote storage auth and TLS settings versioned with Prometheus config
  • Use recording rules to reduce expensive broad reads
  • Validate remote label and external-label conventions after migrations