What's Actually Happening

Prometheus targets show as down in the UI. Metrics are not being collected from the target endpoints.

The Error You'll See

In Prometheus UI at /targets: `` State: DOWN Error: Get "http://target:9090/metrics": dial tcp: connection refused

Why This Happens

  1. 1.Target service not running
  2. 2.Wrong scrape URL
  3. 3.Network connectivity issues
  4. 4.Firewall blocking
  5. 5.Authentication required

Step 1: Check Target Status

bash
curl http://localhost:9090/api/v1/targets
curl http://target-host:9090/metrics

Step 2: Check Prometheus Config

yaml
scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['target-host:9090']

Step 3: Check Network

bash
nc -zv target-host 9090
ping target-host

Step 4: Check Target Service

bash
systemctl status myapp
docker ps | grep myapp

Step 5: Check Logs

bash
journalctl -u prometheus -f
docker logs prometheus

Step 6: Verify Endpoint

bash
curl -v http://target-host:9090/metrics
curl http://target-host:9090/-/healthy

Step 7: Check Firewall

bash
iptables -L -n | grep 9090
firewall-cmd --list-ports

Step 8: Reload Config

bash
curl -X POST http://localhost:9090/-/reload
systemctl reload prometheus

Step 9: Check Service Discovery

bash
curl http://localhost:9090/api/v1/targets

Step 10: Monitor Targets

bash
watch -n 5 'curl -s http://localhost:9090/api/v1/targets | jq ".data.activeTargets[] | {job: .labels.job, health: .health}"'
  • [Fix Prometheus Query Timeout](/articles/fix-prometheus-query-timeout)
  • [Fix Grafana Dashboard Not Loading](/articles/fix-grafana-dashboard-not-loading)