The Problem

Prometheus fails to discover targets from file-based service discovery, or targets are not updated when the file changes:

bash
level=error ts=2026-04-04T19:25:10.123Z caller=file.go:234 msg="Error reading file" path="/etc/prometheus/targets.json" err="open /etc/prometheus/targets.json: no such file or directory"
level=error ts=2026-04-04T19:25:11.234Z caller=file.go:235 msg="Failed to parse file SD config" err="json: cannot unmarshal string into Go value of type []targetgroup.Group"
level=warn ts=2026-04-04T19:25:12.345Z caller=file.go:236 msg="File SD refresh failed" path="/etc/prometheus/targets.json"

File service discovery errors prevent dynamic target updates without Prometheus restart.

Diagnosis

Check File Existence and Permissions

```bash # Check file exists ls -la /etc/prometheus/targets.json

# Check permissions stat /etc/prometheus/targets.json

# Check file content cat /etc/prometheus/targets.json

# Check Prometheus can read file # (from Prometheus process/container) kubectl exec -it prometheus-pod -- cat /etc/prometheus/targets.json ```

Check Prometheus Configuration

bash
# View file SD config
curl -s http://localhost:9090/api/v1/status/config | jq '.data.scrape_configs[] | select(.file_sd_configs)'

Check Discovered Targets

bash
# Check targets from file SD
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.discoveredLabels.__meta_filepath)'

Validate JSON Format

```bash # Validate JSON syntax jq . /etc/prometheus/targets.json

# Check structure python -m json.tool /etc/prometheus/targets.json ```

Solutions

1. Fix File Path Issues

File not found or wrong path:

yaml
# prometheus.yml
scrape_configs:
  - job_name: 'file-sd-targets'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.json'  # Use glob pattern
          - '/etc/prometheus/targets.json'
        refresh_interval: 5m

Check file location:

```bash # Verify path from Prometheus working directory ls -la /etc/prometheus/targets/

# Create directory if needed mkdir -p /etc/prometheus/targets

# Create targets file cat > /etc/prometheus/targets.json << 'EOF' [ { "targets": ["localhost:9090"], "labels": { "job": "prometheus" } } ] EOF ```

2. Fix JSON Format

Invalid JSON format:

```json # WRONG: Incorrect format # { # "targets": ["host1:9090", "host2:9090"] # }

# CORRECT: Proper file SD format (array of groups) [ { "targets": [ "host1:9090", "host2:9090" ], "labels": { "job": "myapp", "environment": "production" } }, { "targets": [ "host3:9090" ], "labels": { "job": "another-app", "environment": "staging" } } ] ```

Alternative YAML format:

```yaml # targets.yaml - targets: - host1:9090 - host2:9090 labels: job: myapp environment: production

  • targets:
  • - host3:9090
  • labels:
  • job: another-app
  • environment: staging
  • `

Use YAML file in config:

yaml
scrape_configs:
  - job_name: 'file-sd-targets'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.yaml'
          - '/etc/prometheus/targets/*.yml'
        refresh_interval: 5m

3. Fix File Permissions

Prometheus cannot read file:

```bash # Set correct permissions chmod 644 /etc/prometheus/targets.json

# Set ownership (if running as prometheus user) chown prometheus:prometheus /etc/prometheus/targets.json

# Check user running Prometheus ps aux | grep prometheus ```

4. Update Targets Dynamically

Targets not updating when file changes:

yaml
scrape_configs:
  - job_name: 'file-sd-targets'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets.json'
        refresh_interval: 30s  # Check every 30 seconds

Manual refresh:

```bash # Trigger config reload curl -X POST http://localhost:9090/-/reload

# Or restart Prometheus systemctl restart prometheus ```

5. Use Multiple Files

Organize targets in multiple files:

```bash # Create directory structure mkdir -p /etc/prometheus/targets/{production,staging}

# Production targets cat > /etc/prometheus/targets/production/apps.json << 'EOF' [ { "targets": ["prod-app1:9090", "prod-app2:9090"], "labels": { "environment": "production", "datacenter": "dc1" } } ] EOF

# Staging targets cat > /etc/prometheus/targets/staging/apps.json << 'EOF' [ { "targets": ["staging-app1:9090"], "labels": { "environment": "staging", "datacenter": "dc2" } } ] EOF ```

Configuration with glob patterns:

yaml
scrape_configs:
  - job_name: 'file-sd-targets'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/**/*.json'
          - '/etc/prometheus/targets/**/*.yaml'
        refresh_interval: 1m

6. Add Labels to Targets

Targets missing required labels:

json
[
  {
    "targets": ["app1:9090", "app2:9090"],
    "labels": {
      "job": "myapp",
      "environment": "production",
      "region": "us-east",
      "team": "platform"
    }
  }
]

Add labels via relabeling:

```yaml scrape_configs: - job_name: 'file-sd-targets' file_sd_configs: - files: - '/etc/prometheus/targets.json' relabel_configs: - source_labels: [__address__] target_label: instance action: replace

# Add default label if missing - source_labels: [environment] regex: '' target_label: environment replacement: 'unknown' ```

7. Generate Targets Dynamically

Script to generate targets file:

```bash #!/bin/bash # generate_targets.sh

TARGETS_FILE="/etc/prometheus/targets.json"

# Discover running instances INSTANCES=$(aws ec2 describe-instances \ --filters "Name=tag:Monitoring,Values=prometheus" "Name=instance-state-name,Values=running" \ --query 'Reservations[*].Instances[*].[PrivateIpAddress]' \ --output text)

# Build JSON echo "[" > $TARGETS_FILE echo " {" >> $TARGETS_FILE echo " \"targets\": [" >> $TARGETS_FILE for ip in $INSTANCES; do echo " \"$ip:9090\"," >> $TARGETS_FILE done echo " ]," >> $TARGETS_FILE echo " \"labels\": {" >> $TARGETS_FILE echo " \"job\": \"discovered-apps\"" >> $TARGETS_FILE echo " }" >> $TARGETS_FILE echo " }" >> $TARGETS_FILE echo "]" >> $TARGETS_FILE

# Signal Prometheus curl -X POST http://localhost:9090/-/reload ```

Run periodically:

bash
# Cron job to update targets
*/5 * * * * /etc/prometheus/generate_targets.sh

Verification

Check File SD Working

bash
# Check discovered targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.discoveredLabels.__meta_filepath) | {file: .discoveredLabels.__meta_filepath, targets: .discoveredLabels.__address__}'

Verify Targets Are Up

```promql # Count targets from file SD count by (job) (up{job="file-sd-targets"})

# Check target status up{job="file-sd-targets"} ```

Test File Updates

```bash # Add new target echo ' [ { "targets": ["host1:9090", "host2:9090", "new-host:9090"], "labels": { "job": "myapp" } } ]' > /etc/prometheus/targets.json

# Wait for refresh (or reload) sleep 30

# Check new target discovered curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.instance == "new-host:9090")' ```

Prevention

Add monitoring for file SD:

```yaml groups: - name: file_sd_alerts rules: - alert: FileSDReadFailed expr: increase(prometheus_sd_file_read_errors_total[5m]) > 0 for: 5m labels: severity: warning annotations: summary: "File service discovery read error" description: "Failed to read file {{ $labels.file }}"

  • alert: FileSDTargetsMissing
  • expr: count(up{job="file-sd-targets"}) == 0
  • for: 10m
  • labels:
  • severity: warning
  • annotations:
  • summary: "No targets from file SD"
  • description: "File service discovery returned no targets"
  • alert: FileSDStaleTargets
  • expr: count(up{job="file-sd-targets"}) != count(up{job="file-sd-targets"} and up == 1)
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Some file SD targets are down"
  • description: "{{ $value }} targets from file SD are unreachable"
  • `

File SD Best Practices

  1. 1.Use glob patterns: Allow multiple files for organization
  2. 2.Set refresh interval: Balance between responsiveness and load
  3. 3.Include labels: Add job, environment, and other labels in file
  4. 4.Validate JSON: Use jq to validate format before deployment
  5. 5.Monitor file changes: Alert when targets go missing
  6. 6.Use YAML for readability: YAML is easier to maintain manually