Fix Prometheus Alertmanager Not Sending Alerts

What's Actually Happening

Prometheus Alertmanager receives firing alerts but does not send notifications to configured receivers. Alerts accumulate without notification delivery.

The Error You'll See

```bash $ curl http://alertmanager:9093/api/v2/alerts

[{"status":"firing","labels":{"alertname":"HighCPU"}...}] # Alerts exist but no notifications sent ```

Alertmanager logs:

level=error msg="Error sending notification" receiver=email err="SMTP connection failed"

UI shows silences:

Alertmanager UI shows alerts as suppressed or silenced

No notification received:

# No email, Slack, PagerDuty notifications received despite firing alerts

Why This Happens

1. 1.Receiver not configured - Missing or wrong receiver config
2. 2.Routing mismatch - Alert labels don't match routes
3. 3.Notification integration failed - Email/Slack/PagerDuty connection issues
4. 4.Alerts silenced - Silence matching active alerts
5. 5.Alerts inhibited - Inhibition rules suppressing alerts
6. 6.Prometheus not sending - Prometheus not forwarding to Alertmanager

Step 1: Check Alertmanager Status

```bash # Check Alertmanager process: systemctl status alertmanager

# Or Docker: docker ps | grep alertmanager

# Check Alertmanager API: curl http://alertmanager:9093/api/v2/status

# Check Alertmanager version: curl http://alertmanager:9093/api/v2/buildinfo

# Check cluster status: curl http://alertmanager:9093/api/v2/cluster

# Check UI: # Open http://alertmanager:9093

# Check logs: docker logs alertmanager journalctl -u alertmanager -f

# Check listening: ss -tlnp | grep 9093 ```

Step 2: Check Alert Rules

```bash # Check Prometheus rules: curl http://prometheus:9090/api/v1/rules

# Check alerting rules: curl http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.type=="alerting")'

# Check firing alerts in Prometheus: curl http://prometheus:9090/api/v1/alerts

# Check alert state: curl http://prometheus:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'

# Check Prometheus config: curl http://prometheus:9090/api/v1/status/config | jq '.config.alerting'

# Check Alertmanager target in Prometheus: curl http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="alertmanager")'

# Prometheus config must have: alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] ```

Step 3: Check Alertmanager Configuration

```bash # Check Alertmanager config file: cat /etc/alertmanager/alertmanager.yml

# Or via API: curl http://alertmanager:9093/api/v2/status | jq '.configJSON'

# Basic config: global: resolve_timeout: 5m

route: receiver: 'default-receiver' group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 4h

receivers: - name: 'default-receiver' email_configs: - to: 'admin@example.com' from: 'alertmanager@example.com' smarthost: 'smtp.example.com:25'

# Validate config: amtool check-config /etc/alertmanager/alertmanager.yml

# Reload config: curl -X POST http://alertmanager:9093/api/v2/reload

# Or restart: systemctl restart alertmanager ```

Step 4: Check Routing Configuration

```bash # Check route tree: amtool config routes --config.file=/etc/alertmanager/alertmanager.yml

# Test route matching: amtool config routes-test --config.file=/etc/alertmanager/alertmanager.yml --labels.alertname=HighCPU

# Check specific route: curl http://alertmanager:9093/api/v2/routes

# Route must match alert labels: route: receiver: 'team-backend' match: team: backend match_re: severity: critical|warning

# Alerts need matching labels: # alert with labels: {team="backend", severity="critical"} # Matches route to team-backend receiver

# Check receiver for alert: amtool alert query --alertmanager.url=http://alertmanager:9093 ```

Step 5: Check Receiver Configuration

```bash # List receivers: amtool config receivers --config.file=/etc/alertmanager/alertmanager.yml

# Check email receiver: receivers: - name: 'email-receiver' email_configs: - to: 'admin@example.com' from: 'alertmanager@example.com' smarthost: 'smtp.example.com:587' auth_username: 'alertmanager' auth_password: 'password'

# Check Slack receiver: receivers: - name: 'slack-receiver' slack_configs: - api_url: 'https://hooks.slack.com/services/xxx' channel: '#alerts'

# Check PagerDuty receiver: receivers: - name: 'pagerduty-receiver' pagerduty_configs: - service_key: 'your-service-key'

# Check webhook receiver: receivers: - name: 'webhook-receiver' webhook_configs: - url: 'http://webhook-server/alerts'

# Test notification manually: amtool alert add --alertmanager.url=http://alertmanager:9093 alertname=Test severity=test ```

Step 6: Check Email Notification

```bash # Check SMTP config: curl http://alertmanager:9093/api/v2/status | jq '.configJSON.global.email_configs'

# Test SMTP connection: telnet smtp.example.com 587

# Check SMTP auth: # Try sending email manually: # Use swaks tool: swaks --to admin@example.com --from alertmanager@example.com --server smtp.example.com:587 --auth LOGIN --auth-user alertmanager

# Check Alertmanager logs for SMTP: grep -i smtp /var/log/alertmanager.log grep -i "email" /var/log/alertmanager.log

# Common issues: # SMTP port wrong (25 vs 587) # TLS not configured # Auth credentials wrong

# Add SMTP TLS: email_configs: - to: 'admin@example.com' smarthost: 'smtp.example.com:587' auth_username: 'user' auth_password: 'password' require_tls: true ```

Step 7: Check Slack Notification

```bash # Check Slack webhook URL: curl http://alertmanager:9093/api/v2/status | jq '.configJSON.receivers[].slack_configs'

# Test Slack webhook: curl -X POST -H 'Content-type: application/json' \ --data '{"text":"Test alert"}' \ https://hooks.slack.com/services/T00/B00/XXX

# Check Slack response: # Should return "ok"

# Check Alertmanager logs for Slack: grep -i slack /var/log/alertmanager.log

# Check Slack API URL format: # https://hooks.slack.com/services/TXXXXXXXX/BXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX

# Verify channel exists: # Check Slack channel #alerts exists

# Common issues: # Webhook URL invalid or revoked # Channel doesn't exist # Slack integration disabled

# Recreate Slack webhook: # In Slack: Apps -> Manage -> Custom Integrations -> Incoming Webhooks ```

Step 8: Check Silences

```bash # List silences: curl http://alertmanager:9093/api/v2/silences | jq .

# Or amtool: amtool silence query --alertmanager.url=http://alertmanager:9093

# Check active silences: amtool silence query --alertmanager.url=http://alertmanager:9093 --active

# Check if alert matches silence: amtool silence query --alertmanager.url=http://alertmanager:9093 --labels.alertname=HighCPU

# Silence details: curl http://alertmanager:9093/api/v2/silence/silence-id | jq .

# Delete silence: amtool silence delete --alertmanager.url=http://alertmanager:9093 silence-id

# Or via API: curl -X DELETE http://alertmanager:9093/api/v2/silence/silence-id

# Create silence for maintenance: amtool silence add --alertmanager.url=http://alertmanager:9093 --comment="Maintenance" --duration=2h alertname=HighCPU ```

Step 9: Check Inhibitions

```bash # Check inhibition rules: curl http://alertmanager:9093/api/v2/status | jq '.configJSON.inhibit_rules'

# Inhibition suppresses alerts when other alert fires: inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'instance']

# If critical alert fires, warning alert inhibited

# Check inhibited alerts: curl http://alertmanager:9093/api/v2/alerts | jq '.[] | select(.status.inhibitedBy != null)'

# List inhibition sources: curl http://alertmanager:9093/api/v2/alerts | jq '.[] | .status.inhibitedBy'

# Remove inhibition rules if not needed: # Edit alertmanager.yml, remove inhibit_rules section

# Check why alert inhibited: amtool alert query --alertmanager.url=http://alertmanager:9093 --inhibited ```

Step 10: Alertmanager Verification Script

```bash # Create verification script: cat << 'EOF' > /usr/local/bin/check-alertmanager.sh #!/bin/bash

AM_URL="http://localhost:9093"

echo "=== Alertmanager Status ===" curl -s $AM_URL/api/v2/status | jq '{uptime, clusterStatus}'

echo "" echo "=== Active Alerts ===" curl -s $AM_URL/api/v2/alerts | jq 'group_by(.status.state) | map({state: .[0].status.state, count: length})'

echo "" echo "=== Firing Alerts ===" curl -s $AM_URL/api/v2/alerts | jq '.[] | select(.status.state=="active") | {labels: .labels, startsAt: .startsAt}'

echo "" echo "=== Silences ===" curl -s $AM_URL/api/v2/silences | jq '.[] | select(.status.state=="active") | {id: .id, comment: .comment, matchers: .matchers}'

echo "" echo "=== Receivers ===" amtool config receivers --alertmanager.url=$AM_URL

echo "" echo "=== Route Tree ===" amtool config routes --alertmanager.url=$AM_URL

echo "" echo "=== Prometheus Alerts ===" curl -s http://prometheus:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing") | {labels: .labels, state: .state}'

echo "" echo "=== Recent Logs ===" docker logs alertmanager --tail 20 2>&1 | grep -E "error|notify|failed" EOF

chmod +x /usr/local/bin/check-alertmanager.sh

# Run: /usr/local/bin/check-alertmanager.sh

# Test notification: amtool alert add --alertmanager.url=http://alertmanager:9093 alertname=TestNotification --comment="Testing notifications" ```

Alertmanager Checklist

Verify the Fix

```bash # After fixing Alertmanager

# 1. Check alerts curl http://alertmanager:9093/api/v2/alerts // Firing alerts present

# 2. Test notification amtool alert add alertname=Test // Notification received

# 3. Check route amtool config routes-test --labels.alertname=HighCPU // Route matched to receiver

# 4. Check no silences amtool silence query --active // No active silences

# 5. Verify email/Slack # Check inbox/Slack channel // Notifications received

# 6. Monitor logs docker logs alertmanager -f // Notification sent successfully ```

Related Issues

• [Fix Prometheus Scrape Error](/articles/fix-prometheus-scrape-error)
• [Fix Grafana Dashboard Not Loading](/articles/fix-grafana-dashboard-not-loading)
• [Fix Prometheus Query Failed](/articles/fix-prometheus-query-failed)