Introduction

Notification channels (now called Contact Points in Grafana 8+) deliver alerts to external systems like Slack, email, PagerDuty, and webhooks. When notifications fail, teams miss critical alerts. Common failures include authentication errors, endpoint unreachability, invalid payload formats, and throttling by notification providers.

Symptoms

  • Alert fires but no notification is received
  • Error: "failed to send notification" in Grafana logs
  • Test notification fails with "connection refused" or "timeout"
  • Slack notifications stopped working after token rotation
  • Email notifications are not delivered or go to spam
  • Webhook notifications return 4xx or 5xx errors

Common Causes

  • Slack webhook URL expired or was revoked
  • SMTP server is unreachable or credentials are wrong
  • Webhook endpoint is down or returns errors
  • PagerDuty/Opsgenie integration key is invalid
  • Notification rate limiting by external providers
  • Network firewall blocks outbound connections
  • Invalid JSON payload format for webhook

Step-by-Step Fix

Slack Notification Issues

  1. 1.Test Slack webhook URL directly:
  2. 2.```bash
  3. 3.curl -X POST -H 'Content-type: application/json' \
  4. 4.--data '{"text":"Test notification from Grafana"}' \
  5. 5.https://hooks.slack.com/services/TXXXXX/BXXXXX/XXXXXXXX
  6. 6.`
  7. 7.If webhook returns "invalid_token" or "channel_not_found":
  8. 8.- Recreate the webhook in Slack app settings
  9. 9.- Update the webhook URL in Grafana contact point
  10. 10.- Ensure the Slack app has permission to post to the channel
  11. 11.For Slack app integration (OAuth):
  12. 12.```ini
  13. 13.# In grafana.ini
  14. 14.[slack]
  15. 15.enabled = true
  16. 16.client_id = your-slack-app-client-id
  17. 17.client_secret = your-slack-app-client-secret
  18. 18.`
  19. 19.Verify Slack app has required OAuth scopes:
  20. 20.- chat:write
  21. 21.- chat:write.public
  22. 22.- channels:read

Email Notification Issues

  1. 1.Test SMTP connectivity:
  2. 2.```bash
  3. 3.telnet smtp.example.com 587
  4. 4.`
  5. 5.Test SMTP authentication:
  6. 6.```bash
  7. 7.openssl s_client -connect smtp.example.com:587 -starttls smtp
  8. 8.`
  9. 9.Verify Grafana SMTP configuration:
  10. 10.```ini
  11. 11.# In grafana.ini
  12. 12.[smtp]
  13. 13.enabled = true
  14. 14.host = smtp.example.com:587
  15. 15.user = noreply@example.com
  16. 16.password = your-smtp-password
  17. 17.from_address = grafana@example.com
  18. 18.from_name = Grafana Alerts
  19. 19.skip_verify = false
  20. 20.`
  21. 21.For Gmail SMTP, use app-specific password:
  22. 22.```ini
  23. 23.[smtp]
  24. 24.host = smtp.gmail.com:587
  25. 25.user = your-email@gmail.com
  26. 26.password = your-app-specific-password
  27. 27.`
  28. 28.Check email delivery logs:
  29. 29.```bash
  30. 30.journalctl -u grafana-server | grep -i "mail|smtp|email"
  31. 31.`

PagerDuty Notification Issues

  1. 1.Verify PagerDuty integration key:
  2. 2.```bash
  3. 3.curl -X POST https://events.pagerduty.com/v2/enqueue \
  4. 4.-H 'Content-Type: application/json' \
  5. 5.-d '{
  6. 6."routing_key": "your-routing-key",
  7. 7."event_action": "trigger",
  8. 8."payload": {
  9. 9."summary": "Test Alert",
  10. 10."severity": "critical",
  11. 11."source": "Grafana"
  12. 12.}
  13. 13.}'
  14. 14.`
  15. 15.Check for correct event action values:
  16. 16.- trigger for firing alerts
  17. 17.- resolve for resolved alerts
  18. 18.- acknowledge for acknowledged alerts

Opsgenie Notification Issues

  1. 1.Verify Opsgenie API key:
  2. 2.```bash
  3. 3.curl -X POST https://api.opsgenie.com/v2/alerts \
  4. 4.-H 'Authorization: GenieKey your-api-key' \
  5. 5.-H 'Content-Type: application/json' \
  6. 6.-d '{
  7. 7."message": "Test Alert",
  8. 8."priority": "P1"
  9. 9.}'
  10. 10.`
  11. 11.Check Opsgenie integration configuration in Grafana:
  12. 12.- API URL: https://api.opsgenie.com (US) or https://api.eu.opsgenie.com (EU)
  13. 13.- API key should have correct permissions

Webhook Notification Issues

  1. 1.Test webhook endpoint directly:
  2. 2.```bash
  3. 3.curl -X POST https://your-webhook.example.com/alert \
  4. 4.-H 'Content-Type: application/json' \
  5. 5.-d '{
  6. 6."title": "Test Alert",
  7. 7."state": "alerting",
  8. 8."message": "Test notification"
  9. 9.}'
  10. 10.`
  11. 11.Check webhook response for errors:
  12. 12.```bash
  13. 13.curl -v -X POST https://your-webhook.example.com/alert \
  14. 14.-H 'Content-Type: application/json' \
  15. 15.-d '{"test": true}'
  16. 16.`
  17. 17.Verify webhook configuration:
  18. 18.```yaml
  19. 19.# Contact point configuration
  20. 20.type: webhook
  21. 21.settings:
  22. 22.url: https://your-webhook.example.com/alert
  23. 23.httpMethod: POST
  24. 24.authorization_scheme: bearer
  25. 25.authorization_credentials: your-token
  26. 26.`

Network and Firewall Issues

  1. 1.Check outbound connectivity:
  2. 2.```bash
  3. 3.# Test connectivity to common notification services
  4. 4.curl -v https://hooks.slack.com
  5. 5.curl -v https://events.pagerduty.com
  6. 6.curl -v https://api.opsgenie.com
  7. 7.`
  8. 8.Check if proxy is required:
  9. 9.```ini
  10. 10.# In grafana.ini
  11. 11.[server]
  12. 12.http_proxy = http://proxy.example.com:8080
  13. 13.https_proxy = http://proxy.example.com:8080
  14. 14.`

Rate Limiting Issues

  1. 1.Check for rate limiting errors in logs:
  2. 2.```bash
  3. 3.journalctl -u grafana-server | grep -i "rate limit|429|throttl"
  4. 4.`
  5. 5.Configure notification grouping to reduce volume:
  6. 6.```yaml
  7. 7.# Notification policy
  8. 8.group_by: ['alertname', 'severity']
  9. 9.group_wait: 30s
  10. 10.group_interval: 5m
  11. 11.repeat_interval: 4h
  12. 12.`

Verification

  1. 1.Test all contact points:
  2. 2.- Navigate to Alerting > Contact points
  3. 3.- Click "Test" on each contact point
  4. 4.- Verify test notifications are received
  5. 5.Check notification logs:
  6. 6.```bash
  7. 7.curl -s -u admin:password http://localhost:3000/api/v1/notifications | jq .
  8. 8.`
  9. 9.Send a test alert and verify delivery:
  10. 10.- Create a test alert rule
  11. 11.- Trigger the alert condition
  12. 12.- Confirm notification is received in all configured channels