Introduction When PagerDuty fails to create incidents from monitoring alerts, on-call engineers are not notified of critical issues. This is a severe monitoring gap that can extend incident resolution times.

Symptoms - Alert fires in monitoring system but no PagerDuty incident - PagerDuty API returns "Not Acceptable" or "Forbidden" - Error: "Invalid routing key" in monitoring system logs - Incident created but not routed to correct escalation policy - Duplicate alerts suppressed by deduplication

Common Causes - Integration key expired or deleted - Event rules filtering out the alert - Escalation policy has no targets - API rate limiting - Alert deduplication key causing suppression

Step-by-Step Fix 1. **Test PagerDuty integration key**: ```bash curl -X POST https://events.pagerduty.com/v2/enqueue \ -H 'Content-Type: application/json' \ -d '{ "routing_key": "<integration-key>", "event_action": "trigger", "payload": {"summary": "Test alert", "severity": "critical", "source": "test"} }' ```

  1. 1.**Check escalation policy':
  2. 2.Verify the escalation policy has active targets and the policy is assigned to the service.
  3. 3.**Check event rules for filtering':
  4. 4.Review PagerDuty event rules to ensure alerts are not being dropped.

Prevention - Test PagerDuty integration weekly with synthetic alerts - Monitor PagerDuty API response rates - Use multiple integration methods (API, email, webhook) - Document all integration keys and their services - Set up dead-man switch alerts for monitoring system health