Introduction Pub/Sub messages that are not acknowledged within the ack deadline are redelivered. After the maximum delivery attempts, they move to the dead letter topic, indicating persistent processing failures.

Symptoms - Dead letter topic accumulating messages - Subscription metrics show high ack_message_count but also high nack/expire - Subscriber logs show processing errors or timeouts - Message age increasing beyond ack deadline

Common Causes - Processing time exceeds ack deadline (default 10 seconds) - Subscriber application crashing before acknowledging - Dead letter policy maximum delivery count too low - Message schema mismatch causing deserialization failure - Network issues preventing ack/nack delivery to Pub/Sub

Step-by-Step Fix 1. **Check subscription configuration**: ```bash gcloud pubsub subscriptions describe <sub-name> \ --format="value(ackDeadlineSeconds,messageRetentionDuration,deadLetterPolicy)" ```

  1. 1.Increase ack deadline for slow processing:
  2. 2.```bash
  3. 3.gcloud pubsub subscriptions update <sub-name> --ack-deadline=60
  4. 4.`
  5. 5.Enable exactly-once delivery:
  6. 6.```bash
  7. 7.gcloud pubsub subscriptions update <sub-name> --enable-exactly-once-delivery
  8. 8.`
  9. 9.Process dead letter messages:
  10. 10.```bash
  11. 11.gcloud pubsub subscriptions pull <dlq-subscription> --auto-ack --limit=10
  12. 12.`
  13. 13.Implement proper ack handling in subscriber:
  14. 14.```python
  15. 15.from google.cloud import pubsub_v1
  16. 16.subscriber = pubsub_v1.SubscriberClient()
  17. 17.def callback(message):
  18. 18.try:
  19. 19.process_data(message.data)
  20. 20.message.ack()
  21. 21.except Exception:
  22. 22.message.nack()
  23. 23.subscription = subscriber.subscribe(sub_path, callback=callback)
  24. 24.`

Prevention - Set ack deadline to 2x the p99 processing time - Use exactly-once delivery for critical messages - Monitor dead letter topic size with alerts - Implement idempotent message processing - Set appropriate message retention duration (7 days minimum)