Introduction

When a CDN edge server receives a request for content that is not in its cache (a cache miss), it pulls the content from the origin server. During a traffic spike, if many requests are for uncached content, the origin server can become overwhelmed by simultaneous pull requests from multiple CDN edge locations. This causes origin pull timeouts (504 Gateway Timeout) at the CDN level, making the site appear down even though the origin server may still be partially responsive.

Symptoms

  • CDN returns 504 Gateway Timeout for uncached resources
  • Origin server shows high CPU and memory from CDN pull requests
  • Cached content works but new or dynamic content fails
  • CDN dashboard shows high origin response times and timeout errors
  • Site works for returning visitors (cached) but fails for new visitors (cache miss)

Common Causes

  • Traffic spike generating more cache misses than the origin can handle
  • CDN cache TTL too short, causing frequent re-validation pulls
  • Origin server undersized for the concurrent pull request load
  • CDN not configured to serve stale content during origin errors
  • No rate limiting on origin pull requests from CDN edge locations

Step-by-Step Fix

  1. 1.Check CDN origin pull status:
  2. 2.```bash
  3. 3.# Check origin response time
  4. 4.curl -w "Origin Response Time: %{time_starttransfer}s\n" -o /dev/null -s https://example.com/resource
  5. 5.# Check CDN headers
  6. 6.curl -sI https://example.com/resource | grep -E "X-Cache|cf-cache-status|Age"
  7. 7.`
  8. 8.Increase CDN cache TTL for static content:
  9. 9.```nginx
  10. 10.# On the origin server, set longer cache headers
  11. 11.location ~* \.(css|js|png|jpg|jpeg|gif|ico|woff2)$ {
  12. 12.expires 30d;
  13. 13.add_header Cache-Control "public, immutable";
  14. 14.}
  15. 15.`
  16. 16.Configure CDN to serve stale content during origin errors:
  17. 17.`
  18. 18.# Cloudflare: Enable "Serve stale content while origin is down"
  19. 19.# Fastly: Use stale-if-error and stale-while-reuse
  20. 20.# CloudFront: Configure origin error caching
  21. 21.`
  22. 22.Pre-warm the CDN cache before expected traffic spikes:
  23. 23.```bash
  24. 24.# Purge and re-cache all important content
  25. 25.# This ensures the CDN has fresh copies before the spike hits
  26. 26.for url in $(cat urls-to-cache.txt); do
  27. 27.curl -s -o /dev/null "https://example.com$url"
  28. 28.done
  29. 29.`
  30. 30.Scale the origin server to handle increased pull requests:
  31. 31.```bash
  32. 32.# Temporarily increase web server workers
  33. 33.sudo systemctl edit nginx
  34. 34.# Add:
  35. 35.# [Service]
  36. 36.# LimitNOFILE=65536
  37. 37.sudo systemctl daemon-reload
  38. 38.sudo systemctl restart nginx
  39. 39.`
  40. 40.Implement origin request coalescing:
  41. 41.Configure the CDN to collapse concurrent requests for the same uncached resource into a single origin request, serving the result to all waiting clients.

Prevention

  • Set appropriate cache TTLs to minimize origin pulls during traffic spikes
  • Pre-warm CDN cache before anticipated high-traffic events
  • Configure CDN to serve stale content when the origin is slow or down
  • Implement origin request rate limiting at the CDN level
  • Use a CDN with request coalescing to reduce duplicate origin pulls
  • Monitor origin response times and set alerts before they approach CDN timeout thresholds