Introduction

A single misconfigured firewall rule can instantly make an entire website unreachable by blocking all inbound traffic on ports 80 and 443. This is one of the fastest and most complete types of site outages - the server is running, the application is healthy, but no external traffic can reach it. Recovery requires out-of-band access to the server (console, cloud provider management console) because SSH may also be blocked.

Symptoms

  • Site completely unreachable from any external network
  • curl and browser both time out (not refused - packets are being dropped)
  • Server is running and application is healthy when accessed locally
  • iptables -L shows a DROP rule matching all inbound traffic
  • Recent firewall change coincides with the outage start time

Common Causes

  • Default firewall policy changed to DROP without explicit ALLOW rules for HTTP/HTTPS
  • Firewall rule with wrong subnet mask blocking a wider range than intended
  • Security group rule deleted in cloud provider console
  • iptables flush (iptables -F) executed without re-adding necessary rules
  • Terraform/infrastructure-as-code change overwriting firewall rules

Step-by-Step Fix

  1. 1.Access the server via console (out-of-band management):
  2. 2.- AWS EC2: Use EC2 Instance Connect or Session Manager
  3. 3.- GCP: Use Serial Console
  4. 4.- Azure: Use Serial Console or Run Command
  5. 5.- Physical: Use IPMI/iDRAC/KVM
  6. 6.Review current firewall rules:
  7. 7.```bash
  8. 8.sudo iptables -L INPUT -n -v --line-numbers
  9. 9.sudo iptables -L FORWARD -n -v --line-numbers
  10. 10.# For firewalld:
  11. 11.sudo firewall-cmd --list-all
  12. 12.# For ufw:
  13. 13.sudo ufw status verbose
  14. 14.`
  15. 15.Restore HTTP/HTTPS access immediately:
  16. 16.```bash
  17. 17.sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
  18. 18.sudo iptables -I INPUT 2 -p tcp --dport 443 -j ACCEPT
  19. 19.# Also ensure SSH is not blocked
  20. 20.sudo iptables -I INPUT 3 -p tcp --dport 22 -j ACCEPT
  21. 21.`
  22. 22.Save the corrected rules:
  23. 23.```bash
  24. 24.# Debian/Ubuntu
  25. 25.sudo iptables-save | sudo tee /etc/iptables/rules.v4
  26. 26.# RHEL/CentOS
  27. 27.sudo service iptables save
  28. 28.`
  29. 29.For cloud provider security groups, fix via the console or CLI:
  30. 30.```bash
  31. 31.# AWS
  32. 32.aws ec2 authorize-security-group-ingress \
  33. 33.--group-id sg-xxxxx \
  34. 34.--protocol tcp --port 80 --cidr 0.0.0.0/0
  35. 35.aws ec2 authorize-security-group-ingress \
  36. 36.--group-id sg-xxxxx \
  37. 37.--protocol tcp --port 443 --cidr 0.0.0.0/0
  38. 38.`
  39. 39.Verify connectivity is restored:
  40. 40.```bash
  41. 41.# From an external machine
  42. 42.curl -I https://example.com
  43. 43.# Should return HTTP/1.1 200 OK
  44. 44.`

Prevention

  • Use infrastructure-as-code (Terraform) with peer review for all firewall changes
  • Test firewall changes in a staging environment before production
  • Implement a firewall change management process with pre and post verification
  • Always maintain console access as a fallback when firewall changes are made
  • Use configuration drift detection to alert on unauthorized firewall changes
  • Keep a rollback script ready before making any firewall modifications