Introduction

Firewall and network security group issues occur when network traffic is blocked or dropped by host-based firewalls (iptables, nftables, firewalld, Windows Firewall) or cloud security groups (AWS Security Groups, Azure NSG, GCP Firewall Rules). Common causes include overly restrictive rules blocking legitimate traffic, rule ordering causing unexpected matches, stateful inspection dropping return traffic, NAT/masquerade misconfiguration, port conflicts, security group ingress/egress rules misconfigured, network ACL conflicts, connection tracking table exhaustion, and ICMP blocking preventing path MTU discovery. The fix requires understanding firewall rule evaluation order, connection tracking, stateful vs stateless filtering, and debugging tools for network packet analysis. This guide provides production-proven troubleshooting for firewall issues across on-premises Linux/Windows servers and cloud deployments (AWS, Azure, GCP).

Symptoms

  • Connection times out with no response: curl: (7) Failed to connect
  • Connection refused immediately: curl: (7) Connection refused
  • telnet shows Trying... then hangs indefinitely
  • SSH connection fails: ssh: connect to host port 22: Connection timed out
  • Application listens on port but external clients cannot connect
  • Internal network works, external network fails (or vice versa)
  • Intermittent connection failures under load
  • dmesg shows nf_conntrack: table full, dropping packet
  • Security group shows traffic allowed but connection still fails
  • NAT port forwarding doesn't work
  • ICMP ping fails but TCP works (or vice versa)
  • Return traffic blocked causing one-way communication

Common Causes

  • Firewall rule explicitly blocking port/protocol
  • Default DROP/REJECT policy with no allow rule
  • Rule ordering: DROP rule before ALLOW rule
  • Security group missing ingress rule for required port
  • Egress rules blocking outbound connections
  • Connection tracking (conntrack) table full
  • NAT/masquerade not configured for forwarding
  • Return traffic blocked due to stateless filtering
  • Network ACL overrides security group rules
  • Firewall zone misconfiguration (firewalld)
  • ICMP blocked preventing path MTU discovery
  • TCP wrapper (/etc/hosts.allow, /etc/hosts.deny) blocking access
  • SELinux/AppArmor network restrictions
  • Cloud firewall service (AWS NACL, Azure Firewall) at network level

Step-by-Step Fix

### 1. Diagnose firewall issues

Check if firewall is running:

```bash # Check iptables status (Linux) iptables -L -n -v # Shows all chains with packet/byte counters

# Check current chain policies iptables -L -n | grep Chain # Chain INPUT (policy ACCEPT) # Chain FORWARD (policy DROP) # Chain OUTPUT (policy ACCEPT)

# If policies show DROP or REJECT, default is to block

# Check for rules affecting specific port iptables -L -n -v | grep -E "22|80|443"

# Check NAT rules iptables -t nat -L -n -v

# Check raw connection tracking conntrack -L

# Number of tracked connections cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_max

# If count approaches max, connections will be dropped ```

Check firewalld (RHEL/CentOS/Fedora):

```bash # Check firewalld status systemctl status firewalld firewall-cmd --state

# List all rules firewall-cmd --list-all

# List all zones firewall-cmd --get-zones

# Show current zone configuration firewall-cmd --zone=public --list-all

# Output: # public (active) # target: default # icmp-block-inversion: no # interfaces: eth0 # sources: # services: ssh dhcpv6-client # ports: 80/tcp 443/tcp # protocols: # masquerade: no # forward-ports: # source-ports: # icmp-blocks: # rich rules: ```

Check nftables (newer systems):

```bash # Check nftables status nft list ruleset

# Show tables, chains, and rules nft list tables nft list chain inet filter input nft list chain inet filter output

# nftables replaces iptables in newer distributions # Rules are more compact and efficient ```

Check cloud security groups:

```bash # AWS: List security groups aws ec2 describe-security-groups --group-ids sg-xxxxxxxxx

# Check specific group rules aws ec2 describe-security-group-rules --filters "Name=group-id,Values=sg-xxxxxxxxx"

# Azure: List NSG rules az network nsg rule list --resource-group myRG --nsg-name myNSG

# GCP: List firewall rules gcloud compute firewall-rules list --network=default ```

### 2. Test connectivity

Network debugging tools:

```bash # Test TCP connection telnet hostname 22 # Or modern alternative nc -zv hostname 22 nc -vz --udp hostname 53

# Test with timeout timeout 5 bash -c 'cat < /dev/null > /dev/tcp/hostname/22'

# Check if port is listening locally ss -tlnp | grep :22 netstat -tlnp | grep :22

# Check which process owns port lsof -i :22 fuser 22/tcp

# Trace network path traceroute hostname tracepath hostname

# Test with curl (HTTP/HTTPS) curl -v http://hostname/ curl -v --connect-timeout 5 https://hostname/

# Check DNS resolution nslookup hostname dig hostname host hostname

# Test from different network locations # If internal works but external fails, likely firewall/security group ```

Packet capture for debugging:

```bash # Capture packets on interface tcpdump -i eth0 -n port 22

# Show full packet details tcpdump -i eth0 -n -e -s 0 port 22

# Capture to file for analysis tcpdump -i eth0 -w /tmp/capture.pcap port 22 # Analyze with Wireshark

# Filter by IP tcpdump -i eth0 -n host 10.0.0.1

# Filter by source/destination tcpdump -i eth0 -n src 10.0.0.1 and dst port 22

# See packets being dropped iptables -A INPUT -j LOG --log-prefix "DROPPED: " --log-level 4 # Then check: dmesg | grep DROPPED # Or: tail -f /var/log/kern.log | grep DROPPED

# Using nftables nft add rule inet filter input log prefix \"DROPPED: \" ```

### 3. Fix iptables rules

Add allow rule for specific port:

```bash # Allow SSH (port 22) iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP and HTTPS iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT

# Allow from specific IP only iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 22 -j ACCEPT

# Allow established/related connections (CRITICAL for return traffic) iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback iptables -A INPUT -i lo -j ACCEPT

# Insert rule at specific position (before a DROP rule) iptables -I INPUT 3 -p tcp --dport 8080 -j ACCEPT

# Delete a rule iptables -D INPUT -p tcp --dport 8080 -j ACCEPT

# Or by rule number iptables -L INPUT --line-numbers iptables -D INPUT 3 # Delete rule #3

# Save rules (Debian/Ubuntu) iptables-save > /etc/iptables/rules.v4

# Save rules (RHEL/CentOS) service iptables save # Or /sbin/iptables-save > /etc/sysconfig/iptables ```

Complete iptables configuration example:

```bash #!/bin/bash

# Flush existing rules iptables -F iptables -X iptables -t nat -F iptables -t nat -X iptables -t mangle -F iptables -t mangle -X

# Default policies (deny all incoming, allow outgoing) iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT

# Allow loopback iptables -A INPUT -i lo -j ACCEPT iptables -A OUTPUT -o lo -j ACCEPT

# Allow established and related connections iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT

# Allow ICMP (ping) - rate limited iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT

# Log dropped packets iptables -A INPUT -j LOG --log-prefix "IPTABLES_DROPPED: " --log-level 4

# Drop everything else (implicit from policy, but explicit for clarity) iptables -A INPUT -j DROP ```

NAT/port forwarding configuration:

```bash # Enable IP forwarding echo 1 > /proc/sys/net/ipv4/ip_forward # Or permanently in /etc/sysctl.conf: # net.ipv4.ip_forward = 1

# NAT port forwarding: external 8080 -> internal 10.0.0.5:80 iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.0.0.5:80 iptables -A FORWARD -p tcp -d 10.0.0.5 --dport 80 -j ACCEPT

# Masquerade for outbound traffic (like typical router) iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Save NAT rules iptables-save -t nat > /etc/iptables/nat.v4 ```

### 4. Fix firewalld configuration

Firewalld zone and service configuration:

```bash # Add service to zone firewall-cmd --zone=public --add-service=ssh firewall-cmd --zone=public --add-service=http firewall-cmd --zone=public --add-service=https

# Add specific port firewall-cmd --zone=public --add-port=8080/tcp

# Make permanent (survives reboot) firewall-cmd --runtime-to-permanent # Or add --permanent flag to commands: firewall-cmd --permanent --zone=public --add-service=ssh firewall-cmd --reload

# Remove service/port firewall-cmd --zone=public --remove-service=ssh firewall-cmd --zone=public --remove-port=8080/tcp

# Create custom service cat > /etc/firewalld/services/myapp.xml << 'EOF' <?xml version="1.0" encoding="utf-8"?> <service> <short>My Application</short> <description>My custom application ports</description> <port protocol="tcp" port="8080"/> <port protocol="tcp" port="8443"/> </service> EOF

# Add custom service firewall-cmd --zone=public --add-service=myapp

# List available services firewall-cmd --get-services ```

Firewalld rich rules:

```bash # Allow from specific IP range firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" service name="ssh" accept'

# Block specific IP firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.100" reject'

# Rate limit SSH connections firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" service name="ssh" limit value="3/m" accept'

# Port forwarding with masquerade firewall-cmd --zone=public --add-forward-port=port=8080:proto=tcp:toport=80:toaddr=10.0.0.5 firewall-cmd --zone=public --add-masquerade

# List rich rules firewall-cmd --zone=public --list-rich-rules ```

### 5. Fix nftables configuration

Basic nftables ruleset:

```bash #!/usr/sbin/nft -f

# Flush existing rules flush ruleset

# Create table and chain table inet filter { chain input { type filter hook input priority 0; policy drop;

# Allow established connections ct state established,related accept

# Allow loopback iif "lo" accept

# Drop invalid packets ct state invalid drop

# Allow SSH tcp dport 22 accept

# Allow HTTP/HTTPS tcp dport { 80, 443 } accept

# Allow ICMP (rate limited) icmp type echo-request limit rate 1/second accept

# Log dropped packets log prefix "NFTABLES_DROPPED: " counter drop }

chain forward { type filter hook forward priority 0; policy drop; }

chain output { type filter hook output priority 0; policy accept; } }

# NAT table table ip nat { chain prerouting { type nat hook prerouting priority -100;

# Port forwarding tcp dport 8080 dnat to 10.0.0.5:80 }

chain postrouting { type nat hook postrouting priority 100;

# Masquerade oifname "eth0" masquerade } }

# Save configuration nft list ruleset > /etc/nftables.conf ```

### 6. Fix cloud security groups

AWS Security Group rules:

```bash # Allow SSH from anywhere (not recommended for production) aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 22 \ --cidr 0.0.0.0/0

# Allow SSH from specific IP aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 22 \ --cidr 192.168.1.0/24

# Allow HTTP/HTTPS aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 80 \ --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 443 \ --cidr 0.0.0.0/0

# Allow from another security group aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 3306 \ --source-group sg-yyyyyyyyy

# Remove a rule aws ec2 revoke-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp \ --port 22 \ --cidr 0.0.0.0/0

# Security groups are stateful - return traffic automatically allowed # No need for explicit egress rules for response traffic ```

AWS Network ACL (stateless):

```bash # List NACLs aws ec2 describe-network-acls --filters "Name=vpc-id,Values=vpc-xxxxxxxxx"

# NACL rules require both inbound and outbound rules (stateless)

# Inbound rule for HTTP aws ec2 create-network-acl-entry \ --network-acl-id acl-xxxxxxxxx \ --rule-number 100 \ --protocol 6 \ --port-range From=80,To=80 \ --cidr-block 0.0.0.0/0 \ --rule-action allow \ --ingress

# Outbound rule for HTTP response (ephemeral ports) aws ec2 create-network-acl-entry \ --network-acl-id acl-xxxxxxxxx \ --rule-number 100 \ --protocol 6 \ --port-range From=1024,To=65535 \ --cidr-block 0.0.0.0/0 \ --rule-action allow \ --egress

# NACLs are evaluated in order (lowest rule number first) # First match wins ```

Azure NSG rules:

```bash # Allow SSH az network nsg rule create \ --resource-group myRG \ --nsg-name myNSG \ --name AllowSSH \ --protocol Tcp \ --priority 1000 \ --source-address-prefixes 192.168.1.0/24 \ --destination-port-range 22 \ --access Allow

# Allow HTTP/HTTPS az network nsg rule create \ --resource-group myRG \ --nsg-name myNSG \ --name AllowHTTP \ --protocol Tcp \ --priority 1001 \ --source-address-prefixes '*' \ --destination-port-ranges 80 443 \ --access Allow

# List NSG rules az network nsg rule list \ --resource-group myRG \ --nsg-name myNSG \ --output table

# Delete NSG rule az network nsg rule delete \ --resource-group myRG \ --nsg-name myNSG \ --name AllowSSH ```

GCP Firewall rules:

```bash # Allow SSH gcloud compute firewall-rules create allow-ssh \ --network default \ --allow tcp:22 \ --source-ranges 0.0.0.0/0 \ --direction INGRESS

# Allow HTTP/HTTPS gcloud compute firewall-rules create allow-http \ --network default \ --allow tcp:80,tcp:443 \ --source-ranges 0.0.0.0/0 \ --direction INGRESS

# Allow from specific tag gcloud compute firewall-rules create allow-internal \ --network default \ --allow tcp:1-65535 \ --source-tags internal \ --target-tags internal \ --direction INGRESS

# List firewall rules gcloud compute firewall-rules list

# Delete firewall rule gcloud compute firewall-rules delete allow-ssh ```

### 7. Fix connection tracking issues

Connection tracking table exhaustion:

```bash # Check current connection count cat /proc/sys/net/netfilter/nf_conntrack_count

# Check maximum cat /proc/sys/net/netfilter/nf_conntrack_max

# If count is near max, new connections will be dropped # Common symptoms: intermittent connection failures

# Increase max connections temporarily sysctl -w net.netfilter.nf_conntrack_max=262144

# Make permanent echo "net.netfilter.nf_conntrack_max=262144" >> /etc/sysctl.conf

# Reduce timeout to free entries faster sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=432000 sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30 sysctl -w net.netfilter.nf_conntrack_tcp_timeout_close_wait=30

# View connection tracking table conntrack -L

# Count by state conntrack -L | awk '{print $4}' | sort | uniq -c

# Delete stale entries conntrack -D -s 10.0.0.1 # Delete all from specific IP conntrack -D -p tcp # Delete all TCP entries conntrack -F # Flush all entries (use carefully!)

# Monitor connection tracking watch -n 1 'cat /proc/sys/net/netfilter/nf_conntrack_count' ```

### 8. Fix Windows Firewall issues

Windows Firewall PowerShell commands:

```powershell # Check firewall status Get-NetFirewallProfile | Select Name, Enabled

# List all firewall rules Get-NetFirewallRule | Select DisplayName, Enabled, Direction, Action, Profile

# List rules for specific port Get-NetFirewallRule | Where-Object { $_.Enabled -eq True } | Get-NetFirewallPortFilter | Where-Object { $_.LocalPort -eq 22 }

# Create allow rule New-NetFirewallRule -DisplayName "Allow SSH" -Direction Inbound -Protocol TCP -LocalPort 22 -Action Allow

# Create allow rule for application New-NetFirewallRule -DisplayName "Allow MyApp" -Direction Inbound -Program "C:\Apps\myapp.exe" -Action Allow

# Block specific IP New-NetFirewallRule -DisplayName "Block IP" -Direction Inbound -RemoteAddress 192.168.1.100 -Action Block

# Remove rule Remove-NetFirewallRule -DisplayName "Allow SSH"

# Enable/disable firewall profile Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled True

# Export firewall configuration Export-NetFirewallRule -Path C:\firewall-backup.xml

# Import firewall configuration Import-NetFirewallRule -Path C:\firewall-backup.xml ```

Windows Firewall GUI:

```powershell # Open firewall GUI wf.msc

# Or through Control Panel: # Control Panel > Windows Defender Firewall > Advanced Settings ```

### 9. Monitor firewall activity

Logging dropped packets:

```bash # iptables logging iptables -A INPUT -j LOG --log-prefix "FIREWALL_DROP: " --log-level 4 # View logs: tail -f /var/log/kern.log | grep FIREWALL_DROP # Or on RHEL/CentOS: tail -f /var/log/messages | grep FIREWALL_DROP

# firewalld logging firewall-cmd --set-log-denied=all

# nftables logging (already in config above) nft list ruleset | grep log

# Set up dedicated firewall log file # /etc/rsyslog.d/firewall.conf: :msg, contains, "FIREWALL_DROP" /var/log/firewall.log & stop

# Restart rsyslog systemctl restart rsyslog ```

Prometheus/Grafana monitoring:

```yaml # Install node exporter for firewall metrics # https://github.com/prometheus/node_exporter

# iptables metrics via custom exporter # https://github.com/meeDamian/iptables_exporter

docker run -d --name iptables-exporter \ -p 9425:9425 \ meedamian/iptables-exporter \ --iptables.rules="/etc/iptables/rules.v4"

# Prometheus scrape config scrape_configs: - job_name: 'iptables' static_configs: - targets: ['localhost:9425']

# Grafana alert rules groups: - name: firewall_health rules: - alert: FirewallRuleCountLow expr: changes(iptables_rules_total[1h]) > 10 for: 5m labels: severity: warning annotations: summary: "Firewall rules changing frequently" description: "{{ $value }} rule changes in the last hour"

  • alert: ConntrackTableFull
  • expr: nf_conntrack_entries / nf_conntrack_entries_limit > 0.9
  • for: 5m
  • labels:
  • severity: critical
  • annotations:
  • summary: "Connection tracking table nearly full"
  • description: "Table at {{ $value | humanizePercentage }} capacity"
  • `

### 10. Security best practices

Firewall hardening checklist:

```bash # 1. Default deny policy iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT # Or DROP with explicit allow rules

# 2. Allow only required services iptables -A INPUT -p tcp --dport 22 -j ACCEPT # SSH iptables -A INPUT -p tcp --dport 443 -j ACCEPT # HTTPS only, no HTTP

# 3. Rate limit SSH iptables -A INPUT -p tcp --dport 22 -m limit --limit 3/min -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j DROP

# 4. Block specific threats iptables -A INPUT -p tcp --dport 23 -j DROP # Telnet iptables -A INPUT -p tcp --dport 3389 -j DROP # RDP

# 5. Log suspicious activity iptables -A INPUT -p tcp --dport 22 -m limit --limit 1/min -j LOG --log-prefix "SSH_ATTEMPT: "

# 6. Use fail2ban for dynamic blocking apt install fail2ban # or yum install fail2ban

# fail2ban configuration: # /etc/fail2ban/jail.local [sshd] enabled = true bantime = 3600 findtime = 600 maxretry = 3

# 7. Regular rule audit iptables-save > /etc/iptables/backup-$(date +%Y%m%d).rules

# 8. Document all rules # Add comments in rules files or maintain separate documentation ```

Prevention

  • Use infrastructure as code (Terraform, CloudFormation) for firewall rules
  • Implement least privilege: only allow required ports and sources
  • Enable logging for denied traffic and set up alerts
  • Regular firewall rule audits (quarterly minimum)
  • Test firewall changes in staging before production
  • Use connection rate limiting for exposed services
  • Monitor connection tracking table utilization
  • Document all firewall rules with business justification
  • Implement automated security group compliance checking
  • Use network segmentation to limit blast radius
  • **Connection refused**: Service not listening or actively rejected
  • **Connection timed out**: Packet dropped (no response)
  • **No route to host**: Routing issue or host unreachable
  • **Network unreachable**: No route to network
  • **Host unreachable**: ICMP unreachable received