Introduction

VPN tunnel not establishing errors occur when two VPN endpoints fail to complete the tunnel negotiation process, preventing secure encrypted communication between networks or remote users. VPN establishment requires multiple phases: IKE (Internet Key Exchange) phase 1 for peer authentication and secure channel creation, and IKE phase 2 (IPsec) for negotiating encryption parameters for data transfer. Common causes include mismatched IKE/IPsec configuration parameters (encryption algorithms, DH groups, lifetimes), pre-shared key mismatch between peers, certificate authentication failures (expired, untrusted, wrong CN), firewall blocking UDP ports 500 (IKE) or 4500 (NAT-T), NAT traversal not enabled when NAT exists between peers, mismatched Phase 2 proxy IDs (traffic selectors), routing issues preventing IKE packets from reaching peer, MTU issues causing fragmented IKE packets, peer IP address changed but configuration not updated, and cryptographic algorithm incompatibility between vendors. The fix requires understanding IKE/IPsec negotiation, authentication methods, network connectivity requirements, and vendor-specific configuration. This guide provides production-proven troubleshooting for VPN tunnel establishment across IPsec (strongSwan, Libreswan, Cisco ASA/IOS, Palo Alto, Fortinet), OpenVPN, and WireGuard implementations.

Symptoms

  • VPN tunnel status shows "DOWN" or "Connecting"
  • IKE negotiation fails with "authentication failed"
  • Phase 1 establishes but Phase 2 fails
  • Tunnel establishes but no traffic passes
  • Intermittent tunnel establishment
  • One side shows UP, other side shows DOWN
  • IKE packets sent but no response received
  • Certificate validation errors in logs
  • NAT-T detection fails
  • Traffic selector mismatch errors

Common Causes

  • Pre-shared key mismatch between peers
  • IKE encryption/hash algorithm mismatch
  • Diffie-Hellman group mismatch
  • Certificate expired or not trusted
  • Firewall blocking UDP 500/4500
  • NAT traversal not enabled
  • Phase 2 proxy ID mismatch
  • Peer IP/hostname changed
  • Routing issues to peer
  • Vendor interoperability issues

Step-by-Step Fix

### 1. Diagnose VPN tunnel state

Check tunnel status:

```bash # strongSwan sudo ipsec status sudo ipsec statusall

# Output shows: # Security Associations in: # vpn-tunnel[1]: CONNECTING, , rekeying in 179 seconds # vpn-tunnel[1]: IKE_SA established, CHILD_SA installing

# Libreswan sudo ipsec auto --status sudo ipsec whack --trafficstatus

# Cisco ASA show crypto ike sa show crypto ipsec sa

# Palo Alto show vpn ike-sa show vpn ipsec-sa ```

Check IKE daemon logs:

```bash # strongSwan logs sudo tail -f /var/log/charon.log # Or via journalctl sudo journalctl -u strongswan -f

# Libreswan logs sudo tail -f /var/log/pluto.log sudo tail -f /var/log/secure | grep pluto

# Common log messages: # "received notify: AUTHENTICATION_FAILED" - PSK or cert issue # "no proposed cipher acceptable" - Algorithm mismatch # "packet from X:500 - ignoring non-IKE packet" - Wrong port/source # "IKE SA established" - Phase 1 successful # "IPsec SA established" - Phase 2 successful ```

Verify network connectivity:

```bash # Test reachability to peer ping <peer-public-ip>

# Test UDP port 500 (IKE) nc -uzv <peer-public-ip> 500 # Or: nmap -sU -p 500 <peer-public-ip>

# Test UDP port 4500 (NAT-T) nc -uzv <peer-public-ip> 4500

# If ports filtered, check firewall ```

### 2. Fix IKE Phase 1 configuration

Check IKE proposal matching:

```bash # strongSwan configuration # /etc/ipsec.conf

conn vpn-tunnel keyexchange=ikev2 ike=aes256-sha256-modp2048! # Breakdown: # - aes256: Encryption algorithm # - sha256: Integrity/hash algorithm # - modp2048: Diffie-Hellman group # - ! : Require this exact match (no fallback)

# Or multiple proposals (in order of preference) ike=aes256gcm16-sha384-modp3072,aes256-sha256-modp2048,aes128-sha256-modp1536

# Both peers MUST have at least one matching proposal # Common compatible settings: # - AES256 + SHA256 + DH Group 14 (modp2048) # - AES256 + SHA128 + DH Group 2 (modp1024) - legacy ```

Verify Phase 1 parameters:

```bash # Check active IKE proposals sudo ipsec statusall | grep -A10 "IKE proposal"

# strongSwan supported algorithms sudo ipsec algorithms

# Common IKE settings by vendor: # Cisco ASA: # encryption aes-256 # hash sha-256 # group 14 # lifetime 86400

# Palo Alto: # Encryption: AES-256 # Authentication: SHA-256 # DH Group: group14 # Lifetime: 28800 seconds

# Must match on both ends! ```

Fix DH group mismatch:

```bash # DH Groups and vendor equivalents: # Group 2 = modp1024 (legacy, weak) # Group 5 = modp1536 # Group 14 = modp2048 (recommended minimum) # Group 15 = modp3072 # Group 16 = modp4096 # Group 19 = ecp256 (ECDSA) # Group 20 = ecp384

# Update configuration # /etc/ipsec.conf ike=aes256-sha256-modp2048!

# Restart IKE daemon sudo ipsec restart ```

### 3. Fix authentication issues

Fix pre-shared key mismatch:

```bash # Check PSK configuration # strongSwan: /etc/ipsec.secrets : PSK "your-super-secret-key-here" # Or for specific peer: <peer-ip> : PSK "your-super-secret-key-here"

# Libreswan: /etc/ipsec.secrets @tunnel PSK "your-super-secret-key-here"

# PSK must be EXACTLY identical on both ends # Common issues: # - Trailing spaces # - Different quotes # - Copy-paste errors # - Case sensitivity

# Test PSK connectivity sudo ipsec restart sudo ipsec up vpn-tunnel ```

Fix certificate authentication:

```bash # Check certificate validity openssl x509 -in /etc/ipsec.d/certs/local.crt -noout -dates openssl x509 -in /etc/ipsec.d/certs/local.crt -noout -subject -issuer

# Verify certificate chain openssl verify -CAfile /etc/ipsec.d/cacerts/ca.crt /etc/ipsec.d/certs/local.crt

# Check certificate purpose openssl x509 -in /etc/ipsec.d/certs/local.crt -noout -text | grep -A5 "X509v3 extensions"

# Required for IPsec: # - Extended Key Usage: serverAuth, clientAuth (or ipsecEndSystem) # - Subject Alternative Name: FQDN or IP address

# Check CRL (Certificate Revocation List) openssl crl -in /etc/ipsec.d/crls/crl.pem -noout -text

# Disable CRL checking (temporarily for debugging) # /etc/ipsec.conf conn vpn-tunnel leftcert=local.crt leftca=ca.crt revocationcrl=none # Disable CRL checking ```

Configure certificate paths:

```bash # strongSwan certificate locations # /etc/ipsec.d/ # ├── cacerts/ # CA certificates # ├── certs/ # Local/end-entity certificates # ├── crls/ # Certificate revocation lists # ├── ocspcerts/ # OCSP certificates # ├── private/ # Private keys (mode 600) # └── reqs/ # Certificate requests

# Set correct permissions chmod 600 /etc/ipsec.d/private/*.key chmod 644 /etc/ipsec.d/certs/*.crt chmod 644 /etc/ipsec.d/cacerts/*.crt

# Configuration # /etc/ipsec.conf conn vpn-tunnel leftcert=local.crt leftcert2=intermediate.crt # If using chain leftca=ca.crt rightcert=peer.crt auto=add ```

### 4. Fix firewall rules for IKE

Allow IKE traffic:

```bash # IPsec requires UDP ports 500 and 4500 # Add firewall rules

# iptables sudo iptables -I INPUT -p udp --dport 500 -j ACCEPT # IKE sudo iptables -I INPUT -p udp --dport 4500 -j ACCEPT # NAT-T sudo iptables -I INPUT -p esp -j ACCEPT # ESP protocol (IP protocol 50) sudo iptables -I INPUT -p ah -j ACCEPT # AH protocol (IP protocol 51)

# Or allow from specific peer sudo iptables -I INPUT -s <peer-ip> -p udp --dport 500 -j ACCEPT sudo iptables -I INPUT -s <peer-ip> -p udp --dport 4500 -j ACCEPT

# Save rules sudo iptables-save > /etc/iptables/rules.v4 ```

Firewalld configuration:

```bash # RHEL/CentOS with firewalld sudo firewall-cmd --permanent --add-service=ipsec sudo firewall-cmd --reload

# Or add specific ports sudo firewall-cmd --permanent --add-port=500/udp sudo firewall-cmd --permanent --add-port=4500/udp sudo firewall-cmd --permanent --add-protocol=esp sudo firewall-cmd --reload

# Verify sudo firewall-cmd --list-all ```

Check cloud security groups:

```bash # AWS Security Group aws ec2 authorize-security-group-ingress \ --group-id sg-123456 \ --protocol udp --port 500 --cidr <peer-cidr>/32

aws ec2 authorize-security-group-ingress \ --group-id sg-123456 \ --protocol udp --port 4500 --cidr <peer-cidr>/32

aws ec2 authorize-security-group-ingress \ --group-id sg-123456 \ --ip-permissions IpProtocol=50,IpRanges="[{CidrIp=<peer-cidr>/32}]" # ESP

# Azure NSG az network nsg rule create \ --resource-group myRG \ --nsg-name myNSG \ --name AllowIKE \ --protocol Udp --priority 1000 \ --destination-port-ranges 500 4500 \ --source-address-prefix <peer-ip>/32 ```

### 5. Fix NAT traversal issues

Enable NAT-T:

```bash # NAT-T (NAT Traversal) encapsulates ESP in UDP port 4500 # Required when NAT exists between VPN peers

# strongSwan: NAT-T enabled by default # Verify in config # /etc/strongswan.conf charon { ike { nat_t = yes nat_keepalive = 20 seconds } }

# Check if behind NAT # /var/log/charon.log # "local NAT detection" or "remote NAT detection"

# Force NAT-T mode if needed # /etc/ipsec.conf conn vpn-tunnel encapsulation=yes # Force encapsulation forceencaps=yes # Even if no NAT detected ```

Fix NAT-related issues:

```bash # Check for NAT along path # strongSwan logs will show: # "received NAT-T (vendor) me" # "found peer behind NAT"

# If one peer is behind NAT: # - Ensure UDP 4500 is allowed # - Enable NAT keepalive # - Use encapsulation=yes

# For mobile clients (changing IPs): # /etc/ipsec.conf conn vpn-tunnel left=%any # Accept any local IP right=%any # Accept any remote IP leftsubnet=192.168.1.0/24 rightsubnet=10.0.0.0/24 ```

### 6. Fix Phase 2 (IPsec) configuration

Configure Phase 2 proposals:

```bash # Phase 2 (CHILD_SA) configuration # /etc/ipsec.conf

conn vpn-tunnel # Phase 1 (IKE) ike=aes256-sha256-modp2048!

# Phase 2 (IPsec) esp=aes256gcm16,aes256-sha256,aes128-sha256!

# ESP breakdown: # - aes256gcm16: AES-256-GCM with 16-byte ICV (combined mode) # - aes256-sha256: AES-256-CBC + HMAC-SHA256 # - aes128-sha256: AES-128-CBC + HMAC-SHA256

# PFS (Perfect Forward Secrecy) dpdaction=clear dpddelay=300s rekey=yes reauth=yes

# Key exchange lifetime ikelifetime=28800s # 8 hours lifetime=3600s # 1 hour ```

Fix proxy ID mismatch:

```bash # Proxy IDs (traffic selectors) define what traffic goes through tunnel # MUST match on both ends (or be compatible subsets)

# /etc/ipsec.conf conn vpn-tunnel leftsubnet=192.168.1.0/24 # Local network rightsubnet=10.0.0.0/24 # Remote network

# Common issues: # - leftsubnet on peer A != rightsubnet on peer B # - Subnet mask mismatch (/24 vs /25) # - Overlapping subnets

# For host-to-host: left=192.168.1.100 right=10.0.0.100

# For multiple subnets: leftsubnet=192.168.1.0/24,192.168.2.0/24 rightsubnet=10.0.0.0/24,10.0.1.0/24

# Verify proxy IDs sudo ipsec statusall | grep -A5 "CHILD_SA" ```

### 7. Fix routing for VPN

Add routes for VPN traffic:

```bash # Check current routes ip route show ip route get <peer-ip>

# Ensure route to peer public IP exists # Default gateway should handle this for internet peers

# For static routing: sudo ip route add <peer-public-ip>/32 via <gateway-ip>

# Add routes for remote network (after tunnel up) # strongSwan can install these automatically # /etc/ipsec.conf conn vpn-tunnel leftsubnet=192.168.1.0/24 rightsubnet=10.0.0.0/24 leftupdown=/etc/ipsec.d/updown # Installs routes automatically

# Manual route addition sudo ip route add 10.0.0.0/24 via <vpn-gateway-ip> ```

Check for routing conflicts:

```bash # Check for overlapping subnets # Local and remote subnets MUST NOT overlap

# Local subnets ip addr show | grep inet

# If overlap exists: # - Renumber one network # - Use NAT on one side # /etc/ipsec.conf conn vpn-tunnel leftsourceip=192.168.100.0/24 # NAT local subnet ```

### 8. Fix MTU and fragmentation issues

Check MTU along path:

```bash # Test MTU to peer ping -M do -s 1472 <peer-ip> # 1472 + 28 = 1500

# If fails, try smaller ping -M do -s 1400 <peer-ip>

# Find path MTU tracepath <peer-ip>

# IPsec overhead: # - ESP header: 10-73 bytes # - ESP trailer: varies # - Total overhead: ~50-70 bytes

# Recommended: Reduce MTU on tunnel interface ip link set dev vpn0 mtu 1400 ```

Configure MTU/MSS clamping:

```bash # iptables MSS clamping sudo iptables -A FORWARD -m policy --pol ipsec --dir in \ -p tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 \ -j TCPMSS --set-mss 1360

sudo iptables -A FORWARD -m policy --pol ipsec --dir out \ -p tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 \ -j TCPMSS --set-mss 1360

# Or via strongSwan # /etc/strongswan.conf charon { kernel-netlink { install_policy = yes mss_clamp = 1360 } } ```

### 9. Debug IKE negotiation

Enable debug logging:

```bash # strongSwan debug sudo ipsec restart sudo charon -l stderr -d 3 # Debug level 3

# Or via config # /etc/strongswan.conf charon { stdout { default = 3 # Debug level 0-4 } }

# Debug levels: # 0 = Basic # 1 = Control # 2 = More verbose # 3 = Debug # 4 = Full packet dump ```

Capture IKE packets:

```bash # Capture IKE negotiation sudo tcpdump -i any -n -s 0 -w /tmp/ike.pcap \ port 500 or port 4500

# Or with verbose output sudo tcpdump -i any -n -vvv -s 0 \ 'udp port 500 or udp port 4500'

# Analyze with wireshark # Look for: # - IKE_SA_INIT exchange (first 2 packets) # - IKE_AUTH exchange (next packets) # - Notify payloads (error messages) # - Delete payloads (SA teardown) ```

Test tunnel manually:

```bash # strongSwan sudo ipsec init # Initialize daemon sudo ipsec up vpn-tunnel # Bring tunnel up

# Check status sudo ipsec status

# Bring tunnel down sudo ipsec down vpn-tunnel

# Purge and re-establish sudo ipsec purge sudo ipsec restart ```

### 10. Vendor-specific troubleshooting

Cisco ASA IPsec debug:

```bash # Enable debugging debug crypto ike 255 debug crypto ipsec 255

# View debug output terminal monitor

# Check SA status show crypto isakmp sa show crypto ipsec sa

# Clear and rebuild clear crypto isakmp clear crypto ipsec sa ```

Palo Alto VPN debug:

```bash # CLI commands show vpn ike-sa gateway <gateway-name> show vpn ipsec-sa tunnel <tunnel-name>

# Debug commands debug vpn ike <level> debug vpn kmd <level>

# Packet capture test vpn flow vpn-type ike gateway <gateway-name> ```

OpenVPN troubleshooting:

```bash # Enable verbose logging # /etc/openvpn/server.conf verb 4 # Increase for more detail

# Check status sudo systemctl status openvpn@server sudo journalctl -u openvpn@server -f

# Test configuration sudo openvpn --config /etc/openvpn/server.conf --test-crypto

# Check certificate sudo openssl verify -CAfile /etc/openvpn/easy-rsa/pki/ca.crt \ /etc/openvpn/easy-rsa/pki/issued/server.crt ```

Prevention

  • Document IKE/IPsec parameters for all VPN peers
  • Use standardized encryption profiles across organization
  • Monitor tunnel status with automated alerting
  • Test certificate renewal before expiration
  • Maintain backup tunnel paths
  • Regular audit of firewall rules for VPN ports
  • Use configuration management for consistency
  • Document NAT-T requirements for each peer
  • **AUTHENTICATION_FAILED**: PSK mismatch or certificate issue
  • **NO_PROPOS_CHOSEN**: Algorithm mismatch between peers
  • **TIMEOUT**: Firewall blocking IKE packets or peer unreachable
  • **TS_UNACCEPTABLE**: Phase 2 proxy ID/traffic selector mismatch
  • **INVALID_KE_PAYLOAD**: DH group mismatch or corruption