IPsec troubleshooting: why your VPN is not connecting, phase by phase

Most IPsec failures are a mismatch between two peers. Work through the phases in order, read the right log, and the cause usually shows itself within minutes.

Branch firewall198.51.100.20IPsec VPN server203.0.113.101. ReachabilityUDP 500 / 4500 and ESP open2. IKE SAestablished, proposals match3. Child SATS_UNACCEPTABLE: subnets differ4. Trafficroutes, NAT exemption, MTU / MSS

The short answer

IPsec troubleshooting goes fastest when you work through the phases in order and stop at the first one that fails: can the peers reach each other, does the IKE SA come up, does the child SA come up, and does traffic actually flow through it. Each phase has its own typical errors, and the log on the responder usually says more than the initiator's.

The examples use the shared test network: the IPsec server is 203.0.113.10 with 10.10.0.0/16 behind it, the branch is 198.51.100.20 with LAN 192.168.10.0/24.

IPsec troubleshooting commands for each platform

Run these on both peers. They show whether an IKE SA and child SAs exist, and the debug output says why a negotiation failed.

strongSwan (swanctl)

swanctl --list-sas                    # IKE and child SAs, bytes, rekey times
swanctl --list-conns                  # what is configured
swanctl --initiate --child branch-lan # bring it up and print the negotiation
journalctl -u strongswan -f           # live log
ip -s xfrm state                      # kernel SAs with packet counters
tcpdump -ni eth0 "host 198.51.100.20 and \
  (udp port 500 or udp port 4500 or ip proto 50)"

Cisco IOS XE

show crypto ikev2 sa detailed
show crypto ipsec sa peer 198.51.100.20
show crypto session remote 198.51.100.20 detail
debug crypto ikev2
debug crypto ipsec
undebug all

FortiGate (FortiOS 7.4)

get vpn ipsec tunnel summary
diagnose vpn ike gateway list name 
diagnose vpn tunnel list name 
diagnose vpn ike log filter rem-addr4 203.0.113.10
diagnose debug application ike -1
diagnose debug enable
diagnose debug disable                # when done

Palo Alto (PAN-OS 11)

show vpn ike-sa gateway 
show vpn ipsec-sa tunnel 
show vpn flow
test vpn ike-sa gateway 
test vpn ipsec-sa tunnel 
less mp-log ikemgr.log

MikroTik (RouterOS 7)

/ip ipsec active-peers print
/ip ipsec installed-sa print
/ip ipsec policy print
/system logging add topics=ipsec,!packet
/log print where topics~"ipsec"

Phase 0: reachability

If the log on one side shows nothing at all when the other side initiates, the packets are not arriving. IKE needs UDP 500; once NAT is detected both peers switch to UDP 4500 and ESP travels inside it (NAT-T). Without NAT, data uses ESP, IP protocol 50, which many firewalls don't allow unless you name it explicitly.

  • Capture on the receiving side (tcpdump, diagnose sniffer packet, the PAN-OS packet capture). Packets arriving but no log entry: a local firewall drops them, or the remote address doesn't match any configured peer.
  • Check network ACLs and any firewall in front of either peer.
  • IKE_AUTH with certificates can exceed 1500 bytes. If IKE_SA_INIT succeeds and IKE_AUTH times out, fragments are being dropped; enable IKEv2 fragmentation (RFC 7383) on both sides if supported.
  • A peer behind NAT must use the NAT's public address as the remote address on the other side, and its ID usually stays its private address unless you set it explicitly.

Phase 1: IKE SA failures

NO_PROPOSAL_CHOSEN

The peers share no IKE proposal: encryption, integrity/PRF or DH group differ. Line up the exact algorithms on both sides, for example AES-256, SHA-256 and group 20 (ECP-384). Watch for a peer that defaults to group 2 or 14 when you expected 20, and for a GCM proposal on one side while the other only offers CBC with an HMAC. Also check the IKE version: an IKEv1 peer and an IKEv2 peer never agree.

AUTHENTICATION_FAILED

The proposals matched, but authentication didn't. With PSKs, strongSwan logs a line saying the shared key was tried but the MAC mismatched: the keys differ. Copy-paste errors and trailing spaces are common, so set the key again on both sides. With certificates, check the chain, the validity dates and that the CA is trusted on the verifying side.

Identity mismatch

Each side sends an ID (an IP address, FQDN or certificate DN) and the other side uses it to pick the configuration and the key. If 203.0.113.10 sits behind 1:1 NAT and sends its private address as its ID, the branch finds no match. On strongSwan this shows as no matching peer config found; on other platforms it often looks like a generic authentication failure. Set local and remote IDs explicitly on both peers.

Phase 2: child SA failures

TS_UNACCEPTABLE: traffic selector mismatch

The IKE SA is up but no child SA. The selectors must mirror each other exactly: 10.10.0.0/16 ↔ 192.168.10.0/24 on the server must be 192.168.10.0/24 ↔ 10.10.0.0/16 on the branch. A /24 on one side and a /16 on the other, a forgotten second subnet, or a host address instead of a network all cause it. Some vendors create one child SA per subnet pair while others combine them, which also leads to mismatches; route-based tunnels with 0.0.0.0/0 selectors avoid this class of problem.

PFS mismatch

In IKEv2 the first child SA is created inside IKE_AUTH without its own key exchange, so a PFS mismatch often does not show at first. The tunnel works for an hour, then fails at the first child SA rekey with NO_PROPOSAL_CHOSEN. If a tunnel drops at exactly the child SA lifetime, compare the PFS group on both sides.

A tunnel that fails at a regular interval (every hour, every eight hours) is almost always a rekey problem: compare PFS settings and the proposals used for rekeying, not just the initial negotiation.

Tunnel up, but no traffic

Both SAs are established, yet pings fail. Look at the SA counters: are packets being encrypted on one side and decrypted on the other?

  • Routing. Hosts in 10.10.0.0/16 need a route to 192.168.10.0/24 via the IPsec gateway, and the branch hosts the reverse. In a route-based setup, the route must point at the tunnel interface.
  • NAT exemption. If the gateway also NATs to the internet, traffic for the remote subnet must be excluded from that NAT, or its source address no longer matches the selector and it leaves unencrypted. The usual sign: encrypt counters stay at zero.
  • Firewall policy. Many firewalls need an explicit policy between the LAN and the tunnel in each direction. FortiGate and PAN-OS will not pass tunnel traffic without one.
  • Testing from the wrong address. A ping from the gateway's public IP doesn't match the selectors. Test from a host inside the protected subnets.

MTU and TCP MSS clamping

Pings work, small requests work, but file transfers, web pages and SSH sessions stall. That is an MTU problem: encryption adds overhead, full-size packets no longer fit, and the ICMP messages that would make path MTU discovery work are often filtered.

The overhead for a tunnel-mode ESP packet:

PartAES-256-GCMAES-256-CBC + SHA-256
New outer IP header2020
UDP header (NAT-T only)88
ESP header (SPI, sequence)88
IV816
Padding + pad length + next header2 to 52 to 17
Integrity check value1616
Worst case65 bytes85 bytes

On a 1500-byte path the inner packet can be at most about 1415 bytes in the worst case, and less again if the path runs over PPPoE (1492). A tunnel MTU of 1400 leaves room for all of these. TCP MSS is the MTU minus 20 bytes of IP header and 20 bytes of TCP header, so 1400 − 40 = 1360. Clamping rewrites the MSS in TCP SYN and SYN-ACK packets crossing the gateway, so both ends send segments that fit without relying on fragmentation or ICMP.

On Cisco IOS XE, use ip tcp adjust-mss 1360 on the tunnel or inside interface. On Linux with nftables, in the forward chain:

tcp flags syn / syn,rst tcp option maxseg size set 1360

FortiGate, PAN-OS and MikroTik all have an MSS adjustment (in the firewall policy, the interface settings and a mangle rule respectively); check your version's documentation for the exact field. Clamp on at least one gateway in the path, applied to traffic in both directions.

Flapping tunnels

  • DPD too aggressive. Dead peer detection with a short interval and few retries tears down a tunnel on a few seconds of packet loss. Use an interval of around 30 seconds, and if both peers are set to restart on DPD, consider letting only the initiator restart.
  • Lifetimes. In IKEv2 lifetimes are not negotiated; each side rekeys on its own timer. Matching them (for example 28800 s for the IKE SA and 3600 s for child SAs) still makes behaviour predictable. In IKEv1, mismatched lifetimes can cause failures on some platforms.
  • Rekey collisions. If both peers start a rekey at the same moment, IKEv2 resolves the collision, but some implementations end up with duplicate or orphaned SAs. Randomised rekey timing (strongSwan adds jitter by default) and making one side the clear initiator help.
  • Duplicate tunnels. Two IKE SAs to the same peer, each with child SAs, can cause traffic to use one SA while the other side expects the other. Check --list-sas or its equivalent for duplicates.

Decision table

SymptomLikely causeCheck first
Nothing in the responder's logUDP 500/4500 blocked, wrong peer addressPacket capture, security groups
IKE_SA_INIT ok, IKE_AUTH times outFragmented IKE packets droppedIKEv2 fragmentation, path MTU
NO_PROPOSAL_CHOSEN during setupIKE proposals or IKE version differEncryption, integrity, DH group on both sides
AUTHENTICATION_FAILEDWrong PSK, ID mismatch, untrusted certificateLocal and remote IDs, key, CA
TS_UNACCEPTABLESubnets don't mirror each otherLocal and remote subnets on both sides
Drops at the child SA lifetimePFS group mismatchPFS setting on both sides
SAs up, encrypt counter at zeroRouting or missing NAT exemptionRoutes, NAT rules
SAs up, decrypt counter rises, no repliesFirewall policy or return route on the far sidePolicies, routes behind the peer
Ping works, TCP stallsMTUMSS clamping to 1360
Tunnel up and down every few minutesDPD, duplicate SAs, rekey collisionsDPD timers, list of SAs

Getting ahead of it

Most of these failures are configuration mismatches between two devices configured by hand. The IPsec VPN server setup guide and the per-vendor examples (Cisco, FortiGate, Palo Alto, MikroTik, strongSwan) all use the same values, so you can compare line by line.

On TunnelYard, guided tunnel setup with presets per remote device and peer config export reduce the chance of mismatched settings, and live tunnel status, traffic graphs and per-tunnel logs show which phase failed. Every plan sends an email alert when a tunnel has been down for 2 minutes; Business adds webhooks, and Enterprise adds Slack and Teams. See features and use cases.

Bottom line

Work phase by phase, read the responder's log, and compare settings on both sides rather than trusting either one. When an IPsec VPN is not connecting, the cause is almost always one of the ten rows in the table above.

FAQ

IPsec troubleshooting: common questions

What does NO_PROPOSAL_CHOSEN mean in IPsec?

The peers share no matching proposal: encryption, integrity/PRF or DH group differ, or one side uses IKEv1 and the other IKEv2. Line up the exact algorithms on both sides, for example AES-256, SHA-256 and group 20. If it appears only after a tunnel has worked for about an hour, it is usually a PFS mismatch at the first child SA rekey.

How do I fix IPsec AUTHENTICATION_FAILED errors?

The proposals matched but authentication did not. With pre-shared keys the keys differ, often through copy-paste errors or trailing spaces, so set the key again on both sides. Also check the IDs: behind 1:1 NAT a peer that sends its private address as its ID finds no match. With certificates, check the chain, validity dates and that the CA is trusted.

What causes TS_UNACCEPTABLE in IKEv2 phase 2?

TS_UNACCEPTABLE means the traffic selectors do not mirror each other. 10.10.0.0/16 to 192.168.10.0/24 on one peer must be exactly 192.168.10.0/24 to 10.10.0.0/16 on the other. A /24 on one side and /16 on the other, a forgotten second subnet or a host address instead of a network all cause it. Route-based tunnels with 0.0.0.0/0 selectors avoid it.

Why is my IPsec tunnel up but no traffic passes?

Check the SA counters first. If the encrypt counter stays at zero, routing or a missing NAT exemption is the usual cause. If decrypt counters rise but there are no replies, look at the firewall policy or return route on the far side; FortiGate and PAN-OS need explicit policies in each direction. Also test from a host inside the protected subnets, not the gateway's public IP.

What MSS should I clamp to for an IPsec VPN?

Clamp TCP MSS to 1360. ESP tunnel-mode overhead is up to 65 bytes with AES-256-GCM and 85 bytes with AES-256-CBC and SHA-256, so a tunnel MTU of 1400 fits a 1500-byte path, and 1400 minus 40 bytes of IP and TCP headers gives 1360. On Linux use tcp option maxseg size set 1360; on Cisco IOS XE, ip tcp adjust-mss 1360.

Why does my IPsec tunnel keep dropping and reconnecting?

A flapping tunnel usually comes from dead peer detection that is too aggressive, duplicate IKE SAs to the same peer, or rekey collisions. Use a DPD interval of around 30 seconds and consider letting only the initiator restart. Drops at exactly the child SA lifetime point to a PFS mismatch. TunnelYard shows live tunnel status and per-tunnel logs, so you can see which phase failed.