What is IPsec? How IPsec VPN tunnels work
IPsec encrypts traffic between networks at the IP layer. Here is how IKE and ESP build a tunnel, which ports it needs, and where IPsec VPNs usually go wrong.
The short answer
IPsec (Internet Protocol Security) is a suite of protocols that encrypts and authenticates IP packets between two devices. It works at the network layer, so it protects all traffic between two networks at once, whatever the application: file shares, databases, VoIP, monitoring. That makes it the standard way to build a site-to-site VPN between offices, branches and data centers.
An IPsec VPN has two jobs. First the two peers authenticate each other and agree on keys with IKE (Internet Key Exchange). Then they encrypt the actual traffic with ESP (Encapsulating Security Payload). The architecture is defined in RFC 4301, and every major firewall and router implements it, which is why an IPsec tunnel can connect a FortiGate in one office to a MikroTik in another.
The building blocks of the IPsec protocol
IPsec is not one protocol but a small family. You will see these names in every configuration screen:
- IKE (Internet Key Exchange). The control protocol. The peers use it to authenticate each other, negotiate algorithms and derive keys. Use IKEv2, defined in RFC 7296; IKEv1 is deprecated.
- ESP (Encapsulating Security Payload). The data protocol. It encrypts each packet and adds an integrity check so tampering is detected (RFC 4303).
- AH (Authentication Header). Integrity without encryption. It breaks behind NAT and is rarely used for VPNs today.
- Security associations (SAs). The agreed set of keys and algorithms for one direction of traffic. A working tunnel has one IKE SA and at least one pair of child SAs (IPsec SAs).
- Traffic selectors. The source and destination subnets an SA protects, for example 192.168.10.0/24 to 10.10.0.0/16. Older tools call them proxy IDs or encryption domains.
How an IPsec VPN tunnel comes up
With IKEv2, a tunnel is built with just four messages, in two exchanges. Older guides call them phase 1 and phase 2, and the names are still useful when you troubleshoot.
Phase 1: the IKE SA
In the IKE_SA_INIT exchange, the peers agree on an IKE proposal (encryption, integrity or PRF, and a Diffie-Hellman group) and run a Diffie-Hellman key exchange. Both sides now share a secret that was never sent over the network. In IKE_AUTH, which is already encrypted, each peer proves who it is with a pre-shared key or a certificate and sends its identity.
Phase 2: the child SAs
Inside the same IKE_AUTH exchange, the peers also create the first child SA: they agree on an ESP proposal and on the traffic selectors. From this point, packets that match the selectors are encrypted with ESP and sent to the peer. Further child SAs, and every rekey, use a short CREATE_CHILD_SA exchange, optionally with a fresh Diffie-Hellman exchange for perfect forward secrecy (PFS).
Both peers must agree on everything: IKE version, proposals, identities, keys and traffic selectors. Almost every IPsec VPN that won't connect comes down to one setting that differs between the two ends.
Tunnel mode and transport mode
ESP can protect packets in two ways:
- Tunnel mode encrypts the whole original IP packet, headers included, and wraps it in a new IP header between the two gateways. The networks behind each gateway stay hidden. This is what every site-to-site VPN uses.
- Transport mode encrypts only the payload and keeps the original IP header. It is used between two hosts, or underneath another tunnel protocol such as GRE or L2TP.
A packet from 192.168.10.25 in a branch to 10.10.4.7 in the head office leaves the branch firewall as an ESP packet from the branch's public address, 198.51.100.20, to the gateway's public address, 203.0.113.10. Anyone in between sees two public addresses and encrypted bytes, nothing more.
IPsec ports and protocols
Firewalls in front of either peer must allow three things. This is the most common reason a new tunnel shows nothing at all in the logs.
| Traffic | Protocol and port | Used for |
|---|---|---|
| IKE | UDP 500 | Starting the negotiation |
| NAT traversal | UDP 4500 | IKE and ESP once NAT is detected |
| ESP | IP protocol 50 | Encrypted data when there is no NAT |
IKEv2 detects NAT between the peers automatically. When it finds one, both sides move to UDP 4500 and carry ESP inside UDP (RFC 3948), because many NAT devices cannot handle raw ESP. A packet capture filtered on these three is the first thing to run when a tunnel is silent:
tcpdump -ni eth0 "host 198.51.100.20 and (udp port 500 or udp port 4500 or ip proto 50)"
IPsec encryption: what actually protects the data
An IPsec proposal names the algorithms for each job. For new tunnels, a good baseline is AES-256 for encryption, SHA-256 or stronger for integrity and the PRF, and an elliptic-curve Diffie-Hellman group such as group 19 or 20, or group 14 where a peer supports nothing newer. Retire DES, 3DES, MD5, SHA-1 and DH groups 1, 2 and 5. Our guide to choosing IPsec proposals covers each algorithm and how to line them up between vendors.
Keys don't live forever. The IKE SA and the child SAs each have a lifetime, commonly hours for the IKE SA and about an hour for child SAs, after which the peers rekey without dropping traffic. With PFS enabled, each rekey runs a new Diffie-Hellman exchange, so one compromised key can't decrypt traffic from other periods.
How IPsec decides what goes into the tunnel
There are two ways to tell a gateway which traffic to encrypt. A policy-based VPN matches packets against the traffic selectors directly: if a packet goes from 192.168.10.0/24 to 10.10.0.0/16, it is encrypted. A route-based VPN creates a tunnel interface and uses the routing table instead: whatever is routed into the interface is encrypted. Both are standard IPsec and interoperate with care; see policy-based vs route-based IPsec for how to choose.
What IPsec is used for
- Connecting offices and branches. Each site's firewall builds a tunnel to the others, or to a central gateway, so the networks reach each other as if they were one.
- Giving an office a static IP. The office firewall sends its internet traffic through a tunnel to a gateway with a fixed address, so SaaS apps, banks and partners can allow-list one IP. See static IP for SaaS allow-lists.
- Linking OT and IoT sites, where remote equipment needs a fixed, encrypted path back to a control network.
Where IPsec VPNs go wrong
IPsec itself is mature and well understood. The difficulty is that two devices, often from different vendors and run by different teams, must be configured to match exactly. The usual problems are mismatched proposals, wrong pre-shared keys or identities, traffic selectors that don't mirror each other, blocked UDP 500/4500, missing NAT exemptions and MTU issues once the tunnel is up. The IPsec troubleshooting guide walks through each phase with the commands to check it, and the IPsec VPN configuration examples show matching settings for six platforms.
TunnelYard gives you a dedicated IPsec gateway with its own static public IPv4 address in the city you choose; your office's internet traffic leaves from that IP, so services can allow-list it. It runs IKEv2 with AES-256 presets, guided tunnel setup suggests settings that match the device on the other end, and live tunnel status and logs show which phase failed. See features and pricing.
Bottom line
IPsec encrypts traffic between networks at the IP layer. IKE authenticates the peers and agrees on keys, ESP encrypts the packets, and both ends must match on every setting. Get the basics right (IKEv2, modern proposals, UDP 500 and 4500 open, mirrored subnets) and an IPsec VPN is one of the most dependable ways to connect sites.