IPsec VPN server setup: choices and a strongSwan walkthrough

Hosting, IKEv2, policy- or route-based, keys or certificates, proposals: the choices first, then a working strongSwan server with firewall, NAT exemption and checks.

Head office198.51.100.20Branch198.51.100.30Warehouse198.51.100.40Data center10.30.0.0/16VPN serverstrongSwanIKEv2 · UDP 500 / 4500 · ESP

The short answer

An IPsec VPN server is a machine with a static public IP address that runs an IKE daemon and terminates IPsec tunnels from offices, branches and data centers. For a new setup, use IKEv2, give the server its own static IP, pick modern proposals (AES-256, SHA-256, DH group 20), and decide early whether each tunnel is policy-based or route-based.

This guide covers the decisions first, then a complete IPsec VPN server setup with strongSwan on Ubuntu 24.04 for one site-to-site peer: install, swanctl.conf, forwarding, firewall, NAT exemption and verification.

What an IPsec VPN server does

IPsec itself is two things. IKE (Internet Key Exchange, RFC 7296) authenticates the two peers and agrees on keys over UDP 500, moving to UDP 4500 when NAT is detected. ESP (IP protocol 50) then carries the encrypted traffic, or ESP inside UDP 4500 when one side is behind NAT.

A server running IPsec usually acts as a hub. Each peer gets an IKE SA (the control channel) and one or more child SAs (the data channels). Traffic between two sites can be routed through the hub rather than meshing every site with every other site.

Decisions before you build

Where to host it

Options are a rented virtual server, a server in your own data centre, or a managed service. What matters most is a static public IPv4 address that you control: peers pin it in their configuration, and changing it means touching every remote device. Some hosting providers put the public address 1:1 NAT in front of a private interface address. That works fine with NAT-T, but you must set the IKE identity to the public IP explicitly (shown below).

IKEv2, not IKEv1

IKEv2 is faster to negotiate, has NAT traversal and liveness checks built in, and is what every current device supports. Keep IKEv1 only for a peer that cannot do anything else. See IKEv1 vs IKEv2 for the details.

Policy-based or route-based

A policy-based tunnel encrypts traffic that matches a pair of subnets (the traffic selectors, here 10.10.0.0/16 to 192.168.10.0/24). It is simple and interoperates with almost everything. A route-based tunnel uses a virtual tunnel interface (VTI) with selectors of 0.0.0.0/0, and the routing table decides what goes into it. Route-based is the better fit for many subnets, and for dynamic routing such as BGP. This guide builds a policy-based tunnel, because it needs nothing beyond strongSwan and the kernel.

Pre-shared key or certificates

For a handful of site-to-site peers, a long random pre-shared key per peer is acceptable and is what most firewalls and routers expect. X.509 certificates scale better and avoid sharing secrets. Never reuse one PSK across peers.

Proposals

Both peers must agree on at least one proposal for the IKE SA and one for the child SA. A sensible baseline for new tunnels:

SettingIKE SA (phase 1)Child SA (phase 2)
EncryptionAES-256 (CBC or GCM)AES-256-GCM
Integrity / PRFSHA-256Built into GCM
DH group20 (ECP-384)20 for PFS
Lifetime28800 s3600 s

If a peer cannot do group 20, group 14 (2048-bit MODP) or group 19 (ECP-256) are reasonable fallbacks. Avoid group 2 and 5, 3DES and SHA-1.

IPsec VPN server setup with strongSwan on Ubuntu 24.04

The example network: the server has public IP 203.0.113.10 and the networks behind it are 10.10.0.0/16. The branch device has public IP 198.51.100.20 and LAN 192.168.10.0/24.

1. Install strongSwan

Use the swanctl interface and the systemd-native daemon, not the legacy ipsec.conf starter:

sudo apt update
sudo apt install charon-systemd strongswan-swanctl
sudo systemctl enable --now strongswan

2. Write the connection

Example configuration for the strongSwan server, in /etc/swanctl/conf.d/branch.conf:

connections {
  branch {
    version = 2
    local_addrs = 203.0.113.10   # or the private interface IP behind 1:1 NAT
    remote_addrs = 198.51.100.20
    proposals = aes256-sha256-ecp384
    rekey_time = 28800s
    dpd_delay = 30s
    local {
      auth = psk
      id = 203.0.113.10          # always the public IP
    }
    remote {
      auth = psk
      id = 198.51.100.20
    }
    children {
      branch-lan {
        local_ts = 10.10.0.0/16
        remote_ts = 192.168.10.0/24
        esp_proposals = aes256gcm16-ecp384
        rekey_time = 3600s
        dpd_action = restart
        start_action = start
      }
    }
  }
}

secrets {
  ike-branch {
    id = 198.51.100.20
    secret = ""
  }
}

Generate the pre-shared key with something like openssl rand -base64 48, share it with the other side over a separate channel, and keep the file readable by root only (chmod 600).

A few notes on the settings. ecp384 is DH group 20; adding it to esp_proposals turns on PFS for child SA rekeys. dpd_delay sends a liveness check after 30 seconds without traffic, and dpd_action = restart re-establishes the tunnel if the peer stops answering. start_action = start makes the server initiate on load; use trap instead if you want the tunnel to come up on the first matching packet. Load it with:

sudo swanctl --load-all

3. Enable forwarding

The server must route packets between its LAN side and the tunnel. Create /etc/sysctl.d/99-ipsec.conf and apply it with sudo sysctl --system:

net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0

Hosts in 10.10.0.0/16 also need a route to 192.168.10.0/24 via the server, unless the server is already their default gateway.

4. Firewall: UDP 500, UDP 4500 and ESP

Example configuration for /etc/nftables.conf. It allows IKE and ESP from the peer only, forwards traffic between the two subnets, clamps TCP MSS, and exempts tunnel traffic from NAT. Replace eth0 with your public-facing interface (often ens5 on EC2), restrict SSH to your admin addresses, and disable ufw if you use this file instead.

#!/usr/sbin/nft -f
flush ruleset

table inet filter {
  chain input {
    type filter hook input priority filter; policy drop;
    ct state established,related accept
    iif "lo" accept
    meta l4proto { icmp, ipv6-icmp } accept
    tcp dport 22 accept                      # restrict to admin IPs
    ip saddr 198.51.100.20 udp dport { 500, 4500 } accept
    ip saddr 198.51.100.20 meta l4proto esp accept
  }
  chain forward {
    type filter hook forward priority filter; policy drop;
    tcp flags syn / syn,rst tcp option maxseg size set 1360
    ct state established,related accept
    ip saddr 192.168.10.0/24 ip daddr 10.10.0.0/16 meta ipsec exists accept
    ip saddr 10.10.0.0/16 ip daddr 192.168.10.0/24 accept
  }
}

table ip nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 10.10.0.0/16 ip daddr 192.168.10.0/24 accept   # NAT exemption
    oifname "eth0" ip saddr 10.10.0.0/16 masquerade
  }
}

Apply it with sudo nft -f /etc/nftables.conf and sudo systemctl enable nftables. If the server does not NAT the LAN to the internet, drop the masquerade line; the exemption rule is then harmless. If it does, the exemption is essential: without it, traffic for the branch gets its source rewritten to the public IP, no longer matches the 10.10.0.0/16 selector, and leaves unencrypted. The MSS value of 1360 is explained in the troubleshooting guide. If your hosting provider filters traffic in front of the server, open the same ports there as well.

5. Configure the other side

The branch needs the mirror image: local and remote IDs swapped, local and remote subnets swapped, the same proposals and the same key. Vendor-specific examples are in the guides for Cisco, FortiGate, Palo Alto, MikroTik, pfSense and strongSwan, with an overview in IPsec VPN config examples.

Verify the tunnel

sudo swanctl --list-conns        # what is loaded
sudo swanctl --list-sas          # IKE SA and child SA state, bytes in/out
sudo swanctl --initiate --child branch-lan
sudo ip xfrm policy              # kernel policies for the selectors
sudo ip xfrm state               # kernel SAs (SPIs, algorithms)
sudo journalctl -u strongswan -f # live log
ping -I 10.10.0.1 192.168.10.1   # from an address inside the local selector

In --list-sas, look for the IKE SA in state ESTABLISHED and the child SA branch-lan in state INSTALLED, with the expected selectors and counters that increase while you ping. Ping from an address inside 10.10.0.0/16 (the -I address must exist on the server or use a host behind it): a ping from the public IP does not match the selectors and will not be encrypted.

Common problems

  • Nothing in the log when the peer initiates: UDP 500/4500 is blocked by a security group, the local firewall or an upstream device.
  • NO_PROPOSAL_CHOSEN: the proposals differ. Compare encryption, integrity, DH group and PFS on both sides.
  • AUTHENTICATION_FAILED: wrong key, or the IDs do not match. Behind 1:1 NAT, the server must still use its public IP as its ID.
  • TS_UNACCEPTABLE: the subnets are not exact mirror images on both peers.
  • Tunnel up, no traffic: missing return routes, a missing NAT exemption, a forward-chain drop, or MTU problems.

Each of these has its own section, with commands for every major vendor, in IPsec troubleshooting.

Running it managed instead

The setup above is a good way to understand IPsec, and it works. Running it long term also means patching, certificate renewals, firewall changes for every new peer, and watching logs for tunnels that quietly went down.

TunnelYard runs the same kind of server for you: a dedicated IPsec server with its own static public IP, ready in 10–15 minutes, with IKEv2 policy-based tunnels and guided setup with presets for each remote device. It can export the matching peer configuration for common firewalls and routers, and shows live tunnel status and per-tunnel logs. See the features, use cases and pricing.

Bottom line

A working IPsec VPN server needs a static IP, IKEv2, matching proposals and selectors on both sides, forwarding, three firewall openings and a NAT exemption. Get those right and the tunnel comes up; the rest is routing.

FAQ

IPsec VPN server setup: common questions

Which ports does an IPsec VPN server need open?

Open UDP 500 for IKE, UDP 4500 for NAT traversal, and ESP (IP protocol 50) for encrypted traffic when no NAT is involved. Allow them from each peer's address in the server's firewall and in any firewall in front of it. If the log shows nothing when a peer initiates, one of these openings is usually missing.

Does an IPsec VPN server need a static public IP address?

Yes, in practice. Peers pin the server's public IPv4 address in their configuration, so changing it means touching every remote device. If the public address is 1:1 NAT in front of a private interface address, that works with NAT-T, but set the IKE identity to the public IP explicitly.

What IKEv2 proposals should I use for a new IPsec tunnel?

A sensible baseline is AES-256 (CBC or GCM) with SHA-256 and DH group 20 (ECP-384) for the IKE SA, with a 28800 s lifetime, and AES-256-GCM with PFS group 20 for the child SA, with 3600 s. If a peer cannot do group 20, use group 14 or group 19 as a fallback. Avoid DH groups 2 and 5, 3DES and SHA-1.

Should I use a policy-based or route-based IPsec tunnel?

A policy-based tunnel encrypts traffic matching a pair of subnets and interoperates with almost everything. A route-based tunnel uses a virtual tunnel interface (VTI) with 0.0.0.0/0 selectors, and the routing table decides what enters it. Route-based suits many subnets and dynamic routing such as BGP; policy-based is simpler and needs nothing beyond strongSwan and the kernel.

Why do I need a NAT exemption on an IPsec VPN server?

If the server also NATs the LAN to the internet, traffic for the remote subnet must be excluded from that NAT. Otherwise its source is rewritten to the public IP, it no longer matches the local traffic selector (10.10.0.0/16 in the example) and leaves unencrypted. In nftables, add an accept rule for the subnet pair in the postrouting chain before the masquerade rule.

How do I verify my new IPsec VPN server is working?

Run sudo swanctl --list-sas and look for the IKE SA in state ESTABLISHED and the child SA in state INSTALLED, with the expected selectors and byte counters that increase while you ping. Ping from an address inside the local subnet, not the public IP. ip xfrm policy, ip xfrm state and journalctl -u strongswan -f show kernel policies, SAs and the live log.