RSystems

Networking · Security

IPsec

Also known as: Internet Protocol Security

Encrypts and authenticates IP traffic at the network layer — the foundation of site-to-site VPN tunnels.

IPsec is a suite of protocols that encrypts and authenticates IP traffic at the network layer. It's the foundation of most site-to-site VPN tunnels, providing confidentiality, integrity, and authentication for traffic between networks.

IPsec operates at Layer 3, encrypting the IP payload (and optionally the header) before it's sent across an untrusted network. Unlike TLS/SSL which secures individual application connections, IPsec secures all traffic between two endpoints at the IP level — regardless of application.

Flowchart showing the IPsec process between sender and receiver — encryption algorithm, HMAC authentication, symmetric key exchange via Diffie-Hellman, and ICV integrity verification

IPsec operates in two modes:

Tunnel mode — the entire original IP packet is encrypted and encapsulated in a new IP packet. This is how site-to-site VPNs work: traffic between two networks is wrapped in an encrypted tunnel, and the tunnel endpoints are the VPN gateways.

Transport mode — only the payload is encrypted; the original IP header remains. Used for host-to-host encryption, less common in practice.

IPsec uses two core protocols:

AH (Authentication Header) — provides integrity and authentication but no encryption. Rarely used alone.

ESP (Encapsulating Security Payload) — provides encryption, integrity, and authentication. This is what you actually use.

Key negotiation is handled by IKE (Internet Key Exchange) — IKEv1 or IKEv2. IKEv2 is faster, more resilient to network changes (important for mobile VPN clients), and more efficient. Prefer it where supported.

IPsec is the standard for site-to-site VPN between firewalls — SonicWall to SonicWall, Cisco to Fortinet, or any combination. For remote access VPN (individual users), SSL VPN has largely displaced IPsec due to its simpler client deployment and firewall-friendliness.