IPSec (Internet Protocol security) provides message authentication and encryption. This can be used for a Virtual Private Network (VPN). IPSec is based around the concept of Security Associations (SA).

IPSec concepts

Security Association (SA)

A Security Association provides a unidirectional logical connection between two end-points to secure the traffic. In practice, two SAs are established, one in each direction, to secure the traffic flow.

The IP Authentication Header (AH) provides a mechanism for data integrity and authentication. The IP Encapsulating Security Payload (ESP) provides encryption in addition to data integrity and authentication. An SA comprises the AH or ESP security protocol identifier, the IP Address (destination) and a Security Parameter Index (SPI). An IPSec key exchange establishes and maintains the Security Associations.

IPSec supports two types of Security Associations, transport mode SAs and tunnel mode SAs.

Transport mode

A Transport mode SA is a security association between two hosts. This mode is only supported by IPSec hosts. It can be used to protect communication between a computer and a router for example.

Tunnel mode

A Tunnel mode SA is a security association for an IP tunnel. This mode is supported by hosts and security gateways. You can use tunnel mode if you are connected to the Internet and you wish to access computers on your LAN remotely and securely.

Internet Key Exchange (IKE)

The mechanism to establish SAs is known as Internet Key Exchange (IKE). It defines an automatic and secure way to negotiate the details for the SA to be set up. IKE uses the Internet Security Association and Key Management Protocol (ISAKMP), and Oakley and SKEME as the key-exchange protocol. The IKE daemon uses UDP port 500. The protocol has two phase, Phase 1 for authentication and Phase 2 for the Key Exchange.

Two modes are available in Phase 1, main and aggressive mode. Aggressive mode is faster but it does not provide identity protection for the negotiating parties. Aggressive mode may be vulnerable to denial-of-service attacks. Phase 2, also known as Quick mode, establishes the IPSec SA.


Pre-shared keys (PSK) or IKE X.509 certificates can be used for authentication during Phase 1.

Protecting sensitive information

Protocols such as FTP, POP3 and SMTP generally send out information over the wire in plain text. The following data is exchanged during a mail retrieval session using POP3:

+OK read <30195.118491@mail.example.com>
+OK Capa
USER username
+OK Pass
PASS secret
+OK username has 1 visible message (0 hidden) in 924 octets

As you can see, the user name and password was sent out in plain text and is clearly visible to anyone who can capture the traffic; on an open wireless network, for example.

A VPN tunnel can be used to protect the information exchanged between your computer and the mail server. The information sent in plain text gets encrypted at the source and is decrypted at the destination and is thus protected from eavesdroppers.

There are open source software such as isakmpd and ipsec-tools which implement IPSec. They support both transport and tunnel mode and can be used as VPN gateways.

Windows 2000, XP and 2003 also comes with IPSec support. An IPSec tunnel can be established using ipseccmd.exe (Windows XP) which is part of the Windows Support Tools. The following links explain how to set up a VPN tunnel:

Note that IPSec is not the same as L2TP or PPtP.


SA Security Association

SPI Security Parameter Index


If you have any comments, you can contact the author here.