Using IPsec for Network Protection: Part 1 of 2

Security Management

by Steve Riley
Senior Program Manager, Security Business and Technology Unit

See other Security Management columns.

Internet Protocol security (IPsec) is truly one of the most useful security technologies I know of. Yet it remains poorly understood and underused—in no small part because it can be difficult to comprehend and configure. In this month’s article, I’ll introduce you to the technology; next month we’ll explore some very useful scenarios that can help solve security pain today.

IPsec Operation

When two computers (peers) use IPsec to communicate, they create two kinds of security associations. In the first, called main mode or phase one, the peers mutually authenticate themselves to each other, thus establishing trust between the computers. In the second, called quick mode or phase two, the peers will negotiate the particulars of the security association, including how they will digitally sign and encrypt traffic between them. Packet signing ensures that the data hasn’t been tampered with in transit; packet encryption ensures that the data isn’t vulnerable to eavesdropping attacks.

A computer can have only one IPsec policy assigned at a time. The policy can have any number of rules, each of which has a filter list and a filter action. Filter lists contain one or more filters that specify the characteristics of the traffic that the rule should process: source and destination addresses, source and destination port numbers, and protocol types. Filter actions specify the behaviors of the rule: whether to permit traffic, block traffic, or negotiate the pair of IPsec security associations. Actions that specify negotiating security can have many options, including encryption suites, per-packet authentication methods, how often to generate new keys, how to respond to incoming insecure requests, and whether to communicate with computers that don’t support IPsec.

Each rule in an IPsec policy combines one filter list with one filter action. Traffic that matches a particular filter list is processed according to the settings in the linked filter action. Rules also indicate the security association’s mode (transport or tunnel, explained later) and one of three phase one authentication methods:

  • Preshared keys. Included only for RFC conformance, it’s a good idea to use preshared keys only when testing your IPsec policies. Every peer that participates in the same security policy will need the same preshared key. Shared secrets don’t remain secret for very long! Furthermore, they’re stored in the registry and clearly visible to anyone with administrative privileges on the computer.

  • Digital certificates. As long as each peer possesses an IPsec or computer certificate signed by an authority the other peer trusts, the peers will authenticate to each other. Note where the trust lies: in the signer of the certificate. The actual name on the certificate is unimportant in this case. Digital certificates are much preferred over preshared keys because each peer can have its own certificate, and a multilevel certificate hierarchy can help create more granular IPsec policies. For example, super-secure Machine A might accept only certificates signed by high-value Authority X, while sort-of-secure Machine B might accept certificates signed either by high-value Authority X or medium-value Authority Y.

  • Kerberos version 5 protocol. If both peers are in the same Active Directory forest, IPsec on computers running the Windows Server 2003 operating system can also use the Kerberos protocol for the initial computer-to-computer authentication. Kerberos is appropriate if you don’t have a public key infrastructure (PKI) and don’t need to establish IPsec security associations between computers outside a single forest.

IPsec Modes and Methods

There is no such thing as an “IPsec tunnel.” It bears repeating: There is no such thing as an “IPsec tunnel.” Yes, it’s an extremely common phrase, and everyone thinks they know what it means, but it’s actually meaningless because it lacks specificity. Two phase one security association modes are available:

  • Transport mode. This is the more common of the two and is often what people have in mind when thinking of an “IPsec tunnel.” In transport mode, two peers authenticate each other (phase one) and establish the traffic signing and encryption parameters (phase two). Any traffic between the computers that matches the characteristics specified in the filter list will be signed and/or encrypted according to the details of the linked filter action. Transport mode ensures that communications between two computers remains tamper-free and private. Transport mode doesn’t create new packets; rather, it secures existing packets. Interestingly enough, in L2TP+IPsec VPNs, it’s transport mode, not tunnel mode, that secures the L2TP traffic between a client and a VPN server.

  • Tunnel mode. Tunnel mode is intended for secure site-to-site communications over an untrusted network. Each site has an IPsec gateway configured to route traffic to the other site. When a computer in one site needs to communicate with a computer in the other site, the traffic passes through the IPsec gateways (and possibly through intervening routers in each site before reaching the local gateway). At the gateway the outbound traffic is encapsulated inside another complete packet and secured according to the details of the filter action in the rule. Of course, the gateways have already performed their phase one authentication and established their phase two signing/encryption security association. In IPsec for Windows Server 2003, tunnel mode is supported only for site-to-site VPNs on Routing and Remote Access Service (RRAS) gateways and not for any kind of client-to-client or client-to-server communications.

A filter action can specify one of three behaviors: permit traffic, block traffic, or negotiate security. The first two don’t really do any kind of security processing: If traffic matches a filter list that’s linked to a “permit” filter action, the traffic is allowed to pass; if traffic matches a filter list that’s linked to a “block” filter action, the traffic is dropped. Filter actions that negotiate security can choose one or both of two different phase two security methods:

  • AH (authentication header). AH security associations are useful when the requirement is only for integrity but no confidentiality. AH computes a SHA1 or MD5 digital signature across the entire packet (including the IP header which contains the source and destination addresses) and adds this signature to the packet. The receiver computes its own version of the signature and compares that with the signature stored in the header; if they match, the packet hasn’t been modified.

  • ESP (encapsulated security payload). Use ESP security associations when you have a need for confidentiality. ESP will negotiate a DES or 3DES (triple DES) session key that’s exchanged between the peers and used for encrypting the traffic between them. You can also specify a SHA1 or MD5 digital signature in ESP. Note that both the ESP encryption and signature computations include the payload and the TCP/UDP header portions of each packet but not the IP header; compare to AH whose digital signature covers the entire packet.

There are no dependencies between modes and methods. Both transport mode and tunnel mode security associations can use AH, ESP, or AH and ESP together.

IKE

The Internet Key Exchange (IKE) protocol is the mechanism by which IPsec security associations negotiate their protection suites and exchange signing or encryption keys. IKE defines how the peers communicate policy information and how authentication messages are constructed and exchanged. A hybrid of three other protocols (ISAKMP, Oakley, and SKEME), IKE is ideal for IPsec’s requirements. IKE is fairly complicated; to fully understand it, it’s helpful to possess multiple advanced degrees in mathematics and cryptography and to have copious amounts of spare time to read many detailed yet highly valuable resources.

IPsec over NAT

One of the biggest deployment blockers for IPsec is the presence of a Network Address Translator (NAT). IPsec authenticates computers; a NAT hides them. So, if you think about it, the purposes of IPsec and NAT are often at odds. Yet because of the heavy reliance on NATs in most IPv4 networks, there’s a huge demand for getting IPsec to work correctly over NAT devices. Indeed, making the two work together will rapidly accelerate the adoption of IPsec.

Getting IPsec to traverse a NAT is more difficult than you might think. Three problems loom over IPsec and NAT:

  • AH integrity violation. AH computes its digital signature of the packet before it leaves the sending peer. If that packet then passes through a NAT (either local or remote), its IP header gets modified. The receiving peer, when it computes its own version of the packet’s signature, will generate a different result because the NAT modified the source address. Therefore the receiving peer will drop the packet.

  • IPsec helper. Many small/home office NAT gateways include a feature called “IPsec helper” (or “IPsec passthrough”). Originally designed for tunnel mode, the feature also acts on transport mode. If multiple computers behind the gateway create IPsec security associations to destinations outside the gateway, the gateway will forward all incoming IPsec traffic to the first computer that created its security association. The helper function simply remembers which computer on the inside initiated an IPsec conversation and forwards all incoming traffic there, without modification.

  • IKE fragmentation. It’s common that the payload of a digital certificate will exceed the size of an IP frame. Whenever an application generates a data packet larger than an IP frame, IP fragments the packet so that each fragment fits in a single frame. While this is fine inside a local network, network border devices (including many NATs) drop fragments because carefully crafted malicious fragments are a popular way to circumvent some firewalls. NATs that drop fragments will prevent IKE from working properly.

Turns out, though, that by defining a mechanism to encapsulate ESP (but not AH) inside UDP, it’s possible to forward IPsec traffic through a NAT without it getting rejected. Each side sends some discovery packets to the other to determine if there is a local NAT present and if both sides are capable of performing NAT traversal (NAT-T). If one or both sides are behind a NAT and both can perform NAT-T, IPsec will first shift the IKE exchange to UDP port 4500 (to avoid IPsec helper conflicts) and then encapsulate the entire IPsec security association in the same UDP conversation. To the NAT the traffic looks like ordinary UDP traffic, so the NAT can handle it just fine. Each side also exchanges its NAT details with the other so that they can properly reassemble the IP headers of received traffic. Remember that NATs change the headers, so receivers will need to “remanufacture” the original header before decrypting traffic and checking signatures. The NAT-T setup process provides each side with the necessary information to do this.

UDP-ESP (as the specification is called) knows how to handle multiple IPsec security associations behind a single NAT device. When multiple computers make outgoing connections using the same protocol, NAT devices will use unique source port numbers for each computer—this is how the NAT knows where to forward incoming return traffic. In each client, NAT-T maintains a table of source port–security association pairs and matches up traffic accordingly as it flows in and out.

As of this writing, NAT-T is still an Internet draft. The authors continue to make minor changes to improve functionality and interoperability. NAT-T is not defined for AH because there’s no way to effectively work around the AH integrity violation problem. And IETF decided not to address the third problem, IKE fragmentation; if one side is blocking fragments, that side’s administrator will need to change the NAT programming or the side simply won’t be able to participate in NAT-T. The Microsoft implementation of NAT-T includes a pre-fragmentation workaround that engages if both sides support it. IKE will fragment the key into smaller portions before sending the data to the IP layer; each portion then is placed into a fully formed IP packet and sent on its way. The receiving end will be expecting this and will reassemble the portions back into the complete key. The NAT discovery phase also checks for support of IKE fragmentation and will use it only if both devices are capable.

Next Month: Scenarios

There’s a lot more to IPsec than this—but now you know enough to think about where it might be useful in your organization. Next month we’ll explore three scenarios perfectly tailored for IPsec: stopping worms, protecting servers, and achieving domain isolation.