The Cable Guy - June 2002

IKE Negotiation for IPSec Security Associations

By The Cable Guy

To ensure successful and secure communication for Internet Protocol Security (IPSec), the Internet Key Exchange (IKE) protocol, a combination of the Internet Security Association Key Management Protocol (ISAKMP) and Oakley Key Determination protocol, performs a two-phase negotiation. For the IPSec implementation for Windows 2000 and Windows XP, the two phases are Main Mode and Quick Mode.

• Main Mode

  Main Mode (also known as Phase 1) IKE negotiation establishes a secure channel, known as the ISAKMP Security Association (SA), between two computers. The ISAKMP SA is used to protect security negotiations. To achieve this, Main Mode negotiation determines a specific set of cryptographic protection suites, exchanges keying material to establish the shared secret key, and authenticates computer identities.

• Quick Mode

  Quick Mode (also known as Phase 2) IKE negotiation establishes a secure channel between two computers to protect data. Because this phase involves the establishment of SAs that are negotiated on behalf of the IPSec service, the SAs created during Quick Mode are called the IPSec SAs. During Quick Mode, keying material is refreshed or, if necessary, new keys are generated. A protection suite that protects specific IP traffic is also selected. Quick Mode is not considered a complete exchange because it is dependent upon a Main Mode exchange.

ISAKMP Messages

An ISAKMP message is the payload of a UDP message with the source and destination UDP ports set to 500. It consists of an ISAKMP header and one or more ISAKMP payloads. ISAKMP payloads contain negotiation information and are encrypted for most ISAKMP messages. Encryption protects most of the negotiation from being viewed by malicious users who capture ISAKMP traffic.

The ISAKMP payloads that are used for Main Mode and Quick Mode negotiation by IPSec for Windows 2000 and Windows XP are:

• Security Association

  Contains either a list of proposed protection suites or a selected protection suite for an SA.

• Vendor ID

  Contains either a string or a number that is defined by a vendor so that an IPSec implementation can recognize an IPSec peer running the same implementation. The Vendor ID value must be unique and is typically a hash of well-known text created by the designers of an IPSec implementation. In IPSec for Windows XP and Windows 2000, the Vendor ID value is: 0x1E-2B-51-69-05-99-1C-7D-7C-96-FC-BF-B5-87-E4-61-00-00-00-02.

• Key Exchange

  Contains Diffie-Hellman public key information for the Diffie-Hellman key exchange process.

• Nonce

  Contains a pseudorandom number that is only used once. Nonces provide replay protection.

• Kerberos Token

  Contains an IPSec peer's current Kerberos token, which is used to by an IPSec peer to authenticate another IPSec peer, by using a Kerberos Key Distribution Center (KDC).

• Identification

  Contains information to identify and authenticate an IPSec peer.

• Hash

  Contains a hash value, which is the result of a hash function computed over a set of fields or other parameters. The Hash payload is used to provide integrity or authentication of IPSec peers.

• Certificate Request

  Contains a list of trusted root certification authorities (CAs) from which the IPSec peer will accept either a certificate or certificate chain for authentication.

• Certificate

  Contains a certificate or certificate chain of an IPSec peer.

• Signature

  Contains a digital signature, which is calculated over a set of fields or parameters. The Signature payload provides data integrity and nonrepudiation services during the authentication phase of Main Mode negotiation.

• Notification

  Contains control information, such as an error condition.

The payloads of all messages beyond the first four are encrypted and vary based on the selected authentication method. These messages are discussed in the "Main Mode Negotiation, Part 3: Authentication" section.

Main Mode Negotiation

Main Mode negotiation occurs in three parts:

1. Negotiation of protection suites
2. Diffie-Hellman exchange
3. Authentication

The Main Mode negotiation consists of a series of six ISAKMP messages. The initiator and responder each send three messages. The initiator is the IPSec peer that initiates secure communications by sending the first ISAKMP message. The responder, which sends the second message, is the IPSec peer with which the initiator is requesting secure communications.

The first four Main Mode messages are not encrypted and are shown in the following table (Table 1).

Table 1 Main Mode messages 1 through 4

Main Mode Negotiation Part 1: Negotiation of Protection Suites

When initiating an IKE exchange, the initiator proposes protection suites that are based on the applied security policy. Each proposed protection suite includes attributes for encryption algorithms, hash algorithms, authentication methods, and Diffie-Hellman Oakley groups. Main Mode negotiation part 1 is contained in Main Mode messages 1 and 2.

The following table (Table 2) lists the protection suite attribute values that are supported by the current implementation of IPSec in Windows XP and Windows 2000.

Table 2 Main Mode protection suite attribute values

The encryption algorithm, integrity algorithm, and Diffie-Hellman group are configured as one of multiple key exchange security methods. These methods are available from the Methods button on the Key Exchange Settings dialog box, which is available from the Advanced button on the General tab in the properties of an IPSec policy in the IP Security Policies snap-in. The authentication methods are configured on the Authentication Methods tab in the properties of an IPSec rule.

The initiating IPSec peer proposes one or more protection suites in the same order in which appear in the applied security policy. If one of the protection suites is acceptable, the responder selects it for use and responds to the initiator with its choice. Because the responding IKE peer is potentially not running Windows XP or Windows 2000 and is selecting the first proposed protection suite that is acceptable, the protection suites in the applied security policy should be configured in the IP Security Policies snap-in in the order of most secure to least secure. Due to advances in cryptanalysis, DES should not be used to provide confidentiality for IKE or for Encapsulating Security Payload (ESP)-protected data.

Main Mode Negotiation Part 2: Diffie-Hellman Exchange

After a protection suite has been negotiated, the IPSec peer uses the CryptoAPI to generate a Diffie-Hellman public and private key pair based on the negotiated Diffie-Hellman Oakley group. The Diffie-Hellman public key is sent to the other IPSec peer in an ISAKMP Key Exchange payload. Main Mode negotiation part 2 is contained in Main Mode messages 3 and 4.

The cryptographic strength of a Diffie-Hellman key pair is related to its prime number length (also known as key size). The Windows 2000 IKE module currently supports only Diffie-Hellman Oakley Group 1 (768 bits) and Group 2 (1024 bits). Although both groups provide security from conventional attacks, Group 2 is considered more secure because of its larger key size. However, computations involving Group 1-based keys might occur slightly faster because of their smaller prime number size.

After the Diffie-Hellman public keys have been exchanged, the IKE module uses the CryptoAPI to compute the Main Mode master key keying material based on the mutually agreeable authentication method.

Main Mode Negotiation Part 3: Authentication

Windows XP and Windows 2000 support three methods of authentication:

1. Kerberos (the default)
2. Certificate-based digital signature
3. Preshared key

The authentication that occurs for Main Mode negotiation is a computer-based authentication (also known as machine-based authentication). The authentication process verifies only the identity of the computers, not the individual using the computer when the authentication process occurs.

Kerberos Authentication

Kerberos authentication is provided mainly for client-to-server IPSec machine authentication inside the corporate network where clients and servers are members of a Windows 2000 domain or mutually trusted domains. In Windows XP and Windows 2000, Kerberos authentication is based on the Generic Security Service (GSS) API IKE authentication method, which is described in the Internet draft titled "A GSS-API Authentication Method for IKE." The following table (Table 3) lists the ISAKMP messages and their payloads, which are exchanged during a Kerberos authentication Main Mode negotiation.

Table 3 Main Mode messages for the Kerberos authentication method

Authentication occurs when:

1. Each peer verifies the other peer's Kerberos token. The responder verifies the initiator's Kerberos token and the initiator verifies the responder's Kerberos token.
2. Each peer's hash is verified. The hash payload is encrypted using the Kerberos session key, in addition to being encrypted with the Main Mode master key. After the hash payload is decrypted with the Main Mode master key, the receiver decrypts the payload with the Kerberos session key and verifies the hash.

Certificate Authentication

In compliance with Internet standards, Windows XP and Windows 2000 IPSec peers perform certificate-based digital signature authentication during Main Mode negotiation. The IKE module uses the CryptoAPI to retrieve the certificate chain that will be sent, verify peer certificates and certificate chains, check certificate revocation, and create and verify digital signatures. All certificate, certificate chain, and digital signature information is exchanged in the Main Mode messages that are listed in the following table (Table 4).

Table 4 Main Mode messages for the Certificate authentication method

Windows XP and Windows 2000 only support RSA-based certificates and digital signatures.

By sending the Certificate Request payload, each IPSec peer is specifying to the other IPSec peer that, for authentication, it will accept only a certificate from an issuing CA that can follow a chain back to a CA in the list of trusted root CAs. The list of trusted root CAs is configured on the Authentication Methods tab in the properties of an IPSec rule in the IP Security Policies snap-in.

When submitting a certificate, each IPSec peer uses the CryptoAPI to retrieve a certificate chain from its local certificate store. Each IPSec peer must have the private key for the certificate and the certificate must follow a chain back to one of the listed trusted root CAs that are sent by the other IPSec peer in the received Certificate Request payload. The certificate chain (without the root certificate) is inserted into a Certificate payload and sent to the other IPSec peer. Each IPSec peer also includes a digital signature in the ISAKMP Signature payload, providing proof that the submitting IPSec peer has access to the private signing key of the submitted certificate.

Certificate authentication is successful when each peer verifies the other peer's certificate and digital signature.

For L2TP over IPSec connections, the list of trusted root CAs is not configurable. Instead, the trusted root CA list sent in the Certificate Request payload of Main Mode messages 4 and 5 contains the trusted root CAs for which a corresponding computer certificate is installed. For L2TP over IPSec connections, the IPSec peers must have computer certificates issued through a common trusted root CA, or by using cross-certificates that allow the client and server certificates to be trusted by each other. Each peer must be able to build the trust chain using the other peer's certificate to it's own machine certificate root CA.

Preshared Key Method

Preshared key IKE authentication requires that each IKE peer use a predefined and shared key to authenticate the IKE exchange. Both Windows XP and Windows 2000 support the preshared key authentication method specified in RFC 2409. A preshared key is configured on the Authentication Methods tab in the properties of an IPSec rule in the IP Security Policies snap-in.

The Main Mode messages for preshared key IKE authentication are listed in the following table (Table 5).

Table 5 Main Mode messages for the preshared key authentication method

The initiator and responder hash calculation incorporates the value of the preshared key. Each IPSec peer calculates the other peer's hash and compares it to the value in the Hash payload in the received message. If both hashes are valid, the authentication is successful.

Quick Mode Negotiation

Upon either the completion of Main Mode negotiation or the expiration of an existing IPSec SA, Quick Mode negotiation is initiated based on the appropriate filter actions. These include specific encryption and hashing algorithms, and specification of whether the link is tunnel or transport and the protocol is ESP and/or AH. All Quick Mode negotiation messages are protected with the ISAKMP SA established during Main Mode. Each successful Quick Mode negotiation establishes two IPSec SAs. One SA is for inbound traffic and the other is for outbound traffic.

The following table (Table 6) lists the Quick Mode messages that are exchanged by two IPSec peers running Windows XP or Windows 2000.

Table 6 Quick Mode messages

The four Quick Mode messages are the following:

1. Includes a Security Association payload, which contains a list of proposals for encryption and hashing algorithms to secure traffic (AH vs. ESP, DES vs. 3DES, and MD5 vs. SHA), an Identification payload, which contains a description of the traffic that is protected (IP addresses, IP protocols, TCP ports, or UDP ports), and a Nonce payload.

   Encryption and hashing algorithms are configured on the Security Methods tab in the properties of an IPSec filter action in the IP Security Policies snap-in.

   The IP traffic description is configured as an IPSec filter list in the IP Security Policies snap-in.

2. Has the Commit bit set in the ISAKMP header and includes a Security Association payload that contains the selected method for securing traffic, an Identification payload, and a Nonce payload. The Commit bit is used to ensure that the SA completes its negotiation before protected data is sent.

3. Includes a Hash payload that provides verification and replay protection.

4. Includes a Notification payload that contains the "Connected" notify message. This message, which is used by IPSec peers running Windows XP or Windows 2000, is not required by the IKE standard. It is used to prevent the initiator from sending IPSec-protected packets to the responder before the responder is ready to receive them.

If session key perfect forward secrecy is enabled on the filter action, Quick Mode messages 1 and 2 also contain Key Exchange payloads to perform a Diffie-Hellman key exchange from which the Quick Mode session key is derived.

After the Quick Mode negotiation is complete, there are three SAs:

1. The ISAKMP SA, which is used by both peers to protect future Main Mode or Quick Mode negotiations.
2. The outbound IPSec SA, which is used by an IPSec peer to protect data that is outbound to the other IPSec peer.
3. The inbound IPSec SA, which is used by an IPSec peer to protect data that is inbound from the other IPSec peer.

The outbound IPSec SA of one IPSec peer is the inbound IPSec SA of the other IPSec peer.

For More Information

For more information about IPSec, consult the following resources:

• Windows 2000 Server Documentation (Networking\Internet Protocol Security)
• Internet Protocol Security (Windows 2000 Server Resource Kit chapter)
• IP Security for Windows 2000 Server (white paper)
• Step-by-Step Guide to Internet Protocol Security (IPSec)
• Exploring Peer-to-Peer IPSec in Windows 2000 (The Cable Guy column for May 2001)
• IETF IP Security Protocol working group (IPSec RFCs and Internet drafts)
• Using IPSec in Windows 2000 and XP (SecurityFocus.com article)
• Using IPSec in Windows and XP, Part Two (SecurityFocus.com article)
• Using IPSec in Windows 2000 and XP: Part Three (SecurityFocus.com article)

For a list of all The Cable Guy articles, click here.