Security information for IPSec

  • Article

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

Security information for IPSec

Before deploying IPSec in your organization, consider the following security issues:

  • 3DES and computers running Microsoft® Windows® 2000

  • Authentication method

  • Firewall packet filtering

  • Protected traffic (using AH or ESP)

  • Unprotected traffic (using the default response rule)

  • Diffie-Hellman groups (using Diffie-Hellman Group 1, 2, or 2048)

3DES and computers running Windows 2000

IPSec policies allow the choice of a strong encryption algorithm, 3DES, which provides stronger encryption than DES for higher security. Computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 (or later) installed in order to perform the 3DES algorithm. If a computer running Windows 2000 receives a 3DES setting, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, then the 3DES setting is set to the weaker DES, to provide some level of confidentiality for communication, rather than blocking all communication. However, you should use DES only as a fallback option if not all computers in your environment support the use of 3DES. Computers running Windows XP and the Windows Server 2003 family support 3DES and do not require installation of the High Encryption Pack.

Authentication method

If the computers in your organization are part of an Active Directory® domain, IPSec main mode authentication can be accomplished with the default authentication method (Kerberos V5). You do not have to deploy public key certificates for intranet communication. However, computers running Windows XP Home Edition do not support the Kerberos V5 authentication method. Also, if you have computers connected to the Internet, it is recommended that you do not use Kerberos V5 as an authentication method. When Kerberos authentication is used, during main mode negotiation, each IPSec peer sends its computer identity in unencrypted format to the other peer. The computer identity is unencrypted until encryption of the entire identity payload takes place during the authentication phase of the main mode negotiation. An attacker can send an Internet Key Exchange (IKE) packet that causes the responding IPSec peer to expose its computer identity and domain membership. To secure computers that are connected to the Internet, certificate authentication is therefore recommended.

For enhanced security, the use of preshared key authentication is not recommended because it is a relatively weak authentication method. In addition, preshared keys are stored in plaintext. Preshared key authentication is provided for interoperability purposes and to adhere to IPSec standards. It is recommended that you use preshared keys only for testing.

For more information, see Authentication methods and Preshared key authentication.

Firewall packet filtering

For a firewall, security gateway, router, or any other server or device that is connected to the Internet and providing packet filtering capabilities to computers on a perimeter network (also known as a demilitarized zone or DMZ), special filtering must be enabled on that computer to ensure that packets secured with IPSec are allowed to be forwarded to computers on the perimeter network. The firewall or other device should normally allow the following types of traffic to pass through it:

  • IP Protocol ID of 50 (0x32) for IPSec Encapsulating Security Payload (ESP) traffic

  • IP Protocol ID of 51 (0x33) for IPSec Authentication Header (AH) traffic

  • UDP port 500 (0x1F4) for Internet Key Exchange (IKE) negotiation traffic

Most packet filtering software allows you to be more specific. You can define separate packet filters for inbound traffic (inbound filters), for outbound traffic (outbound filters), and for each interface. Additionally, you can specify IP addresses for the IPSec computers on the perimeter network. The most restrictive packet filters for IPSec traffic exchanged with a single IPSec computer on the perimeter network are outlined in the following sections.

Filters on the Internet interface

Configure the following inbound packet filters on the Internet interface of the firewall to allow the specific types of traffic:

  • Destination IP address of the IPSec computer's perimeter network interface and UDP destination port of 500 (0x1F4)

    This filter allows IKE traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    This filter allows IPSec ESP traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    This filter allows IPSec AH traffic to be sent to the IPSec computer on the perimeter network.

Configure the following outbound packet filters on the Internet interface of the firewall to allow the specific types of traffic:

  • Source IP address of the IPSec computer's perimeter network interface and UDP source port of 500 (0x1F4)

    This filter allows IKE traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    This filter allows IPSec ESP traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    This filter allows IPSec AH traffic to be sent from the IPSec computer on the perimeter network.

Filters on the perimeter network interface

Configure the following inbound packet filters on the perimeter network interface of the firewall to allow the specific types of traffic:

  • Source IP address of the IPSec computer's perimeter network interface and UDP source port of 500 (0x1F4)

    This filter allows IKE traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    This filter allows IPSec ESP traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    This filter allows IPSec AH traffic to be sent from the IPSec computer on the perimeter network.

Configure the following outbound packet filters on the perimeter network interface of the firewall to allow the specific types of traffic:

  • Destination IP address of the IPSec computer's perimeter network interface and UDP destination port of 500 (0x1F4)

    This filter allows IKE traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    This filter allows IPSec ESP traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    This filter allows IPSec AH traffic to be sent to the IPSec computer on the perimeter network.

Protected traffic (using AH or ESP)

When deciding whether to use AH or ESP to protect your IP traffic, consider the following:

  • AH provides both data authentication and integrity services through the computation and inclusion of a keyed hash for each packet. With AH, the hash calculation includes the entire IP packet and header. Some fields that are allowed to change in transit are excluded. Packet replay protection services are provided through the inclusion of a sequence number for each packet.

  • ESP provides data authentication and integrity services through the computation and inclusion of a keyed hash for each packet. With ESP, the hash calculation only includes the ESP header, trailer, and payload. The IP header is not protected with the hash. ESP provides data confidentiality services by encrypting the ESP payload with the DES (Data Encryption Standard) or 3DES (triple DES) encryption algorithms. Packet replay protection services are provided through the inclusion of a sequence number for each packet.

It is less CPU-intensive to verify the hash for each packet than it is to encrypt and decrypt each packet. If performance is a significant consideration, you can use AH to protect most of your traffic. When confidentiality is required, you can use ESP instead. For example, you can use AH to protect traffic on your intranet and ESP for traffic that is sent over the Internet.

Note

  • This performance consideration assumes that you are not using IPSec offload network adapters in your organization. Offload network adapters perform cryptographic calculations, such as the calculation and verification of the hash and the encryption and decryption services, on the adapter itself, enhancing performance for IPSec-secured traffic.

Unprotected traffic (using the default response rule)

You can easily implement an IPSec deployment scenario that secures traffic from a specific set of servers, as follows:

  1. Place the servers in an organizational unit and assign an IPSec policy to a Group Policy object of that organizational unit with a rule that requires secured traffic between the servers and any other computer. Within the filter action for the rule, enable the secure servers to accept unsecured communication but always respond using IPSec.

  2. Place the client computers in an organizational unit and assign an IPSec policy to a Group Policy object of that organizational unit that uses only the default response rule. For more information, see IPSec Policy Rules.

In this configuration, the following traffic between the client computers and the secure servers is unsecured:

  • The initial request (for example, a TCP SYN packet) sent by the client computer to establish communication is unsecured because the client does not have a rule in its policy to initiate secured communication with the secure server.

  • Although ongoing communications are secured after secured communication is negotiated between the client and secure server computers, the quick mode security association (SA) that provides security settings for traffic can eventually time out and be removed from both the secure server and the client computer. This occurs when a TCP connection exists and no traffic is sent between the client computer and the secure server for an extended period of time. The dynamic filter created on the client computer by the default response rule is also removed after the dynamic filter has been timed out.

    If the secure server sends new data on the existing connection, it renegotiates the quick mode SA before sending this data to the client computer because a rule exists on the secure server to secure traffic between itself and all other computers. However, if the client computer sends new data after the quick mode SAs and dynamic filter time out, the new data is sent unsecured because the client does not have a rule to initiate secure communications with the secure server. For TCP connections, one or multiple TCP segments can be sent unsecured by the client computer. To prevent client computers from sending unsecured data to secure servers, you must configure your client computer IPSec policy with additional rules that initiate secured communications to secure servers.

Diffie-Hellman groups (using Diffie-Hellman group 1, 2, or 2048)

Diffie-Hellman groups are used to determine the length of the base prime numbers that are used during the key exchange process. Group 2048 (high) is stronger (more secure) than Group 2 (medium), which is stronger than Group 1 (low). Group 1 provides 768 bits of keying strength, Group 2 provides 1,024 bits, and Group 2048 provides 2,048 bits.

Strong Diffie-Hellman groups combined with longer key lengths increase the computational difficulty of determining a secret key.

Important

  • For enhanced security, do not use Diffie-Hellman Group 1. For maximum security, use Group 2048 whenever possible. Use Group 2 when required for interoperability with Windows 2000 and Windows XP.

Note

  • Diffie-Hellman Group 2048 is provided only with the Windows Server 2003 family.

For more information about Diffie-Hellman groups and key exchange, see Key exchange methods and Key management and protection.