Special IPSec considerations

  • Article

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

Special IPSec considerations

The following special IPSec considerations can help simplify the administration of IPSec policies.

Following are recommended scenarios and scenarios that are not recommended for the Windows Server 2003 family implementation of IPSec.

The Windows Server 2003 family implementation of IPSec is recommended for use as follows:

  • Packet filtering. IPSec provides limited firewall capabilities for end-systems. You can also use IPSec with Internet Connection Firewall, Windows Firewall, and Routing and Remote Access to permit or block inbound or outbound traffic.

  • Securing host-to-host traffic on specific paths. You can use IPSec to provide mutual authentication and cryptographic protection for traffic between servers or other static IP addresses or subnets. For example, IPSec can secure traffic between domain controller sites or forests, or between Web servers and database servers.

  • Securing traffic to servers. You can use IPSec to require mutual authentication for all client computers that access a server. In addition, you can set restrictions on which computers are allowed to connect to a server running Windows Server 2003 family.

  • Using L2TP/IPSec tunneling for VPN connections. You can use the combination of the Layer Two Tunneling Protocol (L2TP) and IPSec (L2TP/IPSec) for all virtual private network VPN scenarios.

  • Using IPSec in tunnel mode for gateway-to-gateway tunnels. For interoperability with other routers, gateways, or end-systems that do not support L2TP/IPSec or PPTP VPN tunneling, you can use IPSec in tunnel mode for gateway-to-gateway tunnels.

Notes

  • Windows Firewall is not included in the original release of the Windows Server 2003 operating systems.

  • Internet Connection Firewall is included only in the original releases of Windows Server 2003, Standard Edition, and Windows Server 2003, Enterprise Edition.

When using IPSec authentication and data protection features, the policy management model in Windows 2000, Windows XP, and Windows Server 2003 family lends itself best to server-to-server and client-to-server scenarios where one end-point has a static address. In large network deployments, when policy involves dynamic addresses at both end–systems, and in some mobility cases, policy management complexity can make IPSec deployment difficult. Because of this, IPSec is not recommended for use as follows:

  • Securing communication between domain members and their domain controllers. Because of the complexity of the IPSec policy configuration and management required for this scenario, using IPSec to secure communication between domain members and their domain controllers is not recommended.

  • Securing all traffic in a network. Configuring IPSec communications between all client computers and all servers in order to secure many or all computers in a network is difficult. IPSec cannot negotiate security for multicast and broadcast traffic. Traffic from real-time communications, applications that require ICMP, and peer-to-peer applications might be incompatible with IPSec. For these reasons, using IPSec to secure all traffic in a network is not recommended.

Additionally, the IPSec protocol and implementation has characteristics that require special consideration in other scenarios:

  • Securing traffic over wireless 802.11 networks. IPSec policies are not optimized for configuration for mobile clients. Because of this, IPSec is not recommended as the only method to secure traffic sent over wireless 802.11 networks. Instead, it is recommended that you use IEEE 802.1X authentication. 802.1X enhances security and ease of deployment by providing support for centralized user identification, authentication, dynamic key management, and accounting. In cases where clients roam between access points on the same network, IPSec can be used in combination with 802.11 and 802.1x. In cases where roaming causes the client IP address to change, IPSec security associations become not valid and new security associations are renegotiated.

  • Using IPSec in tunnel mode for remote access VPN connections. Using IPSec in tunnel mode is not recommended for remote access VPN scenarios. Instead, use L2TP/IPSec or PPTP for remote access VPN connections.

Managing policies that use new features available only in the Windows Server 2003 family

The following features are available only in the Windows Server 2003 family implementation of IPSec:

  • Support for new source and destination address options for defining filters. For example, you can now create:

    • Filters with a source address and destination address of Any IP Address.

    • Filters that use a non-standard subnet class as a source address or a destination address.

    • Filters that use a multicast or a broadcast address as a destination address.

    • Filters with a source address or a destination address that corresponds to the local IP configuration for the DNS servers, WINS servers, and the DHCP server and default gateway for the computer to which the filter applies.

  • The ability to exclude the name of the certification authority (CA) from certificate requests.

  • Certificate-to-account mapping for network access control.

  • Support for root CA names that are represented in extended ASCII character sets.

  • A stronger Diffie-Hellman group (2048-bit Diffie-Hellman key exchange).

Computers running Windows XP and Windows 2000 cannot use these new policy settings. If you plan to apply Active Directory-based IPSec policies that use features that are only available in the Windows Server 2003 family, use the Windows Server 2003 family version of the IP Security Policy Management console to manage them. If you use the Windows XP or Windows 2000 version of the IP Security Policy Management console to manage these policies, the earlier versions of the IP Security Policy Management console will delete the new feature settings.

In addition, if you want to apply the same IPSec policy to computers running different versions of Windows, test the policy to ensure that it functions as expected on both servers running Windows Server 2003 and computers running Windows XP or Windows 2000.

For information about new features available with the Windows Server 2003 family implementation of IPSec, see New features for IPSec. For more information about best practices for using IPSec, see IPSec Best practices.

IP filters

Consider the following when configuring IP filters:

  • Use general filters when you want a single filter to apply to a group of computers. For example, when configuring the filter, use My IP address, Any IP Address, or a subnet address instead of specifying a specific computer's source and destination IP addresses.

  • Define filters that allow you to group and secure traffic from logically associated segments or subnets within your network.

  • Be aware that when you are viewing the IPSec policy, the order in which the filters apply is not in the order displayed. When the IPSec Policy Agent reads an IPSec policy, filters are processed into one ordered list, sorted from most to least specific. You can use the IP Security Monitor console to view filters sorted by weight. Filters that have the same weight might appear in any order relative to each other.

    During policy updates, the IPSec Policy Agent recalculates the sorted order and updates the IPSec driver with the changes only. If a filter is deleted, or if the filter action is changed for a filter, all security associations related to that filter are deleted. As a result, some packets are lost when a new security association is negotiated. TCP connections, however, should immediately recover by retransmitting the lost packets.

  • If you are configuring an IPSec policy to secure Kerberos traffic, IPSec can only negotiate security associations if your authentication method does not use Kerberos. If Kerberos is required for authentication between domain members, special consideration must be given to the traffic required by Kerberos between domain members and domain controllers, and the following requirements must be met:

    • If a domain member and a domain controller are members of different domains, a two-way trust must exist between the domains or Internet Key Exchange (IKE) rekeys will fail.

    • If a client computer or a server in the domain uses Kerberos authentication to negotiate IPSec with all peers, you must create mirrored permit filters for all Kerberos-dependent traffic to all domain controllers. It is recommended that you create permit filters to exempt all traffic to each domain controller IP address in the domain member's domain. You do not need to create permit filters to exempt all traffic to all trusted domain controller IP addresses. Instead, create permit filters to exempt only Kerberos traffic and domain controller site location (LDAP) queries to all domain controller IP addresses in the trust path between the computers. If DNS service is not provided by the domain controller (for which you already permit all traffic), you might also need to exempt DNS traffic to the DNS server IP addresses so that DNS can be used for domain controller discovery.

      To exempt these traffic types in the IPSec policy, create the following filters, each with a permit filter action:

      Filter Filter Action

      All traffic to my domain controllers
      • Mirrored

      • Source Address = My IP Address

      • Destination Address = A specific IP Address (where A specific IP Address is the static IP address of each domain controller in the domain)

      • Protocol = Any

      • This filter permits LDAP site location queries and ICMP, DNS, and Kerberos traffic. In addition, it permits the remote procedure call (RPC) traffic and Group Policy downloads required to manage Active Directory-based IPSec policies.

      Kerberos
      • Mirrored

      • Source Address = My IP Address

      • Destination Address = A specific IP Address (where A specific IP Address is the static IP address of each domain controller in the domain)

      • Protocol = Any

      • This filter permits LDAP site location queries and ICMP, DNS, and Kerberos traffic. In addition, it permits the remote procedure call (RPC) traffic and Group Policy downloads required to manage Active Directory-based IPSec policies.

      Domain controller site location (LDAP) queries
      • Mirrored

      • Source Address = My IP Address

      • Destination Address = A specific IP Address (where A specific IP Address is the static IP address of each domain controller in trusted domains)

      • Protocol = UDP

      • Source port = Any

      • Destination port = 88

      • After you create this filter, create another filter with the same settings, but specify TCP for the protocol.

      DNS client queries
      • Mirrored

      • Source Address = My IP Address

      • Destination Address = DNS Servers <dynamic> (or A specific IP Address, where the IP address is the static IP address of each DNS server in the network)

      • Protocol = UDP

      • Source port = Any

      • Destination port = 53

      • After you create this filter, create another filter with the same settings, but specify TCP for the protocol.

    Caution

    • To minimize the security issues, you should allow these exemptions only for the specific IP addresses of the domain controllers and the DNS servers.

When designing an IPSec policy to secure or block ICMP traffic, ensure that you plan carefully and test the policy before implementing it in a production environment. Be aware that ICMP is required for the following scenarios:

  • Outbound TCP connections that traverse networks with different link frame sizes require path maximum transmission unit (PMTU) discovery. For PMTU discovery to function correctly, inbound ICMP traffic from routers and other gateways must be received.

  • The tracert command requires ICMP traffic to determine the path taken to a destination. If IPSec protects ICMP traffic, however, tracert output shows only the destination computer--not the path of intermediate routers.

  • File sharing and other applications use ICMP traffic to determine whether a destination IP address can be reached.

  • Network logon and Group Policy measure ICMP response times to each domain controller IP address. This measurement is performed to both aid in the selection of a domain controller to service a network logon and determine whether to apply Group Policy settings over slow links.

In some cases, even the three-second delay before a soft SA (an unsecured connection) is established might impact upper-layer communication that requires fast ICMP responses.

  • In Windows 2000 and Windows XP, by default, broadcast, multicast, Kerberos, Internet Key Exchange (IKE), and Resource Reservation Protocol (RSVP) traffic are exempt from filter matches. In the Windows Server 2003 family, broadcast, multicast, and Kerberos, are not exempt from filter matches by default (only IKE traffic is exempt). In additon, Resource Reservation Protocol (RSVP) has been turned off. Broadcast and multicast packets will be dropped if they match a filter with a filter action to negotiate security. By default, the Windows Server 2003 family provides limited support for filtering broadcast and multicast traffic. A filter with a source address of Any IP Address will match multicast and broadcast addresses. A filter with a source address of Any IP Address and a destination address of Any IP Address will match inbound and outbound multicast addresses. You can use such a filter to block all traffic. One-way filters that would be used to block or permit specific multicast or broadcast traffic, however, are not supported.

    As a result of the change in default exemption behavior for the Windows Server 2003 family implementation of IPSec, you should verify the behavior of IPSec policies designed for Windows 2000 or Windows XP, and determine whether to configure explicit permit filters to permit specific traffic types. To restore the default Windows 2000 and Windows XP behavior for IPSec policies, you can use the netsh ipsec dynamic set config command, or you can modify the registry. For more information, see the section "Using Netsh to change the IPSec configuration on computers running the Windows Server 2003 family" in IPSec troubleshooting tools.

    Caution

    • The Windows 2000 and Windows XP default exemption settings for IPSec are designed for corporate LAN environments with a low risk of attack. For this reason, you should only use the Windows 2000 and Windows XP default exemption settings when necessary for troubleshooting, in low-risk environments, or when you cannot solve program compatibility issues by configuring explicit filters in IPSec policies.

Safe Mode with Networking

If you start a computer in Safe Mode with Networking, consider the following:

  • The IPSec service cannot apply filters from an IPSec policy or negotiate IP security when the computer is in Safe Mode with Networking. As a result, when Directory Services Restore Mode operations are in progress, communication does not occur with remote domain controllers or network backup devices that enforce an IPSec policy to require IPSec.

  • When a computer loads the Windows operating system during startup, the IPSec driver is loaded at the same time as the TCP/IP driver. If an IPSec policy is assigned to the computer and the IPSec service is configured for automatic startup, by default, the IPSec driver startup mode is changed to perform stateful filtering. When stateful filtering is performed, all outbound traffic is permitted, but all inbound traffic is blocked, except for DHCP traffic and traffic that is sent in response to the outbound traffic initiated during computer startup. (The IPSec driver automatically creates inbound filters to permit responses to outbound traffic.) The IPSec service itself, however, does not start. As a result, no persistent, local, or Active Directory-based IPSec policies can be applied.

  • When the computer is in Safe Mode with Networking, you can allow the IPSec driver to continue to provide stateful filtering protection by configuring inbound exemptions for specific traffic. Or, you can force the IPSec driver to permit all traffic.

  • To configure inbound exemptions for specific traffic, restart the computer normally (not in Safe Mode), use the netsh ipsec dynamic set config bootexemptions command to specify the traffic types to exempt during computer startup, and then restart the computer. The new exemptions will take effect when the computer is restarted and will remain in effect until they are deleted or changed.

  • If you allow the IPSec driver to configure inbound exemptions for specific traffic, keep in mind that unless you configure the appropriate exemptions, the following will occur:

    • Inbound connections will be blocked to network services, such as Remote Desktop Connection and Remote Assistance, which open ports for receiving connections. To allow clients to use Remote Desktop Connection or Remote Assistance, you must configure an exemption to allow inbound traffic to TCP port 3389. To configure this exemption, use the following Netsh command:

      netsh ipsec dynamic set config bootexemptions TCP:any:3389:inbound

    • Some outbound TCP connections, such as those requiring PMTU discovery, might not work. To enable TCP PMTU discovery, you must configure an exemption to allow inbound ICMP traffic. To configure this exemption, use the following Netsh command:

      netsh ipsec dynamic set config bootexemptions ICMP:inbound

      Important

    • Modifying IPSec traffic exemptions from startup security (that is, modifying the bootexemptions= parameter) will overwrite all previous exemptions from startup security.

For more information about how to specify traffic types to exempt during computer startup, see Netsh commands for Internet Protocol security.

To force the IPsec driver to permit all inbound traffic while the computer is in Safe Mode with Networking, you must either configure the IPSec service for manual startup or disable it, and then restart the computer in Safe Mode with Networking.

Filter actions

Consider the following special requirements when configuring IP filter actions:

  • If there is specific traffic that you always want to block, such as traffic to and from either not valid addresses or specific TCP ports, create a blocking filter action with the Security Method set to Block.

  • If there is specific traffic that you always want to be unsecured, create a permit filter action with the Security Method set to Permit.

  • When you are configuring custom security methods for encrypted traffic, you should only set the ESP confidentiality selection to <None> when a higher layer protocol provides data encryption.

  • When you are planning Internet scenarios, such as IPSec tunneling between branch offices, consider using a list of security methods that specifies high levels of security (for example, 3DES only), short key lifetimes (less than 50 MB), and perfect forward secrecy (PFS) for the session keys. This helps protect against known key attacks.

Note

  • Computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 (or later) installed in order to use the 3DES algorithm. If a computer running Windows 2000 receives a 3DES setting, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, the 3DES setting in the security method is set to the weaker DES, to provide some level of confidentiality for communication, rather than blocking all communication. However, you should only use DES as a fallback option if not all computers in your environment support the use of 3DES. Computers running Windows XP or a Windows Server 2003 operating system support 3DES and do not require installation of the High Encryption Pack.

VPN connections using L2TP

When you are using Layer Two Tunneling Protocol (L2TP) for your remote access or router-to-router VPN connections, certificates are the default authentication method for main mode security associations. Computer certificates must be installed either on both the VPN server and remote access client (for remote access connections), or on both the calling router and answering router (for router-to-router VPN connections). The simplest method for installing computer certificates is to configure your Windows 2000 and Windows Server 2003 domains to auto-enroll computer certificates for domain members. For more information, see Computer certificates for L2TP/IPSec VPN connections.

When you are using the certificate authentication method for L2TP connections, the list of certification authorities (CAs) is not configurable. Instead, each computer in the L2TP connection sends a list of root CAs to its IPSec peer from which it accepts a certificate for authentication. The root CAs in this list correspond to the root CAs that issued computer certificates to the computer. For example, if Computer A was issued computer certificates by root CAs CertAuth1 and CertAuth2, it notifies its IPSec peer during main mode negotiation that it will accept certificates for authentication from only CertAuth1 and CertAuth2. If the IPSec peer, Computer B, does not have a valid computer certificate issued from either CertAuth1 or CertAuth2, IPSec main mode negotiation fails. Therefore, ensure one of the following before attempting an L2TP connection:

  • Both the VPN client and VPN server were issued computer certificates from the same CA.

  • Both the VPN client and VPN server were issued computer certificates from CAs that follow a certificate chain up to the same root CA.

In general, the VPN client must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain from the issuing CA up to a root CA that the VPN server trusts. Additionally, the VPN server must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain from the issuing CA up to a root CA that the VPN client trusts.

For example, if the VPN client was issued computer certificates by root CAs CertAuth1 and CertAuth2, it notifies the VPN server during main mode negotiation that it will accept certificates for authentication from only CertAuth1 and CertAuth2. If the VPN server does not have a valid computer certificate issued from a CA that follows a certificate chain to either CertAuth1 or CertAuth2, IPSec main mode negotiation fails.

A single CA commonly issues computer certificates to all computers in an organization. Because of this, all computers within the organization both have computer certificates from a single CA and request certificates for authentication from the same single CA.

Windows XP and Windows Server 2003 family VPN clients also support the use of preshared key authentication to create L2TP/IPSec connections.

Important

  • The use of preshared key authentication is not recommended because it is a relatively weak authentication method. Preshared key authentication creates a master key that is less secure (that might produce a weaker form of encryption) than certificates or the Kerberos V5 protocol. In addition, preshared keys are stored in plaintext. Preshared key authentication is provided for interoperability purposes and to adhere to IPSec standards. It is recommended that you use preshared keys only for testing or as an interim measure in a production environment, when your public key infrastructure (PKI) is being deployed to obtain computer certificates, or when preshared key authentication is required by your VPN server. For long-term use in production environment, it is recommended instead that you use certificates or Kerberos V5.

For a remote access VPN client running Windows XP or a Windows Server 2003 operating system, you can enable preshared key authentication and configure a preshared key in IPSec settings on the Security tab in the properties of a VPN connection in Network Connections.

Domain controllers and DHCP, DNS, or WINS servers

Before enabling IPSec for a computer that functions as a domain controller or DHCP, DNS, or WINS server, determine whether all of the clients are IPSec-enabled. If the IPSec policy assigned to the domain controller or DHCP, DNS, or WINS server is not configured to allow fall back to clear, which permits unsecured communication with non-IPSec-capable clients, secure negotiation will fail and access to these network services will be blocked.

SNMP

If you are using Simple Network Management Protocol (SNMP) and your SNMP management systems or agents include computers running Windows 2000, Windows XP, or a Windows Server 2003 operating system that are configured with IPSec policy, you can add a rule to the appropriate IPSec policy to prevent SNMP messages from being secured. The filter list must specify the source and destination addresses of the SNMP management systems and agents for all UDP traffic that uses ports 161 and 162. The filter action should be set to Permit, which passes through any traffic that matches the IP filter list.