Simplifying IPsec Policy with the Simple Policy Update

Published: October 20, 2006 | Updated: February 12, 2008

Writer: Joe Davies

On This Page

IPsec Negotiation Behavior
Changes in IPsec Policy
Deployment Guidance
For More Information

Common Internet Protocol security (IPsec) scenarios require the configuration of an IPsec policy that contains rules for protected and permitted traffic. For some enterprise deployments, the IPsec policy rules can require hundreds of IP filter definitions that must be maintained over time. The Simple Policy Update for Microsoft® Windows® XP and Windows Server® 2003 changes the behavior of IPsec negotiation so that the IPsec policy rules can be simplified, in some cases drastically reducing the number of required IP filters and their ongoing maintenance.

Note

This article assumes knowledge of IPsec policy configuration and operation. For more information, see Description of an IPsec Policy.

Important

The Simple Policy Update has been incorporated into Windows Server 2003 Service Pack 2 and Windows XP Service Pack 3. All of the behaviors for the Simple Policy Update described in this article also apply to computers running Windows Server 2003 with Service Pack 2 or Windows XP with Service Pack 3.

IPsec Negotiation Behavior

For computers running Windows XP or Windows Server 2003 without the Simple Policy Update, negotiation for IPsec protection for outgoing traffic occurs in the following way:

  • If the outgoing traffic matches an IPsec rule that requires IPsec protection, the IPsec components determine if IPsec protection has already been negotiated. If so, IPsec protects the packet.

  • If protection for the outgoing traffic has not already been negotiated and is required, IPsec on the initiator begins the negotiation process by sending an initial Internet Key Exchange (IKE) message.

    • If the node receiving the initial IKE message supports IPsec, it responds with an IKE message and the negotiation proceeds. For more information, see IKE Negotiation for IPsec Security Associations.

      If the IPsec negotiation fails (for example, due to credential verification or mismatched policy settings), the communication between the initiator and the responder fails and the error is reported to the application on the initiator.

    • If the node receiving the initial IKE message is not using IPsec, it discards the IKE message. The initiator waits one second for a response and sends an additional initial IKE negotiation message. After waiting two additional seconds for a response to either initial IKE message, IPsec on the initiator checks the rule settings to determine if the initiator can send unprotected traffic, a behavior known as fallback to clear. For the outgoing unprotected traffic to be allowed, the rule in the active IPsec policy on the initiator for the outgoing traffic must be using a filter action with the Allow unsecured communication with non-IPsec-aware computer setting enabled.

      • If fallback to clear is allowed, the initiator begins sending unprotected traffic.

      • If fallback to clear is not allowed, the communication between the initiator and the responder fails and the error is reported to the application on the initiator.

For the Server and Domain Isolation scenarios, IPsec computers are configured to require protection for inbound communication attempts and to request (but not require) protection for outbound communication attempts. Therefore, trusted computers in Server and Domain Isolation scenarios use fallback to clear to initiate communication with computers on their intranets that do not use IPsec.

The Simple Policy Update changes the IPsec negotiation process in the following way:

  • It takes only 500 milliseconds (ms) for an initiator to fallback to clear.

  • IPsec negotiation failures will fallback to clear.

Because IPsec negotiation failures will now fallback to clear, it is possible for two Windows-based IPsec peers who cannot validate each other's credentials or have mismatched policy to fallback to clear in the following configuration:

  • The initiator is using fallback to clear (the rule in the active IPsec policy on the initiator for the outgoing traffic must be using a filter action with the Allow unsecured communication with non-IPsec-aware computer setting enabled).

  • The responder is using both inbound passthrough (the rule in the active IPsec policy on the responder for the incoming traffic must be using a filter action with the Accept unsecured communication, but always respond using IPsec setting enabled) and fallback to clear.

The following section describes how these changes can simplify the configuration of IPsec policy for Domain Isolation.

Changes in IPsec Policy

A typical IPsec policy for Domain Isolation without the Simple Policy Update contains the following rules:

  • Protect intranet traffic

    This rule requires inbound protection and requests outbound protection and contains a series of IP filters for all traffic for the address space of an intranet. This rule is restricted to the intranet address space to prevent inbound connection problems when a computer with this policy goes to another network. If this rule contained a single any-to-any IP filter, the computer would not respond to inbound connection attempts unless the initiating computer had matching IPsec policy settings and could perform authentication with the appropriate credentials. Additionally, for outbound initiated communications with an any-to-any IP filter, the IPsec computer would attempt to negotiate IPsec protection with every destination, incurring a 3-second fallback to clear delay when connected to networks other than the intranet.

    By specifying the address space of the intranet, the computer will not attempt to negotiate IPsec protection for outbound traffic or require protected traffic for inbound communication attempts when it is on other networks, such as the Internet or a home network.

    This rule has a filter list consisting of IP filters that correspond to the address prefixes that summarize the entire address space of the intranet. For example, if an organization is only using a single public address prefix for their intranet, the filter list has a single IP filter for the public address prefix.

  • Permit traffic to infrastructure servers

    This rule is to permit traffic and contains a series of IP filters for my IP address to the IP address of an infrastructure server. Without this rule, assuming that the infrastructure servers have an address from the address space in the IP filter lists of the “Protect intranet traffic” rule, IPsec computers would attempt to negotiate IPsec with infrastructure servers. IPsec computers that attempt to negotiate protection with an infrastructure server will incur the 3-second fallback to clear delay, which can cause timeout problems for some programs or services. Therefore, this rule is needed to prevent the 3-second fallback to clear delay.

    This rule has a filter list that contains an IP filter for every infrastructure server on the intranet, including Dynamic Host Configuration Protocol (DHCP) servers, Domain Name System (DNS) servers, Windows Internet Name Service (WINS) servers, and Active Directory® directory service domain controllers. For some organizations, this could correspond to hundreds of IP filters. This rule requires ongoing maintenance as infrastructure servers are either added or removed from the intranet.

  • Permit traffic to exempted computers

    This rule is to permit traffic and contains a series of IP filters for my IP address to the IP address of an exempted computer that cannot or should not use IPsec protection. Without this rule, assuming that the exempted computers have an address from the address space in the IP filter lists of the “Protect intranet traffic” rule, IPsec computers would attempt to negotiate IPsec protection to exempted computers.

    Exempted computers can be computers that run services that cannot tolerate the 3-second fallback to clear delay or computers that do not use IPsec but must be able to initiate inbound communications to protected computers, such as computers running operating systems other than Windows that perform system monitoring.

    This rule has a filter list that contains an IP filter for every exempted computer on the intranet. This rule requires ongoing maintenance as exempted computers are either added or removed from the intranet.

  • Permit ICMP traffic

    This rule is to permit traffic and contains a single IP filter for my IP address to any IP address for the Internet Control Message Protocol (ICMP). ICMP is needed for troubleshooting and for network processes such as path maximum transmission unit (PMTU) discovery. This rule does not require ongoing maintenance.

This set of rules applied to an intranet computer produces the following behaviors when the computer is connected to the intranet:

  • Traffic initiated by computers with this policy to other computers with this policy is protected. Traffic initiated by a computer that does not use IPsec to a computer with this policy is discarded, resulting in failed communication. A failed IPsec negotiation results in failed communication. This is the result of the “Protect intranet traffic” rule.

  • Traffic initiated to and from infrastructure servers is permitted (no IPsec negotiation or protection). This is the result of the “Permit traffic to infrastructure servers” rule.

  • Traffic initiated to and from exempted computers is permitted. This is the result of the “Permit traffic to exempted computers” rule.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

This set of rules applied to an intranet computer produces the following behaviors when the computer is connected to a network other than the intranet (such as the Internet or a home network):

  • Traffic initiated to and from all locations that do not use the same address space as the intranet is permitted. This traffic does not match any rule.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

When you deploy the Simple Policy Update, you can change your policies using the following options:

  • Option 1: Use the “Protect intranet traffic” rule

    With this option, you keep the “Protect intranet traffic” rule and simplify policy by eliminating the "Permit traffic to infrastructure servers" rule.

  • Option 2: Modify the “Protect intranet traffic” rule

    With this option, you modify the “Protect intranet traffic” rule to protect all traffic and further simplify policy by eliminating the "Permit traffic to infrastructure servers" rule.

Option 1: Use the “Protect intranet traffic” Rule

Due to the changes in IPsec negotiation behavior with the Simple Policy Update, IPsec policy for Domain Isolation with the Simple Policy Update deployed for this option can be simplified to contain the following rules:

  • Protect intranet traffic

    This rule is unchanged from the previously defined “Protect intranet traffic” rule.

  • Permit traffic to exempted computers

    This rule can be simplified by removing IP filters from the IP filter list that correspond to computers running services that cannot tolerate the 3-second fallback to clear delay.

  • Permit ICMP traffic

    This rule is unchanged from the previously defined “Permit ICMP traffic” rule.

Note that there is no longer a "Permit traffic to infrastructure servers" rule. Without the Simple Policy Update, the rule for the infrastructure servers was needed to prevent the 3-second fallback to clear delay. With the update, the fallback to clear time is 500 ms, a short enough time to prevent most application timeout problems. Therefore, the rule to permit traffic with infrastructure servers is no longer needed. This is the principal benefit of the Simple Policy Update.

The new negotiation behavior with the Simple Policy Update and this new set of rules applied to an intranet computer produces the following behaviors when the computer is connected to the intranet:

  • Traffic initiated by computers with this policy to other computers with this policy is protected.

  • Traffic initiated by computers with this policy to computers that do not use IPsec (including infrastructure servers) will fallback to clear after 500 ms.

  • Traffic initiated by a computer that does not use IPsec to a computer with this policy is discarded, resulting in failed communication.

  • Traffic initiated by exempted computers is permitted. This is the result of the “Permit traffic to exempted computers” rule.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

The new negotiation behavior with the Simple Policy Update and this new set of rules applied to an intranet computer produces the following behaviors when the computer is connected to a network other than the intranet (such as the Internet or a home network):

  • Traffic initiated to and from all locations that do not use the same address space as the intranet is permitted. This traffic does not match any rule.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

Option 2: Modify the “Protect intranet traffic” Rule

As another option, IPsec policy for Domain Isolation with the Simple Policy Update deployed can also be simplified to contain the following rules:

  • Protect all traffic

    This rule is to request outbound protection and require inbound protection and contains a single IP filter for any IP address to any IP address. Because the fallback to clear time has been reduced to 500 milliseconds, an any-to-any filter can now be used. Additionally, because a failed IPsec negotiation now falls back to clear, a computer with this policy will now be able to communicate with Windows-based IPsec-capable computers that have mismatched policy settings or improper credentials.

  • Permit traffic to exempted computers

    This rule can be simplified in the same way as described in the "Option 1: Use the “Protect intranet traffic” Rule" section.

  • Permit ICMP traffic

    This rule is unchanged from the previously defined “Permit ICMP traffic” rule.

Just as for Option 1, there is no longer a "Permit traffic to infrastructure servers" rule.

The new negotiation behavior with the Simple Policy Update and this new set of rules applied to an intranet computer produces the following behaviors when the computer is connected to the intranet:

  • Traffic initiated by computers with this policy to other computers with this policy is protected.

  • Traffic initiated by computers with this policy to computers that do not use IPsec (including infrastructure servers) will fallback to clear after 500 ms.

  • Traffic initiated by a computer that does not use IPsec to a computer with this policy is discarded, resulting in failed communication.

  • Traffic initiated by exempted computers is permitted. This is the result of the “Permit traffic to exempted computers” rule.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

The new negotiation behavior with the Simple Policy Update and this new set of rules applied to an intranet computer produces the following behaviors when the computer is connected to a network other than the intranet (such as the Internet or a home network):

  • Traffic initiated by computers with this policy will attempt IPsec protection to all destinations. Failed outbound IPsec negotiations will fallback to clear after 500 ms. This is the result of the “Protect all traffic” rule.

  • Traffic initiated by a computer that does not use IPsec to a computer with this policy is discarded, resulting in failed communication.

  • All ICMP traffic sent or received is permitted. This is the result of the “Permit ICMP traffic” rule.

For Option 2, the principal difference for communicating on networks other than the intranet is the following:

  • Without the Simple Policy Update, computers accept inbound communication attempts that are unprotected.

  • With the Simple Policy Update and the rule modifications, computers no longer accept inbound communication attempts that are unprotected.

This difference in behavior can cause problems for computers with this IPsec policy for server, peer, or listening programs or services that rely on inbound communications that are used on the Internet or a home network. IT departments can evaluate whether to allow selected unprotected inbound traffic for these programs or services and add the traffic as an additional IP filters in the “Permit traffic to exempted computers” rule. Additional exceptions for Windows Firewall or other host-based firewalls might also be needed.

Deployment Guidance

To simplify your Domain Isolation IPsec policy and take advantage of the behaviors in the Simple Policy Update, you must install the update on all of the computers on your intranet that are running Windows XP or Windows Server 2003 (computers running Windows 2000 should be upgraded) and then configure the IKEFlags registry key with its appropriate value. For information about the IKEFlags setting, see the Simple Policy Update. For example, you could use Computer Configuration Group Policy and an .ADM file to set the IKEFlags registry setting.

Next, you can transition your IPsec policy from the current rules to the new rules by doing the following (using the example rules described in this article):

  1. Analyze the set of computers in the IP filter list of your current “Permit traffic to exempted computers” rule to determine which computers can be removed because they cannot tolerate the 3-second fallback to clear delay.

  2. Remove the IP filters in the IP filter list of your current “Permit traffic to exempted computers” rule for those computers that cannot tolerate the 3-second fallback to clear delay.

  3. Remove the “Permit traffic to infrastructure servers” rule.

  4. If you are using Option 2:

    • Perform an analysis of the traffic for allowed server, peer, and listener programs and services when the computer is not connected to the intranet. Add IP filters for the traffic of allowed server, peer, and listener programs and services to the “Permit traffic to exempted computers” rule. Configure firewall exceptions as needed.

    • Change the “Protect intranet traffic” rule to the “Protect all traffic” by removing all of the IP filters for the intranet address space and replacing them with a single any-to-any IP filter.

For More Information

See the following resources for more information: