The policy store organizes IPSec policy data and stores it in a format that the Policy Agent can use. In Windows Server 2003, policy data can be stored in the following:

• Active Directory
  
  
• Local and remote registry
  
  
• A file (for exporting and importing only)
  
  

In addition to providing an interface that UI services can use to store policy in each of these media, the policy store does the following:

• Provides policy data for default IPSec policies
  
  
• Checks policy information for consistency
  
  
• Retrieves policy version information
  
  

The policy store reads and writes policy information both to and from persistent storage and is aware of shared policy-setting dependencies. This ensures that all policies using shared settings are marked as changed when they are modified and that Windows Server 2003 IPSec components download the modified policies.

The Policy Agent retrieves IPSec policy information and delivers it to other IPSec components that require this information to perform security services, as shown in the following illustration.

Policy Agent Service Retrieving and Delivering IPSec Policy Information

The Policy Agent performs the following tasks:

• Retrieves the appropriate IPSec policy (if one has been assigned) from Active Directory if the computer is a domain member or from the local registry if the computer is not a member of a domain
  
  
• Determines filter list order
  
  
• Delivers the assigned IPSec policy information (IP filters) to the IPSec driver
  
  
• Delivers both main mode and quick mode settings to IKE
  
  
• Polls for changes in policy configuration. If the computer is a member of a domain, policy retrieval occurs when the computer starts, at the interval specified in the IPSec policy, and at the default Winlogon polling interval. You can also manually poll Active Directory for policy by using the gpupdate /target:computer command.
  
  

If there are no IPSec policies in Active Directory or the registry, or if the IPSec Policy Agent cannot connect to Active Directory, the IPSec Policy Agent waits for policy to be assigned or activated.

The Policy Agent appears in the list of computer services in the Services snap-in under the name IPSEC Services and starts automatically as part of the initialization of the Local Security Authority (LSA) service.

Each computer running Windows XP or a Windows Server 2003 has only one local Group Policy object, often called the local computer policy. Using this local Group Policy object allows Group Policy settings to be stored on individual computers regardless of whether they are members of an Active Directory domain.

The local registry maintains the IPSec policy configuration in the following registry key and its subkeys: HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\IPSec.

If you assign local IPSec policies and you do not assign Active Directory-based IPSec policies, the local policies are stored in the following registry key: HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\IPSec\Policy\Local.

If you assign Active Directory-based IPSec policies, the policies are read from Active Directory and stored in the local cache.

The local cache is part of the registry. If you assign an IPSec policy in Active Directory, the policy is stored in and read from Active Directory. A copy of the current policy in Active Directory is maintained in a cache in the local registry at: HKEY_LOCAL_MACHINE\Software\Policies \Microsoft\Windows\IPSec\Policy\Cache.

If a computer to which an IPSec policy in Active Directory is assigned cannot connect to the domain, the cached copy of the policy in Active Directory is applied. When the computer reconnects to the domain, new policy information for that computer replaces old, cached information. You cannot configure or manage the cached copy of an IPSec policy in Active Directory.

Interface Manager maintains a list of physical and logical network adapters on the computer and notifies the Policy Agent when interface and address changes occur. Interface Manager also maintains a complete list of generic filters. Generic filters are filters that are configured to use My IP Address either as a source address or as a destination address. Generic filters are saved in the appropriate IPSec policy storage location with either a source address or a destination address of 0.0.0.0 and a corresponding subnet mask of 255.255.255.255.

Each policy has a default response rule. The default response rule has the IP filter list of <Dynamic> and the filter action of Default Response when the list of rules is viewed with the IP Security Policies snap-in. The default response rule cannot be deleted, but it can be deactivated. It is activated for all of the default policies and in the IP Security Policy wizard.

IPSec uses the default response rule to ensure that the computer responds to requests for secure communication. If an active policy does not include a rule for a computer requesting secure communication, IPSec applies the default response rule and negotiates security. For example, if Host A intends to communicate securely with Host B, but Host B does not have an inbound filter defined for Host A, then IPSec uses the default response.

For the defense response rule, you can configure only the security methods for secure traffic and the authentication methods. The following table lists the default security methods for the default response rule.

Default Security Methods for the Default Response Rule

You can configure the security methods and their preference order on the Security Methods tab in the properties of the default response rule in the IP Security Policies snap-in.

The default response rule works in the following way:

1. If the IKE module receives a request to negotiate security, it queries the Policy Agent for a matching filter for traffic to and from the source and destination address of the ISAKMP message. If a matching filter is explicitly configured, the IKE negotiation is based on the settings of the associated rule.
   
   
2. If no matching filter is found and the default response rule is not activated, IKE negotiation fails.
   
   
3. If no matching filter is found and the default response rule is activated, then IKE dynamically creates an IP filter within the Policy Agent that corresponds to the traffic specification of the incoming ISAKMP message. IKE authenticates and negotiates security based on the settings on the Authentication Methods and Security Methods tabs for the default response rule.
   
   

You configure the default authentication method for the default response rule by using the IP Security Policy wizard.

Typically, the default response rule is used when a group of servers are configured with policy to secure communications between themselves and any IP address and to accept unsecured communication, but respond using secured communications. The client computers are configured with the default response rule. When the clients communicate with each other, the traffic is not secured. When the clients communicate with the server, the traffic is secured (with the exception of the initial packet sent by the client to the server).

For example, a client computer can reliably exchange data with a server by using Transmission Control Protocol (TCP). A TCP connection is established through the exchange of three TCP segments: SYN (synchronize), SYN-ACK (synchronize-acknowledgement), and ACK (acknowledgement).

Because the client computer does not have an explicit rule to secure traffic to the server, the client computer sends an unsecured SYN segment to the server. The server receives the SYN segment and checks its filters. It finds the filter that requires secure traffic to and from any IP address. However, the filter action settings also allow the server to accept unsecured communication, responding with secured communication. Consequently, the SYN segment sent by the client and received by the server is sent to the TCP/IP protocol on the server.

The TCP/IP protocol on the server responds by sending a SYN-ACK segment back to the client. This IP packet is sent to the IPSec driver on the server. Checking the filter settings to require secured communications, the IPSec driver notes that there are no active SAs between itself and the client. The IPSec driver requests that the IKE module negotiate SAs for secure communication.

The IKE module on the server sends an ISAKMP message to the client to begin the process of negotiating SAs for secure communications. When the ISAKMP message is received on the client computer, the IKE module on the client computer queries the filter lists in the Policy Agent. Because it finds no explicit filter match and the default response rule is activated, the IKE module creates a dynamic filter for traffic to and from the client and server computer.

IKE negotiation continues based on the policy settings of the server and the client. After IKE negotiation is finished, the server sends the secured TCP SYN-ACK segment to the client. The client responds with a secured TCP ACK segment and secured TCP data can be exchanged between the client and the server.

The AH protocol provides data origin authentication, data integrity, and anti-replay protection for the entire packet (both the IP header and the data payload carried in the packet), except for the fields in the IP header that are allowed to change in transit. AH does not provide data confidentiality, which means that it does not encrypt the data. The data is readable, but protected from modification and spoofing.

AH Transport Mode Packet Structure

As shown in the figure, data integrity and authentication are provided by the placement of the AH header between the IP header and the IP packet payload. The AH protocol uses keyed hash algorithms to sign the packet for integrity. The AH protocol is identified in the IP header with an IP protocol ID of 51. This protocol can be used alone or with the ESP protocol.

The following table describes the AH header fields.

AH Header Fields

The ESP protocol provides data origin authentication, data integrity, anti-replay protection, and the option of confidentiality for the IP payload only. ESP in transport mode does not protect the entire packet with a cryptographic checksum nor does it protect the IP header.

ESP Transport Mode Packet Structure

As shown in the figure, the ESP header is placed before the IP payload and an ESP trailer and ESP trailer and authentication data field are placed after the IP payload. The ESP protocol is identified in the IP header with the IP protocol ID of 50.

The following table describes the ESP header fields.

ESP Header Fields

The following table describes the ESP trailer fields.

ESP Trailer Fields

The following table describes the ESP authentication trailer field.

ESP Authentication Trailer Field

AH tunnel mode encapsulates an IP packet with an AH and IP header and signs the entire packet for integrity and authentication.

AH Tunnel Mode Packet Structure

ESP tunnel mode encapsulates an IP packet with both an ESP and IP header and an ESP authentication trailer.

ESP Tunnel Mode Packet Structure

The signed portion of the packet indicates where the packet has been signed for integrity and authentication. The encrypted portion of the packet indicates what information is protected with confidentiality.

Because a new header for tunneling is added to the packet, everything that comes after the ESP header is signed (except for the ESP authentication trailer) because it is now encapsulated in the tunneled packet. The original header is placed after the ESP header. The entire packet is appended with an ESP trailer before encryption occurs. All data that follows the ESP header, except for the ESP authentication trailer, is encrypted. This includes the original header, which is now considered to be part of the data portion of the packet.

The entire ESP payload is then encapsulated within the new tunnel header. The tunnel header is not encrypted because it is used only to route the packet from origin to tunnel endpoint.

If the packet is being sent across a public network, it is routed to the IP address of the gateway for the receiving intranet. The gateway decrypts the packet, discards the ESP header, and uses the original IP header to route the packet to the intranet computer.

When initiating an IKE exchange, IKE proposes protection mechanisms based on the applied security policy. Each proposed protection mechanism includes attributes for encryption algorithms, hash algorithms, authentication methods, and Diffie-Hellman groups. The first part of the main mode is contained in main mode messages 1 and 2.

The following table lists the protection mechanism attribute values that are supported by Windows Server 2003 IKE. These values are described in more detail in later sections.

Main Mode Attribute Values Supported by IKE

The encryption algorithm, integrity algorithm, and Diffie-Hellman group are configured as one of multiple key exchange security methods.

The initiating IKE peer proposes one or more protection suites in the same order as they appear in the applied security policy. If one of the protection suites is acceptable to the responding IKE peer, the responder selects it for use and responds to the initiator with its choice. Because the responding IKE peer might not be running Windows Server 2003 or Windows 2000 and is selecting the first proposed protection suite that is acceptable, the protection suites in the applied security policy should be configured in the order of most secure to least secure.

After a protection suite is negotiated, IKE queries a Diffie-Hellman CSP through CryptoAPI to generate a Diffie-Hellman public and private key pair based on the negotiated Diffie-Hellman group. The Diffie-Hellman public key is sent to the IKE peer in an ISAKMP Key Exchange payload. Main mode negotiation part 2 is contained in main mode messages 3 and 4.

The cryptographic strength of a Diffie-Hellman key pair is related to its prime number length (key size). Windows Server 2003 IKE supports the following Diffie-Hellman groups:

• Group 1 (768 bits)
  
  
• Group 2 (1024 bits)
  
  
• Group 14 (2048 bits)
  
  Note
  
  
  • For enhanced security, Windows Server 2003 IPSec includes Diffie-Hellman Group 14, which provides 2048 bits of keying strength. However, Diffie-Hellman Group 14 is not currently supported in Windows 2000 or Windows XP for general IPSec policy use. For updated information about the availability of Diffie-Hellman Group 14 for L2TP/IPSec connections for Windows XP and Windows 2000, search for article 818043, L2TP/IPSec NAT-T Update for Windows XP and Windows 2000, in the Microsoft Knowledge Base.
    
    

After the Diffie-Hellman public keys are exchanged, IKE accesses CryptoAPI to compute the shared key based on the mutually agreeable authentication method.

The following figure shows the Diffie-Hellman exchange between the IPSec peers and the relationship between IKE, CryptoAPI, and the Diffie-Hellman CSP.

IKE Diffie-Hellman Key Exchange

The Diffie-Hellman exchange occurs in the following steps:

1. On each IPSec peer, IKE requests that the first Diffie-Hellman CSP generate a Diffie-Hellman public and private key pair based on the Diffie-Hellman group selected during main mode messages 1 and 2.
   
   
2. The public portion of the Diffie-Hellman public and private key pair is returned to IKE.
   
   
3. The initiator sends the Diffie-Hellman public key to the responder in a Key Exchange payload (main mode message 3).
   
   
4. The responder sends its Diffie-Hellman public key to the initiator in a Key Exchange payload (main mode message 4).
   
   
5. On each IPSec peer, the IKE component sends the other IPSec peer’s Diffie-Hellman public key to the Diffie-Hellman CSP.
   
   
6. The Diffie-Hellman CSP computes the shared secret and returns the value to the IKE component.
   
   

IKE supports three methods of authentication:

• Kerberos v5
  
  
• Public key certificate
  
  
• Preshared key
  
  

The authentication that occurs is a computer-based authentication, also known as machine-based authentication. The authentication process verifies only the identity of the computers, not the individual using the computer when the authentication process occurs.

Kerberos v5 authentication

Kerberos v5 authentication is the default authentication standard in Windows Server 2003 and Windows 2000 domains. Any computer in the domain or a trusted domain can use this method of authentication.

Note

• In Windows Server 2003, the Kerberos protocol is no longer a default exemption. Therefore, if you want to enable Kerberos authentication, you must create filters in the IPSec policy that explicitly allow such traffic. For more information, see article 810207, “IPSec Default Exemptions Are Removed in Windows Server 2003” in the Microsoft Knowledge Base.
  
  

Windows IKE Kerberos authentication is based on the Generic Security Service (GSS) API IKE authentication method, which is described in the Internet draft entitled “A GSS-API Authentication Method for IKE.”

The following table lists the ISAKMP messages exchanged during a Kerberos authentication main mode negotiation.

Kerberos Authentication Method Main Mode Messages

 

Main Mode Message	Sender	Payload
1	Initiator	ISAKMP header, Security Association (contains proposals)
2	Responder	ISAKMP header, Security Association (contains a selected proposal)
3	Initiator	ISAKMP header, Key Exchange (contains Diffie-Hellman key), Nonce, Initiator Kerberos Token
4	Responder	ISAKMP header, Key Exchange, Nonce, Responder Kerberos Token
5*	Initiator	ISAKMP header, Identification, Initiator Hash
6*	Responder	ISAKMP header, Identification, Responder Hash

* ISAKMP payloads of message are encrypted.

Authentication occurs when:

• Each peer authenticates the other peer’s Kerberos token: The responder verifies the initiator’s Kerberos token and the initiator verifies the responder’s Kerberos token.
  
  
• Each peer’s hash is calculated and verified: The responder verifies the initiator’s hash and the initiator verifies the responder’s hash.
  
  

The hash calculation is performed over the following:

• The Diffie-Hellman public values of the initiator and responder
  
  
• The initiator and responder cookies
  
  
• The ISAKMP payloads of message 2
  
  
• The identity name string for the IPSec peer
  
  
• The IPSec peer’s Kerberos token
  
  

To perform Kerberos authentication, Windows Server 2003 uses a Kerberos SSP that is accessible through the SSPI. The following figure shows the exchange of Kerberos tokens between the IKE peers and the relationship between the IKE, SSPI, and the Kerberos SSP.

IKE Authentication with Kerberos

Kerberos authentication is performed in the following steps:

1. IKE on the initiator accesses the SSPI to initialize a security context.
   
   
2. The initiator’s Kerberos SSP creates a Kerberos token and returns it to the IKE component.
   
   
3. The initiator sends the initiator’s Kerberos token and computer identity to the responder in main mode message 3.
   
   
4. The responder IKE component accesses the SSPI to acquire a security context.
   
   
5. The responder’s Kerberos SSP creates a Kerberos token and returns it to IKE.
   
   
6. The responder sends the responder Kerberos token and computer identity to the sender in main mode message 4.
   
   
7. On each IPSec peer, IKE creates the hash for the next main mode message to be sent and then requests that SSPI sign the hash with the Kerberos session key.
   
   
8. The Kerberos SSP returns the signature to the IKE component.
   
   
9. The initiator sends the signed initiator hash to the responder in main mode message 5.
   
   
10. The responder sends the signed responder hash to the initiator in main mode message 6.
    
    
11. On each IPSec peer, IKE accesses the SSPI to compute the hash for the other peer. The initiator accesses the SSPI to compute the responder hash. The responder accesses the SSPI to compute the initiator hash.
    
    
12. The Kerberos SSP returns the computed hash to the IKE component where the hash value is verified to complete IKE authentication. The initiator compares its calculated responder hash with the responder hash received in main mode message 6. The responder compares its calculated initiator hash with the initiator hash received in main mode message 5.
    
    

The following sections describe the IKE certificate selection and acceptance process. If you decide to use certificates for IKE authentication, understanding this process and its requirements is integral to ensuring proper deployment.

Public key certificate authentication

                  netsh ipsec dynamic set config strongcrlcheck value={0 | 1 | 2}
                

Note

• Incorrectly editing the registry might severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.
  
  

1. Under the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\ key, add a new Oakley subkey, with a DWORD entry named StrongCRLCheck.
   
   
2. Assign this entry any value from 0 through 2, where:
   
   
   • A value of 0 disables CRL checking (this is the default for Windows 2000).
     
     
   • A value of 1 causes CRL checking to be attempted and certificate validation to fail only if the certificate is revoked (this is the default for Windows XP and Windows Server 2003). Other failures that are encountered during CRL checking (such as the revocation URL not being reachable) do not cause certificate validation to fail.
     
     
   • A value of 2 enables strong CRL checking, which means that CRL checking is required and that certificate validation fails if any error is encountered during CRL processing. Set this registry value for enhanced security.
     
     
3. Do one of the following:
   
   
   • Restart the computer.
     
     
   • Stop and then restart the IPSec service by running the net stop policyagent and net start policyagent commands at the command prompt.
     
     

Note that IPSec CRL checking does not guarantee that certificate validation fails immediately when a certificate is revoked. There is a delay between the time that the revoked certificate is placed on an updated and published CRL and the time when the computer that performs the IPSec CRL checking retrieves this CRL. The computer does not retrieve a new CRL until the current CRL has expired or until the next time the CRL is published. By default, IKE requests that CryptoAPI wait 15 seconds to complete the CRL retrieval. If the CRL cannot be retrieved at that time, IKE either ignores the error (if the value of the StrongCRLCheck registry subkey is set to 1, or it causes authentication to fail (if the value of StrongCRLCheck is set to 2). CRLs are cached in memory and in \Documents and Settings\UserName\Local Settings\Temporary Internet Files by CryptoAPI. Because CRLs persist across computer restarts, if a CRL cache problem occurs, restarting the computer does not resolve the problem.

Excluding the CA name from certificate requests

If you use certificate authentication to establish trust between IPSec peers, you can also use Windows Server 2003 to exclude CA names from certificate requests. Excluding the CA name prevents a malicious user from learning sensitive information about the trust relationships of a computer, such as the name of the company that owns the computer and the domain membership of the computer (if an internal PKI is being used). Although excluding the CA name from certificate requests enhances security, computers with multiple certificates from different roots might require the CA root names to select the correct certificate. Also, some non-Microsoft IKE implementations might not respond to a certificate request that does not include a CA name. For these reasons, excluding the CA name from certificate requests might cause IKE certificate authentication to fail in certain cases.

Certificate-to-account mapping

In Windows Server 2003, a specific group of computers can be authorized to use IPSec when either Kerberos v5 or certificates are used for IKE authentication. This capability enables much stronger peer authentication and allows IPSec to be used to restrict network access to a server. When you enable IPSec certificate-to-account mapping, the IKE protocol associates (that is, maps) a computer certificate to a computer account in an Active Directory domain or forest, and then retrieves an access token, which includes the list of computer security groups. This process ensures that the certificate offered by the IPSec peer corresponds to an active computer account in the domain, and that the certificate is one that should be used by that computer.

Certificate-to-account mapping can be used only for computer accounts that are in the same forest as the computer performing the mapping. This provides much stronger authentication than simply accepting any valid certificate chain. For example, you can use this capability to restrict access to computers that are within the same forest. Certificate-to-account mapping, however, does not ensure that a specific trusted computer is allowed IPSec access.

If the certificate-to-account mapping process is not completed properly, authentication will fail and IPSec-protected connections will be blocked.

Preshared key authentication

The quick mode negotiation process is implemented as defined in RFC 2409. All quick mode negotiation messages are protected with the main mode SA that was established during the main mode negotiation. Each successful quick mode negotiation establishes two quick mode SAs. One SA is inbound and the other SA is outbound.

The following table lists the quick mode messages exchanged by two IPSec peers running Windows IPSec.

Quick Mode Messages

* ISAKMP payloads of message are encrypted.

The four quick mode messages contain the following payloads:

• Security Association. This payload contains a list of proposals and encryption and hashing algorithms (AH or ESP, DES or 3DES, and HMAC-MD5 or HMAC-SHA1) for securing the traffic and a description of the traffic that is protected. This description might include IP addresses, IP protocols, TCP ports, or UDP ports, and is based on the matching filter of the initiator. The Security Association in the second quick-mode message includes a Security Association payload that contains the chosen method of securing the traffic.
  
  
• Hash. This payload provides verification and replay protection.
  
  
• Notification. This payload has a connected notify message. This message is requested and sent between two IPSec peers running Windows Server 2003. Quick mode message 4 with the Notification payload is not required by the IKE standard and is used to prevent the initiator from sending IPSec-protected packets to the responder before the responder is ready to receive them.
  
  
• For more information about the contents of ISAKMP payloads for quick mode exchanges, see RFC 2407 and RFC 2408 in the IETF RFC Database.
  
  

Windows Server 2003 IPSec supports the filter action choices listed in the following table.

Filter Action Choices

Note

• Computers running Windows Server 2003 and Windows XP support the 3DES and DES algorithms and do not require installation of additional components. However, computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 (or later) installed in order to use 3DES. If a computer running Windows 2000 is assigned a policy that uses 3DES encryption, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, the security method defaults to the weaker DES algorithm.
  
  

IKE generates session keys for both the inbound and outbound quick mode SAs based on the main mode shared master key and nonce material exchanged during the quick mode negotiation. Additionally, Diffie-Hellman key exchange material can also be exchanged and used to enhance the cryptographic strength of the IPSec session key.

In Windows Server 2003 IPSec, you can configure the number of times quick mode SAs can be created based on a single main mode SA. If you enable master key PFS, the IKE allows only a single quick mode SA for each main mode SA. By default, master key PFS is disabled, so there is no limit to the number of quick mode SAs that can be created from one main mode SA. To derive a new quick mode SA, a new main mode negotiation is performed, which includes a new Diffie-Hellman exchange and a new authentication process.

Whenever a quick mode SA requires renegotiation, IKE determines whether a session key PFS is specified in the corresponding filter rule. If it is, IKE additionally generates a new Diffie-Hellman key and exchanges it with the IKE peer during quick mode negotiation. By performing another Diffie-Hellman key exchange, IKE provides additional cryptographic strength to quick mode key generation beyond that already contributed by the main mode SA. Performing additional Diffie-Hellman exchanges requires additional computational resources and might affect IPSec performance.

The following figure illustrates this process.

Basic Inbound Packet Process

Note

• The inbound packet process applies only to local host unicast traffic (traffic with the unicast destination address of the host) when there is an active IPSec policy.
  
  

Basic inbound packet processing for transport mode occurs in the following sequence:

1. IP packets are sent from the network interface to the TCP/IP driver.
   
   
2. The TCP/IP driver sends the IP packet to the IPSec driver.
   
   
3. If the inbound packet is IPSec-protected, the IPSec driver looks up the SA in the SAD.
   
   
4. If the inbound packet is not IPSec-protected, the IPSec driver checks the packet for a filter match by looking up the filters in the SPD.
   
   
5. After the IPSec-protected inbound packet is authenticated and decrypted, the AH or ESP or both headers are removed and the packet is sent to the TCP/IP driver. If a packet that is not IPSec-protected is permitted by policy, that packet is sent to the TCP/IP driver.
   
   
6. The TCP/IP driver performs IP packet processing as needed and sends the application data to the TCP/IP application.
   
   

Detailed inbound packet processing for transport mode occurs in the following sequence:

1. The TCP/IP driver sends the unicast packet to the IPSec driver.
   
   
2. If the packet is ISAKMP, the unmodified packet is sent back to the TCP/IP driver.
   
   Note
   
   
   • To modify the default filtering behavior for Windows Server 2003 IPSec, you can use the Netsh IPSec context or modify the registry. For more information, see “Default exemptions to IPSec filtering” later in this section.
     
     
3. If hardware offload processing was performed, the IPSec driver checks to determine whether the hardware processing was successful.
   
   
4. If the hardware processing was not successful, an event is logged and the packet is discarded.
   
   
5. The packet is parsed to determine whether an AH or ESP header or both are present.
   
   
6. If the packet does not contain an AH or ESP header, the packet is compared to the filter list for a match.
   
   
7. If a filter match is not found, the unmodified packet is sent to the TCP/IP driver.
   
   
8. If a filter match is found, the IPSec driver attempts to find an SA based on the packet contents.
   
   
9. If an SA is not found, the matching filter is checked to determine if it is an inbound permit filter.
   
   
10. If the matching filter is an inbound permit filter, the unmodified packet is sent to the TCP/IP driver.
    
    
11. If the matching filter is not an inbound permit filter, the packet is discarded.
    
    
12. If an SA is found, it is checked to determine whether it is a soft SA. A soft SA is one in which the Negotiate security filter action is enabled, but there is no authentication or encryption being performed because the computer with which communication occurs is not running IPSec. This process is also known as fallback to clear. Even though the packet is not being protected, an SA without an AH or ESP header is still maintained in the SAD. Soft SAs and fallback to clear are possible only when Allow unsecured communication with non IPSec-aware computer is selected on the Security methods tab in the properties of a filter action.
    
    
13. If the SA is a soft SA, the unmodified packet is sent to the TCP/IP driver.
    
    
14. If the SA is not a soft SA, the packet is discarded.
    
    
15. If the packet contains an AH or ESP header (or both), the header is parsed for the SPI.
    
    
16. The SPI is used to look up the SA in the SAD.
    
    
17. If the SA corresponding to the SPI is not found in the SAD, a Bad SPI event is logged and the packet is discarded.
    
    
18. If the SA corresponding to the SPI is found in the SAD, the current time is used to update the SA’s last used time. The time is used for aging the SA.
    
    
19. The SA is checked to determine whether cryptographic processing for the SA was offloaded to hardware. For packets that have been processed by hardware offload, steps 20 and 21 are skipped.
    
    
20. The packet is authenticated or decrypted or both. This process involves verifying the HMAC in the AH or ESP header, processing the other fields in the AH and ESP headers and trailer, and decrypting the ESP payload.
    
    
21. If cryptographic processing is unsuccessful, an event is logged and the packet is discarded.
    
    
22. The AH or ESP headers and ESP trailer are removed.
    
    
23. If the SA for this packet is a tunnel SA (using either AH or ESP tunnel mode), the decapsulated packet is reinjected into the TCP/IP driver and the original packet is discarded. By reinjecting the decapsulated packet, the TCP/IP driver treats it as if it were received from the network adapter.
    
    
24. If the SA for this packet is not a tunnel SA, the IP packet, with the AH and ESP headers removed, is sent back to TCP/IP driver for additional processing.
    
    

Basic outbound packet processing is shown in the following figure.

Basic Outbound Packet Process

Basic outbound packet processing for transport mode occurs in the following sequence:

1. Application data is sent to the TCP/IP driver from the TCP/IP application.
   
   
2. The TCP/IP driver sends an IP packet to the IPSec driver.
   
   
3. The IPSec driver checks the packet for a filter match by looking up the filters in the SPD.
   
   
4. The IPSec driver checks the packet for an active SA by looking up the SAs in the SAD. Based on the SA, the traffic is authenticated or encrypted or both.
   
   
5. If the traffic must be protected and there is not an SA, the IPSec driver requests that IKE create the appropriate SAs. The IP packet is then held until the SA is established and can be IPSec framed.
   
   
6. The IP packet is sent back to the TCP/IP driver.
   
   
7. The TCP/IP driver sends the IP packet to the network interface.
   
   

Detailed outbound packet processing for transport mode occurs in the following sequence:

1. The TCP/IP driver sends the unicast outbound packet to the IPSec driver.
   
   
2. If the packet is ISAKMP, the unmodified packet is sent back to the TCP/IP driver.
   
   
3. The IPSec driver attempts to find a filter that matches the packet. If a filter is not found, the unmodified packet is sent back to the TCP/IP driver.
   
   
4. If a filter match is found, the IPSec driver attempts to find an SA that matches the packet.
   
   
5. If an SA is not found, the filter action is checked. If the filter action is set to Negotiate security, the IPSec driver requests that the IKE module negotiate the appropriate SAs.
   
   
6. If the IKE negotiation is successful, the IKE module informs the IPSec driver of the new SA and the IPSec driver looks up the SA again.
   
   
7. If the IKE negotiation is not successful, the packet is discarded.
   
   
8. If the filter action is set to Permit, the unmodified packet is sent back to the TCP/IP driver. Otherwise, the packet is discarded.
   
   
9. If an SA is found in the SAD, the current time is used to update the SA’s last used time. The time is used for aging the SA.
   
   
10. The SA is checked to determine whether it is about to expire. If the SA is about to expire, the IPSec driver informs the IKE module to initiate a quick mode or Phase 2 rekey of the quick mode SA.
    
    
11. The SA is checked to determine whether it has expired. If the SA has expired, the packet is discarded.
    
    
12. The Don’t Fragment (DF) flag in the IP header of the packet is checked. If the DF flag is set to 1, the size of the IP packet with the proposed AH or ESP or both headers and trailer is calculated.
    
    
13. If the size of the IP packet with the proposed IPSec overhead is larger than the path maximum transmission unit (PMTU) for the destination IP address, the IPSec driver indicates a packet-too-large condition for the packet and the unmodified packet is sent back to the TCP/IP driver. The packet-too-large condition allows the TCP/IP driver to either adjust the PMTU for the destination or, in the case of transit traffic, inform the sending host with an Internet Control Message Protocol (ICMP) Destination Unreachable-Fragmentation Needed and DF Set message that includes the new PMTU. The packet is eventually discarded by the TCP/IP driver.
    
    
14. If the DF flag is not set to 1, or if it is set to 1 and the additional IPSec overhead is not greater than the current PMTU for the destination, blank AH or ESP both headers and trailer are constructed (based on the settings for the SA).
    
    
15. The IPSec driver checks to determine whether the hardware offload is capable of offloading the SA for this packet. If so, the IPSec driver checks to determine whether the SA for the packet was offloaded to the hardware.
    
    
16. If the SA was offloaded to the hardware, an offload status is set on the packet and the modified packet with blank AH or ESP or both headers and trailer is sent to the TCP/IP driver.
    
    
17. If the SA has not been offloaded to the hardware, the IPSec driver accesses NDIS with instructions to add the SA to the hardware offload network interface.
    
    
18. If hardware offload is not enabled or the SA has not been offloaded to the hardware, the IPSec driver performs the cryptographic processing and adds the appropriate values in the fields of the AH or ESP or both headers and trailer.
    
    
19. The IPSec driver sends the modified packet to the TCP/IP driver.
    
    

In Windows Server 2003, the default filtering exemptions have been removed for Kerberos, Resource Reservation Setup Protocol (RSVP), and multicast and broadcast traffic, but remain for ISAKMP traffic, and inbound multicast and broadcast traffic.

To modify the default filtering behavior for Windows Server 2003 IPSec, you can use the Netsh IPSec context or modify the registry.

To modify the default filtering behavior using Netsh, use the following command:

                netsh ipsec dynamic set config ipsecexempt value={ 0 | 1 | 2 | 3} 

Depending on which exemptions you want, specify the appropriate values as follows:

• A value of 0 specifies that multicast, broadcast, RSVP, Kerberos, and ISAKMP traffic are exempt from IPSec filtering. This is the default filtering behavior for Windows 2000 (with Service Pack 3 and earlier service packs) and Windows XP.
  
  Note
  
  
  • Use this setting only if it is required for compatibility with Windows 2000 and Windows XP. If Kerberos traffic is exempted from filtering, an attacker can bypass other IPSec filters by using either UDP or TCP source port 88 to access any open port. Many port scan tools will not detect this because these tools do not allow setting the source port to 88 when checking for open ports.
    
    
• A value of 1 specifies that Kerberos and RSVP traffic are not exempt from IPSec filtering (multicast, broadcast, and ISAKMP traffic are exempt).
  
  
• A value of 2 specifies that multicast and broadcast traffic are not exempt from IPSec filtering (RSVP, Kerberos, and ISAKMP traffic are exempt).
  
  
• A value of 3 specifies that only ISAKMP traffic is exempt from IPSec filtering. This is the default filtering behavior for Windows Server 2003, Windows 2000 (with Service Pack 4 and later service packs) and Windows XP (with Service Pack 1 and later service packs).
  
  

If you change the value for this setting, you must restart the computer for the new value to take effect.

1. In Regedit, under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC key, add a new DWORD entry named NoDefaultExempt.
   
   
2. Assign this entry any value from 0 through 3.
   
   
3. Restart the computer.
   
   

The filtering behaviors for each value are equivalent to those noted above for the netsh ipsec dynamic set config ipsecexempt value=xcommand.

The following table summarizes the equivalent filters that are implemented if all default exemptions to IPSec filtering are enabled (that is, if NoDefaultExempt is 0). When the IP address is specified, the subnet mask is 255.255.255.255. When the IP address is Any, the subnet mask is 0.0.0.0.

Equivalent Filters When NoDefaultExempt=0

1In order for IPSec transport mode to be negotiated through an IPSec tunnel mode SA, ISAKMP traffic cannot be exempted if it needs to pass through the IPSec tunnel first.

2 When IPSec NAT-T is performed, the filter exemption for UDP port 4500 is automatically generated based on the source and destination IP addresses used during the initial part of the IKE negotiation on UDP port 500. This dynamic permit filter for port 4500 is displayed in the IP Security Monitor snap-in, under Quick Mode\Specific Filters, and in the output for the netsh ipsec dynamic show qmfilter command.

3Multicast traffic is defined as the class D range, with a destination address range of 224.0.0.0 with a 240.0.0.0 subnet mask, which corresponds to the range of addresses from 224.0.0.0 to 239.255.255.255.

4Broadcast traffic is defined as a destination address of 255.255.255.255 (the limited broadcast address) or as having the host ID portion of the IP address set to all 1’s (the subnet broadcast address).

5IPSec does not support filtering for IP version 6 (IPv6) packets, except when IPv6 packets are encapsulated with an IPv4 header.

Windows Server 2003 IPSec does not support specific filters for broadcast protocols or ports, nor does it support multicast groups, protocols, or ports. Because IPSec does not negotiate security for multicast and broadcast traffic, these types of traffic are dropped if they match a filter with a corresponding filter action to negotiate security. A filter with a source address of Any IP Address and a destination address of Any IP Address can block or permit all multicast and broadcast traffic. By default (and if the NoDefaultExempt registry key is set to a value of 2 or 3), outbound multicast or broadcast traffic will be matched against a filter with a source address of My IP Address and a destination address of Any IP Address. More specific unicast IP address filters that block, permit, or negotiate security for unicast IP traffic should be configured in the same IPSec policy to achieve appropriate security.

Hardware acceleration is accomplished by offloading specific processing tasks that are normally completed by an operating system component to the network adapter. Some network adapters can perform IPSec cryptographic functions, such as encryption and decryption of data and the calculation and verification of message authentication codes.

When the NDIS interface binds, the offload capability of the network interface is queried. During outbound packet processing, after an SA is created a check is made to ensure that the network interface can offload cryptographic functions, support transport-over-tunnel functionality, and support IP header options. If not, the packet cannot be offloaded. A check is also made to determine whether the SA for the packet being offloaded is a soft SA. A soft SA is an SA in which no authentication or encryption is being performed because the computer with which communication occurs is not running IPSec. Because no AH or ESP headers need to be processed, hardware offloading is unnecessary.

If hardware offloading is enabled, a check is made per packet to determine whether the SA for an outbound packet has already been offloaded to the offload adapter. If so, the existing offloaded SA is used. If the SA is not yet offloaded and the offload has not previously failed for this SA, an attempt is made to offload the SA. However, the attempt to offload the SA is made asynchronously. The IPSec driver does not wait for the SA offload to be successful before continuing to process the packet. This causes the first packet to always be cryptographically processed by the IPSec driver, with the cryptographic processing of following packets occurring on the hardware offload network adapter.

Typically, IPSec network offload adapters do not accelerate the IKE negotiation. However, some SSL offload adapters might be capable of processing the IKE Diffie-Hellman calculation in hardware. To determine whether your SSL offload adapter can do so, see the manufacturer’s documentation.

Windows 2000, Windows XP, and Windows Server 2003 provide hardware acceleration APIs in the Windows Driver Development Kit (DDK) as part of TCP/IP Task Offload. For more information about these APIs, see Task Offload on MSDN.