IPsec: Frequently Asked Questions

Internet Protocol security (IPsec) is a framework of open standards for ensuring private, secure communications over Internet Protocol (IP) networks, through the use of cryptographic security services. The Internet Engineering Task Force (IETF) IPsec working group defines the IPsec standards.

IPsec is the long-term direction for secure networking. It provides aggressive protection against private network and Internet attacks through end-to-end security. The only computers that must know about IPsec protection are the sender and receiver in the communication. IPsec provides the ability to protect communication between workgroups, local area network computers, domain clients and servers, branch offices (which might be physically remote), extranets, and roving clients.

The Windows Vista, Windows Server 2008, Windows XP, Windows Server 2003, and Windows 2000 implementations of IPsec are IETF standards-based.

IPsec documentation is included with Windows XP(click Start, then click HelpandSupport) and Windows Server 2003 (click Start, then click Help and Support). There are also IPsec sections of the Windows Server 2003 Deployment Guideand the Windows Server 2003 Technical Reference.

For a list of all the resources for IPsec in Windows, see the IPsec Web site .

Windows Vista and Windows Server 2008 include the following improvements to IPsec:

• Integrated firewall and IPsec configuration
• Simplified IPsec policy configuration
• Client-to-DC IPsec protection
• Improved load balancing and clustering server support
• Improved IPsec authentication
• New cryptographic support
• Integration with Network Access Protection
• Additional configuration options for protected communication
• Integrated IPv4 and IPv6 support
• Extended events and performance monitor counters
• Network Diagnostics Framework support

For more information, see the “IPsec Improvements” section of New Networking Features in Windows Server 2008 and Windows Vista.

The following IETF standards define IPsec:

• RFC 4301: Security Architecture for the Internet Protocol
• RFC 4302: IP Authentication Header
• RFC 4303: IP Encapsulating Security Payload (ESP)
• RFC 2407: The Internet IP Security Domain of Interpretation for ISAKMP
• RFC 2408: Internet Security Association and Key Management Protocol (ISAKMP)
• RFC 2409: The Internet Key Exchange (IKE)

Firewalls are designed to monitor incoming and outgoing traffic to determine whether the traffic is allowed. The Windows implementation of IPsec can also perform this function. However, IPsec can also ensure that the incoming and outgoing traffic is secure (protected with cryptography). For example, with the correct IPsec policy settings, you can require that all communications between domain controllers be secured.

Another key difference between IPsec for Windows and firewalls is the following:

• The default behavior of firewalls is to discard incoming or outgoing traffic unless there is an exception to allow it.
• The default behavior of IPsec for Windows is to allow incoming or outgoing traffic, unless there is an exception to discard or secure it.

The following usage scenarios are currently recommended:

• Server and Domain Isolation Using IPsec and Group Policy
• Using Microsoft IPsec for Windows to Help Secure an Internal Corporate Network Server
• Active Directory in Networks Segmented by Firewalls
• Improving Security with Domain Isolation

Because IPsec works at the IP layer of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol stack, you do not have to modify existing applications to use IPsec. All TCP/IP applications can use IPsec, whereas only SSL-enabled TCP/IP applications can use SSL. IPsec is an excellent solution to securing the traffic of legacy applications.

Other points of contrast between IPsec and SSL are the following:

• SSL was designed for client application-to-server application authentication and encryption. IPsec can be used end-to-end or for gateway-to-gateway scenarios.
• SSL only supports the use of digital certificates for authentication. The Windows implementation of IPsec supports the use of Kerberos, preshared key, and digital certificates for authentication.

With IPsec for Windows policy settings, you can block or permit incoming and outgoing traffic based on:

• The source and destination addresses based on IPv4 address ranges expressed as subnets
• The IP protocol number
• The source and destination Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) ports.

In contrast, with Windows Firewall you can only specify exceptions (incoming traffic that is permitted) based on source IPv4 address ranges expressed as subnets and destination TCP and UDP ports.

However, with Windows Firewall, you can do the following:

• Specify exceptions based on program names
• Permit or block Internet Protocol version 6 (IPv6) traffic and specify both port and program-based exceptions

An IPsec Policy is a group of settings that specify IPsec behavior with regard to the types of traffic that are permitted, blocked, or secured. An IPsec policy consists of:

• General IPsec policy settings—Settings that apply regardless of which rules are configured. These settings determine the name of the policy, its description for administrative purposes, how often to check for policy changes, key exchange settings, and key exchange methods.
• IPsec policy rules—One or more IPsec rules that determine which types of traffic IPsec must examine, how traffic is treated, how to authenticate an IPsec peer, and other settings such as the type of network connection to which the rule applies and whether or not to use IPsec tunneling.

After IPsec policies are created, an individual IPsec policy can be assigned (activated) at the domain, site, organizational unit, and local level.

Each IPsec rule contains the following configuration items:

• Filter list—A single filter list is selected that contains one or more predefined packet filters that describe the types of traffic to which the configured filter action for this rule is applied. The filter list is configured on the IP Filter List tab in the properties of an IPsec rule within an IPsec policy.
• Filter action—A single filter action is selected that includes the type of action required (Permit, Block, or Negotiate Security) for packets that match the filter list. For the Negotiate Security filter action, the negotiation data contains one or more security methods that are used (in order of preference) during IKE negotiations and other IPsec settings. Each security method determines the security protocol (such as Authentication Header [AH] or Encapsulating Security Payload [ESP]), the specific cryptographic and hashing algorithms, and session key regeneration settings used. The filter action is configured on the Filter Action tab in the properties of an IPsec rule within an IPsec policy.
• Authentication methods—One or more authentication methods are configured (in order of preference) and used for authentication of IPsec peers during main mode negotiations. The available authentication methods are the Kerberos V5 protocol, use of a certificate issued from a specified certification authority, or a preshared key. The authentication methods are configured on the Authentication Methods tab in the properties of an IPsec rule within an IPsec policy.
• Tunnel endpoint—Specifies whether the traffic is tunneled and, if it is, the IP address of the tunnel endpoint. For outbound traffic, the tunnel endpoint is the IP address of the IPsec tunnel peer. For inbound traffic, the tunnel endpoint is a local IP address. The tunnel endpoint is configured on the Tunnel Setting tab in the properties of an IPsec rule within an IPsec policy.
• Connection type—Specifies whether the rule applies to local area network (LAN) connections, dial-up connections, or both. The connection type is configured on the Connection Type tab in the properties of an IPsec rule within an IPsec policy.

The rules for a policy are displayed in reverse alphabetical order based on the name of the filter list selected for each rule. There is no method for specifying an order in which to apply the rules in a policy. IPsec for Windows automatically creates an IPsec filter list and orders the list based on the most specific to the least specific filter list. For example, a filter that specified individual IP addresses would be applied before a filter that specified all addresses on a subnet.

IPsec policy is configured with the IPsec Policy Management snap-in for the Microsoft Management Console (MMC) on all versions of Windows that support IPsec. This snap-in can be used to configure both local computer and domain-based policy. This snap-in is also available from the Group Policy snap-in in Computer Configuration\Windows Settings\Security Settings.

The command line tool that you can use to configure IPsec policy depends on the version of Windows:

• For Windows XP, use Ipseccmd.exe (for Windows XP Service Pack 2 use the updated version included in the Windows XP SP2 Support Tools for Advanced Users)
• For Windows Vista, Windows Server 2008, or Windows Server 2003, use the commands in the netsh ipsec context

An IP filter defines a specific set of IP traffic. The configuration parameters of an IP filter are the following:

• Source address (individual address or address range)
• Source address mask
• Source TCP port
• Source UDP port
• Destination address (individual address or address range)
• Destination address mask
• Destination TCP port
• Destination UDP port
• IP protocol

The default response rule, which can be used for all policies, has the IP filter list of <Dynamic> and the filter action of Default Response when the list of rules is viewed with the IP Security Policies snap-in. The default response rule cannot be deleted, but it can be deactivated. It is activated by default for all policies.

The default response rule is used to ensure that the computer responds to requests for secure communication. If an active policy does not have a rule defined for a computer that is requesting secure communication, then the default response rule is applied and security is negotiated. For example, when Computer A communicates securely with Computer B, and Computer B does not have an inbound filter defined for Computer A, the default response rule is used.

When enabled on a client computer, the default response rule allows the client to start communicating in the clear to a server with the Accept unsecured communication, but always respond using IPsec option enabled. The server will respond with a negotiation request that, if successful, protects the rest of the traffic. Security methods and authentication methods can be configured for the default response rule. The filter list of <Dynamic> indicates that the filter list is not configured, but that filters are created automatically based on the receipt of IKE negotiation packets. The filter action of Default Response indicates that the action of the filter (Permit, Block, or Negotiate Security) cannot be configured. Negotiate Security will be used. However, you can configure:

• The security methods and their preference order on the Security Methods tab.
• The authentication methods and their preference order on the Authentication Methods tab.

For examples of sets of IPsec rules for various IPsec deployment scenarios, see the following resources:

• Using Microsoft Windows IPSec to Help Secure an Internal Corporate Network Server
• Active Directory in Networks Segmented by Firewalls
• Server and Domain Isolation Using IPsec and Group Policy

The netsh ipsec dump command was never implemented for two main reasons:

• The general expectation is that netsh ipsec commands will be entered through a batch file; therefore most network administrators will already have a copy of their IPsec configuration.
• Because of the way that IPsec filters are expanded and stored, a netsh ipsec dump command could not present the IPsec configuration in the exact form that it was originally entered.

The default exemptions for IPsec is specified by the NoDefaultExempt registry value (located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC), which has the following possible settings:

• 0—Specifies that multicast, broadcast, Resource Reservation Protocol (RSVP), Kerberos, and IKE (ISAKMP) traffic are exempt from IPsec filtering. This is the default filtering behavior for Windows 2000 and Windows XP.
• 1—Specifies that Kerberos and RSVP traffic are not exempt from IPsec filtering, but multicast, broadcast, and IKE traffic are exempt. This is the recommended value for Windows 2000 and Windows XP.
• 2—Specifies that multicast and broadcast traffic are not exempt from IPsec filtering, but RSVP, Kerberos, and IKE traffic are exempt. Supported only in Windows Server 2003.
• 3—Specifies that only IKE traffic is exempt from IPsec filtering. Supported only in Windows Server 2003.

You can change the value of the NoDefaultExempt registry key in Window Server 2003 with the netsh ipsec dynamic set config ipsecexempt value={ 0 | 1 | 2 | 3} command.

For more information, see IPsec default exemptions are removed in Windows Server 2003.

You can view the IPsec filter list with the IP Security Monitor snap-in provided with Windows XP and Windows Server 2003. To add the IP Security Monitor snap-in, do the following:

1. Click Start, click Run, type MMC, and then click OK.
2. Click File, click Add/Remove Snap-in, and then click Add.
3. Click IP Security Monitor, and then click Add.
4. Click Close, and then click OK.

To view the IPsec filter list, you need to open the Main Mode and Quick Mode folders in the console tree. In the Main Mode folder, click Specific Filters to view the filters in the IPsec filter list that require security., In the Quick Mode folder, click Specific Filters to view all of the filters in the IPsec filter list. For more information about the IPsec filter list, see IPsec Filter Ordering.

With the Windows Server 2003 family, if you use either Kerberos V5 or certificate authentication, you can set restrictions on which computers are allowed to connect. This functionality allows you to use IPsec to allow or deny any of the following access to a server running Windows Server 2003:

• Computers that are members of a specific domain.
• Computers that have a certificate from a specific issuing certification authority.
• A specific group of computers.
• A specific computer.

When you enable certificate to account mapping in IPsec, the IKE protocol associates (maps) a computer certificate to a computer account in an Active Directory domain or forest, and then retrieves an access token, which includes the list of the user rights that are assigned to the computer. You can restrict access by configuring Group Policy security settings and assigning either the Access this computer from the network user right or the Deny access to this computer from the network user right to individual or multiple computers, as needed.

For more information about certificate to account mapping for IPsec, see Authentication methods.

You can either add the servers that the visiting computers need to access to the boundary zone, or you can use certificates for IPsec authentication to the specific servers and install computer certificates on the visiting computers. If the visiting computers are running Windows 2000 with Service Pack 3 or earlier, or versions of Windows prior to Windows 2000, you must add the servers that the visiting computers need to access to the boundary zone.

For more information, see the Interoperability Considerations for IPsec Server and Domain Isolation white paper.

IPsec requires the following attributes for certificates used in IPsec authentication:

• Must contain an RSA public key that has a corresponding private key that can be used for RSA signatures
• Cannot be expired
• Must have been issued from a trusted root certification authority

For additional information, see the "IKE Main Mode and Quick Mode Negotiation" section of How IPsec Works.

Results vary because there are many factors affecting the performance of IPsec such as processor speed and the types of network adapters. In Microsoft testing, the following results were achieved on an Intel Pentium III-based computer, running at 993 MHz, and with 384 MB of RAM:

The most time and processor-intensive part of an IPsec-secured connection is the main mode negotiation, from which the master key is derived.

The Oakley log records all IKE (ISAKMP) main mode and quick mode negotiations. To enable Oakley logging, do the following:

• For computers running Windows 2000, set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLogging registry setting to 1. The Oakley key does not exist by default and must be created.
• For computers running Windows XP, use the ipseccmd set logike command.
• For computers running Windows Server 2003, use the netsh ipsec dynamic set config ikelogging 1 command.

The Oakley log is stored in the systemroot\Debug folder. A new Oakley.log file is created each time the IPsec policy agent is started and the previous version of the Oakley.log file is saved as Oakley.log.sav.

You can use the Windows XP Event Viewer snap-in to view the following IPsec-related events:

• IPsec Policy Agent events in the audit log.
• IPsec driver events in the system log. To enable IPsec driver event logging, set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ IPSEC\DiagnosticMode registry setting to 1. You must restart the computer for this change to take effect. The IPsec driver only writes events to the system log once an hour.
• IKE events (SA details) in the audit log. To view these events, enable success or failure auditing for the Audit logon events audit policy for your domain or local computer.
• IPsec policy change events in the audit log. To view these events, enable success or failure auditing for the Audit policy change audit policy for your domain or local computer.

You can remove static local IPsec policy settings with the following:

• The IP Security Policies snap-in for Windows 2000, Windows XP, or Windows Server 2003
• The Ipseccmd.exe tool for Windows XP
• Commands in the netsh ipsec static context for Windows Server 2003

The negotiation of a secured IPsec session has two distinct phases: main mode and quick mode. The main mode negotiation creates a bidirectional main mode SA (also known as an ISAKMP SA), which is a secure channel through which the quick mode negotiation and all future IKE traffic takes place.

Main mode negotiation accomplishes the following:

• Negotiates security parameters for IKE traffic. These parameters include the authentication method, lifetime of the main mode SA, the Diffie-Hellman group to be used to generate a shared secret, and how the IKE traffic is to be protected (encryption and HMAC algorithms).
• Exchanges Diffie-Hellman keying material. For a set of publicly exchanged keys, a mutually determined secret key is calculated.
• Authenticates the identities of the IPsec peers (Kerberos, digital certificates, or preshared key).

Configure your router-based firewall to allow the following:

• UDP port 500 (IKE traffic)
• UDP port 4500 (IPsec NAT-T traffic)
• IP protocol 50 (ESP-protected traffic)
• IP protocol 51 (AH-protected traffic)