Windows Server 2003 IPSec Concepts

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

You can use the Windows Server 2003 implementation of IPSec to compensate for the limited protections provided by applications for network traffic, or as a network-layer foundation of a defense-in-depth strategy. Do not use IPSec as a replacement for other user and application security controls, because it cannot protect against attacks from within established and trusted communication paths. Your authentication strategy must be well defined and implemented for the potential security provided by IPSec to be realized, because authentication verifies the identity and trust of the computer at the other end of the connection. For example, if IPSec is required to encrypt all traffic only to trusted domain member computers, but there are weak controls on how computers can join the domain, the IPSec-protected traffic might be exposed to a trusted man-in-the-middle attack. Similarly, if attackers can join their computers to the domain, they might obtain trusted, encrypted access to IPSec-protected servers. If computers cannot be properly authenticated, IPSec does not provide security.

Table 6.1 outlines the specific security needs that IPSec addresses.

Table 6.1   IPSec Security Solutions

Security Concern IPSec Solution

Network attacks using specific protocols or ports.

Two methods:

  • IPSec traffic blocking, to block specified traffic.

  • IPSec policy filter lists can allow traffic from only trusted senders over particular protocols to particular addresses and ports.

Eavesdropping. Also known as sniffing. Eavesdropping occurs when an attacker uses a frame capture program (also known as a sniffer) to view the data that is placed on the network.

IPSec uses the Encapsulating Security Payload (ESP) protocol to encrypt data by using Triple Data Encryption Standard (3DES) or Data Encryption Standard (DES) so that IP packets cannot be read if intercepted in transit.

Data modification. If an intruder can access the information, they might be able to change it in such a way that the recipients cannot detect that any change has occurred.

IPSec uses a cryptographic checksum that incorporates a secret key to provide data integrity. Although a packet can be modified, the checksum cannot be updated for the proper value without knowledge of the secret key. An invalid checksum indicates to the receiving computer that the packet was modified in transit.

Identity spoofing. Intruders falsely using an IP address to compromise packet filter security.

IPSec uses Kerberos, public key certificates, or preshared key authentication to verify the identity of computer systems before the application level communication can take place.

Denial-of-service attack. By flooding a computer or network with malicious, malformed, or useless information, an attacker can disrupt normal operations.

IPSec uses IP packet filtering to determine whether communication is allowed, secured, or blocked, according to specified IP address ranges, protocols, or TCP or User Datagram Protocol (UDP) ports. IPSec cannot be targeted by denial-of-service attacks.

When communication is secured between computers, IPSec authenticates computer identities and negotiates security options. Security options include aspects such as encryption strength or acceptable authentication methods. Negotiating security options ensures that communication partners agree how to secure the connection. By requiring IPSec for all network access to a server, you can ensure that only trusted computers can access the server. Active Directory can be used to deliver Group Policy security settings for all clients in a domain, as well as for groups of servers in an OU. Thus you can create, distribute, and enforce access rules for groups of computers for any of the following:

  • Computers that are members of a specific domain

  • A specific group of computers with common needs or roles in an OU

  • A specific computer

Compatibility

Operating systems older than Microsoft® Windows® 2000 do not provide built-in support for IPSec. These include Microsoft® Windows® 98, Windows® Millennium Edition, and Microsoft® Windows NT®. If you have computers running these operating systems in your environment, make sure they are not required to use IPSec because the enforcement of IPSec-secured communications denies them access to resources.

Several features in the Windows Server 2003 implementation of IPSec are not provided in Windows 2000 or in Microsoft® Windows® XP Professional. If you plan to apply IPSec policies that use any of the new features that are available only in the Windows Server 2003 implementation of IPSec, do not use the Windows XP or Windows 2000 IP Security Policy Management snap-ins to manage these policies. Using those operating systems to manage Windows Server 2003 policies will result in the new features in the Windows Server 2003 policies being lost. IPSec policies that do not use the new Windows Server 2003 features, however, can be applied to computers running any of the three operating systems.

Remote management and monitoring of IPSec is only supported between computers running the same version of the Windows operating system. For remote management and monitoring of IPSec, it is recommended that you use Remote Desktop Connection.

For central management of IPSec policy on computers running Windows XP, Windows 2000, and Windows Server 2003, use Active Directory.

The IPSec internal infrastructure components were significantly modified for Windows Server 2003. Policies created on computers running Windows XP, Windows 2000, and Windows Server 2003 can be shared between any of those operating systems, but many tools have changed for creating and managing those policies and some tools are not installed by default. In Windows Server 2003, use the IP Security Policy Management snap-in, the Netsh IPSec context, and the Resultant Set of Policy (RSoP) snap-in for management and the IP Security Monitor snap-in for monitoring. Netdiag.exe, which is included in the Support Tools folder of the Windows Server 2003 operating system CD, no longer includes IPSec functionality. IPSecmon.exe, which was included in Windows 2000, is not included with Windows Server 2003 (it has been replaced by the new IP Security Monitor snap-in). IPSecpol.exe, which was included with the Microsoft® Windows®* 2000 Server Resource Kit*, is not included with Windows Server 2003. IPSeccmd.exe, which was included in the Support Tools folder of the Windows XP operating system CD, is not included with Windows Server 2003. For Windows Server 2003, the Netsh IPSec context replaces and enhances the functionality provided by Netdiag.exe and IPSeccmd.exe.

New in Windows Server 2003

IPSec provides new features that enhance security, scalability, availability, ease of deployment, and administration. Although the additions to Windows Server 2003 provide greater security, they are not always compatible with earlier versions of Windows. Ensure that you understand which technologies work with clients in your environment, and then determine whether to upgrade clients, postpone the use of specific IPSec technologies, or a mix of both. Then test your plan in a lab before deploying it in your production environment.

The new technologies highlighted in this section focus on those pertinent to deployment.

Filters update IP configuration of IPSec partners   You can use the IP Security Policy Management snap-in or Netsh to configure the source or the destination address fields that the target computer interprets as the addresses for the Dynamic Host Configuration Protocol (DHCP) server, the Domain Name System (DNS) servers, the Windows Internet Name Service (WINS) servers, and the default gateway. As a result, IPSec policies can now automatically accommodate changes in the IP configuration of the source or destination, by using either DHCP or static IP configurations. Enhanced subnet definitions are also supported by Windows Server 2003 IPSec.

NAT traversal support has been added   Windows Server 2003 IPSec meets new Internet Engineering Task Force (IETF) draft version 2 specifications, "UDP Encapsulation of IPSec Packets" and "Negotiation of NAT-Traversal in the IKE," which describe methods by which ESP-protected IPSec traffic can pass through a network address translator (also known as a NAT). However, some applications might not work when their traffic is first protected with IPSec ESP and then passed through a network address translator. The Internet Key Exchange (IKE) protocol automatically detects network address translators and uses UDP ESP encapsulation to send all user data by using UDP port 4500. Using the Authentication Header (AH) protocol across network address translators is not supported by IPSec NAT traversal (NAT-T).

Netsh enables command-line support for IPSec   Replacing IPSecpol.exe and IPSeccmd.exe, the new IPSec context of the Netsh command-line tool can be used to automate IPSec configuration, deploy IPSec remotely or across many computers, and perform other tasks. For example, domain logon scripts, startup scripts, and other files can be used to configure IPSec policy. For more information, see "Netsh commands for Internet Protocol Security (IPSec)" in Help and Support Center for Windows Server 2003.

Default exemptions have been removed   IPSec provides greater protection because filtering exemptions that existed in Windows XP and Windows 2000 have been removed from Windows Server 2003. In Windows XP and Windows 2000, all broadcast, multicast, Internet Security Association and Key Management Protocol (ISAKMP), Kerberos, and Resource Reservation Protocol (RSVP) traffic was exempt from IPSec filtering. In Windows Server 2003, only ISAKMP traffic is exempt from IPSec filtering. Exemptions can be added by using the IP Security Policy Management snap-in or the Netsh IPSec context. The Kerberos protocol is no longer a default exemption; if you want to enable Kerberos authentication, you must create filters in the IPSec policy that explicitly allow all traffic to each domain controller IP address in the domain member’s domain. If you created policies for Windows XP or Windows 2000, you might need to configure explicit permit filters to maintain the expected behaviors of the IPSec policy on servers running Windows Server 2003. For more information, see "Understanding IP Filters, Filter Actions, and Filter Lists" later in this chapter.

RSoP support has been added   RSoP provides information that can help assess and resolve unexpected IPSec behavior resulting from Group Policy conflicts. For more information, see "Using RSoPd to Analyze IPsec Policy Assignments" later in this chapter.

Other improvements that enhance security and IPSec policy management include:

  • 2048-bit Diffie-Hellman keys, which provide very strong cryptographic key generation to maximize 3DES encryption strength.

  • Persistent IPSec policy, which enhances security during computer startup. Additionally, persistent IPSec policy provides a method for local administrators to force local IPSec policy settings to be applied when Active Directory-based policy is assigned. For more information about persistent IPSec policy, see "Assigning IPSec Policies Locally" later in this chapter.

  • Stateful filtering of network traffic during computer startup, which permits only the following traffic during computer startup: the outbound traffic that the computer initiates during startup, the inbound traffic that is sent in response to the outbound traffic, and DHCP traffic.

  • The IP Security Monitor snap-in, which displays more details about IPSec communication than IPsecmon.exe (used for monitoring IPSec in Windows 2000). These additional details and performance enhancements enable effective monitoring of IPSec-secured servers.

For more information about new features of Windows Server 2003, see "New features for IPSec" in Help and Support Center for Windows Server 2003.

Terms and Definitions

This section provides brief definitions of core IPSec and IPSec-related terms. For more information, see the sources mentioned in "Additional Resources for Deploying IPSec" later in this chapter, or Help and Support Center for Windows Server 2003.

Authentication   The process for verifying that a peer or object is who or what it claims to be. Examples include confirming the source of information, such as verifying a digital signature, or verifying the identity of a computer.

Authentication Header (AH)   An IPSec protocol that provides data origin authentication, data integrity, and anti-replay for the entire packet (the IP header and the data payload carried in the packet, except fields in the IP header that are allowed to change in transit). AH can be used alone, in combination with ESP, or in IPSec tunnel mode. Windows AH functionality complies with RFC 2402.

Cryptography   The process of keeping messages and data secure. Cryptography enables and ensures data confidentiality, data integrity, authentication (peer and data origin), and nonrepudiation.

Default response   An IPSec rule that is used to ensure that the computer responds to requests for secure communication. If an active policy does not have a rule defined for a computer that is requesting secure communication, the default response rule is applied and security is negotiated. The default response rule cannot be deleted, but it can be deactivated. It is activated by default for all policies.

Encapsulating Security Payload (ESP)   An IPSec protocol that provides data confidentiality, data origin authentication, data integrity, and anti-replay for the ESP payload. ESP can be used alone, in combination with AH, or in IPSec tunnel mode. Windows ESP functionality complies with RFC 2406.

Filter action   The part of an IPSec rule that defines the security requirements for the data transmission. A filter action can be configured to permit traffic, block traffic, or negotiate secure communications with IPSec. If security negotiation is selected, you must also configure security methods and their order, whether to accept initial incoming unsecured traffic, whether to allow unsecured communication with computers that do not support IPSec, and whether to use perfect forward secrecy (PFS).

Group Policy   The infrastructure within Active Directory that enables directory-based change and configuration management of user and computer settings, including security and user data. You use Group Policy to define configurations for groups of users and computers. The Group Policy settings that you create are contained in a Group Policy object (GPO). By associating a GPO with selected Active Directory system containers — domains, sites, and OUs — you can assign the GPO’s policy settings to the users and computers in those Active Directory containers. Although Group Policy applies to both users and computers, IPSec policy is a computer configuration Group Policy setting that is only applied to computers.

Group Policy Object (GPO)   A collection of Group Policy settings that are stored at the domain level that affect users and computers contained in domains, sites, and OUs. In addition, each computer has one local GPO. To create an individual GPO, use the Group Policy Object Editor snap-in. Although GPOs can apply to both users and computers, IPSec policy objects can only be applied to computers.

IPSec authentication   The process of verifying trust in an identity. IKE uses mutual authentication between computers to establish trusted communications, and requires the use of one of the following authentication methods: Kerberos V5, a computer X.509 v3 public key infrastructure (PKI) certificate, or a preshared key. The two communication endpoints must have at least one common authentication method, or communication fails.

IPSec filter   A specification in an IPSec rule that is used to match IP packets to filter actions (permit, block, or negotiate security). An IP filter list contains one or multiple filters, and it can be shared among different IPSec policies. For example, one filter list can contain the list of each domain controller IP address in a domain.

IPSec policy   A collection of one or more IPSec rules that determine IPSec behavior.

IPSec rule   A statement in an IPSec policy that associates a filter list with a filter action, an authentication method, and an IPSec mode. An IPSec rule is typically configured for a specific purpose (for example, "Block all inbound traffic from the Internet to TCP port 135"). Many IPSec rules can be defined in a single IPSec policy.

IPSec transport mode   A mode of IPSec that is used to protect host-to-host communications. Transport mode provides security between computers that are on the same local area network (LAN) or connected by private wide area network (WAN) links.

IPSec tunnel mode   A mode of IPSec that is used to protect site-to-site (gateway-to-gateway) traffic between networks, such as site-to-site networking through the Internet. The sending gateway encapsulates the entire IP datagram by creating a new IP packet that is then protected by one of the IPSec protocols. Windows Server 2003 supports IPSec tunnel mode for configurations where Layer Two Tunneling Protocol (L2TP) cannot be used. For more information about recommended uses of tunnel mode, see "Using IPSec in Tunnel Mode" later in this chapter.

Netsh   A command-line scripting tool for configuring networking components on the local computer or on remote computers running Windows 2000, Windows XP Professional, or Windows Server 2003. The Netsh IPSec context is only available on Windows Server 2003, although policies created with Netsh can be used for computers running other versions of Windows that support IPSec.

Perfect forward secrecy (PFS)   A mechanism that determines whether the existing keying material for a master key can be used to derive a new session key. Session key PFS performs a new Diffie-Hellman key exchange to generate new master key keying material instead of using master key keying material to derive more than one session key.

Tunnel endpoint   The tunneling computer that is closest to the IP traffic destination, as specified by the associated IP filter list. Two rules are required to describe an IPSec tunnel. For the outbound traffic rule, the tunnel endpoint is the IP address or subnet of the IPSec peer on the other end of the tunnel. For the inbound traffic rule, the tunnel endpoint is an IP address or subnet configured on the local computer.