Executive Summary

The Microsoft Information Technology organization (known as Microsoft IT) is introducing Domain Isolation to the Microsoft global enterprise network as part of Microsoft's Embody Trustworthy Computing efforts. The purpose of Domain Isolation is to prevent unauthorized access (external and internal) to trusted assets. The technology chosen for isolation is Internet Protocol Security (IPsec). The result of these efforts is a secure, segmented network of trusted computers, which Microsoft calls "SecureNet."

This white paper focuses on Microsoft IT's experiences with planning and deployment of IPsec to create SecureNet—their secure network. The technical elements covered in this white paper include:

• Technical overview of IPsec. Background information for the technology chosen to create a trusted network segment. In addition, other technologies considered for the project, and the reasons IPsec was chosen are detailed.
• Microsoft Security environment. An overview of the specific challenges and considerations of the corporate network that influenced deployment decisions.
• Planning. The steps taken while planning the deployment of IPsec across the enterprise.
• The use of Group Policy for IPsec distribution. Specific architecture of the Group Policy Objects used to deploy IPsec.
• Microsoft IT IPsec Policy settings and filter design. A description of the filters, filter lists, rules, and policies deployed to hosts in the SecureNet, including how Microsoft IT manages computers not capable of using IPsec or not centrally managed.
• Known issues and best practices. Specific problems encountered and lessons learned while planning and deploying IPsec across the enterprise.

This paper is written for enterprise technical decision-makers, IT implementers, and architects who want to gain a better understanding of the processes and implementation of IPsec for isolating a corporate domain. This paper does not include a detailed discussion of other initiatives, such as patch management, security health management, or perimeter defenses. These initiatives are covered in other white papers published at http://www.microsoft.com/technet/itshowcase.

The essential enabling technologies described in this paper run on Microsoft® Windows Server™ 2003. However, most of the technologies were developed when the infrastructure was based on Microsoft Windows® 2000 Server, and can be implemented on Windows 2000 Server SP 3 and Windows XP. All of the technologies and deployments continue to mature, based on evolving security requirements, future strategic plans, and product testing and validation requirements.

This paper is based on Microsoft IT's experience and recommendations as an early adopter. It is not intended to serve as a procedural guide. Each enterprise environment has unique circumstances; therefore, each organization should adapt the plans and lessons learned described in this paper to meet its specific needs.

Introduction

Microsoft IT is introducing Domain Isolation to the Microsoft corporate network as part of its Embody Trustworthy Computing initiative. The purpose of Domain Isolation is to prevent access from untrusted devices to trusted assets on the corporate network.

In the past, Microsoft operated a largely open corporate network. This was not an issue because it was assumed that computers behind the corporate firewall were secure. While Microsoft IT managed approximately 50 percent of the desktop computers on the corporate network through Systems Management Server (SMS), Group Policies, and other domain management tools, computers not joined to a managed domain could still connect to assets inside the firewall and access their resources. These unmanaged machines (not joined to the domain) were by definition untrusted because their security health was largely unknown.

As the corporate network and the Internet matured, Microsoft IT became more aware of the risk that these untrusted computers pose to the organization. Computers connecting to the corporate network that are not joined to the domain provide a relatively easy way for attackers to gain a foothold on the corporate network. Microsoft IT decided to improve the security of the corporate network by using IPsec to enhance the security and isolation of trusted, IT-managed computers inside the corporate firewall.

Levels of Trusted Assets

At Microsoft, there are different levels of trusted assets, each requiring different security policies. At the base level, computers managed by Microsoft IT meet domain, operating system, group, and IPsec policy conditions. Any domain-joined computer meeting these conditions is considered trusted. Computers that do not meet these requirements, such as being joined to the domain—or being joined to the domain but without valid security configurations—are untrusted.

Figure 1 illustrates the overall architecture of the corporate network after logically isolating it using IPsec.

Figure 1. Isolating domains in the Microsoft corporate network.

Some IT assets require more stringent security policies. As the core intellectual property of Microsoft, the central repository for the source code of Microsoft software must be carefully protected. Human Resources data and Accounting systems data with stringent confidentiality requirements must also receive special protection.

Technology Chosen for Isolation: IP Security (IPsec)

IP Security is a suite of IP protocols used to provide secure communication. Within the Microsoft corporate network, peer-to-peer computer communication uses the IPsec Encapsulating Security Payload (ESP) protocol in conjunction with the Internet Key Exchange (IKE) protocol to provide authenticated communication. Group Policy is used to distribute IPsec policies and filters. Domain Controllers and the use of "access this computer from the network" user rights on sensitive resources provide authorization for authenticated users and machines. The encryption capability of IPsec is used only for a limited number of servers in Microsoft's deployment; instead, IPsec is used mainly for authentication. The use of ESP in this unencrypted configuration is called ESP-Null.

IPsec is commonly thought of as a tunneling protocol, providing encrypted access between gateways for the purpose of Virtual Private Networking (VPN). Microsoft IT chose to use IPsec in transport mode, which connects hosts directly to each other without routing traffic through a tunnel. At Microsoft, IPsec is used mainly to authenticate hosts to each other, and setting up tunnels is not needed.

SecureNet is the Resulting Subset of Devices Within the Corporate Network Secured with IPsec

IPsec allows trusted and untrusted hosts to be connected side-by-side on the same physical network and address space, while keeping them logically isolated. Trusted hosts may access anything they could access before the deployment of IPsec. The entire set of trusted hosts on the corporate network comprises the SecureNet.

Hosts that are not properly joined to a Microsoft IT-managed domain are untrusted, and unable to access anything not explicitly made available. Untrusted hosts may not access trusted hosts, but can access specially designated "boundary" computers. Microsoft IT sets up special boundary hosts with policies that permit limited access from untrusted computers.

Business Benefits

By isolating the corporate domains, Microsoft IT gained several key benefits:

Decreased Network Risks

Originally, the main purpose of Domain Isolation was to mitigate risk by segmenting unmanaged computers from Microsoft IT-managed computers at the network level.

Many computers at Microsoft are not managed by Microsoft IT. These include non-Windows computers such as Apple Macintosh, Unix, Xbox, and Pocket PCs. Test machines and computers owned by guests and vendors are also not managed by IT. Although IPsec can be used between these platforms and Windows computers, because Microsoft IT does not manage these computers, isolating them from the majority of domain-managed computers reduces the risk of general network threats from compromising the network as a whole.

These threats include buffer overflow attacks, exploitation of vulnerabilities and misconfigurations, compromised data integrity, and unauthorized access to a computer. These threats come from hackers, worms, and viruses, as well as malware such as spyware and other malicious software. Any network machine can hold valuable data or be a conduit for data.

By preventing unmanaged computers from connecting to trusted assets, Microsoft IT can improve the security for those trusted computers.

Improved Asset Management Information

A second benefit of isolating domains is gaining an understanding of what computers are managed by the domain, and what computers are not. For the unmanaged computers, Microsoft IT gained great insight into the business needs for them being unmanaged. This insight can be used to better refine the security policy to accommodate the business need.

For example, the Office for Mac product development group must use Apple Macintosh computers to develop their product. These Apple computers require access to file servers, but with the deployment of IPsec, this access would be blocked. By creating a special exception policy for specific file servers, Microsoft IT could allow access to file sharing on the server. Microsoft IT still manages the server, but knows that it is used by unmanaged computers and can disconnect access from devices outside of SecureNet by deploying an updated IPsec policy when necessary.

Furthermore, the act of deploying IPsec across the corporate network has resulted in a better understanding of all situations where unmanaged computers exist and why they are used on the corporate network. By testing secure deployments, small subnets, and test domains, most situations requiring exceptions to the IPsec policy were discovered and categorized. The IPsec deployment was publicized internally, and users having problems were instructed to call the help desk. Help desk personnel were assigned to resolve issues related to IPsec, and could further identify any unexpected areas where IPsec conflicted with existing systems. These issues were collected and passed to the IPsec deployment team where solutions and mitigation techniques were developed.

At the start of the project, scans of the corporate network identified approximately 300,000 unauthenticated IP addresses on the network. As of this writing, approximately 208,000 IP addresses are used by Windows computers that are joined to managed domains. About 2.5 percent of these are exceptions to the IPsec policies, called "boundary" computers because they can be accessed by untrusted computers.

While boundary computers can communicate with both SecureNet devices and untrusted hosts and are therefore at higher risk of attack, Microsoft IT now knows where these computers are and can quickly disconnect access from devices outside SecureNet when necessary. These boundary computers are further discussed in the Supporting non-IPsec Communications section. Future phases of the IPsec deployment will address securing these boundary machines further.

"Domain joined machines increased by 30% (208K clients today).  These are now machines that can have policy applied, an SMS agent installed etc with the result a more secure and controlled environment." Bob Davis Senior Director, Windows Services and Storage, Microsoft Corporation

The remaining 95,000 computers are unmanaged for various reasons such as product development, testing, and supporting customers. Microsoft IT can now work at further reducing the number of untrusted computers by solving the issues that prevent these computers from being managed by the domain, or further securing the boundary computers where necessary.

Protection of Intellectual Property

Some IT assets require more stringent security policies. As the core intellectual property of Microsoft, the central repository for the source code of Microsoft software must be carefully protected. Human Resources data and Accounting systems data with stringent confidentiality requirements must also receive special protection. Microsoft IT must ensure that servers containing this data can only be accessed by members of appropriate groups within Microsoft. While group policies can be used to restrict access, it's difficult to enforce these restrictions for non-domain-joined computers. Before the deployment of IPsec, authorized users could log into these restricted servers from unmanaged computers. With IPsec in place, machine accounts can be verified as well as user accounts, and access from unmanaged computers can be completely blocked.

IPsec provides the ability to enforce the domain policies. The IPsec SecureNet initiative provided Microsoft IT the ability to better protect the key intellectual property assets of Microsoft, providing access only to designated groups of users and network hosts.

Increased Policy Compliance

An unexpected benefit of deploying IPsec on the Microsoft corporate network was a marked increase in the percentage of computers joined to managed domains. Because non-domain-joined machines are cut off from domain resources, employees now have a stronger incentive to join their computers to the domain. Microsoft IT has seen an increase of 30 percent in domain-joined machines since deploying IPsec for Domain Isolation.

These additional machines can now have policy applied and be managed by SMS, group policy, and other domain management tools, resulting in a more secure and controlled environment.

Improved Malware Detection

Many spyware applications compromise machines by replacing core network "stack" components (Layered Service Provider). Because IPsec will not work without an intact "network stack," this makes malicious software compromises much more apparent. Deploying IPsec can reveal these types of applications.

Deploying IPsec across the corporate network facilitated the removal of these undesired applications.

Domain Isolation at Microsoft

IPsec allows the creation of logical, secure networks within a larger network. Microsoft IT employed Group Policy to deploy IPsec policies and filters throughout the managed domains. IPsec authenticates traffic between individual hosts on a network. (IPsec can optionally encrypt traffic, though Microsoft uses encryption only for a few servers in this deployment. All of the hosts sharing an IPsec policy on the Microsoft corporate domains form the SecureNet within the network.

Group Policy provides a framework for easily deploying IPsec to all of the hosts on a managed domain. An administrator can use Group Policy to set policies that apply across a given site, domain, or organizational units (OUs) in an Active Directory® directory service. Support for Group Policy is available on machines running Microsoft Windows 2000 Server, Microsoft Windows 2000 Professional, Microsoft Windows® XP Professional, and Windows Server 2003.

Through this Active Directory infrastructure and Group Policy, administrators have everything they need to deploy and administer IPsec across the enterprise. At Microsoft, because Active Directory and Group Policy were already deployed across the enterprise, deploying IPsec did not require additional software distribution. Administrators can take advantage of policy-based management to do the following:

• Enable one-to-many management of users and computers throughout the enterprise.
• Automate enforcement of IT policies.
• Simplify administrative tasks, such as system updates and application installations.
• Consistently implement security settings across the enterprise.
• Efficiently implement standard computing environments for groups of users.

Group Policy can be used to define user-related policies as well as security, networking, and other policies applied at the machine level. In addition, Group Policy enables management of domain controllers and member servers as well as desktop user machines.

Choosing Domain Isolation Technology

There are several different ways of segmenting and securing networks. Microsoft IT considered two leading technologies:

• IPsec
• 802.1x

These technologies have markedly different strengths, and are best applied in different circumstances.

IPsec

IPsec provides end-to-end authentication and (optionally) encryption between hosts on a network. One of its main strengths is that all administration occurs at the edges of the network, on each host. The intervening equipment between hosts and the network itself are essentially transparent. While there used to be problems using older Network Address Translation (NAT) devices with IPsec, these problems have been eliminated with the addition of Network Address Translation-Traversal (NAT-T) support in IPsec, included in Windows 2000 SP3, Windows XP SP2, Windows Server 2003 and later.

IPsec met the requirements for Domain Isolation defined by Corporate Security:

• End-to-end authentication. Traffic is authenticated from end-to-end, regardless of the routers, hubs, and networks in between the hosts.
• Packet integrity. All packets are verified to make sure they have not been changed in transit.
• Optional encryption. Between hosts with sensitive data, encryption may be turned on to ensure the confidentiality of the data.
• Scalable enterprise-wide segmentation. By deploying IPsec policies on individual hosts using Group Policy, IPsec easily scales across the enterprise.
• Uses Windows infrastructure. Support for IPsec is built into Windows 2000 and later, and can be easily deployed and managed using Group Policy, Active Directory, and domain management tools. IPsec uses Kerberos for authentication (optionally certificates), leveraging domain management efforts Microsoft IT already had in place.
• No hardware upgrades necessary. IPsec works "out of the box." Deploying IPsec in the Microsoft environment did not require any additional hardware or network investments.

Encrypting traffic can add a substantial amount of processor overhead, and on busy servers it may be necessary to add hardware cryptographic accelerators.

Because of the large number of concurrent connections to proxy servers, Microsoft deployed IPsec offload cards on these servers, and experienced some performance gains as a result.

802.1x

Another approach to segmenting a network uses 802.1x, certificates, and a public key infrastructure (PKI) to manage devices and users on the network. In contrast to IPsec, 802.1x has more of a network orientation than a host orientation. 802.1x must be supported by each network host, switch, router, or device, enforcing network policies through individual authentication by each device.

Microsoft IT uses 802.1x to secure its wireless access points, and for other specific purposes. It was not chosen for segmenting the entire corporate network because of the following characteristics:

• Hardware investment required. Because all network equipment must support 802.1x, many legacy hubs and switches require replacement or upgrade.
• Control not granular enough. At Microsoft, many employees connect hubs to network ports so they can connect multiple computers to the network. 802.1x becomes cumbersome to manage at this level—all computers connecting to a hub have to be trusted, or none are.
• Limited ability to support sub-segmentation. With certificate-based infrastructures such as 802.1x, at this writing there is no standard way of isolating smaller groups within a domain. A certificate is considered valid or invalid based entirely on whether the designated Certificate Authority (CA) has signed it and it isn't on the Certificate Revocation List (CRL). With Kerberos and Active Directory, smaller groups of machines within a domain can be set up as Security Groups and managed independently.
• No support for data integrity. 802.1x only authenticates connections between directly connected devices. It does not guarantee data integrity.
• No support for encryption. While several encryption technologies make use of 802.1x for authentication, including IPsec, 802.1x (by itself) provides no ability to encrypt data.

802.1x in many circumstances provides a good way for authenticating hardware and users. But authentication is the only thing it provides. IPsec was chosen because it is a complete solution.

Introduction to IPsec

Designed by the Internet Engineering Task Force (IETF) as the security architecture for the Internet Protocol (IP), IPsec defines IP packet formats and related infrastructure to provide end-to-end strong authentication, integrity, anti-replay, and (optionally) confidentiality for network traffic. Internet Key Exchange (IKE), part of IPsec, specifies an on-demand security negotiation and automatic key management service. The Windows Server 2003 operating system simplifies deployment and management of network security with Windows 2003 IP Security, a robust implementation of IP Security (IPsec).

Standards-based Framework of Security Protocols and Cryptographic Services

IPsec refers to a framework of security protocols and cryptographic services for Internet Protocol (IP) packets. IPsec uses rules to specify basic packet filtering. A rule correlates a list of filters with an action. Each filter specifies two end points of a connection, identifying IP, port, protocol or subnets to match. Actions include:

• Permit. Allow traffic between the hosts, ports, protocols, and subnets specified in the filter lists without using IPsec authentication or encryption.
• Block. Drop all packets matching the filter descriptions.
• Negotiate security. Security is negotiated host-to-host using a combination of authentication with Kerberos or digital certificates, data integrity, and optional encryption.

IPsec is a Foundation for a Secure Environment, but is not a Secure Environment Itself

Isolating domains using IPsec makes it much more difficult for an attacker to gain access to a protected machine. IPsec also helps administrators gain a deeper understanding of their network and systems environments, and lays the foundation for future deployments.

IPsec can be deployed as a local policy or a domain-based Group Policy. Local policies can be deployed with a Microsoft Management Console (MMC) snap-in or via command line tools such as netsh.exe (Windows Server 2003), ipseccmd.exe (Windows XP), and ipsecpol.exe (Windows 2000). Microsoft IT uses domain based Group Policy deployments almost exclusively in its global deployment.

IPsec Negotiated Security has Four Modes, Microsoft IT Uses Two

Windows IPsec supports four different behaviors with negotiated security:

• Default Response. A host responds to IPsec requests, but never initiates IPsec.
• Request Mode. A host responds to both IPsec and unauthenticated (non-IPsec) requests. It initiates communications with IPsec, and if that fails, allows unauthenticated communications. Microsoft used Request mode in Phase 1 of the IPsec deployment.
• Secure Request Mode. A host responds to requests secured by IPsec, and ignores unauthenticated requests. It initiates communications with IPsec, and if that fails, falls back to unauthenticated communication. Microsoft uses Secure Request mode on most SecureNet machines.
• Full Require Mode. A host requires IPsec-secured communications for both inbound and outgoing requests.

The two most commonly deployed behaviors are request mode and secure request mode, and these are the two behaviors Microsoft IT used during the IPsec deployment.

The Microsoft Security Environment

Microsoft IT has a broad variety of responsibilities. Its first role is providing IT services that range from end-user support and telecommunications management, to server and network operations. Microsoft IT ensures that approximately 55,000 employees, 5,000 contractors, and 17,000 vendors in more than 400 Microsoft locations around the world are able to access the corporate network 24 hours a day, seven days a week.

Environment

Microsoft operates in an extremely active and challenging security environment. Challenges include the following:

Each month, Microsoft experiences approximately 100,000 intrusion attempts.

Each month Microsoft detects and processes, on average, over one million virus-infected e-mail messages.

Microsoft has unique IT environments for product development, testing, and support, which require special security.

This combination of factors—an evolving security landscape full of potential vulnerabilities operating across a large and dynamic IT environment—presents an array of variables that add complexity.

Challenges

Many aspects of the Microsoft work environment are not typical of a large-scale enterprise. Some of the unique aspects of the desktop environment at Microsoft include:

• Multiple computers per user.  Many Microsoft employees have two or more desktop computers in their offices. Often, one computer is dedicated to a specific configuration to perform a given task such as developing office products, while another is reserved for standard network and e-mail access. The user may purposely have systems running different versions of Windows or Office to test for application compatibilities. Microsoft employees also use Microsoft Virtual PC to emulate several operating systems on one desktop computer.
• Diverse desktop implementations.  Microsoft employees are highly technology-literate, and approximately 97 percent are local administrators on their computers. Employees routinely explore the limits of the tools available to them in order to improve product quality. Therefore, many desktop computers have unique capabilities and software characteristics.
• Frequently rebuilt computers.  To test software functionality, many employees frequently rebuild their systems completely, sometimes daily. Approximately 5,000 computers per month are rebuilt in the data center at the Redmond, Washington, headquarters alone.
• Diverse mix of approved software versions.  When Microsoft releases a new version of an application, departments across the enterprise—such as product development, product support, and sales—do not unilaterally install the software immediately. Like many other enterprises, departments and business units within Microsoft have a generous amount of autonomy and latitude to choose the software versions that best fit their work models. For example, some departments run older versions of software to test version interactions and backward compatibility. Development groups deploy multiple—sometimes up to six—builds of an operating system or an application into the desktop environment before release to manufacturing.

Project Deployment Team Model and Structure

The main business of Microsoft is software development and marketing. Consequently, in addition to running the global IT service internally, Microsoft IT is committed to testing Microsoft enterprise products in production before they are released to customers. Known as "dogfooding" within Microsoft, this early adoption of technologies brings additional challenges to the network environment within Microsoft, but helps ensure that products will scale to meet the business challenges of other large enterprises.

Deploying IPsec in the Microsoft network required coordination between many different groups and people across the Microsoft IT organization. The following teams played key roles:

• Corporate Security. Created the specifications, identifying the requirements for segmenting the corporate network.
• Windows Infrastructure Engineering/ Infrastructure Architecture. Designed the overall solution.
• Windows Infrastructure Engineering. Created the IPsec filters, filter lists, and policies, and deployed them using Group Policy.
• Global Client Technology Team. Assisted people encountering trouble with the deployment of IPsec.

Planning

The planning phase of the Domain Isolation project consisted of the following basic steps:

1.     Determine segmentation requirements. Microsoft IT started by evaluating the security measures already in place, and identified some number of attacks that might be launched from unmanaged computers. The goal of the project was to deploy a mechanism to protect individual hosts managed by Microsoft IT on the corporate network from attacks launched by unmanaged hosts.

2.     Choose technology. IPsec, 802.1x, virtual LANs, and VPNs were considered. IPsec provided a scalable solution that did not require additional hardware or software investment, or the need to change the IP addresses within internal networks. IPsec provided additional abilities to further isolate hosts into security groups and encrypt traffic if desired.

3.     Design IPsec/Group Policies. Microsoft IT first created the security groups and Group Policy Objects (GPOs) for IPsec policy assignment and management. Microsoft IT uses two group and IPsec policies, one for the default "secure request" behavior and the other for the boundary server exceptions using the "request security" behavior. Microsoft IT then worked with the network engineering team to identify all of the IPv4 subnets assigned to the corporate network. This list of subnets was used to create the default rule and filter set for IPsec utilization, called the Secure Subnets rule. IT also worked with the network engineering team to identify exceptions to the secure subnets filter list and set up rules for each of them as needed.

With every IPsec rule, there is a corresponding filter list that includes one or more filters, a filter action, authentication methods, a connection type, and an IPsec encapsulation mode (transport mode or tunnel mode). When designing the IT filter actions, Microsoft IT defined its core filter actions (permit, request security and secure request). With the baseline policies and filters defined, IT had a defined IPv4 environment, a centralized policy structure and common security actions that could then be used for deployment. For more details, refer to the Microsoft IT IPsec Filter Design section.

4.     Test policies/IPsec functionality and behaviors. With the filters designed and GPOs defined, Microsoft IT first enabled policies that had filters with limited subnets listed so only small numbers of machines used IPsec. This allowed Microsoft IT to deploy IPsec on limited machines to tightly control the initial testing. This testing confirmed the need for boundary machines, and helped Microsoft IT refine the filters and filter rules.

5.     Create a rollout schedule. With limited testing complete, Microsoft IT examined the network organization to create a rollout schedule across the enterprise. Because a deployment of IPsec on a large scale was completely unprecedented, Microsoft IT planned to deploy IPsec in a way that would allow unanticipated problems to be resolved with minimal impact to most employees. The deployment would start with the smallest individual subnets, then entire buildings, and eventually the entire network.

Deployment was planned in two phases:

• First phase: Deploying Request mode IPsec.
• Second phase: Deploying Secure Request mode.

6.     Test process and strategy. During the rollout, Microsoft IT planned to carefully monitor traffic on deployed subnets as well as impact to the end user community through helpdesk call volumes and other observations, and reevaluate the rollout process and strategy if necessary. The deployment focused on ensuring minimal impact to users, placing high priority on allowing employees to work uninterrupted.

Microsoft IT used the phased subnet deployment approach for its global request mode deployment with a great deal of success. The Secure Subnets filter list was slowly expanded as the rollout progressed until the full set of corporate network subnets were in the list and nearly all network traffic was encapsulated with IPsec. With request mode complete, all domain-joined machines in the enterprise had the same Secure Subnets filter list.

To support its secure request rollout, Microsoft IT created a new, more specific rule and filter list (with smaller subnet designations) and assigned a secure request filter action to it.

Although deploying by subnet provides a more granular level of control for IPsec deployment and allows an IT organization to gauge the impact from the deployment, it does create some challenges when using a more restrictive filter action. Specifically, with secure request or require mode, users with non-domain joined machines on the deployed subnets will be blocked from initiating sessions to other machines with the policies. While this is the end state of a secure IPsec deployment, the effects can be confusing to end users because many users assume that the changes will not apply to their non-domain joined machines.

Microsoft IT started its secure request deployment with the per-subnet filter list method, but found that the changes were too confusing for the user populations and created a broader impact than Microsoft IT felt was acceptable. In response, the rollout process was changed to deploy to individual domains instead of subnets, starting with the smallest domains. The main corporate domain, containing the majority of Microsoft IT-managed computers, would be deployed last.

7.     Communicate with users. The IPsec deployment was completely transparent to the majority of Microsoft employees, and required no user interaction. At the peak deployment times, the maximum number of Helpdesk calls per month related to IPsec was less than 250. This call volume was extremely low, compared to other systems—the Microsoft help desk gets around 7,000 calls per month for Exchange, and a similar number for PBX.

Helpdesk personnel were trained to help users with network problems that were discovered when IPsec was deployed. Helpdesk personnel were given troubleshooting flowcharts, process articles, and training using WMI, IPsec, Kerberos Authentication, and DNS to resolve issues that arose with the deployment of IPsec. Often employees using computers previously unmanaged by Microsoft IT were cut off from resources they needed, and encouraged to put their unmanaged computer on a managed domain.

Developers and other users of servers containing sensitive intellectual property, or human resources or accounting information also needed to be aware of the changes to the security policy that IPsec enforced. Previously these groups were able to access the sensitive servers from any computer connected to the corporate network, passing their user credentials to the server applications. With the deployment of IPsec, these servers could only be accessed using a computer in the proper domain security group, by an authorized user. Microsoft IT could use the "Access this computer from the network" logon right to restrict access to these domain security groups, and IPsec would block connections from unauthorized computers. The business decision to only allow access to this information to authorized computers needed to be communicated to users of these servers.

The entire company was notified and kept aware of progress on the IPsec deployment through a variety of methods:

• Security bulletins sent through e-mail to all employees
• Internal training sessions
• Informal lunch-time presentations (Brown bag sessions)
• Stories in an internal corporate newsletter
• Self-paced multimedia training
• An internal Web page with technical detail and a Frequently Asked Questions (FAQ) section.

For most employees, no training was necessary. Before requiring IPsec for connections to SecureNet servers, an e-mail was sent to all employees notifying them that their computers must meet two requirements:

A.     Running a minimum Microsoft IT-specified operating system.

B.     Joined to a Microsoft IT-managed domain.

The e-mail warned that computers not meeting these requirements would not be able to gain access to IT-managed servers and internal Web sites after a specified date, and provided links to the internal Web page for more information.

Deployment

This section describes the issues involved in deploying IPsec across the Microsoft corporate network.

Group Policy for IPsec Distribution

Group Policy provides a framework for deploying IPsec within a corporate network. Microsoft IT deployed IPsec settings using Group Policy. The "default" Group Policy is applied to the Domain Computers group. A limited number of computers are denied access to the default policy through the use of security groups. These computers then apply a different policy allowed by the same security group.

Microsoft IT went through the following steps before deploying IPsec:

• Create dedicated Group Policy Objects (GPOs) for IPsec.
• Create universal security groups to exclude hosts from IPsec policies.
• Create universal security groups to control the application of GPOs.
• Create a universal security group to administer the GPOs and IPsec policies.
• Administer Group Policies for IPsec.

Create Dedicated Group Policy Objects (GPOs) for IPsec

Microsoft IT created three GPOs for each domain for IPsec. Two of these are the following primary IPsec policies, accounting for over 99 percent of the computers in each domain:

• Request Mode GPO
• Secure Request Mode GPO

Client computers apply GPOs in a specific, inherited order. The first listed GPO has precedence over the rest. This ordering ensures that the least restrictive IPsec policy is applied if there is an error. Figure 2 illustrates the ordering of GPOs created for domains within Microsoft.

Figure 2. Ordering of GPOs for each corporate domain.

The remaining GPO, IPsec Require Encryption, is used to further secure sensitive resources.

Create Security Groups

Because many computers need special IPsec policies, Microsoft IT created a set of security groups to separate these computers. Each security group can allow or deny the application of a GPO. Some computers, such as many infrastructure servers (Domain Controllers, DNS servers, etc.), cannot provide services to clients if IPsec is enabled with a secure request policy. These servers are added to a "No IPsec" security group.

The security group of the computer and the GPO associated with the domain are used to determine which IPsec policy that computer receives.

Create Universal Security Groups to Control the Application of GPOs

To simplify administration across domains, Microsoft IT made universal security groups that could be applied and managed at the forest level.

Create a Universal Security Groups for Group/IPsec Policy Administration

By defining a security group consisting of designated user accounts, Microsoft IT is able to control who can administer IPsec across the entire forest.

Administer Group Policy

Microsoft IT uses Windows Server 2003 to administer Group Policy for IPsec. While Windows 2000 had support for IPsec, Windows Server 2003 has a number of performance enhancements and better support for specific IPsec rules. Because Windows 2000 computers are still widely used on the corporate network, Microsoft IT defined rules and filters that work for Windows 2000 and later.

Microsoft IT administrators build local IPsec policies, and then export an .IPSEC file. The .IPSEC file is imported into a test environment to be evaluated and validated before deployment. Once Microsoft IT is satisfied with the IPsec policy, it is imported into each domain for a standardized deployment.

The .IPSEC files serve as a disaster recovery backup for IPsec policy settings.

Microsoft IT IPsec Policy Settings and Filter Design

For the majority of the computers on the corporate network, Microsoft IT uses a default policy and a set of standard filter rules.

An IPsec policy consists of a set of rules. Each rule consists of a filter list and an action, and can specify an authentication method, a tunnel, and a connection point. A filter list contains a set of filters. Each filter specifies a source address or network, a destination address or network, and optionally a port and protocol. Filter actions can be Permit, Block, or a customizable Negotiate Security action.

Figure 3 illustrates the relationship of these building blocks of an IPsec policy.

Figure 3. Structure of an IPsec Policy.

Each host may only have a single active IPsec policy. Microsoft IT defined different IPsec policies that apply to computers via different GPOs during different phases of the deployment process. There are cases where IPsec may already be in use for other things. Some businesses use IPsec block and permit filters to secure individual servers. These existing IPsec deployments may directly conflict with a deployment like the one Microsoft IT uses—a single computer cannot have separate policies for being part of a SecureNet-like environment and still use separate IPsec policy to locally secure a server. To address these issues, specific "secure server" type policies can be implemented within the larger context of the domain policies with either more restrictive settings (require encryption, for example), or less restrictive settings (permit all SMTP traffic for monitoring, for example). These individual server policies can be implemented by linking specific GPOs to OUs or by using security group filtering to apply the "secure server" settings to a subset of servers.

Policy Settings

To create the IPsec policy that is imported into the domain, Microsoft IT uses an MMC snap-in to define a local IPsec policy on a Windows Server 2003 machine. Individual rules are managed from the IPsec policy properties sheet. Filter lists and filter actions can be defined directly from the MMC snap-in, or in individual rule definitions. Policy settings are associated with particular elements: the policy itself, individual rules, filters, and filter actions.

Table 1 describes the standard IPsec policy deployed to most computers managed by Microsoft IT.

Table 1. IPsec policy settings in use in the Microsoft IPsec deployment

Policy Setting	Value	Notes
Default Filter Action	Secure Request	At Microsoft, the IPsec policy for most domain-joined computers specifies a Secure Request default filter action. Each computer with this policy uses IPsec on outbound IP requests to all subnets not excluded by another filter. If IPsec is not successfully negotiated within 3 seconds, the computer falls back to unauthenticated traffic for that host and a soft Security Association (SA) is established. All inbound unauthenticated requests to hosts with this policy are denied. Only incoming requests that successfully negotiate IPsec are permitted. This filter action is not present by default, but can be created in a new filter action by unchecking the "Accept unsecured communication" and checking the "Allow unsecured communication" checkboxes. Note that this policy is only supported on Windows 2000 SP3 and greater, Windows XP SP1 and greater, and Windows 2003. On earlier versions of Windows 2000 and Windows XP, this policy behaves like Request Security.
Default Filter List	Any <-> Secure Subnets	A set of filters limit IPsec to corporate subnets. If a mobile computer connects to a network other than the corporate network, it does not use IPsec for outgoing or incoming communications. If a Microsoft IT-managed computer connects to a private network using an RFC 1918 IP range specified in this filter list, the computer cannot connect to computers in the overlapping IP address ranges. For example, the entire 10.0.0.0/8 network space is defined to use IPsec within the Microsoft corporate network. If a Microsoft computer connects to a different corporate network also using this range, it may not be able to successfully establish communications with any other hosts if incompatible IPsec policies are being used, and all incoming communication requests will be blocked. Microsoft has explicitly exempted two small private IP address ranges, to allow employees to connect and use home networks.
Default Authentication Method	Kerberos	Microsoft Windows IPsec can use one of three authentication methods: ·          Kerberos, supplied by Active Directory ·          A certificate signed by a particular certification authority (CA) ·          A specific string used as a preshared key. By using Kerberos for authentication, IPsec can use much of the power in Active Directory to define security groups and further segment the network. At Microsoft, Kerberos is the default authentication method for IPsec. Certificates are supported by particular rules to support computers that are not part of an Active Directory domain—for example, allowing VPN clients to connect without being managed by Active Directory. Pre-shared keys are not recommended for most installations because they can be easily compromised and cannot be easily revoked.
Default Negotiate Security Method	ESP with NULL encryption	IPsec provides for two different security methods: Authentication Header (AH) and Encapsulating Security Payload (ESP). Generally AH is used for authenticating and validating the integrity of data. ESP is generally used when encryption is desired. Even though Microsoft IT does not use encryption for most traffic within the corporate network, ESP was chosen for all IPsec traffic for several reasons: ·          ESP can pass through Network Address Translation (NAT) devices. ·          Encryption can be easily enabled for particular servers, if desired. ·          Additional network bandwidth and server overhead is minimal.
Policy refresh	60 minutes	Each host is set to cache the IPsec policy locally for 60 minutes. After 60 minutes, the host refreshes its policy from Active Directory. This allows changes to IPsec policy to be deployed to the entire corporate network in (at most) an hour, making it possible to quickly respond to any compromises of the corporate network.
Main mode lifetime	2 hours	The default main mode lifetime for Windows is 8 hours. This specifies the length of time a security association (SA) between two machines is valid, before being cleared out of memory and renegotiated. Microsoft IT chose to set a shorter main mode lifetime to reduce memory usage on busy servers.
Quick mode lifetime	1 hour (Windows IPsec default setting)	Length of time a key is valid before being regenerated. Quick mode SAs have a hard coded 5-minute idle timeout—they are removed if no traffic is exchanged for 5 minutes.
NoDefaultExempt	Set to 1	By default, certain types of traffic are exempted from IPsec. Microsoft IT removed the exemption for Kerberos and Resource Reservation Protocol (RSVP) through the use of a Group Policy administrative template. These exemptions are not necessary because Microsoft permits all IP traffic to the domain controllers. The exemptions for broadcast and multicast, as well as for IKE (UDP/500) were left intact since these cannot be secured using IPsec at this time. See the following articles in the Microsoft Knowledge Base (KB) for more information: ·          KB 810207 - IPsec Default Exemptions Are Removed in Windows Server 2003 (http://support.microsoft.com/default.aspx?scid=kb;en-us;810207) ·          KB 811832 - IPsec Default Exemptions Can Be Used to Bypass IPsec Protection in Some Scenarios (http://support.microsoft.com/default.aspx?scid=kb;en-us;811832)
IPsec Key Exchange (IKE) default security method	3DES, SHA1, DH2	The Microsoft IT-preferred security method specifies that the key exchange is encrypted with Triple Data Encryption Standard (3DES), key integrity is checked using Secure Hash Algorithm 1 (SHA1), and the key exchange uses the medium (2) level Diffie-Hellman sequence.
Negotiate Security method preference order	Custom	For filter actions specifying that IPsec must be negotiated to permit traffic, a custom security method was created to specify ESP with no encryption. All keys are set to expire after one hour. Figure 4 illustrates setting up a custom negotiate security method. The default security method for all traffic is ESP without encryption, but with all combinations of security methods specified, computers with IPsec policies using this filter action should be able to use encryption and other IPsec features when required by the other host.

 

Figure 4. Setting up a set of custom security methods that allow IPsec communications to most hosts.

 

Microsoft IT IPsec Filter Design

Microsoft IT uses the basic filter rules outlined in Table 2 as the default policy for most of the computers on the corporate network.

Table 2. Microsoft IT's recommended IPsec filter lists for Request and Require policies.

Filter	Filter Action
Any <-> SecureNet, /18 subnet	Secure
Any <-> Exempted Subnets /23 subnet	Permit
Any <-> Virtual IP addresses (Clusters, NLB)	Permit
Any <-> Domain Controllers	Permit
Any <-> DNS Servers	Permit
Any <-> DHCP Servers	Permit
Any <-> WINS Servers	Permit
Me <-> Any, ICMP	Permit

 

Design Rationale

Each IPsec rule associates a list of filters with an action. An individual filter defines source and destination addresses and the protocol that the filter will match. The most specific filter describing a network connection is the one applied by the rule.

In the Microsoft IPsec deployment, all rules that permit traffic are applied to specific filters. Rules that negotiate security are used for the general SecureNet IP address ranges.

IPsec always applies the most specific filter. For this reason, exception rules that permit traffic must always be more specific than a default rule that negotiates security.

Filter Design Challenges

IPsec is managed and deployed through Group Policy and Active Directory security group membership, while the filters being deployed define IP address ranges. As subnets included in SecureNet were added, removed, or changed, Microsoft IT updated individual filters used in the IPsec rules. Once Phase 1 of the deployment was complete, all of the computers on the corporate network were capable of responding to IPsec-secured communications. For Phase 2, Microsoft IT initially tried to use subnet deployment but encountered significant challenges as described in the Planning section of this document. Because of these challenges, Microsoft IT chose to roll out Phase 2 by domain instead of by subnet.

Having rules that permit unsecured traffic need to be carefully thought out, both from a policy design and security perspective. It is important to be careful about what destination addresses are permitted, because hosts can communicate with those addresses without using IPsec.

Another filter design challenge is managing multi-homed computers—computers that have more than one IP address. For example, some computers in the extranet have two Network Interface Cards (NICs)—one connected to the corporate network, and one connected to the Internet. IPsec filters define IP addresses and networks but have no concept of physical NICs. So any IPsec policy applied on the box considers the source and destination addresses, not the individual NIC.

At Microsoft, the IPsec policy for multi-homed computers does not include filters for the public Internet addresses, so the Internet-facing NIC on these computers does not have active IPsec policies.

For more details, see the IPsec Design Best Practices section.

Non-IPsec Platform Support

One of the more significant benefits of Domain Isolation is the knowledge gained by analyzing the platforms and services being run on the network and identifying any exceptions. Developing a detailed understanding of the exception scenarios provides the IT administrator with the required knowledge for taking additional security steps to strengthen the overall security of the environment.

Non-IPsec Aware Platforms

On the Microsoft corporate networks, there are a number of computers and devices not capable of using IPsec as deployed by Microsoft IT. These primarily include:

• Non-Windows computers and devices. Apple Macintosh computers, Unix computers, Xboxes, and Pocket PCs do not have Windows IPsec installed or easily deployed.
• Legacy Windows operating systems. Some groups in Microsoft have test computers running Windows NT4 or Windows 9x, which are incapable of using Group Policy-based IPsec.
• Windows computers not joined to a managed domain. Some development groups run private domains for testing. Many individuals have test and development machines not connected to a managed domain.
• RAS and VPN clients not owned by Microsoft. Partner and vendor companies need to connect to certain Microsoft resources. Many employees connect to the corporate network using home computers. These computers may not have a machine account in a managed domain.

Historically, all of these computers have had access to domain resources. With IPsec deployed, these computers cannot access computers inside the SecureNet. Windows IPsec is compatible with IPsec implementations available on these other platforms, but Microsoft IT chose to design the IPsec policies to exclude them since they are not managed to the security standards required for SecureNet.

Developing effective ways to connect specific non-Windows devices with a business need to access domain resources is a high priority for Microsoft IT. Currently, these devices must access resources that have special IPsec rules. Microsoft IT calls these exception servers "boundary machines." These boundary machines will be discussed in the next section.

Legacy and test environments are not considered a priority for adding to SecureNet. These computers do not meet the minimum security standards for connecting to the corporate network. If there is a strong business need for people to use a test machine to access corporate resources, users can apply to make the resource a boundary machine.

Supporting non-IPsec Communications

Microsoft has a business need to allow some non-domain-joined computers access to business resources that would otherwise be in SecureNet. In addition to the types of client computers already listed, there are specific infrastructure-types of services that cannot be secured by IPsec.

To establish IPsec in the first place, a computer must have network connectivity and be able to get the IPsec policy from a domain controller. This means that domain controllers must communicate with unauthenticated hosts, and DHCP relay agents must be able to forward responses from DHCP servers so client computers can gain access to the network. In addition, WINS and DNS are needed so that clients can locate the domain controllers, and allow access to boundary machines and the Internet by guest and test computers that do not require access to SecureNet.

Certain other services do not support IPsec, such as network thin clients or other bootstrap clients that need to download an image from a Remote Installation Server (RIS), and Advanced Deployment Services (ADS).

Boundary Computers

To manage exceptions from the general SecureNet deployment of IPsec, Microsoft IT defined different types of boundary computers. A boundary computer is an IPsec-enabled computer that accepts non-IPsec inbound connections, straddling the border between SecureNet and the untrusted corporate network. Boundary computers do not "proxy" communication between SecureNet and the untrusted corporate network; they are simply SecureNet computers that allow access from the untrusted corporate network.

• Boundary computers use Request security behavior, not Secure Request mode.
• Boundary machines give Microsoft great flexibility in defining exceptions to the default IPsec policy using security groups instead of IP addresses.

Managing Boundary Computers

Boundary computers require extra management and security because a boundary computer provides a way for untrusted computers to gain access to SecureNet resources. Microsoft IT has identified specific acceptable reasons for having a boundary computer. All of the computers that share a particular reason for allowing untrusted access are put into a defined security group. In this way, threats to particular services can be quickly blocked, and additional group policies such as firewall rules may be deployed to particular security groups.

For example, all RIS servers are placed in a special security group. As boundary computers, they permit incoming non-IPsec from any computer, which allows the RIS servers to support computers that download operating system images before loading Windows or having any IPsec capabilities. By being part of a RIS server security group, Group Policy denies access to the default corporate IPsec policy and the RIS server security group allows the request mode policy to be applied.

Another common type of boundary machine is a file and print server used by Apple Macintoshes in a particular group or building. File servers are granted boundary status on a case-by-case basis. Several Mac business units require access to particular domain resources for developing their software. Requests from these groups are typically granted, but efforts are underway to consolidate Mac resources to particular machines.

Deploying Boundary Computers

Any employee may request making a particular computer a boundary machine. The employee must provide the reason for asking for the exemption, along with supporting details. Microsoft IT can then evaluate the need, and grant or deny the request.

Deploying IPsec has allowed Microsoft IT to identify where security risks exist on the network and gain a better understanding of the computer population within the corporate network. Microsoft IT can now deny insecure network traffic by default and grant it on a case-by-case basis, after gaining an understanding of the ramifications and the ability to further restrict the insecure traffic, quickly if necessary.

After deploying Secure Request mode to the entire corporate network, fewer than 2 percent of all domain-joined computers were boundary machines. Now that the deployment is complete, Microsoft IT can continue to isolate, consolidate, and take additional security steps to mitigate the risks posed by boundary machines.

Microsoft IT Today

Within Microsoft, IPsec policies were first applied in small subnets where the issues could be discovered and resolved before wider deployment. Phase 1 of the IPsec deployment at Microsoft was completed in March 2003, resulting in more than 160,000 computers running IPsec. Technical issues were identified and corrected. No connectivity was blocked during this phase, and over 70 percent of all Microsoft-internal network traffic was encapsulated in ESP.

Phase 2 of the IPsec deployment involves deploying Secure Request mode across the enterprise. Phase 2 deployment was completed in April 2004, ahead of schedule. As of this writing, approximately 208,000 computers across six forests and 18 domains are successfully using secure request mode. The Microsoft corporate network is now logically segmented using IPsec. Computers identified as trusted are protected from untrusted computers.

Known Issues and Problem Applications

Many of the issues surrounding the deployment of IPsec have already been covered in this white paper, including primarily non-IPsec aware platforms and RAS/VPN clients. During development and deployment of the Domain Isolation solution, Microsoft IT encountered the following additional issues.

Performance - LAN

ESP adds a brief header and trailer to every packet crossing the network. In deployment, this additional packet information adds an overhead of three-to-nine percent to bandwidth consumption, depending on the application. Across all applications for the Microsoft enterprise environment, the mean increase was approximately 3 percent.

Different enterprise environments might experience a different amount of increase, depending on the mix of applications present. The increase in bandwidth is likely to have a more significant effect in a WAN scenario where the traffic is already near capacity, than in a LAN or over Fast Ethernet connections.

Microsoft has noted this effect in its enterprise environment. It was not judged to be enough of an impact to compare the actual cost for the increase of bandwidth against the potential security value. This could be a concern for other enterprises.

Performance - CPU

The additional overhead of adding ESP to every packet is negligible on most client computers. For servers with a large number of concurrent network connections, this additional overhead adds up, typically due to the demands of the increased numbers of Diffie-Hellman exchanges taking place during main mode negotiation requiring slightly more CPU time to manage. Microsoft IT is now working on deeper performance analyses of core IT services and server scenarios including analysis of both IPsec and SSL/encryption offload cards.

Full ESP encryption demands more CPU time. For servers containing extremely sensitive data requiring fully encrypted connections, special 10/100 MBPS network cards are available that can offload this work from the CPU. These cards are not yet available for Gigabit Ethernet.

Microsoft did not use ESP encryption for the initial IPsec deployment, but is investigating it for future secure server and possible Extranet security scenarios.

IPsec and Windows VPN Servers

For IPsec deployments using Kerberos, VPN servers must have special IPsec policies to allow traffic to remote clients and support unauthenticated traffic for the tunnel itself. The IPsec Policy used in the Microsoft deployment is incompatible with default Layer 2 Tunneling Protocol (L2TP) filters.

For deployments that rely upon certificates with a single trusted root for the entire enterprise, a separate IPsec policy for VPN servers may not be necessary. Microsoft IT decided to use Kerberos for the majority of SecureNet, and only uses certificates for VPN machines that are not joined to the domain.

RFC 1918 Private IP ranges

Request For Comment (RFC) 1918 specifies private IP ranges that anybody may use for computers on a private network. These ranges are used by Microsoft, as well as its employees and vendors connecting to the corporate network through a VPN.

Because Microsoft has designated most of this private network address space as a secure subnet, any hosts using an IP address in a secure subnet must be joined to the domain and recognized by Kerberos and IPsec to be able to connect to SecureNet. This condition causes problems for home network users and vendors using private addresses who connect to SecureNet.

This issue is most noticed by mobile users between different networks. For example, a corporate user with a domain-joined laptop on a business trip may try to use a hotel network. If the network space used by the hotel overlaps one of the secured subnets defined in the IPsec policy, the mobile user may experience problems if IPsec policies are in use in that subnet. This is because the laptop has the IPsec policy cached on it, and will now attempt to negotiate IPsec with IP addresses on the hotel's network, because there is a matching filter. Depending on the IPsec policy, this may cause a 3 second delay in initiating network connections while the laptop attempts to initiate IPsec communications before falling back to unauthenticated connections, or completely failed connections when the other computer is using an incompatible IPsec policy. To date, Microsoft IT has not received reports of failed communications from its mobile users, so this remains a theoretical problem.

Microsoft has designated two specific private subnets to be excluded from the list of secure subnets:

• 192.168.0.0/24
• 192.168.1.0/24

Any vendor or employee using a computer on their own network and wishing to connect to SecureNet resources over the VPN can use one of these approved private address subnets to avoid these types of issues.

To resolve this issue, limit the number of private subnets used in secure subnets, and exempt other private subnets from this list. Ideally, do not use a private IP address space for your corporate network.

Network Device Issues

Within a TCP/IP packet, IPsec changes the offsets for the destination ports and protocols. These changes disable many applications running on network devices like routers, especially those that monitor and manage traffic through the network.

Over time these network applications are getting updated to support IPsec, but many are not yet compatible:

• Router ACLs cannot examine protocol and port fields in ESP-protected packets. ACLs based only on IP address are unaffected.
• Cisco NetFlow on routers cannot analyze packets between IPsec members based on protocol or port. Analysis based only on IP address is unaffected.
• Router-based Quality of Service (QoS) cannot use ports to classify packets. Host-based classification of packets supported by Windows is unaffected, as is router-based QoS using only IP addresses.
• Weighted Fair Queuing (WFQ) and other flow-based router traffic prioritization methods may not queue packets as intended resulting in suboptimal network utilization.
• Network monitors may be unable to parse ESP packets.

In general, IPsec defeats network-based prioritization and port or protocol-based traffic management. For encrypted packets, there is no workaround—the host itself must handle any traffic management functions. For unencrypted, authenticated-only packets, the devices and applications must be aware of how IPsec changes packets to be able to do anything with them other than route them to the correct host.

The Microsoft Network Monitor added an ESP parser in version 2.1 to aid troubleshooting of unencrypted IPsec packets. Network Monitor is available in a light version in Windows Server 2003, and a full version for SMS. The SMS version adds formatting of data, advanced diagnostic capabilities, and other features for capture management.

To resolve these issues, upgrade to the latest versions of these types of tools that have support for IPsec where available.

IPsec Adds to Use of System Resources

On Windows 2000, applying at least Service Pack 3 reduces memory consumption for computers with many Security Associations (SAs). Windows 2000 expands "Me" filters for each static IP on the machine, which can cause large filter lists and increased CPU/memory usage. Microsoft IT uses "Any" filters (which are not expanded) instead of "Me" filters to avoid this issue.

A computer using IPsec establishes an SA for every authenticated connection. For most computers in normal use, there are not enough active connections for this memory use to be significant. For servers with thousands of active connections, these SAs can add up to a few megabytes—Microsoft IT estimates that each active SA consumes about 5 kilobytes of memory. With Windows Server 2003, the additional CPU time involved is not enough to have any impact on traffic over 100 MBps interfaces. On Gigabit Ethernet connections, throughput on extremely busy servers has decreased slightly due to the additional overhead involved in authenticating traffic.

With encryption enabled, performance on busy servers can substantially decline. To resolve these types of issues, install an IPsec offload card.

Filter Processing Issues

On busy servers, it is essential to be as frugal as possible with the number of filters. The IPsec driver caches filters that match a particular connection. If filters are defined that never match a connection, they are not cached and must be loaded every time a security association needs to be checked.

Wherever possible, write IPsec filters and rules that cover multiple cases, and avoid using filters that specify particular ports.

IPsec and NLB Clusters

There are challenges implementing IPsec on Network Load Balancing (NLB) clusters. IPsec authenticates individual hosts to each other, so if one server in the cluster goes down, clients currently connected to that server need to renegotiate the connection. NLB under Windows Server 2003 resolves specific IPsec problems that were present in Windows 2000, when used in a cluster that shares a virtual IP address.

NLB allows multiple servers to be clustered together to provide high availability for a service by providing automatic failover to other nodes in the cluster if an individual server fails. IPsec directly opposes NLB in that it prevents different computers from handling the same client connection. Because IPsec is negotiated between individual hosts before any data is transmitted, the same node in the cluster must respond to established IPsec connections or the traffic will be dropped.

If the particular node in the cluster fails, existing IPsec connections to that server cannot detect the failover and rebuild the security association until the preset time-out period. In Windows 2000 and Windows XP, IPsec must be idle for 5 minutes before the client attempts to re-authenticate. This results in a two-to-six minute interruption in service for that client.

Windows XP SP1 (with the NAT-T update), Windows XP SP2, and Windows Server 2003 reduce the idle timer to one minute. There will always be a delay if a client happens to be connected to a cluster node that fails.

With Windows XP and earlier, NLB only worked with IPsec in clusters that use complete subnets—using DNS to distribute the load among several individual IP addresses in a class C subnet. Windows Server 2003 adds support for IPsec with NLB using a single virtual IP address for the entire cluster. Each node in the cluster negotiates a list of source IP ranges that it handles, dynamically adjusting the list based on traffic and failover state of other nodes in the cluster.

Network Address Translation Traversal (NAT-T)

Network Address Translation (NAT) has historically caused problems for IPsec. If a NAT device changes the port or source address of a packet, the checksum no longer verifies the packet and is rejected. Test labs and other test environments at Microsoft sometimes make use of NAT within the corporate network.

The IETF has created a NAT-T specification to address this issue, and a supported fix is available for Windows 2000 SP3 and SP4, and Windows XP SP1 and later. NAT-T is built into Windows XP SP2 and Windows Server 2003.

Troubleshooting Issues

Because IPsec encapsulates regular IP traffic inside ESP, troubleshooting network issues becomes challenging. Usually IPsec itself does not have any issues, but it depends upon the correct configuration of all of the supporting technologies. If there are any misconfigurations in a supporting technology, it can be difficult to find the source of the problem.

Troubleshooting IPsec involves checking for connectivity at several different layers. The basic process involves:

8.     Check for basic network connectivity, using utilities like ping, ipconfig, and route.

9.     Turn on verbose logging and use Net View to capture the creation of security associations and diagnose IKE issues.

10.   Check standard client logs to verify core networking, and netdiag.exe to look at DNS issues.

11.   Verify Group Policy and IPsec policy versions using gpresult, ipseccmd, ipsecpol, and netdiag.

12.   Review security events using logman.exe.

13.   Verify necessary service functionality and look for known problem services and applications using netsh, srvinfo, and tlist.

To diagnose these issues, Microsoft IT enables auditing using domain-based group policies. If an issue is not resolved using these basic troubleshooting steps, the helpdesk personnel gather all associated data escalates to Tier 2, 3, and 4 support.

Most of the Microsoft tools mentioned are available natively in the operating system, in the Support Tools provided with the operating system, or included in a resource kit.

For advanced diagnostics, it may be necessary to enable Oakley logging, which can provide details about the negotiations used to create a Security Association. For more information, see the KB article #257225, "Basic IPsec Troubleshooting in Microsoft Windows 2000 Server," available at http://support.microsoft.com/default.aspx?scid=kb;en-us;257225.

Best Practices

Successful isolation of a corporate domain using IPsec requires a number of different decisions. Companies deploying an IPsec solution need to consider Group Policy design, IPsec filter design, individual deployment steps, individual treatment for different segments, how to handle non-domain-joined computers, and high-availability (NLB) clusters. This section contains recommendations from Microsoft IT to help companies design their own IPsec Domain Isolation solution.

Group Policy Design

When designing a Group Policy for Domain Isolation using IPsec on a corporate network, Microsoft IT recommends:

• Set up group policies for all behavior types to support IT and Business Unit IPsec testing. On a large network, exception policies take additional management time. Wherever possible, group exceptions together to simplify management. Identify as few different types of security groups and corresponding policies as necessary for business and IT needs.
• Filter the "Apply Group Policy" Access Control Entry (ACE) for each policy to only the limited security user groups. Because people who can change IPsec policy can drastically affect the deployment of IPsec across the entire enterprise, it is critical to limit Group/IPsec administration to a small number of administrators.
• Use a naming convention that covers the policy and group function for easier management and troubleshooting. With different policies and security groups, it is easy to get confused about which policy was created for which security group. For example, create a "Secure Request" group policy that corresponds to a "Secure Request" security group.

IPsec Design

When creating and deploying IPsec policies, filters, and rules, Microsoft IT learned a lot about IPsec design.

Filter weighting is the most sensitive part of filter design. The filters cannot overlap, so plan subnets accordingly. It is best to make the secure subnets list use the largest subnets possible and the permitted subnets use the smallest subnets possible to minimize the overall number of filters. Ideally there should be a subnet gap between two filter lists—for example, make the smallest secure subnet filters a /18 network (a subnet mask of 255.255.192.0) and the largest permitted subnet filters a /23 network (a subnet mask of 255.255.254.0). Then experiment with deploying request mode and piloting secure request mode to specific subnets. This can be done by adding a Secure Request Subnets Pilot filter where the subnets used are between /19 and /22 (subnet masks of 255.255.224.0 and 255.255.252.0). That way all of the filters will work together and not cancel each other out. See the Filter Design Challenges section for more details about designing filters.

Here are other specific best practices.

• Use "Any" instead of "Me" as the base approach to filter design. "Any" filters behave the same on all computers, even those with exempted IP addresses. "Me" filters can fail if the IP address of the local computer changes. "Me" filters have been optimized on Windows XP and Windows Server 2003, but may cause performance issues in Windows 2000.
• Create "Any <-> Corporate subnet" rules instead of "Me <-> Any" for secure subnets. These rules can be applied evenly across all Windows platforms, including platforms with virtual IP configurations. In addition Microsoft IT found that mobile computers with "Me <-> Any" rules may fail to correctly negotiate IPsec when they connected to other networks where incompatible IPsec policies are in use.
• Manage permitted subnets. Exempted subnets can be a security hole in an IPsec policy. For subnets that must be excluded from the IPsec policy, use other technologies to secure the traffic. For example, Microsoft IT allows unauthenticated traffic to and from its Extranet subnets. These subnets have been specifically created to provide access to particular resources for vendors and partners. To secure traffic between these subnets and the corporate network, Microsoft IT relies on router ACLs.
• Use "Any" rules for virtual IP addresses used by Clusters. A rule specifying "Me <-> Virtual IP address" results in a policy mismatch on servers in the cluster. Using "Any <-> Virtual IP address" allows IPsec to be used on the dedicated IP address while not using IPsec on the virtual IP address.
• Permit unsecured traffic to infrastructure servers. Because a client must connect to a domain controller to get a Kerberos ticket, it cannot use IPsec until after it has authenticated. Domain Controllers and DHCP servers must permit unsecured traffic. Microsoft IT also allows unsecured traffic to WINS and DNS servers and proxy servers connecting to the Internet, allowing untrusted computers to connect to boundary computers and Internet resources. There is less management overhead to exempt infrastructure by IP addresses and subnets in global settings than to exempt each required protocol.
• Use Kerberos as the default authentication mechanism. Kerberos involves less configuration administration than certificates. Kerberos works between forests if a two-way trust exists between the forests or domains, and if the name on the trust link is a fully qualified DNS domain name, instead of the NetBIOS name. If a network must support non-domain joined computers or non-Windows computers, it will need to use certificates.
• Set NoDefaultExempt = 1 via group policy ADM template. This setting removes the exemptions for Kerberos and RSVP. These protocols do not typically need to be exempted. It leaves the exemption for broadcast and multicast traffic, and also allows IKE.
• Permit the ICMP protocol. IPsec troubleshooting can be difficult. By allowing ICMP traffic, it is easier to verify basic network connectivity before having to troubleshoot more specific IPsec issues. ICMP is also required for PMTU detection, a low-level negotiation of packet size between hosts.
• Minimize securing by port or protocol. Use "All IP" filters wherever possible. When defining filters with multiple ports specified, falling back to clear communications sometimes fails. This is a result of the IPsec specification—if one computer has a policy that secures all IP traffic communicates with another that only secures a particular port; connections to other ports without a defined policy will fail. Securing all IP traffic to particular addresses without specifying ports greatly simplifies the policy and increases security.
• Avoid "Any <-> Any" filters. These types of filters do not work on Windows 2000.
• Don't use IPsec Default Response rule with custom policy. The Default Response rule does not allow specifying particular ports or protocols, including ICMP.

Deployment Options

There are many different ways of deploying IPsec to segment a corporate network. Microsoft IT recommends evaluating different methods and choosing the most appropriate for a corporate network environment. Microsoft IT found that different deployment options were appropriate for different stages of deployment in the corporate network.

• Deploy by subnet. Microsoft IT originally deployed IPsec policies to individual subnets. When switching to Secure Request mode, these policies did not work correctly because workgroup computers and other non-IPsec clients on the same subnet were no longer able to connect to required resources. Although this was the expected end state, the trouble was with communicating this level of impact to the end user community during the rollout. To successfully deploy IPsec policies by subnet, a complete understanding of data flows in the corporate network is necessary.
• Deploy by security group. In the Secure Request policy pilot stage, Microsoft IT chose to deploy IPsec on the corporate network using both security groups and subnet-specific deployments. Security groups make it relatively straightforward to create different IPsec policies for different types of computers although they do add overhead for computer account management. Boundary computers can be managed as one security group. Sensitive business resources can be protected by a more restrictive IPsec policy. Businesses that want to segment resources into different grades or levels of protection should consider deploying by security group.
• Deploy by domain. Adding an IPsec Group Policy to a domain is the simplest approach to deploying and managing IPsec on a network. If a single policy can be constructed to cover all computers on a domain, allowing access to infrastructure servers and securing other types of access, deploying by domain is the best approach. However, due to the number of legacy issues discovered during deployment, Microsoft IT recommends using the other deployment options first to uncover and resolve these issues and to give support teams ample time to develop proficiency with handling IPsec-related support issues. The big advantage of managing a single IPsec policy for a domain is that it becomes very easy to deploy a new policy to the entire domain if necessary. This ability could prove to be useful in the event of a new virus or exploit, making it possible to block propagation very quickly.

Recommended Deployment Steps

When deploying IPsec on a corporate network, Microsoft IT recommends the following overall steps:

• Pilot Request Mode IPsec. Deploy an IPsec policy with only exceptions, no secure rules to all domain computers. This ensures that any previously existing and potentially incompatible local IPsec policies are removed. Then add a secure rule on a single subnet in the corporate network. For example, make a rule that specifies: "Any <-> Subnet #1, All IP, Request Security."
• Deploy Request Mode IPsec. After resolving any issues discovered in the request mode pilot, continue to add additional subnets used on the corporate network to the policy until the deployment is completed.
• Pilot Secure Request IPsec policy. Create a security group for a few computers to use in the pilot. Apply the Secure Request IPsec policy to the GPO for the security group, allow read and apply permissions of the GPO to the security group, and test. Microsoft IT recommends using security groups for this step because subnets are harder to troubleshoot.
• Deploy Secure Request IPsec policy. Apply the Secure Request policy to more and more computers by adding them to a security group GPO that allows the application of the secure request policy. When most of the deployment and legacy issues have been resolved, the policy can be applied to all computers in a domain.

Microsoft IT recommends staging deployments gradually across an enterprise network. Start with a few subnets or security groups, and identify the technical issues. Take time to correct or consider the issues before deploying to the entire network. Do not attempt to deploy to entire domains until most issues have been resolved.

Non-domain Joined Clients

Use Kerberos exclusively for an IPsec deployment, if possible. Kerberos simplifies the administrative work involved in isolating a corporate domain. If a network is already managed by Active Directory, Kerberos, and Group Policy, adding a Public Key Infrastructure to manage non-domain joined computers adds administrator overhead.

As described earlier, Microsoft supports several different groups of non-domain joined clients. For many specific cases already described, Microsoft IT created specific security groups to provide access to necessary resources by making them boundary computers. Microsoft IT recommends carefully evaluating the need to create exceptions to global IPsec policies. Most corporate networks have some devices that cannot be joined to a domain. For these computers, creating special boundary computers may be the best way to manage access to resources from untrusted computers.

Microsoft chose to use Kerberos as the main IPsec authentication process. This allows the entire network to be managed using domain management tools such as Active Directory and Group Policy, providing granularity for defining security groups within individual domains for more restrictive security.

Even so, Microsoft decided to support Windows-based computers not joined to a corporate domain connecting over VPN and provide access to SecureNet. These non-domain joined computers are primarily employees, partners, and vendors connecting to the corporate network with computers not owned by Microsoft. To support these computers, the IPsec deployment has to support certificates for authentication, in addition to Kerberos.

IPsec and NLB

If high availability for a cluster is more important than securing it with IPsec, consider exempting the cluster's virtual IP address in the IPsec policy, making clients connect to the cluster without IPsec.

As described earlier, IPsec on Windows Server 2003 does work with NLB, but only to the extent that new connections can always be made to a cluster. Existing connections can experience dropped connections if one of the servers in a cluster goes down.

Microsoft IT exempted business-critical services that required high availability from its IPsec deployment.

Conclusion

Within Microsoft, IPsec policies were first applied in point solutions based on business requirements. When considering Domain Isolation as a way to improve security on the corporate network, these point solutions were considered and small subnets were chosen for initial deployment where problems could be discovered and resolved prior to wider deployment. Phase 1 of the IPsec deployment at Microsoft was completed in March 2003, resulting in more than 160,000 computers running IPsec. Technical issues were identified and corrected, or considered. No connectivity was blocked during this phase, and over 70 percent of all Microsoft-internal network traffic was encapsulated in ESP.

Phase 2 of the IPsec deployment involved deploying Secure Request mode across the enterprise. By deploying IPsec using domains instead of subnets, Microsoft IT was able to complete Phase 2 ahead of schedule, in April 2004. The impact on the Helpdesk has been minimal—less than 90 calls per week related to IPsec, well within the original expected impact. The number of calls is expected to decrease quickly now that the deployment is complete.

Now, approximately 208,000 computers on the Microsoft corporate network are running in Secure Request mode. These computers can no longer be accessed from untrusted computers, greatly reducing their exposure to worms and attackers.

Moving forward, the IPsec team plans to further review the categories of boundary machines in detail, determining if further mitigation is necessary and is also considering the use of encryption in specific "secure server" scenarios. Microsoft IT is also researching the use of IPsec in its Extranet environment. Otherwise, the project is considered complete, and moving into a maintenance state.

For More Information

For scenario-based guidance on domain and server isolation, refer to "Server and Domain Isolation Using IPsec and Group Policy" at http://go.microsoft.com/fwlink/?linkid=33945.

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

http://www.microsoft.com/

http://www.microsoft.com/technet/itshowcase

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Microsoft grants you the right to reproduce this White Paper, in whole or in part, specifically and solely for the purpose of personal education.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, email address, logo, person, place or event is intended or should be inferred.

© 2004 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.