Security WatchIsolate Servers With IPsec

Joseph Davies

We live in an age of always-connected communication and are able to send e-mail messages or gain access to files, databases, and Web pages at any time, through a proliferation of networking technologies. However, the ease and ubiquity of constant connection has also exposed computers to risks like never before. That same connectivity also allows malicious programs (viruses and worms) and users to attack computers or their resources 24 hours a day from anywhere in the world. A more secure world of computing needs to allow this constant access, but only to authenticated and authorized users and computers. The resources of your network must be isolated, not only from the Internet, which is commonly done today, but also from unauthorized and unmanaged computers on your intranet.

For example, the data on sensitive servers is typically protected by access control security at the application layer. Before accessing the files on a sensitive file server, a user must typically provide security credentials, such as a user name and proof of knowledge of a password, before being granted access. However, specifying access control lists (ACLs) at the application layer does not protect the server from various kinds of attacks launched from unauthorized or unmanaged computers connected to your intranet.

To provide an additional layer of protection for sensitive servers, you can isolate them from unauthorized and unmanaged computers by implementing additional authentication and security at the Internet layer with Internet Protocol security (IPsec). With IPsec-based server isolation, unauthorized and unmanaged computers cannot initiate any type of IP-based communication with isolated servers. This additional level of protection is needed for organizations (such as those in the financial service or health care industries) that routinely send sensitive data on their networks and must provide extra protection for sensitive data assets.

The Process

To isolate servers, you need to first create a set of credentials so that computers that are attempting to communicate with isolated servers can prove their identity. When a computer joins an Active Directory® domain, an Active Directory domain controller creates a computer account with a set of credentials. In subsequent connections, the initiating domain member computer sends its credentials to authenticate a communication attempt, which can then be verified by any Active Directory domain controller.

Next, distribute network policy settings to configure the isolated servers and the computers that are allowed to communicate with them. By being a member of an Active Directory domain, the computer can receive centrally configured network policies through Group Policy. Network administrators use Group Policy to distribute computer and user settings to the member computers of an Active Directory domain.

All of the elements to isolate servers are already built into computers running Windows® XP, Windows Server™ 2003, or Windows 2000. The only requirement is to ensure that computers are members of your domain and to configure the appropriate Group Policy setting to enforce IPsec authentication and protection for traffic to and from the isolated server. No new hardware equipment is required.

Server isolation provides an extra layer of security and access control that operates independently and compatibly with security technologies such as IEEE 802.1X and Secure Sockets Layer (SSL). IEEE 802.1X requires a computer to authenticate itself prior to being allowed to send any frames on a network, but it does not protect data sent on the network. SSL provides computer authentication and data confidentiality (encryption), but only for SSL-enabled client and server applications. IPsec and server isolation operate either independently or in combination with 802.1X and SSL, providing end-to-end security services for IP traffic sent between computers.

Benefits of Server Isolation

Let's look at some of the benefits of server isolation. Server isolation restricts incoming communications to managed computers. A Windows-based managed computer is defined as a computer that is a member of an Active Directory domain. Managed computers can receive centrally configured network policies through Group Policy and other types of security updates, such as operating system updates and antivirus software signatures. Managed computers use their domain credentials to authenticate communication attempts with isolated servers. Unmanaged computers, consisting of standalone, unknown, and guest computers, do not have domain credentials, and therefore cannot authenticate communication attempts with isolated servers.

Server isolation also works to complement other security mechanisms designed to prevent unwanted communications. It provides additional security above and beyond existing security mechanisms deployed on your network, such as ACLs, 802.1X, SSL, and firewalls. For example, if your Internet firewall is somehow compromised, malicious users from the Internet will not be able to initiate communications with isolated servers.

Server isolation also encourages domain membership. By isolating critical organization servers such as e-mail servers, users on the organization network are no longer able to access the critical resource from an unmanaged computer. To receive valid domain credentials for performing IPsec authentication with isolated servers, users must join their computers to the domain. Once this occurs, the IT department of the organization can also ensure that the user's computer has the latest operating system and antivirus updates.

Another benefit of server isolation is that it protects traffic sent both to and from isolated servers. This traffic is cryptographically protected in such a way that the receiving computer can verify that an authenticated computer sent the packet and that it was not modified in transit. Additionally, the isolated server traffic can be encrypted, providing protection from malicious users on your organization network who are attempting to capture and interpret network traffic. By encrypting your traffic, your organization can comply with governmental regulations and business partner requirements to encrypt sensitive traffic on the network.

For improved security, isolated servers protect applications that cannot protect themselves. Applications running on servers that do not have facilities for enforcing access control or security at the application layer can use server isolation to enforce authentication, authorization, and communication security at the Internet layer.

To perform server isolation, you must configure Group Policy settings to require that all communication with isolated servers be authenticated and protected with IPsec. With IPsec protection, traffic is protected from data tampering attacks such as address spoofing, data injection, session hijacking, and replay attacks. Packets can be protected with data confidentiality. You can also configure exceptions so that specific computers that are not domain members are allowed to initiate unprotected communications with the isolated servers.

Configuration of Server Isolation

The components of server isolation consist of the Active Directory domain, member computers, Group Policy, and IPsec policy. The Active Directory domain includes domain controllers and the appropriate trust relationships to trust other domains or directory trees within an organization network. Also included here are member computers that have joined an Active Directory domain and have been assigned domain credentials.

Group Policy will be used to specify computer and user settings that are automatically downloaded to member computers and active IPsec policy determines the server isolation behavior of managed computers. To show how server isolation works in a simplified example deployment, configure and activate IPsec policy with rules that require secured traffic from domain members and permit unsecured traffic from specific excepted computers.

Once the policy settings are complete, the IPsec policy is activated for the appropriate Active Directory system containers such as sites, domains, and organizational units. The member computers in the Active Directory system containers to which the Group Policy settings apply automatically download the Group Policy settings.

After the IPsec Group Policy settings are downloaded and applied, managed computers that have both the correct IPsec policy for supporting server isolation and domain credentials are able to communicate with isolated servers. Unmanaged computers that do not have the correct IPsec policy for supporting domain isolation and domain credentials are unable to communicate with isolated servers.

Group-Specific Server Isolation

The server isolation configuration described in this article assumes that all managed computers have the ability to communicate with an isolated server. However, servers differ in the level of sensitivity of their data and whether they allow universal access or access only from specific computers. For example, e-mail servers typically have to be available to all the managed computers to allow someone on an arbitrary managed computer to be able to read their e-mail. However, finance or legal department servers should only be available to a specific subset of computers.

To further isolate sensitive servers and prevent managed computers that do not have authorization from communicating with them, there are several steps you could take. First, you could separate the sensitive servers and the authorized computers by IP address onto a separate subnet. You could then create a new subnet-based IPsec rule to require secure communication between the computers sharing the same subnet address prefix. However, this method requires additional network planning to get all of the servers and their authorized computers on a specific subnet. In addition, this method does not allow users' computers to roam off of their subnet, such as with a wireless laptop.

Another step you could take is separating the sensitive servers and the authorized computers by using a different authentication method. For example, you could specify an IPsec rule that uses an authentication method that differs from the method being used to access the other isolated servers. If the other isolated servers are using Kerberos authentication, the sensitive servers can use certificate-based authentication. Once this is configured, only the computers that have been allocated certificates can access the sensitive servers. This method requires the installation of a certification authority whose sole purpose is to assign certificates to access the sensitive servers.

Both of these methods use the parameters of IPsec policy to further isolate a subset of servers that are more sensitive and enforce an additional layer of authorization that prevents access except to a specified subset of computers.

For Windows-based servers, you can use another method to enforce authorization for access that does not rely on IPsec policy settings. With the "Access this computer from the network" access right (a local Group Policy setting), you can specify the set of computer accounts or Active Directory security groups that are allowed to access a server over the network. When IPsec processes the credentials of the computer requesting communications, Windows checks this access right.

Resources

For Information About Active Directory

For Information About Group Policy

For Information About Windows IPsec

To further isolate sensitive servers based on Active Directory group membership, you must do three things. First, ensure that the sensitive servers have been added to the existing isolated servers IPsec policy. Second, create an Active Directory security group and add the computer accounts of the authorized computers to the group. Third, administer the local Group Policy settings of each sensitive server and change the "Access this computer from the network" access right so that it only contains the Active Directory security group that you created in the second step. (This access right is found in the local Group Policy at Computer Configuration\Windows Settings\Security Settings\Local Policies\ User Rights Assignment\Access this computer from the network.)

For example, for a file server that contains sensitive financial data, add the financial server to the isolated servers IPsec policy, create a ConfidentialFinancial security group, and then add the authorized computer accounts. Next, change the "Access this computer from the network" access right on the financial file server so that it contains only the ConfidentialFinancial security group.

By leveraging Active Directory's authentication and Group Policy infrastructure, IPsec server isolation can easily provide an extra layer of protection for sensitive servers. This extra layer ensures that unmanaged computers on your intranet cannot initiate any type of IP-based communications with any isolated servers. With group-specific server isolation, you can specify that managed computers on your network must also belong to a specific Active Directory security group that you designate, further reducing the number of computers that are going to be able to communicate with specific servers that contain sensitive data.

How Are You Communicating?

A Managed Computer to an Isolated Server

When a managed computer with both Active Directory credentials and domain isolation IPsec policy settings (for example, Computer 1) initiates communication with an isolated server (for example, Server 1), the following occurs:

1. The initial communication packet being sent by Computer 1 [for example, a TCP Synchronize (SYN) segment destined for the IP address of an e-mail server that is isolated] matches a rule of the active IPsec policy.
2. Because the rule requires secured traffic to isolated servers, Computer 1 uses IPsec to authenticate itself to Server 1 and to negotiate the use of IPsec protection.
3. Computer 1 uses its domain credentials to perform IPsec authentication with Server 1.
4. Computer 1 then uses its IPsec policy settings to negotiate the cryptographic settings for IPsec-protected communication with Server 1.
5. Computer 1 sends the IPsec-protected initial communication packet (the TCP SYN segment) to Server 1.
6. Server 1 sends the IPsec-protected response to the initial communication packet [for example, a TCP SYN-Acknowledgement (SYN-ACK) segment], to Computer 1.
7. Subsequent packets that are sent between Computer 1 and Server 1 are encrypted.

As a result of the IPsec policy and domain membership, managed computers authenticate and IPsec-protect communications to all isolated servers.

An Unmanaged Computer to an Isolated Server

When an unmanaged computer (for example, Computer 2) initiates communication with an isolated server (for example, Server 1), the following occurs:

1. Because it does not have any IPsec policy settings, Computer 2 sends its initial communication packet (for example, a TCP SYN segment—without encryption) to Server 1.
2. On Server 1, the initial communication packet sent by Computer 2 matches a rule of its IPsec policy, which requires that the incoming communication be IPsec-protected.
3. Because the rule does not allow incoming packets without IPsec protection, Server 1 silently discards the TCP SYN segment sent by Computer 2.
4. Subsequent communication initiation packets sent by Computer 2 to Server 1 are also discarded by Server 1.
5. Eventually, Computer 2 fails in its communication attempt with Server 1.

As a result of the IPsec policy on Server 1, unmanaged computers that attempt to communicate with an isolated server never receive a response, and therefore cannot connect to them. If a user on an unmanaged computer was able to duplicate the IPsec policy settings of a managed computer, it would still be unable to connect to Server 1 because it would not have a set of valid domain credentials with which to authenticate itself.

An Excepted Computer to an Isolated Server

When an excepted computer (for example, Computer 3) initiates communication with an isolated server (for example, Server 1), the following steps occur:

1. Because it lacks IPsec policy settings, Computer 3 sends its initial packet (a TCP SYN segment) to Server 1 without IPsec protection.
2. On Server 1, the initial packet sent by Computer 3 matches a rule of its IPsec policy, which permits unsecured communication to its own IP address from the IP addresses of specific excepted computers.
3. Server 2 sends a response to Computer 3 without IPsec protection.
4. Subsequent packets sent between Computer 3 and Server 1 are sent without IPsec protection.

Excepted computers can now communicate with isolated servers without IPsec protection.

Joseph Davies is a technical writer with Microsoft who has been writing about Windows networking topics for more than ten years. He has written five books for Microsoft Press and is the author of the monthly TechNet Cable Guy column.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.