IP Security (IPSec)

  • Article

This section provides information about using IPSec security with MOM, and includes the procedures needed to configure IPSec for this purpose.

Understanding IPSec

Internet Protocol Security (IPSec) is a framework of open standards for ensuring private, secure communications over Internet Protocol (IP) networks, through the use of cryptographic security services. IPSec supports network-level peer authentication, data origin authentication, data integrity, data confidentiality (encryption), and replay protection. The Microsoft implementation of IPSec is based on standards developed by the Internet Engineering Task Force (IETF) IPSec working group.

IPSec is supported by the Windows Server 2003, Windows XP, and Windows 2000 operating systems and is integrated with the Active Directory service. IPSec policies can be assigned through Group Policy configuration of Active Directory domains and organizational units. This allows the IPSec policy to be assigned at the domain, site, or organizational unit level, simplifying IPSec deployment.

You can also assign IPSec policies using the IP Security Policies MMC snap-in on the local computer. You must be an administrator on every computer on which you want to assign IPSec policies.

Network administrators and managers benefit from integration of IPSec with Windows 2000 Server and Windows Server 2003 for a number of reasons, including:

  • Open industry standard. IPSec provides an open industry-standard alternative to proprietary IP-based security technologies. Network managers benefit from the resulting interoperability.

  • Transparency. IPSec exists below the transport layer, making it transparent to applications and users, meaning there is no need to change network applications on a user's desktop when IPSec is implemented in the firewall or router.

  • Authentication. Strong authentication services prevent the acceptance of data through the use of falsely claimed identities.

  • Confidentiality. Confidentiality services prevent unauthorized access to sensitive data as it passes between communicating parties.

  • Data origin authentication and integrity. Data origin authentication and integrity is provided by a hashed message authentication code (HMAC) value, which is included in every packet.

  • Dynamic re-keying. Dynamic re-keying during ongoing communications eliminates manual reconfiguration of secret keys and helps protect against secret key determination.

  • Secure links end to end. IPSec for Windows 2000 provides secure links end-to-end for private network users within the same domain or across any trusted domain in the enterprise.

  • Centralized management. Network administrators use IPSec policies to provide appropriate levels of security, based on user, work group, or other criteria. Centralized management reduces administrative overhead costs.

  • Flexibility. The flexibility of IPSec for Windows 2000 allows policies to apply enterprise-wide or to a single workstation.

IPSec is configured by creating global or local policies that define how IP packets are authenticated, signed, and encrypted, or are blocked for specific hosts, a range of hosts or an entire network.

IPSec Through a Firewall

If you have a firewall between two MOM computers, you might need to configure IPSec Tunneling. For more information about this, see article 252735, "How To Configure IPSec Tunneling in Windows 2000" or article 816514, "How To Configure IPSec Tunneling in Windows Server 2003," in the Microsoft Knowledge Base.

IPSec Through a NAT Device

IPSec can be used through a Network Address Translator (NAT device) in the following situations:

  • Windows 2000 to Windows 2000 (only if both have the L2TP/IPSec NAT-T Update installed)

  • Windows 2000 to Windows XP (only if both have the L2TP/IPSec NAT-T Update installed)

  • Windows 2000 (with L2TP/IPSec NAT-T Update installed) to Windows Server 2003

  • Windows Server 2003 to Windows Server 2003

If the L2TP/IPSec NAT-T Update is not installed on Windows 2000 and Windows XP end-points, IPSec cannot be used through a NAT device. This is true because the NAT device cannot translate the IP addresses in the secured packets without this update. For more information about installing the update, see the Microsoft Support article 818043 "L2TP/IPSec NAT-T Update for Windows XP and Windows 2000".

Windows Server 2003 supports IPSec NAT Traversal (NAT-T). IPSec NAT-T allows traffic to be secured by IPSec and also to be translated by a NAT device.

IPSec and MOM

You can use IPSec to add further security to your MOM environment. The connection between the Management Server and the agent is mutually authenticated and encrypted by default, and digitally signed in Active Directory environments.

Important

    A MOM Management Server that is in a non-IPSec domain cannot push install an agent to a target computer that is in a IPSec-enabled domain. A MOM Management Server that is in a non-IPSec domain cannot monitor an agent that is in a IPSec-enabled domain. To get this functionality, you must configure the IPSec-enabled Management Server as a "Boundary Server". For more information, see "Deploying Agents Across IPSec Boundaries" in Discovery-Based Deployment_library in this guide.

You can use this method to enhance the security of data flowing between the Management Server and the MOM Database Server, between the Management Server and the Administrator or Operator console (if you have installed them on a different computer), or between the MOM Database Server and the MOM Reporting Database.

The settings for all of these connections that use IPSec will be the same, except for the IP address, and are summarized below:

  • Filter Action. Require Security

  • Authentication Method. Kerberos V5 Protocol (default)

  • Connection Type. All network traffic (default)

  • Tunnel Setting. This rule does not specify an IPSec tunnel (default)

You can use IPSec between any of the following computer pairs:

  • The Management Servers and the MOM Database Server

  • The MOM Database Server and the Reporting Database

  • The Management Server and any agentless-managed computers

  • The Management Server and the Administrator or Operator consoles (if they are on a separate server)

Note

    The communications between the MOM 2005 agent and the Management Server is secured by default and IPSec is not needed.

You can use IPSec between the following computer pairs only if they are within the same domain, or have a trust relationship between their domains:

  • The MCF on the Management Server and the destination MCF.

  • The Reporting Database and the Reporting console.

You cannot use IPSec between any of the following computer pairs if they are accessed over the Internet. However, you can use Secured Sockets Layer (SSL) encryption for these:

  • The MCF on the Management Server and the destination MCF.

  • The Reporting Database and the Reporting console.

Using IPSec with Multihomed Agents or Redundant Management Servers

If you have an agent reporting to more than one management group, or if you have more than one Management Server in a management group, you must configure IPSec policies for each connection between the Management Servers and the agent.

Creating IPSec Policy

  • To configure IPSec, you create IPSec policies and then assign (activate) them. The IPSec policy includes one or more rules. These rules are a collection of filter lists, the actions taken for these filter lists, authentication methods used, any tunneling settings, and the connection type to which the filter list applies. For more information about IPSec Policies, see the Windows documentation. To create an IPSec policy, use the "To create a policy using specific IP address" procedure in this guide.

Important

    Before assigning this policy, you must create another policy on the destination that mirrors this policy, using this computer as the destination and the other computer as the source. You must active the policy on both computers for the policies to work correctly.

Using IPSec with DHCP - Client Reservations

IPSec is based on transport between two computers identified solely by their IP addresses. If you are using Dynamically Host Configuration Protocol (DHCP) assigned IP addresses, you might have to reconfigure the IPSec policies when the IP addresses of your MOM servers are re-assigned or changed.

This means that when the IP address of the source or the destination computer changes, the IPSec policy will no longer be configured for the two original servers. The IPSec policy will continue to function but for whatever computers have the IP addresses defined in the policy.

Even if you use the My IP Address or A Specific DNS Name (Windows Server 2003 only) options to set the IP address for the Filter, the actual IP address is used and remains static. IPSec will not do a DNS look-up for these options.

You can use client reservations to reserve a specific IP address for permanent use by a DHCP client. If multiple DHCP servers are configured with a scope that covers the range of the reserved IP address, the client reservation must be made and duplicated at each of these DHCP servers. Otherwise, the reserved client computer can receive a different IP address, depending on the responding DHCP server. For more information about using client reservations, see the Windows documentation.

Using IPSec with Subnet Masks

You can define the filter rules to use a subnet mask instead of a specific IP address. Be sure to specify as narrow of a subnet mask as possible that still encompasses all of the computers needed. If you use a subnet mask, all IP addresses that fall within that IP address range will have the IPSec policy applied. This policy might not work for all the computers in the IP address range. Also be aware that if you are using DHCP IP address assignments, that any of the MOM computers that are assigned address outside the subnet mask no longer have the policy applied to them.

You must still create a policy on each destination computer for its connection to the Management Server, even if using subnet masks.

Using IPSec on Non-Trusted Domains or Workgroups

You can use IPSec between MOM computers in a workgroup, between a MOM computer in a trusted domain and a MOM computer in a non-trusted domain, and between a MOM computer in a trusted domain and a MOM computer in a workgroup by using either the shared key method or by using certificates. To do this, use the To create a policy using specific IP address procedure, and on step 19, choose one of these other options for the authentication method.

IPSec and SNMP

If a computer is running an SNMP service, you must add a rule to prevent SNMP messages from being blocked:

  • The IP Filter List should specify the source and destination addresses of the SNMP management systems and agents. The Protocol type should be set to UDP, to and from ports 161 and 162. This requires two filters: one for UDP, to and from port 161, and the other for UDP, to and from port 162.

  • Set the Filter Action to Permit, which blocks negotiation for security and passes through any traffic that matches the IP Filter List.

Disabling IPSec

You can disable an IPSec policy by un-assigning it. The policy remains on the computers but is not active. You can assign or un-assign the policy at any time. To disable an IPSec policy, use the "To un-assign (deactivate) IPSec policies" procedure in this guide.