IPsec Security Requirements

  • Article

The solution in this appendix addresses the following security requirements:

Name resolution attacks must be avoided by using HOSTS and LMHOSTS files instead of DNS and WINS lookups.

Only trusted members of the SMS IPsec group can authenticate IKE with each other.

All IP traffic between the SMS IPsec group member IP addresses is required to be secured using IPsec ESP strong encryption and authentication.

All IP traffic to or from SMS IPsec group members and non-SMS IPsec group members does not require IPsec encapsulation or filtering. IPsec is not required to protect against inbound network attacks on ports and protocols that are listening on the SMS site systems, including the site server.

Important

If you require IPsec to protect against inbound network attacks from computers that are not part of the SMS IPsec group, refer to the Using Microsoft Windows IPsec to Help Secure an Internal Corporate Network Server whitepaper (https://go.microsoft.com/fwlink/?LinkId=33894) on the Microsoft Download site and see the Windows Help and Support Center.

Choosing an Authentication Method

Peer authentication is the process of ensuring that an IPsec peer is who it claims to be. By using peer authentication, IPsec can determine whether or not to communicate with another computer before the communication begins. IPsec performs peer authentication, but requires only mutual trust of the identities exchanged. It does not verify that the identity that is received is authorized to use a particular IP address.

You can use IPsec policies and IKE authentication to limit network access to the trusted computers in the SMS IPsec group. In most scenarios, successful IKE authentication results in successful network access to a computer. IKE is not aware of the identity or the public key that is expected from the peer. Therefore, if the certificate private key or domain password for a computer were compromised, an attacker might be able to use that computer to successfully authenticate and gain access to another IPsec-protected computer or to conduct trusted man-in-the-middle attacks on IPsec communication. To ensure that IPsec provides the appropriate network access controls, limit access to the SMS IPsec computers by limiting the computers that can be authenticated by IKE with the SMS IPsec policy. The following methods are recommended as authentication methods for the SMS IPsec policy:.

Public Key Certificate

When certificates are used, the rule specifies a root CA name. Each computer in the SMS IPsec group would need a digital certificate in the computer store that has a certificate path up to this root CA name. This does not have to be a Microsoft Windows® 2000 or Microsoft Windows Server™ 2003 CA. Refer to the detailed IPsec documentation for the type of certificate and non-Microsoft Public-Key Infrastructure (PKI) support. The most secure environment should have an offline root CA that is dedicated only to the purpose of issuing these certificates for use with IPsec. However, an existing PKI can be used if it is capable of allowing a computer to get a certificate (instead of a user). The cheapest method of deploying this scenario with certificates is to install the CA service on one of SMS servers in the group as its own root CA. Certificates eventually need to be refreshed. The root CA can be registered in the domain as an Enterprise CA so that it is capable of automatically enrolling the SMS IPsec group computers with certificates through Group Policy. Configure the access control list (ACL) on the certificate template in the domain so that only the members of the SMS IPsec group can enroll for a certificate.

Preshared Key

There are several risks to using preshared keys:

  • The authentication key is stored in plaintext format in the system registry and hex-encoded in Active Directory®–based IPsec policy. Well-known methods can enable attackers with access to these data stores to discover weak preshared key values.

  • The keys themselves must be transported and stored in clear text, which offers many opportunities for the physical security of the key to be compromised.

  • Users might not understand what constitutes a strong key.

Even with the many risks, the advantage of using preshared keys is that they can be very helpful in providing granular control of specific groups of hosts that you wish to isolate. If you use preshared keys, you should use local IPsec policies on each box and block domain policy override to prevent attackers from viewing the hex-encoded key in Active Directory–based IPsec policy. Using local policy does not mitigate attacks against the registry to view the key stored in plaintext, but it is assumed that if the attacker has sufficient permissions to access the registry on the site system, he already has sufficient access to perform other attacks with high privilege. Using public key certificates is preferable to using a preshared key, but if your organization cannot support PKI or a separate CA in your PKI deployment, preshared keys are an alternative.

SMS IPsec Policy

The high level IPsec policy design that implements these requirements is simple. It has just one rule:

From Me to Each SMS IPsec Server IP, all traffic -> require ESP/3DES -> Certification authority (preferred) or preshared key (with understanding of the risks and limitations)

Default Response Rule = disabled

The following steps walk through the details of building this policy.