Publishing VPN Protocols in ISA Server

Publishing VPN Protocols in ISA Ser...

Microsoft Internet Security and Acceleration Server 2004

Microsoft® Internet Security and Acceleration (ISA) Server functions as a virtual private network (VPN) server. In addition to running ISA Server as a VPN server at the edge of a network, you may need to provide access to an ISA Server VPN that is not at the edge of a network, or to a VPN server not running on ISA Server in an internal network.

ISA Server uses server publishing rules to provide access to internal resources such as VPN servers. This document provides an overview of publishing VPN protocols with ISA Server 2004 and ISA Server 2000.

On This Page

VPN Protocols
Publishing VPN Protocols with ISA Server
PPTP vs. L2TP over IPsec in ISA Server 2004

VPN Protocols

This section provides a brief overview of VPN protocols and the features they provide:

Point-to-Point Tunneling Protocol (PPTP). PPTP is a VPN tunneling protocol based on Point-to-Point Protocol (PPP) that enables IP traffic to be encrypted, and then encapsulated in an IP header to be sent across a corporate IP network or a public IP network such as the Internet. PPTP encapsulates PPP frames in IP datagrams for transmissions. PPTP can be used for a remote access client-to-site VPN, and for VPN site-to-site connections. PPTP uses a TCP connection for tunnel management, and a modified version of Generic Routing Encapsulation (GRE) to encapsulate PPP frames for tunneled data. Payloads of encapsulated PPP frames can be encrypted, compressed, or both. Frames are encrypted using Microsoft Point-to-Point Encryption (MPPE). MPPE uses the encryption keys generated from the specified authentication method. PPTP does not provide encryption services. It simply encapsulates a previously encrypted PPP frame.
Layer Two Tunneling Protocol (L2TP). L2TP is a VPN tunneling protocol that, like PPTP, is also based on PPP. L2TP allows traffic to be encrypted, and then sent over any medium that supports point-to-point datagram delivery, such as IP, X.25, frame relay, or asynchronous transfer mode (ATM). When configured to use IP as its datagram transport, L2TP can be used as a tunneling protocol over the Internet. L2TP over IP networks uses User Datagram Protocol (UDP) and a series of L2TP messages for tunnel management. L2TP also uses UDP to send L2TP-encapsulated PPP frames as tunneled data.
IPsec. IPsec can be used in transport mode or in tunnel mode. In transport mode, application headers, TCP or UDP headers, and packet data are encrypted, but the IP header is not. IPsec can also be used as a tunnel mode, which encrypts the entire packet. IPsec protects data by encapsulating the original payload with either an IPsec header for transport mode, or an IPsec header and an additional IP header for tunnel mode. There are two IPsec protocols, Authentication Header (AH) and Encapsulating Security Payload (ESP), with some differences in the authentication, integrity, and confidentiality they provide. ESP is the only protocol that can be used with IPsec network address translation traversal (NAT-T). There are two modes:
- IPsec tunnel mode. With Microsoft Windows® installations, the IPsec protocol is commonly used for encryption in conjunction with L2TP. However, IPsec can be used as a tunneling protocol. IPsec in tunnel mode allows IP packets to be encrypted and then encapsulated in an IP header to be sent across a corporate network or a public network such as the Internet. It works only in IP-based networks. The primary reason for using IPsec tunnel mode is for interoperability with other routers or gateways that do not support L2TP over IPsec or PPTP tunneling.
- L2TP over IPsec. Unlike PPTP, the Microsoft implementation of L2TP does not use MPPE for encryption. Instead, it uses a combination of L2TP as the tunneling protocol, and IPsec (ESP) in transport mode for encryption.

There are a number of basic requirements that VPN protocols should fulfill:

Connection type. Specifies if the protocol can be used in a remote access (client-to-site) VPN connection, or in a site-to-site VPN connection.
User authentication. Specifies whether the protocol can verify the client’s identity, and restrict access to authorized users only. May also provide audit and accounting records to track connection characteristics.
Computer authentication. Specifies the ability to verify the identity of the client computer.
Address allocation and management. Specifies whether the protocol can manage address allocation for clients, to ensure that addresses are private.
Data encryption. Provides for encryption of data, to ensure that data carried over a public network such as the Internet must be rendered unreadable.
Key management. Specifies the ability to generate and refresh encryption keys for the encrypted data.

These features are summarized for each VPN protocol in the following table.

Feature	PPTP	L2TP over IPsec	IPsec Transport	IPsec Tunnel mode
VPN client-to-Site site connection	Yes	Yes	No	No
VPN site-to-site connection	Yes	Yes	No	Yes
Data encryption	Yes	Yes	Yes	Yes
User authentication	Yes	Yes	No	No
Computer authentication	No	Yes	Yes	Yes
Address assignment	Yes	Yes	No	No
Key management	No	Yes	Yes	Yes

Publishing VPN Protocols with ISA Server

The following table summarizes publishing support for VPN tunneling protocols in ISA Server 2004 and ISA Server 2000.

Protocol to be published	ISA Server 2000	ISA Server 2004	Details
PPTP	Not supported	Supported	The PPTP filter in ISA Server 2004 supports incoming connections, unlike in ISA Server 2000, where it supports only outgoing connections.
IPsec tunnel	Not supported, unless the IPsec tunnel mode VPN server supports NAT traversal (NAT-T)	Supported between networks with a route relationship, but not supported between networks with a NAT relationship, unless the IPsec tunnel mode VPN server supports NAT traversal (NAT-T)	The main support issue is that a NAT device such as ISA Server invalidates IPsec packets during the address translation process. In an ISA Server 2004 perimeter network configuration, if the perimeter network contains only public addresses and there is a route relationship between the front-end ISA Server computer connected directly to the Internet and the perimeter network, you can publish VPN servers behind the front-end server with IPsec in tunnel mode without going through NAT. If the perimeter network contains private addresses, you can only configure a NAT relationship between networks.
L2TP over IPsec (without NAT-T)	Not supported	Not supported between networks with a NAT relationship, unless the IPsec tunnel mode VPN server supports NAT traversal (NAT-T)
L2TP over IPsec NAT-T	Supported	Supported

NAT Traversal

There are a number of problems with using IPsec over NAT devices. A NAT device changes packet information during the address translation process. The process can either fail because information needed by the NAT device for address translation is encrypted, or the address translation process can cause the packet to be considered invalid by IPsec. NAT traversal (NAT-T) overcomes these issues to allow IPsec peers behind NAT devices to detect the presence of NAT devices, negotiate IPsec security associations (SAs), and send ESP-protected data, despite the fact that the addresses in the IPsec-protected packets are changed by NAT. For more information, see IPsec NAT Traversal Overview.

To allow ISA Server 2004 and ISA Server 2000 to pass IPsec traffic to a VPN server behind the ISA Server computer, the following is required:

The VPN server must be running Microsoft Windows Server™ 2003.
The L2TP over IPsec VPN protocol must be used.

All VPN clients must be using the IPsec NAT-T VPN client.

Note:
An IPsec NAT-T client update is available, with improvements to IPsec to better support VPN clients behind NAT devices. For computers running Microsoft Windows® XP Service Pack 1 (SP1) and Windows 2000, a download is available from article 818043, "L2TP/IPSec NAT-T update for Windows XP and Windows 2000," in the Microsoft Knowledge Base. By default, Windows XP Service Pack 2 (SP2) no longer supports establishing IPsec NAT-T connections to servers that are located behind NAT computers. For more information, see article 885407, "The default behavior of IPSec NAT traversal (NAT-T) is changed in Windows XP SP2, in the Microsoft Knowledge Base."

Note:

An IPsec NAT-T client update is available, with improvements to IPsec to better support VPN clients behind NAT devices. For computers running Microsoft Windows® XP Service Pack 1 (SP1) and Windows 2000, a download is available from article 818043, "L2TP/IPSec NAT-T update for Windows XP and Windows 2000," in the Microsoft Knowledge Base. By default, Windows XP Service Pack 2 (SP2) no longer supports establishing IPsec NAT-T connections to servers that are located behind NAT computers. For more information, see article 885407, "The default behavior of IPSec NAT traversal (NAT-T) is changed in Windows XP SP2, in the Microsoft Knowledge Base."

PPTP vs. L2TP over IPsec in ISA Server 2004

Using the PPTP filter in ISA Server 2004, you can publish PPTP. Considering this feature, you can decide whether to deploy L2TP over IPsec or PPTP. Compare the following:

IPsec provides per-packet authentication of the data source, to prove that data was sent by the authorized user. It also provides data integrity, replay protections, and data confidentiality. By contrast, PPTP provides only per-packet data confidentiality.
L2TP over IPsec connections provide stronger authentication by means of both certificate-based computer authentication, and user-level authentication. PPTP provides only user-level authentication.
L2TP over IPsec requires a certificate infrastructure to issue certificates to the VPN server and all VPN client computers for computer authentication. PPTP can use password-based authentication and does not require an installed certificate.
L2TP over IPsec must be deployed with NAT-T to work through a NAT device such as ISA Server.
Although it is possible to configure L2TP over IPsec VPN client computers using preshared key authentication, it is not recommended.

In summary, L2TP over IPsec is a more secure VPN protocol than PPTP. However, PPTP is still widely used. When using Microsoft Challenge Handshake Authentication Protocol version 2 (MS-CHAP v2) and strong passwords, PPTP provides enough security for many situations.

Publish a PPTP VPN Server with ISA Server 2004

To publish a PPTP VPN server, do the following.

Configure the VPN server.
Create a server publishing rule for the predefined PPTP server protocol, and then select the network that should listen for VPN requests. (For example, if you are publishing to the Internet, select the External network.) The PPTP server protocol defines TCP port 1723, with the PPTP filter enabled.

Publish an L2TP over IPsec VPN Server with ISA Server 2004

To publish an L2TP over IPsec with NAT-T VPN server, do the following.

Configure the VPN server.
Create a server publishing rule to publish Internet Key Exchange (IKE) negotiation. Select the predefined IKE server protocol, and then select the network that should listen for VPN requests. (For example, if you are publishing to the Internet, select the External network.) The IKE server protocol defines UDP port 500 (Receive Send).
Create a server publishing rule to publish NAT-T. Select the IPsec NAT-T server, and then select the network that will listen for VPN requests. The IPsec NAT-T server protocol defines UDP port 4500 (Receive Send).

Note:
You can publish L2TP without IPsec, but this is not recommended for publishing VPN resources to a public network such as the Internet.

Note:
You can publish L2TP without IPsec, but this is not recommended for publishing VPN resources to a public network such as the Internet.

For detailed instructions on publishing these protocols, see the article Publishing a VPN Server in ISA Server 2004, on the ISA Server VPN Guidance Center.