Configuring IPSec Tunnel Mode VPN Between ISA Server 2004 and Cisco PIX v6.3.1

  • Article

Firewall administrators attempting to implement Internet Protocol security (IPSec) in tunnel mode with Microsoft® Internet Security and Acceleration (ISA) Server 2000 were unsuccessful due to an incompatibility between the network address translation (NAT) driver of ISA Server and IPSec. (This same problem was also encountered when using NAT within Routing and Remote Access). This interrupted IPSec only in tunnel mode. Using Layer Two Tunneling Protocol (L2TP) was the suggested solution because L2TP uses a transport mode policy and does not encounter this problem.

With ISA Server 2004, the NAT interaction incompatibility has been removed, and IPSec tunnel mode is now possible. Note that in both Microsoft Windows Server™ 2003 and Windows® 2000 Server, there is still an incompatibility with Routing and Remote Access NAT.

For additional information about this scenario, refer to the following articles:

This guide avoids using the term "IPSec tunnel" to refer to the encapsulation between the two networks. Referring to an IPSec tunnel may cause confusion because the term is used when referring to any type of IPSec protection—either transport mode or tunnel mode. More properly, and to avoid confusion, this guide uses the term "IPSec tunnel mode policy" when referencing the configuration.

Network Captures of IPSec in Tunnel Mode

This section briefly describes how IPSec works in tunnel mode. For a diagram of the network topology, see Figure 4 later in this document.

In this example, traffic is transmitted from the client on the Cisco Pix Internal network, traverses the IPSec tunnel mode policy, and is then received on the ISA Server network. When using Encapsulating Security Payload (ESP), traffic is typically encrypted using Data Encryption Standard (DES) or Triple DES (3DES) and authenticated with SHA1 or MD5. However, you can specify to use Null (no) Encryption so that the packets can be seen. An IPSec tunnel mode policy with Encryption is configured initially, and then Null Encryption is specified, so that the packet structure with ESP can be seen as it traverses the network.

Figure 1 shows a client, 172.25.3.10, using the PING protocol to search for a server, 172.25.10.10, which is located across the IPSec tunnel mode policy. This is what the packet looks like before IPSec protection. The data in the right side of the bottom pane, abcdefghijklmnop..., is the payload that a Windows client uses for Internet Control Message Protocol (ICMP).

Figure 1   Capture taken from the network card on 172.25.3.10

Figure 2 shows the results when the PING is protected by IPSec in tunnel mode with ESP encrypted with 3DES. In this image, in Source Address and Destination Address, the original client source address and server destination address are replaced. The source is now the external address of the PIX system, 192.168.55.1, and the destination is the ISA Server system, 192.168.55.100. The client source IP address, destination IP address, and the data, abcdefghijklmnop..., below the IP header are encrypted, so you cannot decipher the packet structure further.

Figure 2   Capture taken from the external interface of ISA Server (192.168.55.100)

Figure 3 shows the results of the PING when using ESP with null encryption and SHA1 for authentication. The figure shows the IPSec IP header (highlighted with a solid black line) that was added, which contains the tunnel mode policy endpoints as the source and destination, the ESP header, the original IP header (highlighted in the black dash line), and the ICMP payload. Also, you can read the data in the bottom pane, abcdefghijklmnop..., even though it is within ESP.

**Figure 3   Capture taken from the external interface of ISA Server (192.168.55.100)**IPSec accomplishes this in two steps. The first step is called Main Mode and the second step is called Quick Mode. (There is another mode that replaces Main Mode called Aggressive Mode, but this is not included in any Windows operating system.) Comprehensive explanations of what Main Mode and Quick Mode accomplish are beyond the scope of this document, but are explained in detail in the Windows Server 2003 Resource Kit (https://go.microsoft.com/fwlink/?LinkId=32054).

Main Mode is responsible for authenticating both sides of the IPSec tunnel mode policy (either using certificates or a preshared key) and generating a Diffie-Hellman key used to secure the second portion (Quick Mode). There are additional parameters negotiated during Main Mode, but these two tasks are the primary functions.

Quick Mode is responsible for negotiating the specific protocols, and source and destination addresses that will be included in the IPSec tunnel mode policy. Additionally, Quick Mode negotiates how this traffic will be protected (using the encryption algorithms DES or 3DES and the authentication algorithms SHA1 or MD5). There are other settings negotiated, but these are the primary tasks.

Network Topology

The scenario described in this document is based on the network topology shown in this figure.

Figure 4   Network topology

References

For more information, see the Cisco PIX Firewall and VPN Configuration Guide (Version 6.3)