Configuring IPSec Tunnel Mode VPN Between ISA Server 2004 and Netopia R9100 4.11.3

Article
02/29/2008

Firewall administrators attempting to implement Internet Protocol security (IPSec) in tunnel mode with Microsoft Internet Security and Acceleration (ISA) Server 2000 were unsuccessful due to an incompatibility between the network address translation (NAT) driver of ISA Server and IPSec. (This same problem was also encountered when using NAT within Routing and Remote Access.) This interrupted IPSec only in tunnel mode. Using Layer Two Tunneling Protocol (L2TP) was the suggested solution because L2TP uses a transport mode policy and does not encounter this problem.

With ISA Server 2004, the NAT interaction incompatibility has been removed, and IPSec tunnel mode is now possible. Note that in both Microsoft Windows Server™ 2003 and Windows® 2000 Server, there is still an incompatibility with Routing and Remote Access NAT.

For additional information about this scenario, refer to the following articles:

HOW TO: Configure IPSec Tunneling in Windows Server 2003 (https://go.microsoft.com/fwlink/?LinkId=32056)
How to Configure IPSec Tunneling in Windows 2000 (https://go.microsoft.com/fwlink/?LinkId=32055)
Using Internet Protocol Security with Network Address Translation and Internet Security Acceleration Server (https://go.microsoft.com/fwlink/?LinkId=32057)

This guide avoids using the term "IPSec tunnel" to refer to the encapsulation between the two networks. Referring to an IPSec tunnel may cause confusion because the term is used when referring to any type of IPSec protection—either transport mode or tunnel mode. More properly, and to avoid confusion, this guide uses the term "IPSec tunnel mode policy" when referencing the configuration.

Configuring ISA Server 2004

Configuring the Netopia R9100 Device

Testing

Network Captures of IPSec in Tunnel Mode

This section briefly describes how IPSec works in tunnel mode. For a diagram of the network topology, see Figure 4 later in this document.

In this example, traffic is transmitted from the client on the Netopia R9100 Internal network, traverses the IPSec tunnel mode policy, and is then received on the ISA Server network. When using Encapsulating Security Payload (ESP), traffic is typically encrypted using Data Encryption Standard (DES) or Triple DES (3DES) and authenticated with SHA1 or MD5. However, you can specify to use Null (no) Encryption so that the packets can be inspected. An IPSec tunnel mode policy with Encryption is configured initially, and then Null Encryption is specified, so that the packet structure with ESP can be seen as it traverses the network.

Figure 1 shows a client, 172.25.3.10, using the PING protocol to search for a server, 172.25.10.10, which is located across the IPSec tunnel mode policy. This is what the packet looks like before IPSec protection. The data in the right side of the bottom pane, abcdefghijklmnop..., is the payload that a Windows client uses for Internet Control Message Protocol (ICMP).

Figure 1 Capture taken from the network card on 172.25.3.10

Figure 2 shows the results when the search is protected by IPSec in tunnel mode with ESP encrypted with 3DES. In this image, in Source Address and Destination Address, the original client source address and server destination address are replaced. The source is now the external address of the Netopia R9100 device, 192.168.55.1, and the destination is ISA Server, 192.168.55.100. The client source IP address, destination IP address, and the data, abcdefghijklmnop..., below the IP header are encrypted, so you cannot decipher the packet structure further.

Figure 2 Capture taken from the external interface of ISA Server (192.168.55.100)

Figure 3 shows the results of the search when using ESP with null encryption and MD5 for authentication. The figure shows the IPSec IP header (highlighted with a solid black line) that was added, which contains the tunnel mode policy endpoints as the source and destination, the ESP header, the original IP header (highlighted in the black dash line), and the ICMP payload. Also, you can read the data in the bottom pane, abcdefghijklmnop..., even though it is within ESP.

Figure 3 Capture taken from the external interface of ISA Server (192.168.55.100)

Note

Figure 3 is from Network Monitor, which is built in to Windows Server 2003. Network Monitor versions prior to this are unable to decipher ESP when using Null Encryption.

IPSec accomplishes this in two steps. The first step is called Main Mode and the second step is called Quick Mode. (There is another mode that replaces Main Mode called Aggressive Mode, but this is not included in any Windows operating system.) Comprehensive explanations of what Main Mode and Quick Mode accomplish are beyond the scope of this document, but are explained in detail in the Windows Server 2003 Resource Kit (https://go.microsoft.com/fwlink/?LinkId=32054).

Main Mode is responsible for authenticating both sides of the IPSec tunnel mode policy (either using certificates or a preshared key) and generating a Diffie-Hellman key used to secure the second portion (Quick Mode). There are additional parameters negotiated during Main Mode, but these two tasks are the primary functions.

Quick Mode is responsible for negotiating the specific protocols, and source and destination addresses that will be included in the IPSec tunnel mode policy. Additionally, Quick Mode negotiates how this traffic will be protected (using the encryption algorithms DES or 3DES and the authentication algorithms SHA1 or MD5). There are other settings negotiated, but these are the primary tasks.

Diagram

The scenario described in this document is shown in the following figure.

Figure 4 Network topology

Configuring ISA Server 2004

After the ISA Server installation is complete, perform the following steps on the ISA Server computer to set up the IPSec tunnel mode configuration:

Create a remote site network that defines the IP subnet behind the Netopia R9100 system and IPSec settings for the IPSec tunnel mode configuration.
Create a network rule that defines how the traffic is passed to the Netopia R9100 network (either using NAT or routing the traffic).
Create a firewall policy access rule that defines which traffic is allowed to pass to the Netopia R9100 network, and also defines which traffic is allow from the Netopia R9100 network.

Create a Remote Site Network

A remote site network defines the network behind the Netopia device, and also defines the IPSec settings for the tunnel mode configuration. The New Site-to-Site Network Wizard creates a policy of IPSec settings that are not visible in the IPSec Policy Management console. The Main Mode and Quick Mode settings are dynamically inserted into the IPSec driver by the wizard. To create a remote site network, perform the following steps.

To start the wizard, select the Virtual Private Networks (VPN) node in the ISA Server console, and then select the Remote Sites tab. On the Tasks tab, click Add Remote Site Network.
In this example, a network definition that will specify the range of IP addresses that are accessible behind the Netopia R9100 device through the IPSec tunnel mode configuration will be created. Enter the name NetopiaNet, and then click Next.
Select IP Security protocol (IPSec) tunnel mode, and then click Next.
Enter the tunnel mode endpoint addresses. The Netopia R9100 device is the remote VPN gateway and ISA Server is the local VPN gateway. Then, click Next.
Select the type of authentication you want to perform for Main Mode negotiations. For this example, select Use pre-shared key for authentication and enter 123456789 for initial testing. Then, click Next.
Click Add to add the range of IP addresses that will be accessible through the tunnel mode configuration (the subnet that is behind the Netopia R9100 system).
If you want traffic destined for the Netopia R9100 device’s external interface included, specify its address. In the following example, the subnet 172.25.3.0 is defined as behind the Netopia R9100 device. Click OK.
Click Next.
Click Finish to complete the wizard. After the wizard is finished, click Apply to make the configuration change active.

After the changes are applied, you can view the IPSec settings from ISA Server or by using a command-line utility. There are two methods to view the settings from ISA Server. To use the first method to view the IPSec settings, perform the following steps.

On the Remote Sites tab, select the remote site network object you just created.
On the Tasks tab, click View IPSec Policy. The following dialog box appears.

Or, to use another method to view the IPSec settings, perform the following steps.

On the Tasks tab, click Configure Remote Site.
Select the Connection tab, and then click IPSec Settings.
Phase I (Main Mode) settings appear.
Click the Phase II tab. Phase II (Quick Mode) settings appear.

You can also use the command-line utility NETSH to view these Main Mode and Quick Mode policies and filters:

Main Mode Policy "c:\netsh ipsec dynamic show mmpolicy all"
IKE MM Policy Name         : ISA Server NetopiaNet MM Policy
IKE Soft SA Lifetime       : 28800 secs
Encryption     Integrity        DH    Lifetime (Kb:secs)    QM Limit Per MM
-------------------------------------------------------------------------------------------------------------------------------------------------------
3DES           SHA1              2             0:28000              0
Main Mode Filters "c:\netsh ipsec dynamic show mmfilter all"
Main Mode Filters: Generic
-------------------------------------------------------------------------------
Filter name                    : IPSec{4ECE7FAD-F0A7-45FB-BAF7-4E193EB814F6}
Connection Type            : ALL
Source Address             : <My IP Address>   (255.255.255.255)
Destination Address        : 192.168.55.100       (255.255.255.255)
Authentication Methods     : Preshared key
Security Methods           : 1  3DES/SHA1/DH2/28000/QMlimit=0
-------------------------------------------------------------------------------
Filter name                    : IPSec{163EABB5-9F2B-44ED-B80E-4D7C462E4846}
Connection Type            : ALL
Source Address            : <My IP Address>   (255.255.255.255)
Destination Address        : 192.168.55.1      (255.255.255.255)
Authentication Methods     :        Preshared key
Security Methods           : 1         3DES/SHA1/DH2/28000/QMlimit=0
2 Generic Filter(s)
Quick Mode Policy "c:\netsh ipsec dynamic show qmpolicy all"
QM Negotiation Policy Name : ISA Server NetopiaNet QM Policy
Security Methods Lifetime (Kb:secs) PFS DH Group
-------------------------------------------------------------------------------------------------------------------
ESP[3DES,SHA1] 0:3600 Medium (2)
Quick Mode Filters "c:\>netsh ipsec dynamic show qmfilter all"
Quick Mode Filters(Tunnel): Generic
-------------------------------------------------------------------------------
Filter name                    : IPSec{F886828B-A23B-4659-9F29-0B6129A3C9F8}
Connection Type            : ALL
Source Address             : 172.25.10.0       (255.255.255.0  )
Destination Address        : 172.25.3.0        (255.255.255.0  )
Tunnel Source              : <Any IP Address>
Tunnel Destination         : 192.168.55.1
Protocol                       : ANY     Src Port: 0      Dest Port: 0
Mirrored                       : no
Quick Mode Policy          : ISA Server NetopiaNet QM Policy
Inbound Action             : Negotiate
Outbound Action            : Negotiate
-------------------------------------------------------------------------------
Filter name                    : IPSec{DBC53B1F-5A48-47BF-9A2E-081793CE6555}
Connection Type            : ALL
Source Address             : 172.25.3.0        (255.255.255.0  )
Destination Address        : 172.25.10.0       (255.255.255.0  )
Tunnel Source              : <Any IP Address>
Tunnel Destination         : 192.168.55.100
Protocol                       : ANY     Src Port: 0      Dest Port: 0
Mirrored                       : no
Quick Mode Policy          : ISA Server NetopiaNet QM Policy
Inbound Action             : Negotiate
Outbound Action            : Negotiate
-------------------------------------------------------------------------------
Filter name                    : IPSec{34C43528-2089-4CA8-B801-4D2A822F38C2}
Connection Type            : ALL
Source Address            : 192.168.55.100       (255.255.255.255)
Destination Address       : 172.25.3.0        (255.255.255.0  )
Tunnel Source              : <Any IP Address>
Tunnel Destination         : 192.168.55.1
Protocol                       : ANY     Src Port: 0      Dest Port: 0
Mirrored                       : no
Quick Mode Policy          : ISA Server NetopiaNet QM Policy
Inbound Action             : Negotiate
Outbound Action            : Negotiate
-------------------------------------------------------------------------------
Filter name                    : IPSec{EA90C1F4-4CC2-44E4-BB88-D4B1E89B953C}
Connection Type            : ALL
Source Address             : 172.25.3.0        (255.255.255.0  )
Destination Address       : 192.168.55.100       (255.255.255.255)
Tunnel Source              : <Any IP Address>
Tunnel Destination         : 192.168.55.100
Protocol                       : ANY     Src Port: 0      Dest Port: 0
Mirrored                       : no
Quick Mode Policy          : ISA Server NetopiaNet QM Policy
Inbound Action             : Negotiate
Outbound Action            : Negotiate
4 Generic Filter(s)

You have now created a remote site network, and viewed the IPSec settings. Now that the remote site network has been defined, the next step is to define a relationship between the ISA Server Internal network and the Netopia R9100 remote network. In the next section, you will define whether you want the traffic to use NAT or be routed to the remote network.

Create a Network Rule

To create a network rule, perform the following steps.

In the ISA Server console, select Configuration, select Networks, select the Network Rules tab, and then on the Tasks tab, click Create a New Network Rule.
For this scenario, enter the name NetopiaNet to ISANet - Route, and then click Next.
On the Network Traffic Sources page, click Add.
Expand the Networks node.
Select the Internal network, click Add, and then click Close.
On the Network Traffic Sources page, click Next.
On the Network Traffic Destinations page, repeat the same procedure as before, but select the network object NetopiaNet.
On the Network Traffic Destinations page, click Next.
On the Network Relationship page, select Route, and then click Next.

Note

In this example, traffic is routed between the two networks. This is because the IP subnets are different. If your scenario has two IP subnets that overlap (both local and remote subnets are 192.168.0.x), you should consider either using NAT for the traffic or redefining one of the IP subnets so that there is no overlap.
On the summary page, review the rule details and then click Finish.
After the wizard is complete, click Apply to make the configuration changes effective.

You have now created a network rule. The next step is to create an access rule.

Create an Access Rule

Now that you have defined the remote site and the network rule, you need to define which traffic will pass through the IPSec tunnel mode configuration. You control this through the firewall policy by creating an access rule specifying the traffic you want to allow. To create an access rule, perform the following steps.

In the ISA Server console, select Firewall Policy, right-click, select New, and then click Access Rule.
Provide a name that describes accurately the source and destination networks, and the traffic allowed. For this scenario, enter the name NetopiaNet to ISANet – Allow All, and then click Next.
On the Rule Action page, select Allow, and then click Next.
On the Protocols page, in This rule applies to, select All outbound protocols, and then click Next.
Click Add.
Expand Networks.
Click Internal. You could optionally include Local Host if you want to allow ISA Server to send traffic to the remote network. Click Add, click Close, and then click Next.
Click Add, and then click Networks.
Click NetopiaNet for the destination network. Click Add, click Close, and then click Next.
Select which users to allow, and then click Next.
Review the settings in the summary screen, and then click Finish to complete the wizard.
After the wizard is complete, click Apply to make the configuration changes effective.

Note

You must complete the same procedure to allow traffic from the NetopiaNet subnet to the ISANet subnet. Routing rules (which you created earlier in this document) are mirrored, but access rules are "one-way."

You have now created a remote site network, a network rule, and an access rule. Now that ISA Server is configured, you will configure the Netopia R9100 device.

Configuring the Netopia R9100 Device

This procedure is based on the Netopia Technical Notes and Quick Guides article NQG_053: Configuring the Netopia router for IPSec with IKE (https://www.netopia.com/en-us/support/technotes/hardware/NQG\_05dware/NQG\_053.html). A few changes are incorporated, such as IP address changes, but the procedure is the same.

To configure the Netopia R9100 device, perform the following steps.

Access the Netopia R9100 device through a Console or Telnet connection. At the main console, press the down arrow until the Quick Menus option is selected, and then press ENTER.
On the Quick Menus screen, select Add Connection Profile.
In Profile Name, enter a descriptive name (for example, ISANet), and then press ENTER.
The Profile Enabled field is now selected. Press ENTER.
In Encapsulation Type, press ENTER. Select IPSec from the menu, and then press ENTER.
Encapsulation Options should now be selected. Press ENTER.
This opens the IPSec Tunnel Options screen. Key Management should be selected and IKE is set to the default. Leave IKE specified and press ENTER.
IKE Phase 1 Profile is selected. Press ENTER to create a new Phase 1 profile.
In the Phase 1 Profile screen, select the <<ADD PH1 PROFILE>> option, and then press ENTER.
The Add IKE Phase 1 Profile screen appears. In Profile Name, supply a descriptive name for the profile (for example, ISANetTM), and then press ENTER.
In Mode, you can choose between Main Mode and Aggressive Mode. Windows operating systems do not provide support for Aggressive Mode, so leave this as Main Mode. Press ENTER to continue.
The Authentication Method remains set as Shared Secret, and Shared Secret is selected automatically. Type the Shared Secret specified earlier in the ISA Server Remote Site Wizard. In this example, 123456789 is used. Then, press ENTER.
Change the Encryption Algorithm to 3DES and change the Hash Algorithm to SHA1.
The Diffie-Hellman Group setting defaults to Group 2 (1024 bits). Leave this option set to the default. There is a stronger Diffie-Hellman group available in the Netopia R9100 configuration, Group 5 (1536 bits), but Windows does not provide support for this Diffie-Hellman group. Windows Server 2003 does provide support for Diffie-Hellman Group 14 (2048 bits), but the Netopia R9100 device used in testing did not have this available.
Leave the Advanced IKE Phase 1 Options setting and select Add IKE Phase 1 Profile.
This should return you to the IPSec Tunnel Options screen. Go to the Encapsulation option and ensure ESP only is specified. Do not choose AH or ESP+AH.
Go to the ESP Encryption Transform option and change it to 3DES.
Change the ESP Authentication Transform option to HMAC-SHA1-96.
Advanced IPsec Options should now be selected. Press ENTER. Change the SA Lifetime seconds value to 3600, press ENTER, and then press ESC. This change mirrors the option that the ISA Server Remote Site Wizard chooses for the Phase II lifetime.
Go to the Commit option and press ENTER.
Go to the IP Profile Parameters option and press ENTER. In Remote Tunnel Endpoint, enter the ISA Server address. The Remote Member Format option should be set to Subnet. The Remote Member Address should be 172.25.10.0, and the Remote Member Mask should be 255.255.255.0.
The Local Member Format should be Subnet, the Local Member Address should be 172.25.3.0, and the Local Member Mask should be 255.255.255.0.
Address Translation Enabled should be set to No.
Leave the other options set to their defaults, select the Commit option, and then press ENTER.
Leave the Interface Group set to Any Port, select the Commit option, and then press ENTER.

This completes the configuration of the Netopia R9100 device. From the client behind the ISA Server computer, you should be able to search the client behind the Netopia R9100 device. You will receive a few time-outs with the initial search attempts, because there will be a slight delay as the IPSec security associations are negotiated. After the security associations are online, you will start to receive replies.

You can add the IPSec Monitor snap-in and view the settings. To do this, perform the following steps.

On the computer running Windows Server 2003, click Start, click Run, type mmc, and then click OK. (On Windows 2000, click Start, click Run, and then open IPSECMON.exe.)
On the File menu, click Add/Remove Snap-in, and add the IPSec Monitor snap-in.
Expand Console Root to view the Main Mode and Quick Mode security associations. Under Main Mode, click Security Associations. You should see the following screen, which details the Main Mode (Phase I) security association.
Under Quick Mode, click Security Associations. You should see the following screen, which details the Quick Mode (Phase II) security association.

Note

Quick Mode (Phase II) actually has two security associations—Inbound and Outbound—but IPSec Monitor only shows the Outbound security association.

Clients from behind each system should be able to access the remote site through the IPSec tunnel mode policy. (If not, you will need to consider routing tables on the clients.)

Testing

The testing process uses different application layer and transport layer protocols to ensure that data is encrypted and decrypted correctly as it passes through the IPSec tunnel. The following data transfer tests can be used to determine the success of the IPSec tunnel connection:

FTP Transfer
The FTP process uses an FTP GET of a single 100 megabyte (MB) file, renames the file, and then uses an FTP PUT to transfer the new file back to the FTP server. After the two transfers are completed, a comparison is performed, using Windiff.exe from the Windows 2000 Server Resource Kit, at the FTP server to ensure the two files are identical.
TFTP Transfer
The TFTP copy process replicates the FTP tests, with the only difference being that a 20 MB file is transferred rather than the 100 MB file transferred using FTP. Because Windows Server 2003, Windows XP, and Windows 2000 Server do not include a TFTP server, a third-party TFTP server (SolarWinds TFTP Server https://www.solarwinds.com) is used as a TFTP server for the tests. A Windows XP host is the client using the command-line utility TFTP.exe.
CIFS Transfer
The CIFS copy process transfers a folder structure with three subfolders containing a total of 311 files approximately 50 MB in size between the two computers. The data is transferred from the source computer to the target computer using the Resource Kit utility ROBOCOPY.exe and by copying within Windows Explorer. The files are then copied from the target computer to the source computer into a different folder structure. The folders are then compared using Windiff.exe from the Windows 2000 Resource Kit to ensure that the data is not corrupted during transmission.
PING with specific sizes
PING packets are sent from the target to the source computer using specific packet sizes to test packet fragmentation and reassembly through the IPSec tunnel. Specifically, packets sizes of: 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 40, 80, 160, 320, 640, 1280, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1500, 3000, 6000, 12000, 24000, 48000, and 65500 bytes.

Configuring IPSec Tunnel Mode VPN Between ISA Server 2004 and Netopia R9100 4.11.3

Network Captures of IPSec in Tunnel Mode

Diagram

Configuring ISA Server 2004

Create a Remote Site Network

Create a Network Rule

Create an Access Rule

Configuring the Netopia R9100 Device

Testing

Additional resources