Configuring IPSec Tunnel Mode VPN Between ISA Server 2004 and Astaro Security Linux

Firewall administrators attempting to implement Internet Protocol security (IPSec) in tunnel mode with Microsoft® Internet Security and Acceleration (ISA) Server 2000 were unsuccessful due to an incompatibility between the network address translation (NAT) driver of ISA Server and IPSec. (This same problem was also encountered when using NAT within Routing and Remote Access). This interrupted IPSec only in tunnel mode. Using Layer Two Tunneling Protocol (L2TP) was the suggested solution because L2TP uses a transport mode policy and does not encounter this problem.

With ISA Server 2004, the NAT interaction incompatibility has been removed, and IPSec tunnel mode is now possible. Note that in both Microsoft Windows Server™ 2003 and Windows® 2000 Server, there is still an incompatibility with Routing and Remote Access NAT.

For additional information about this scenario, refer to the following articles:

• HOW TO: Configure IPSec Tunneling in Windows Server 2003 (https://go.microsoft.com/fwlink/?LinkId=32056)
• How to Configure IPSec Tunneling in Windows 2000 (https://go.microsoft.com/fwlink/?LinkId=32055)
• Using Internet Protocol Security with Network Address Translation and Internet Security Acceleration Server (https://go.microsoft.com/fwlink/?LinkId=32057)

This guide avoids using the term "IPSec tunnel" to refer to the encapsulation between the two networks. Referring to an IPSec tunnel may cause confusion because the term is used when referring to any type of IPSec protection—either transport mode or tunnel mode. More properly, and to avoid confusion, this guide uses the term "IPSec tunnel mode policy" when referencing the configuration.

Configuring ISA Server 2004

Configuring Astaro Security Linux

Testing the ISA Server Tunnel Mode Policy

Network Captures of IPSec in Tunnel Mode

This section briefly describes how IPSec works in tunnel mode. For a diagram of the network topology, see Figure 4 later in this document.

In this example, traffic is transmitted from the client on the Astaro Security Linux system Internal network, traverses the IPSec tunnel mode policy, and is then received on the ISA Server network. When using Encapsulating Security Payload (ESP), traffic is typically encrypted using Data Encryption Standard (DES) or Triple DES (3DES) and authenticated with SHA1 or MD5. However, you can specify to use Null (no) Encryption so that the packets can be seen. An IPSec tunnel mode policy with Encryption is configured initially, and then Null Encryption is specified, so that the packet structure with ESP can be seen as it traverses the network.

Figure 1 shows a client, 172.25.25.10, pinging a server, 172.25.1.10, which is located across the IPSec tunnel mode policy. This is what the packet looks like before IPSec protection. The data in the right side of the bottom pane, abcdefghijklmnop..., is the payload that a Windows client uses for Internet Control Message Protocol (ICMP).

Figure 1   Capture taken from the network card on 172.25.25.10

Figure 2 shows the results when the search is protected by IPSec in tunnel mode with ESP encrypted with 3DES. In this image, in Source Address and Destination Address, the original client source address and server destination address are replaced. The source is now the external address of the ISA Server 2004 system, 192.168.100.25, and the destination is the Astaro Security Linux system, 192.168.100.1. The client source IP address, destination IP address, and the data, abcdefghijklmnop..., below the IP header are encrypted, so you cannot decipher the packet structure further.

Figure 2   Capture taken from the external interface of ISA Server (192.168.100.25)

Figure 3 shows the results of the PING when using ESP with null encryption and MD5 for authentication. The figure shows the IPSec IP header (highlighted with a solid black line) that was added, which contains the tunnel mode policy endpoints as the source and destination, the ESP header, the original IP header (highlighted in the black dash line), and the ICMP payload. Also, you can read the data in the bottom pane, abcdefghijklmnop..., even though it is within ESP.

Figure 3   Capture taken from the external interface of ISA Server (192.168.100.25) using Null Encryption

IPSec accomplishes this in two steps. The first step is called Main Mode and the second step is called Quick Mode. (There is another mode that replaces Main Mode called Aggressive Mode, but this is not included in any Windows operating system.) Comprehensive explanations of what Main Mode and Quick Mode accomplish are beyond the scope of this document, but are explained in detail in the Windows Server 2003 Resource Kit (https://go.microsoft.com/fwlink/?LinkId=32054).

Main Mode is responsible for authenticating both sides of the IPSec tunnel mode policy (either using certificates or a preshared key) and generating a Diffie-Hellman key used to secure the second portion (Quick Mode). There are additional parameters negotiated during Main Mode, but these two tasks are the primary functions.

Quick Mode is responsible for negotiating the specific protocols, and source and destination addresses that will be included in the IPSec tunnel mode policy. Additionally, Quick Mode negotiates how this traffic will be protected (using the encryption algorithms DES or 3DES and the authentication algorithms SHA1 or MD5). There are other settings negotiated, but these are the primary tasks.

Diagram

The scenario described in this document is shown in the following figure.

Figure 4   Network topology

Configuring ISA Server 2004

After the ISA Server installation is complete, perform the following steps on the ISA Server computer to set up the IPSec tunnel mode configuration:

1. Create a remote site network that defines the IP subnet behind the Astaro Security Linux system and IPSec settings for the IPSec tunnel mode configuration.
2. Create a network rule that defines how the traffic is passed to the Astaro Security Linux private network (either using NAT or routing the traffic).
3. Create a firewall policy access rule that defines which traffic is allowed to pass to the Astaro Security Linux network.

Create a Remote Site Network

A remote site network defines the network behind the Astaro Security Linux system, and also defines the IPSec settings for the tunnel mode configuration. The New Site-to-Site Network Wizard creates a policy of IPSec settings that are not visible in the IPSec Policy Management console. The Main Mode and Quick Mode settings are dynamically inserted into the IPSec driver by the wizard. To create a remote site network, perform the following steps.

1. To start the wizard, select the Virtual Private Networks (VPN) node in the ISA Server console, and then select the Remote Sites tab. On the Tasks tab, click Add Remote Site Network.
   
2. In this example, a network definition that will specify the range of IP addresses that are accessible behind the Astaro Security Linux system through the IPSec tunnel mode configuration will be created. Enter the name AstaroNet, and then click Next.
3. Select IP Security protocol (IPSec) tunnel mode, and then click Next.
   
4. Enter the tunnel mode endpoint addresses. The Astaro Security Linux system is the remote VPN gateway and ISA Server is the local VPN gateway. Then, click Next.
   
5. Select the type of authentication you want to perform for Main Mode negotiations. For this example, select Use pre-shared key for authentication and enter 123456789 for initial testing. Then, click Next.
   
6. Click Add to add the range of IP addresses that will be accessible through the tunnel mode configuration (the subnet that is behind the Astaro Security Linux system).
   
7. If you want traffic destined for the Astaro Security Linux system’s external interface included, specify its address. In the following example, the subnet 172.25.1.0-172.25.1.255 is defined as behind the Astaro Security Linux system. Click OK.
   
8. Click Next.
   
9. Click Finish to complete the wizard. After the wizard is finished, click Apply to make the configuration change active.  
   

Now that this is complete, change the configuration slightly to use ESP with 3DES for encryption and MD5 for authentication. The default is to use ESP with 3DES and SHA1. To make these changes, perform the following steps.

1. Select the AstaroNet Remote Site Network and click Configure Remote Site.
2. On the Connections tab, click IPSec Settings.
3. On the Phase I tab, change the Integrity Algorithm to MD5. Perform the same action on the Phase II tab.
4. Click OK and then click Apply to make the change active.

After the changes are applied, you can view the IPSec settings from ISA Server or by using a command-line utility. There are two methods to view the settings from ISA Server. To use the first method to view the IPSec settings, perform the following steps.

1. On the Remote Sites tab, select the remote site network object you just created.
   
2. On the Tasks tab, click View Remote Site IPSec Policy. The following dialog box appears.
   

Or, to use another method to view the IPSec settings, perform the following steps.

1. On the Tasks tab, click Configure Remote Site.
   
2. Select the Connection tab, and then click IPSec Settings.
   

Phase I and Phase II (Main Mode and Quick Mode) settings appear.

You can also use the command-line utility NETSH to view these Main Mode and Quick Mode policies and filters:

• Main Mode Policy "c:\netsh ipsec dynamic show mmpolicy all"
  IKE MM Policy Name         : ISA Server AstaroNet MM Policy
  IKE Soft SA Lifetime       : 28800 secs
   
  Encryption     Integrity        DH    Lifetime (Kb:secs)    QM Limit Per MM
  -------------------------------------------------------------------------------------------------------------------------------------------------------
  3DES           MD5              2             0:28800               0
• Main Mode Filters "c:\netsh ipsec dynamic show mmfilter all"
  Main Mode Filters: Generic
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{4ECE7FAD-F0A7-45FB-BAF7-4E193EB814F6}
  Connection Type            : ALL
  Source Address             : <My IP Address>   (255.255.255.255)
  Destination Address        : 192.168.100.1       (255.255.255.255)
  Authentication Methods     : Preshared key
  Security Methods           : 1  3DES/MD5/DH2/28800/QMlimit=0
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{163EABB5-9F2B-44ED-B80E-4D7C462E4846}
  Connection Type            : ALL
  Source Address            : <My IP Address>   (255.255.255.255)
  Destination Address        : 192.168.100.25      (255.255.255.255)
  Authentication Methods     :        Preshared key
  Security Methods           : 1         3DES/MD5/DH2/288000/QMlimit=0
   
  2 Generic Filter(s)
• Quick Mode Policy "c:\netsh ipsec dynamic show qmpolicy all"
  QM Negotiation Policy Name : ISA Server AstaroNet QM Policy
   
  Security Methods           Lifetime (Kb:secs)       PFS DH Group
  -------------------------------------------------------------------------------------------------------------------
  ESP[3DES,MD5]         0:3600            Medium (2)
• Quick Mode Filters "c:\>netsh ipsec dynamic show qmfilter all"
  Quick Mode Filters(Tunnel): Generic
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{F886828B-A23B-4659-9F29-0B6129A3C9F8}
  Connection Type            : ALL
  Source Address             : 172.25.1.0       (255.255.255.0  )
  Destination Address        : 172.25.25.0        (255.255.255.0  )
  Tunnel Source              : <Any IP Address>
  Tunnel Destination         : 192.168.100.1
  Protocol                       : ANY     Src Port: 0      Dest Port: 0
  Mirrored                       : no
  Quick Mode Policy          : ISA Server AstaroNet QM Policy
  Inbound Action             : Negotiate
  Outbound Action            : Negotiate
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{DBC53B1F-5A48-47BF-9A2E-081793CE6555}
  Connection Type            : ALL
  Source Address             : 172.25.1.0        (255.255.255.0  )
  Destination Address        : 172.25.25.0       (255.255.255.0  )
  Tunnel Source              : <Any IP Address>
  Tunnel Destination         : 192.168.100.25
  Protocol                       : ANY     Src Port: 0      Dest Port: 0
  Mirrored                       : no
  Quick Mode Policy          : ISA Server AstaroNet QM Policy
  Inbound Action             : Negotiate
  Outbound Action            : Negotiate
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{34C43528-2089-4CA8-B801-4D2A822F38C2}
  Connection Type            : ALL
  Source Address            : 192.168.100.25       (255.255.255.255)
  Destination Address       : 172.25.1.0        (255.255.255.0  )
  Tunnel Source              : <Any IP Address>
  Tunnel Destination         : 192.168.100.1
  Protocol                       : ANY     Src Port: 0      Dest Port: 0
  Mirrored                       : no
  Quick Mode Policy          : ISA Server AstaroNet QM Policy
  Inbound Action             : Negotiate
  Outbound Action            : Negotiate
  -------------------------------------------------------------------------------
  Filter name                    : IPSec{EA90C1F4-4CC2-44E4-BB88-D4B1E89B953C}
  Connection Type            : ALL
  Source Address             : 172.25.1.0        (255.255.255.0  )
  Destination Address       : 192.168.100.25       (255.255.255.255)
  Tunnel Source              : <Any IP Address>
  Tunnel Destination         : 192.168.100.25
  Protocol                       : ANY     Src Port: 0      Dest Port: 0
  Mirrored                       : no
  Quick Mode Policy          : ISA Server AstaroNet QM Policy
  Inbound Action             : Negotiate
  Outbound Action            : Negotiate
   
  4 Generic Filter(s)

You have now created a remote site network, and viewed the changes to the IPSec settings. Now that the remote site network has been defined, the next step is to define a relationship between the ISA Server Internal network and the Astaro private network. In the next section, you will define whether you want the traffic to use NAT or be routed to the remote network.

Create a Network Rule

To create a network rule, perform the following steps.

1. In the ISA Server console, select Configuration, select Networks, select the Network Rules tab, and then on the Tasks tab, click Create a New Network Rule.
   

2. For this scenario, enter the name ISANet to AstaroNet - Route, and then click Next.

3. On the Network Traffic Sources page, click Add and then expand the Networks node.
   

4. Select the Internal network, click Add, and then click Close.

5. On the Network Traffic Sources page, click Next.

6. On the Network Traffic Destinations page, repeat the same procedure as before, but select the network object AstaroNet.
   

7. On the Network Traffic Destinations page, click Next.

8. On the Network Relationship page, select Route, and then click Next.

   Note

   In this example, traffic is routed between the two networks. This is because the IP subnets are different. If your scenario has two IP subnets that overlap (both local and remote subnets are 192.168.0.x), you should consider either using NAT for the traffic or redefining one of the IP subnets so that there is no overlap.

9. On the summary page, review the rule details and then click Finish.

10. After the wizard is complete, click Apply to make the configuration changes effective.
    

You have now created a network rule. The next step is to create an access rule.

Create an Access Rule

Now that you have defined the remote site and the network rule, you need to define which traffic will pass through the IPSec tunnel mode configuration. You control this through the firewall policy by creating an access rule specifying the traffic you want to allow. To create an access rule, perform the following steps.

1. In the ISA Server console, select Firewall Policy, right-click, select New, and then click Access Rule.
   

2. Provide a name that describes accurately the source and destination networks, and the traffic allowed. For this scenario, enter the name ISANet to AstaroNet – Allow All, and then click Next.

3. On the Rule Action page, select Allow, and then click Next.
   

4. On the Protocols page, in This rule applies to, select All outbound protocols, and then click Next.
   

5. Click Add, and then click Networks.
   

6. Click Internal. You could optionally include Local Host if you want to allow ISA Server to send traffic to the remote network. Click Add, click Close, and then click Next.
   

7. Click Add, and then click Networks.
   

8. Click AstaroNet for the destination network. Click Add, click Close, and then click Next.
   

9. Select which users to allow, and then click Next.

10. Review the settings in the summary screen, and then click Finish to complete the wizard.

11. After the wizard is complete, click Apply to make the configuration changes effective.
    

    Note

       You must complete the same procedure to allow traffic from the AstaroNet subnet to the ISANet subnet. Routing rules (which you created earlier in this document) are mirrored, but access rules are "one-way."

You have now created a remote site network, a network rule, and an access rule. Now that ISA Server is configured, you will configure the Astaro Security Linux system.

Configuring Astaro Security Linux

Configuring Astaro is a straightforward process, comprised of the following tasks:

1. Define the network behind ISA Server (ISANet) and the external interface of ISA Server.
2. Define a preshared key that you will use to authenticate the peers.
3. Define an IPSec policy that specifies encryption or authentication algorithms and lifetimes.
4. Create an IPSec connection that brings the preceding tasks into a single group.
5. Create rules to allow traffic from ISANet.

Create Network Definitions for ISANet and ISA Server

To define ISANet and ISA Server, perform the following steps.

1. Access the Astaro configuration Web site by entering https://172.25.1.254 and you will see the following screen. Type the appropriate credentials.
   
2. Click the Definitions link.
   
3. Click Networks.
   
4. From the Definitions page, select Networks.
   
5. To define the network behind ISA Server, in Name, type ISANet, in IP address, type 172.25.25.0, and in Subnet mask, type 255.255.255.0 or type 24. Then click Add.
   The ISANet definition will be added to the list of network definitions. You have now defined the network behind ISA Server (ISANet). The next step is to define the external interface of ISA Server.
6. In IP address, type the ISA Server external address 192.168.100.25, and in Subnet mask, type 255.255.255.255 or 32. Then click Add to have the ISA Server definition moved to the network definitions list.
   

You have now defined ISA Server and ISANet.

Define a Remote Key

To define a preshared key that you will use to authenticate the peers, perform the following steps.

1. Click the IPSec VPN link and you will see the following screen.
   
2. Click Remote Keys.
   
3. In Name, type Pre-Shared Key for ISA, change Auto packet filter to On, and leave the Virtual IP box empty. In Key Type, select PSK. After you select PSK, the Preshared Key box will appear. Enter the preshared key 123456789, and then select Add.
   

You have now defined a preshared key.

Define IPSec Policy

To define an IPSec policy that specifies encryption or authentication algorithms and lifetimes, perform the following steps.

1. From the IPSec VPN menu on the left, click Policies.
   
2. Select New in the upper-right corner.
   
3. In the boxes provided, enter the following information.
   In Name, type ISANet to AstaroNet Policy.
   In ISAKMP(IKE) Settings, enter the following information.
   • In IKE mode, select Main Mode.
   • In Encryption Algorithm, select 3DES-CBC.
   • In Authentication Algorithm, select MD5 160bit.
   • In IKE DH Group, select DH Group 2 (MODP1024).
   • In SA lifetime (secs), type 28800.
     In IPSec Settings, enter the following information.
   • In IPSec mode, select Tunnel.
   • In IPSec protocol, select ESP.
   • In Encryption Algorithm, select 3DES.
   • In Enforce Algorithms, select Off.
   • In Authentication Algorithm, select MD5 160bit.
   • In SA lifetime (secs), type 3600.
   • In PFS, select PFS Group 2 (MODP1024).
   • In Compression, select Off.
     

You have now defined IPSec policy.

Now that you have defined the network behind ISA Server (ISANet), the ISA Server external interface, the preshared key, and the IPSec policy to use, the next step is to gather these individual components into an IPSec connection.

Define IPSec Connection

To define an IPSec connection, perform the following steps.

1. From the IPSec VPN menu on the left, select Connections.
   
2. The following page will change dynamically as you enter information in the boxes. In the Name box, type ISANet to AstaroNet.
   
3. In Type, select Standard.
   
4. In IPSec Policy, select the policy you created earlier (ISANet to AstaroNet Policy).
   
5. In Local Endpoint, select External.
   
6. In Remote Endpoint, select the ISA Server network definition that you created earlier.
   
7. In the Local Subnet, select Internal_Network__.
   
8. In Remote Subnet, select ISANet.
   
9. On the Authentication of remote station(s) screen, in Key, select Pre-Shared Key for ISA.
   
10. Select Add, and the following screen will appear. The connection you just created is highlighted with a red indicator. Select the option button to the left of the red button to make the connection active (as indicated by the arrow).
    The connection is now active as shown by the green option button.
    

You have now defined an IPSec connection.

Create Rules to Allow Traffic from ISANet

To create rules to allow traffic from ISANet, perform the following steps.

1. From the Packet Filter menu, select Rules.
   The following screen appears.
   You will add a simple rule to test connectivity, only allowing PING search requests from ISANet (ICMP Echo Request). In the following screen shot, the client 172.25.25.10, who is behind ISA Server, is using the PING protocol to search for a client Astaro 172.25.1.10. (For more information about this scenario, see Figure 4 earlier in this document.) There currently is no rule to allow ISANet traffic into the Internal network. To make the PING search constant, append -t to the end of the command.
   
2. On the Add Rule page, in the From (Client) box, select ISANet.
   
3. In the To (Server) box, select Internal_Network_.
   
4. In the Service box, select ping-request.
   
5. Make sure that Action is set to Allow, and then click Add.
   
6. The rule will show deactivated (red). Click the option button to the left of the red button to make it active (green).
   

Now that the rule is active, you can verify that the client is receiving replies.

Testing the ISA Server Tunnel Mode Policy

To test the ISA Server tunnel mode policy, perform the following steps.

1. Open the IPSec Monitor snap-in. Click Start, click Run, type mmc, and then click OK. Expand Console Root to view the Main Mode and Quick Mode security associations. Under Main Mode, click Security Associations.
   
2. Under Quick Mode, click Security Associations.
   

The testing process uses different application layer and transport layer protocols to ensure that data is encrypted and decrypted correctly when it passes through the IPSec tunnel. The following data transfer tests can be used to determine the success of the IPSec tunnel mode policy:

• FTP Transfer
  The FTP process uses an FTP GET of a single 100 megabyte (MB) file, renames the file, and then uses an FTP PUT to transfer the new file back to the FTP server. After the two transfers are completed, a comparison is performed, using Windiff.exe from the Windows 2000 Server Resource Kit, at the FTP server to ensure the two files are identical. Both the command-line FTP.exe utility (used for Active Mode connections) and Internet Explorer (used for Passive Mode) are used as the client application. An FTP server running Windows Server 2003 is on the network behind the Check Point NG system.
• TFTP Transfer
  The TFTP copy process replicates the FTP tests, with the only difference being that a 20 MB file is transferred rather than the 100 MB file transferred using FTP. Because Windows Server 2003, Windows XP, and Windows 2000 Server do not include a TFTP server, a third-party TFTP server (SolarWinds TFTP Server https://www.solarwinds.com) is used as a TFTP server for the tests. A Windows XP host is the client using the command-line utility TFTP.exe.
• CIFS Transfer
  The CIFS copy process transfers a folder structure with three subfolders containing a total of 311 files approximately 50 MB in size between the two computers. The data is transferred from the source computer to the target computer using the Resource Kit utility ROBOCOPY.exe and by copying within Windows Explorer. The files are then copied from the target computer to the source computer into a different folder structure. The folders are then compared using Windiff.exe from the Windows 2000 Resource Kit to ensure that the data is not corrupted during transmission.
• PING with specific sizes
  PING packets are sent from the target to the source computer using specific packet sizes to test packet fragmentation and reassembly through the IPSec tunnel. Specifically, packets sizes of: 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 40, 80, 160, 320, 640, 1280, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1500, 3000, 6000, 12000, 24000, 48000, and 65500 bytes.