Walk-through Part 3: Reconciling the IPSec Policies on ISA Server and PIX

  • Article

There are a few modifications that need to be made in the ISA Server and PIX IPSec tunnel mode configuration. The ISA Server wizard defaults to different rekey values than the PIX VPN Wizard. In this scenario, you will make both sides match for ease of setup.

On the PIX, you need to change the following items to match the ISA Server configuration:

  1. In Mode (IKE)
    Change IKE SA Lifetime to 28800 seconds (from the VPN Wizard value of 86400).
  2. Quick Mode (IPSec)
    Change IPSec SA Lifetime to 100000 Kbytes and 1 Hour.
    Enable Perfect Forward Secrecy.

Complete these tasks by performing the following steps.

  1. In the PDM, on the toolbar, select the Configuration button, and then select the VPN tab. Under Categories, expand IPSec, and then select Tunnel Policy. Find the policy that was just created, select it, and then click Edit (or double-click the policy).
    You should see the following dialog box. The PIX VPN Wizard defaults to 4608000 kilobytes (KB), 8 hours for the Quick Mode security association lifetime, and Perfect Forwarding Secrecy not enabled.
    The ISA Server wizard defaults to rekeying every 100000 seconds and enabling Perfect Forward Secrecy (PFS) without enabling rekey based on the amount of traffic.
  2. Change the data and time lifetimes to 100000 KB and 1 hour respectively. Select Enable Perfect Forwarding Secrecy and Diffie-Hellman Group 2. Click OK.
  3. Select IKE, select Policies, select the policy just created, and then select Edit to display the Edit IKE Policy page.
  4. Change the IKE Lifetime from 86400 seconds to 28800 seconds, and then select OK.

Now that you have configured the PIX, perform the following steps to change one setting on ISA Server.

  1. Select the Virtual Private Networks (VPN) node in the ISA Server console, and then click the Remote Sites tab. Select the PIXNet remote site object, and then on the Tasks tab, click Configure Remote Site.
  2. Click the Connection tab, and then click IPSec Settings. Phase I settings are displayed.
  3. Click the Phase II tab. Select Generate a new key every, and in seconds, type 100000.

You should be able to PING from a client behind ISA Server and have the IPSec security associations established. If you want to PING from ISA Server and have the IPSec security associations established, you must add an IPSec rule into the PIX. You specify the external IP address of ISA Server as part of the tunnel mode configuration. Currently, only the subnet behind ISA Server is defined. When ISA Server PINGs the PIXNet, it will source from its external address, which is not defined in the PIX.

To do this, perform the following steps.

  1. In the PDM, on the toolbar, select the Configuration button, and then select the VPN tab. Expand IPSec, select IPSec Rules, right-click in an empty area, and then click Add.
  2. In IP address, type 172.25.3.0 as the local network and under Remote Side Host/Network, enter the ISA Server external address as shown in the following screen.

After this is added into the IPSec rules, traffic initiated from ISA Server destined for the 172.25.3.0 subnet will be included in the tunnel mode configuration.

This completes the procedure for configuring ISA Server and the Cisco PIX501 to use preshared key authentication for IPSec tunnel mode.

You should be able to send and receive traffic from both sides of the IPSec tunnel mode policy. There will be a brief delay in communications as the security associations come online, but this will only last a few moments.

If you encounter problems with communications with the configuration, refer to the following Microsoft Knowledge Base (KB) articles to aid in troubleshooting:

  • 259335 Basic L2TP/IPSec Troubleshooting in Windows (https://go.microsoft.com/fwlink/?LinkId=32058)
  • 314831 Basic L2TP/IPSec Troubleshooting in Windows XP (https://go.microsoft.com/fwlink/?LinkId=32059)
  • A helpful technique in troubleshooting IPSec is determining which mode of IPSec negotiations is failing—either Main Mode or Quick Mode. Using the information in the preceding KB articles, enable auditing so that IKE events are written to the security log, to determine which mode is failing.

If Main Mode fails, it can be mismatched preshared keys, mismatched Main Mode lifetimes, or mismatched Diffie-Hellman Group settings.

If Quick Mode fails, it can be an invalid network filter list (or ACL) specified on one side or mismatched Diffie-Hellman settings, Quick Mode security association lifetimes, or encryption settings