The next step is to reconcile the IPSec tunnel mode policy. This involves changing one setting on the ISA Server computer. To reconcile the IPSec tunnel mode policy, perform the following steps.
-
Select the Virtual Private Networks (VPN) node in the ISA Server console, and then click the Remote Sites tab. Select SmoothwallNet, and then on the Tasks tab, click Configure Remote Site.
-
Verify that Enable the VPN site-to-site connection is selected.
-
Click the Connection tab, and then click IPSec Settings.
The Phase I settings are displayed.
-
Click the Phase II tab.
-
Under Generate a new key every, change the seconds to 28800, to match the SmoothWall Express settings.
This setting is not visible in the SmoothWall Express Web interface. Examining the Oakley logs on the computer running Windows Server 2003 shows that the SmoothWall Express system sends a Session Key Lifetime setting that is different from the setting in the ISA Server wizard. The following is an explanation of an excerpt from the Oakley log, showing the Quick Mode failure.
-
Line 1 is the Incoming Quick Mode offer from the SmoothWall system with the Initiator and Responder cookies set in lines 3 and 4.
-
Line 12 is Proposal 0, which specifies the use of ESP. (Proposals can contain many different transforms that are combinations of encryption algorithms DES or 3DES, hashing algorithms MD5 or SHA1, Diffie-Hellman settings, and IP addresses pertinent to the IPSec tunnel mode policy.)
-
Lines 13 through 18 contain the settings for Transform 0 and have the following settings:
-
3DES (line 13)
-
Perfect Forward Secrecy (PFS) is group 2, which is the Diffie-Hellman group (line 14)
-
Tunnel mode is specified (line 15)
-
Session key lifetime is 28800 seconds (lines 16 and 17)
-
MD5 (line 18)
-
Lines 19 through 24 contain settings for Transform 1, which has all the same settings except for the hashing algorithm that is specified as SHA1 (line 24).
The following is an excerpt from the Oakley log, showing the Quick Mode failure:
12-09: 22:36:20:818:fec Receive: (get) SA = 0x0137ec60 from 192.168.55.1.500
12-09: 22:36:20:818:fec ISAKMP Header: (V1.0), len = 316
12-09: 22:36:20:818:fec I-COOKIE 5e20729eafbcf84d
12-09: 22:36:20:818:fec R-COOKIE d02a42ac5868c79c
12-09: 22:36:20:818:fec exchange: Oakley Quick Mode
12-09: 22:36:20:818:fec flags: 1 ( encrypted )
12-09: 22:36:20:818:fec next payload: HASH
12-09: 22:36:20:818:fec message ID: d4629cf2
12-09: 22:36:20:858:fec Negotiated Proxy ID: Src 172.25.3.0.0 Dst 172.25.10.0.0
12-09: 22:36:20:858:fec Src id for subnet. Mask 255.255.255.0
12-09: 22:36:20:858:fec Dst id for subnet. Mask 255.255.255.0
12-09: 22:36:20:858:fec Checking Proposal 0: Proto= ESP(3), num trans=2 Next=0
12-09: 22:36:20:858:fec Checking Transform # 0: ID=Triple DES CBC(3)
12-09: 22:36:20:858:fec group description for PFS is 2
12-09: 22:36:20:858:fec tunnel mode is Tunnel Mode(1)
12-09: 22:36:20:858:fec SA life type in seconds
12-09: 22:36:20:858:fec SA life duration 28800
12-09: 22:36:20:858:fec HMAC algorithm is MD5(1)
12-09: 22:36:20:858:fec Checking Transform # 1: ID=Triple DES CBC(3)
12-09: 22:36:20:858:fec group description for PFS is 2
12-09: 22:36:20:858:fec tunnel mode is Tunnel Mode(1)
12-09: 22:36:20:858:fec SA life type in seconds
12-09: 22:36:20:858:fec SA life duration 28800
12-09: 22:36:20:858:fec HMAC algorithm is SHA(2)
12-09: 22:36:20:858:fec Finding Responder Policy for SRC=172.25.3.0.0000 DST=172.25.10.0.0000, SRCMask=255.255.255.0, DSTMask=255.255.255.0, Prot=0 InTunnelEndpt 6437a8c0 OutTunnelEndpt 137a8c0
12-09: 22:36:20:868:fec Failed to get TunnelPolicy 13015
12-09: 22:36:20:868:fec Responder failed to match filter(Phase II) 13015
12-09: 22:36:20:908:fec Data Protection Mode (Quick Mode)
12-09: 22:36:20:908:fec Source IP Address 172.25.10.0 Source IP Address Mask 255.255.255.0 Destination IP Address 172.25.3.0 Destination IP Address Mask 255.255.255.0 Protocol 0 Source Port 0 Destination Port 0 IKE Local Addr 192.168.55.100 IKE Peer Addr 192.168.55.1 IKE Source Port 500 IKE Destination Port 500 Peer Private Addr
12-09: 22:36:20:908:fec Preshared key ID. Peer IP Address: 192.168.55.1
12-09: 22:36:20:908:fec Me
12-09: 22:36:20:908:fec No policy configured
12-09: 22:36:20:908:fec ISAKMP Header: (V1.0), len = 68
12-09: 22:36:20:908:fec I-COOKIE 5e20729eafbcf84d
12-09: 22:36:20:918:fec R-COOKIE d02a42ac5868c79c
12-09: 22:36:20:918:fec exchange: ISAKMP Informational Exchange
12-09: 22:36:20:918:fec flags: 1 ( encrypted )
12-09: 22:36:20:918:fec next payload: HASH
12-09: 22:36:20:918:fec message ID: ab623176
12-09: 22:36:20:918:fec Ports S:f401 D:f401
If you compare these settings to the Phase I and Phase II settings on the ISA Server IPSec policy, the session key for Phase II is the only mismatch that needs to change, as shown previously.
After you make this change, you can add the IPSec Monitor snap-in and view the settings. Perform the following steps.
-
On the computer running Windows Server 2003, click Start, click Run, type mmc, and then click OK.
-
On the File menu, click Add/Remove Snap-in, and add the IPSec Monitor snap-in.
-
Expand Console Root to view the Main Mode and Quick Mode security associations. Under Main Mode, click Security Associations. You should see the following screen, which details the Main Mode (Phase I) security association.
-
Under Quick Mode, click Security Associations. You should see the following screen, which details the Quick Mode (Phase II) security association.
Note: |
|---|
|
Quick Mode (Phase II) actually has two security associations—Inbound and Outbound, but the IPSec Monitor only shows the Outbound security association. Clients from behind each system should be able to access the remote site through the IPSec tunnel mode policy. (If not, you will need to consider routing tables on the clients.)
|