Configuring ISA Server 2004

This section contains an overview of configuring ISA Server preshared secrets, a checklist for configuring preshared secrets, walk-throughs with procedures for configuring preshared secrets and certificates, and troubleshooting information.

ISA Server Configuration: Preshared Secret Overview

The following IPSec settings will be used in this section of this configuration document.

• Phase I
  • Main mode
  • 3DES
  • SHA-1
  • MODP Group 2 (1024 bits) for DH
  • SA lifetime of 28,800 seconds
  • Preshared Secret Authentication
• Phase II
  • 3DES
  • SHA-1
  • PFS & MODP Group 2 (1024 bits) for DH
  • SA lifetime of 3600 seconds
  • ESP tunnel mode

ISA Server Configuration: Preshared Secret Checklist

Use the following checklist for preshared secrets.

For installation and configuration information and documentation, refer to the documents found on the Microsoft website (www.microsoft.com).

ISA Server Configuration Walk-through Procedure 1: Configuring the Preshared Secret Solution

This topic describes in detail the process to configure the ISA Server computer to successfully establish a site-to-site IPSec tunnel using a third-party gateway with the settings specified in ISA Server Configuration: Preshared Secret Overview. This section includes tips that can be used to improve the functionality of the IPSec tunnel, performance of the device, or the security of the device.

Configure VPN Site-to-Site Network

Use the following steps to configure a VPN site-to-site network.

1. Launch the ISA Server Management Console from the Start menu.
2. In the ISA Server Management Console, right-click Networks from the left menu and select New Network to start the New Network Wizard. Provide the name Internal_BL for the network, and then click Next.
   Select VPN Site-to-Site Network, and then click Next.
   
   1. Select IP Security protocol (IPSec) tunnel mode, and then click Next.
      
   2. On the Connection settings page, enter 22.23.24.2 as the remote gateway’s external IP address and select 14.15.16.17 (External) as the Local VPN gateway IP address from the drop-down list. Click Next.
      
   3. On the IPSec Authentication page, select Use pre-shared key for authentication as the authentication and enter Cool-Dude! as the preshared key, and then click Next.
   4. On the Network Addresses page, click Add.
      
   5. Enter 172.23.9.0 as the Start Address of the third-party gateway’s protected network.
   6. Enter 172.23.9.255 as the End Address of the third-party gateway’s protected network.
      
   7. Review the configuration of the VPN Site-to-Site Network and click Finish to create the VPN Site-to-Site Network.
      

The new VPN Site-to-Site Network will appear under the Network tab in the Network screen.

Modify IPSec Settings

Use the following steps to modify the IPSec settings.

1. In the ISA Server Management Console, select Virtual Private Networks (VPN) from the left menu, select Remote Sites, and double-click Internal_BL to display the properties page for the Remote Site VPN.

   Note

   This VPN entry was created at the time the VPN Site-to-Site Network was created. You cannot create a Remote Site VPN using the Virtual Private Networks screen. These can only be created through the New Network Wizard.

2. In the Internal_BL Properties window, select the Connection tab to display the VPN site properties pertaining to the remote gateway, the local gateway, and the IPSec settings. Click IPSec Settings.
   

3. Modify the Phase I Settings:

   • Change the Encryption algorithm to 3DES.
   • Change the Integrity algorithm to SH1.
   • Change the Diffie-Hellman group to Group 2 (1024 bit).
   • Enter 28800 as the number of seconds under Authenticate and generate a new key every.
     

4. Select the Phase II tab to modify the Phase II settings:

   • Change the Encryption algorithm to 3DES.
   • Change the Integrity algorithm to SH1.
   • Select Generate a new key every.
   • Enter 3600 in the space provided for the number of seconds under Generate a new key every.
   • Select Use Perfect Forward Secrecy (PFS).
   • Change the Diffie-Hellman group to Group 2 (1024 bit).
     

Configure Access Rule

Use the following steps to configure an access rule.

1. In the ISA Server Management Console, right-click Firewall Policy from the left menu and select New Access Rule to start the New Access Rule Wizard. Provide a name for the rule, such as Site-to-site access, and then click Next.

2. On the Rule Action page, select Allow, and then click Next.
   

3. On the Protocols page, select All outbound traffic, and then click Next.

   Note

   In this example All outbound traffic is selected, and the access rule will apply to all protocols. For a more secure configuration, an alternate method should be selected that will either define the protocols and ports the access rule will apply to or will allow all protocols and ports expect for the ones you specifically deny in the access rule.

   

4. On the Access Rule Sources page, click Add to open the Add Network Entities dialog box.
   

5. Under Networks select Internal_BL as the source network, and click Add to add the network.
   

6. The selected networks are displayed. Click Next.
   

7. Under New Access Rule Wizard, on the Access Rule Destination page, click Add to open the Add Network Entities dialog box.
   

8. Under Networks select Internal as the destination network, and click Add to add the network.
   

9. The selected networks are displayed. Click Next.
   

10. On the User Sets page, leave the default All Users, and then click Next.
    

11. Review the configuration of the access rule and click Finish to create the access rule.
    

12. Repeat the procedure to create an access rule with the following settings:

    • Action: Allow
    • Traffic: All Outbound Protocols
    • Source: Internal
    • Destination: Internal_BL

The new access rules will appear under the Firewall Policy screen.

Configure Network Rule

A network rule that defines a route for the communication between the Internal network and the Internal_BL network is required to ensure this communication does not use network address translation (NAT), which will cause the VPN to fail.

1. In the ISA Server Management Console, right-click Networks from the left menu and select New Network Rule to start the New Network Rule Wizard.
2. Enter Site-to-Site VPN Route as the Network Rule name, and then click Next.
   
3. On the Network Traffic Sources page, click Add to open the Add Network Entities dialog box.
4. Under Networks, select Internal_BL as the source network.
5. Select Add to add the selected network.
   
6. On the Network Traffic Destinations page, click Add to open the Add Network Entities dialog box.
7. Under Networks, select Internal as the destination network.
8. Select Add to add the selected network.
9. On the Network Relationship page, select Route.
10. Review the configuration of the network rule and click Finish to create the network rule.
    

The newly created network rules will appear under the Network Rules tab in the Network screen. Right-click the new network rule and select Move up. Continue until the created network rule is above all network rules that have NAT as the network relation.

1. In the ISA Server details pane, click Apply to apply the changes you made.

   Note

   ISA Server processes the network rules in order and will stop processing after an attempted communication has satisfied all the defined parameters.

2. Test the IPSec tunnel after the third-party gateway peer has been configured by sending icmp traffic to the remote internal network through the IPSec tunnel using the ping utility.

ISA Server: Configuring the Certificate Solution

The following IPSec settings will be used in this section of this configuration document:

• Phase I
  • Main mode
  • 3DES
  • SHA-1
  • MODP Group 2 (1024 bits) for DH
  • SA lifetime of 28,800 seconds
  • Certificate Authentication
• Phase II
  • 3DES
  • SHA-1
  • PFS & MODP Group 2 (1024 bits) for DH
  • SA lifetime of 3600 seconds
  • ESP tunnel mode

ISA Server: Certificate Checklist

Use the following checklist for certificates.

ISA Server Configuration Walk-through Procedure 2: Configuring the Certificate Solution

This topic describes in detail the process to configure the ISA Server computer to successfully establish a site-to-site IPSec tunnel with third-party gateways using the settings specified in ISA Server: Configuring the Certificate Solution. This section includes tips that can be used to improve the functionality of the IPSec tunnel, performance of the device, or the security of the device.

Install Certificates

Use the following steps to install certificates.

1. Create an identity certificate from a certification authority (CA).
2. Copy the CA’s certificate revocation list (CRL), and the identity certificate from the Certificate Authority to the local machine.
3. After all the certificates have been imported:
   • Launch the Microsoft Management Console (MMC).
   • Select Add/Remove Snap-in to add the certificate snap-in.
     
4. Select Computer account.
   
5. Select Local computer to manage.
   
6. Import the CA, CRL, and the identity certificate using the Microsoft Certificate Import Wizard by right-clicking the following certificate stores and selecting Import:
   • Trusted root Certification Authorities for the CA
   • Trusted Root Certification Authorities for the CRL
   • Personal for the identity certificate

Configure VPN Site-to-Site Network

Use the following steps to configure a VPN site-to-site network.

1. Launch the ISA Server Management Console from the Start menu.
2. In the ISA Server Management Console, right-click Networks from the left menu and select New Network.
3. The New Network Wizard will be launched. Enter Internal_BL as the Network name.
   
4. Select VPN Site-To-Site Network.
   
5. Select IP Security protocol (IPSec) tunnel mode.
   
6. In the New Network Wizard, on the Connection Settings page:
   • Enter 22.23.24.2 as the Remote VPN gateway IP address.
   • Select 14.15.16.17 (External) as the Local VPN gateway IP address from the drop-down list.
     
7. In the New Network Wizard, on the IPSec Authentication page:
   • Select Use a certificate from this certificate authority (CA).
   • Select Browse.
     
8. Select the appropriate certificate for authentication. For this configuration document, the certificate issued to Testlab will be used.
   
9. Under the New Network Wizard, on the Network Addresses page, select Add.
   
10. Enter 172.23.9.0 as the Starting address of third-party gateway’s protected network.
11. Enter 172.23.9.255 as the Ending address of the third-party gateway’s protected network.
    
12. Review the configuration of the VPN site-to-site network and select Finish to create the VPN site-to-site network.
    

The created VPN site-to-site network will appear under the Network tab in the Network screen.

Modify IPSec Settings

Use the following steps to modify IPSec settings.

1. In the ISA Server Management Console, select Virtual Private Networks (VPN) from the left menu of the management console. In the details pane, select Remote Sites.

2. Double-click Internal_BL to display the properties page for the remote site VPN.

   Note

   This VPN entry was created at the time the VPN site-to-site network was created. You cannot create a remote site VPN using the Virtual Private Networks screen. This can only be created through the New Network Wizard.

3. In the Internal_BL Properties window, select the Connection tab to display the VPN site properties pertaining to the remote gateway, the local gateway, and the IPSec settings. Select IPSec Settings.
   

4. Modify the Phase I settings:

   • Change the Encryption algorithm to 3DES.
   • Change the Integrity algorithm to SH1.
   • Change the Diffie-Hellman group to Group 2 (1024 bit).
   • Enter 28800 as the number of seconds under Authenticate and generate a new key every.
     

5. Select the Phase II tab to modify the Phase II settings:

   • Change the Encryption algorithm to 3DES.
   • Change the Integrity algorithm to SH1.
   • Select Generate a new key every.
   • Enter 3600 in the space provided for the number of seconds under Generate a new key every.
   • Select Use Perfect Forward Secrecy (PFS).
   • Change the Diffie-Hellman group to Group 2 (1024 bit).
     

Configure Access Rule

Use the following steps to configure an access rule.

1. In the ISA Server Management Console, right-click Firewall Policy from the left menu and select New Access Rule.

2. In the New Access Rule Wizard, on the Rule Action page, select Allow.
   

3. Under New Access Rule Wizard, on the Protocols page,

   • Select the method that the rule will apply to individual protocols from the drop-down list.

   Note

   In this configuration example, All outbound protocols will be selected. The access rule will apply to all protocols. For a more secure configuration, an alternate method should be selected that will either define the protocols and ports the access rule will apply to or will define which protocols and ports the access rule will not apply to, by specifying exceptions.

   If another method is chosen other than All outbound protocols, select Add to select the individual protocol and ports.

4. Under New Access Rule Wizard, on the Access Rule Sources page, select Add to open the Add Network Entities dialog box.
   

5. Under Networks select Internal_BL as the source network.

6. Select Add to add the network.
   All selected networks will be displayed.
   

7. Under New Access Rule Wizard, on the Access Rule Destination page, select Add to open the Add Network Entities dialog box**.**
   

8. Under Networks select Internal as the destination network.

9. Select Add to add the network.
   All selected networks will be displayed.
   

10. On the User Sets page, you can select what this access rule will apply to. Leave the default as All Users.
    

11. Review the configuration of the access rule and select Finish to create the access rule.
    

12. Repeat the steps to create an access rule with the following settings:

    • Action: Allow
    • Traffic: All Outbound Protocols
    • Source: Internal
    • Destination: Internal_BL

The created access rules will appear under the Firewall Policy screen.

Configure Network Rule

A network rule that defines a route for the communication between the Internal network and the Internal_BL network is required to ensure this communication does not use network address translation (NAT),which will cause the VPN to fail. Use the following steps to configure a network rule.

1. In the ISA Server Management Console, right-click Networks from the left menu and select New Network Rule.

2. The New Network Rule Wizard will be launched. Enter Site-to-Site VPN Route as the Network Rule name.
   

3. Under New Network Rule Wizard, on the Network Traffic Source page, select Add to open the Add Network Entities dialog box.
   

4. Under Networks select Internal_BL as the source network.

5. Select Add to add the selected network.
   All selected networks will be displayed.
   

6. Under New Network Rule Wizard, on the Network Traffic Destination page, repeat step 4, selecting Internal.
   

7. Under New Access Rule Wizard, on the Network Rules page, select Route.
   

8. Review the configuration of the network rule and select Finish to create the network rule.
   

9. The newly created network rules will appear under the Network Rules tab in the Networks screen. Right-click the created network rule and select Move up.

   Note

   ISA Server processes the access rules in order and will stop processing after an attempted communication has satisfied all the defined parameters.

10. Continue until the created network rule is above all network rules that have NAT as the network relation.

11. Test the IPSec tunnel after the third-party gateway peer has been configured by sending icmp traffic to the remote internal network through the IPSec tunnel using the ping utility.

Troubleshooting ISA Server

The following section contains troubleshooting tips. For additional troubleshooting information, refer to the Microsoft Knowledge Base articles on the Microsoft website (www.microsoft.com).

Configuration

Review the configuration for accuracy:

• Local IP settings
• Remote IP settings
• IPSec Phase 1 settings
• IPSec Phase 2 settings

The IPSec properties may be viewed by right-clicking the Remote VPN definition and selecting IPSec summary.

Logs

Review the log files for any errors:

• The default ISA Server logs are accessed through the ISA Server Management Console by selecting Monitoring.
• IKE Debugging can be enabled by modifying the following registry key:

(HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLogging (DWORD) ==1)

or by using the command line:netsh ipsec

Modifying these will create the log file: %systemroot5\debug\Oakley.log

• IPSec information can also be found using the ipsecmon snap-in for the Microsoft Management Console (MMC).