Troubleshooting Network Load Balancing in ISA Server 2004 Enterprise Edition

This troubleshooting guide describes common issues encountered when using Network Load Balancing (NLB) for Microsoft Internet Security and Acceleration (ISA) Server 2004 Enterprise Edition. It also details actions you can take to avoid or resolve these issues.

This document will be updated frequently with new troubleshooting tips. If you have a suggestion for a troubleshooting tip, contact ISA Server Documentation Feedback at Microsoft.

NLB configuration is described in Network Load Balancing in ISA Server 2004 Enterprise Edition at the Microsoft TechNet Web site.

How to Use This Document

NLB Issues

NLB Configuration Fails

Appendix A: Using RemoveAllNLBSettings.cmd

Additional Information

How to Use This Document

Follow this approach to use this document:

1. Review background information about NLB in ISA Server before beginning troubleshooting. For details, see the document Network Load Balancing in ISA Server 2004 Enterprise Edition at the Microsoft TechNet Web site.
2. Navigate down the list of issues. If any issue matches the problem you are seeing, read about the possible causes, which include corrective action where possible.
3. If you cannot find a matching issue, try doing a text search in this document for a relevant keyword or error text. For example, you can search for "connectivity" or "switch" or other terms.
4. If still unsuccessful, sequentially scan all of the items in the NLB Issues sections. If relevant information is not found, pose your troubleshooting question to the Microsoft online community newsgroup microsoft.public.isaserver.

NLB Issues

This section describes specific NLB issues, causes, and solutions. For some issues, examples are provided.

NLB Configuration Fails

Issue: NLB configuration fails with this error message:

The network adapter specified for Network Load Balancing (NLB) for network '%1' may be used for intra-array communication. Since there is no suitable network adapter available, NLB cannot be configured properly and will be stopped on this server. An additional network adapter is required for NLB.

Cause: Intra-array communication is not properly configured.

Solution: When you use ISA Server integrated NLB on Microsoft Windows Server 2003 without Service Pack 1 installed, each computer running ISA Server services requires an additional network adapter, for intra-array communication. This is because Windows Server 2003 does not allow NLB cluster members to communicate with each others' dedicated Internet Protocol (IP) addresses over the NLB-enabled adapters. The requirement of an additional network adapter does not exist if you have installed Windows Server 2003 Service Pack 1.

We recommend that these network adapters be physically connected to each other (for example, through a single switch), and not to other network segments, to ensure that they receive only intra-array communication. You should then configure intra-array communication to use the IP address of the new adapter on each server. The network adapters used for intra-array communication should not be NLB-enabled.

Valid Traffic Fails Without Deny Entries in ISA Server Logs

Issue: Valid traffic fails without deny entries in ISA Server logs, and only one server gets all of the traffic destined for the array's virtual IP address. You can diagnose this condition by checking the number of connections each server is handling (on the Sessions tab of the ISA Server Monitoring node), to see if the distribution is imbalanced.

Cause: The Layer 2 Network switch has learned the virtual media access control (MAC) address of the cluster. This means that it has associated the address with one port on the switch, and will send all of the traffic to that one port, so only one array member will receive the traffic.

Solution: Reset the network switch.

How to avoid this issue: When you are adding a computer to the ISA Server array, attach it to a dummy switch, configure NLB on the computer, and then connect it to the true switch.

Bidirectional Affinity Settings Remain After NLB Is Disabled

Issue: If you use ISA Server with integrated NLB mode, and then stop using integrated mode or uninstall ISA Server, the bidirectional affinity setting for NLB remains in place. This can interfere with future NLB configurations. Evidence of this issue can be seen in the system log events. For examples, see Bidirectional Affinity Conflict Scenarios.

Cause: ISA Server does not remove the bidirectional affinity setting from the registry when you turn off integrated mode.

Solution: There are three possible solutions:

• Run the RemoveAllNLBSettings.cmd script provided at the Microsoft Download Center. Note that the script will clear the setting only from active NLB instances (network adapters on which NLB is enabled). If you enable NLB on an additional adapter, you may have to run the script again. Instructions for using the script are provided in Appendix A: Using RemoveAllNLBSettings.cmd.
• Delete the bidirectional affinity setting from the registry on each array member, for each affected network adapter. This setting is located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WLBS\Parameters\Interface\<Interface GUID>\BDATeaming, where <Interface GUID> is the GUID of the affected network adapter. After you delete the BDATeaming entries, either run wlbs reload at a command prompt, or restart the computer.
• Disable and re-enable each network adapter on which integrated NLB was enabled.

Bidirectional Affinity Conflict Scenarios

Example 1: Adding an array member after discontinuing use of integrated NLB

If you stop using integrated NLB, and then add an array member and configure it to use NLB (without integration), the new array member will not necessarily be configured with bidirectional affinity, resulting in a conflict with the other array members' NLB configuration. The new array member will not receive any of the load balanced traffic.

Example 2: Discontinuing integrated NLB, and then disabling NLB on one network

If you have integrated NLB operating on more than one network, one of the networks will serve as the bidirectional affinity master. If you discontinue use of integrated NLB, the bidirectional affinity setting remains, and the master continues to function. If you disable NLB on one of the networks, and it happens to be the bidirectional affinity master, NLB will fail on the other networks.

Cannot Change Friendly Name of Network Adapter

Issue: When you change the friendly name of an NLB-enabled network adapter, the name change does not take effect.

Solution: Restart the Microsoft Firewall service.

MAC Address Assignment Fails

Issue: The NLB Configuration has failed because the Media access control (MAC) address assignment may have failed. When you enable ISA Server integrated NLB for a network, ISA Server assigns the same MAC address to each ISA Server array network adapter connected to that network. This allows communication to the virtual IP address, which is associated with the newly assigned MAC address. If the MAC address assignment fails, the virtual IP address will not be recognized and communication to those network adapters will fail.

Solution: Disable and re-enable ISA Server integrated NLB.

Issue: If you have NLB configured without ISA Server integrated mode, and then enable integrated mode, the configuration required for integrated mode may not be properly established.

Solution: Reset the Windows NLB configuration so that the ISA Server integrated mode settings can be applied. This can be done using the RemoveAllNLBSettings.cmd script, or manually.

RemoveAllNLBSettings.cmd

Follow the procedures in Appendix A : Using RemoveAllNLBSettings.cmd.

Manual Reset

To manually reset the Windows NLB configuration, follow these steps:

1. If ISA Server is installed, and ISA Server NLB integration is enabled, stop the Firewall service. Note that this will break any active connections through the firewall.
2. Unbind NLB from all network adapters:
   1. Open the properties for each network connection.
   2. Clear the Network Load Balancing check box.
   3. Double-click Internet Protocol (TCP/IP) properties, and on the General tab, click Advanced.
   4. In the Advanced TCP/IP Settings dialog box, remove the virtual IP addresses from the IP addresses list, and then click OK.
   5. Click OK to close the TCP/IP Properties page, and then click OK to close the Connection Properties page.
3. Delete existing Windows NLB registry keys from the registry:

1. Open Registry Editor: click Start, click Run, type regedit, and then click OK.
2. Delete all subkeys of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Wlbs\Parameters\Adapters ("Adapters" node should be left with no subkeys.)
3. Delete all subkeys of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Wlbs\Parameters\ Interface ("Interface" node should be left with no subkeys.)

1. If you stopped the Firewall service, restart it.

Secondary Connections or Repeated Connections Fail

Issue: When you add a computer to the ISA Server array, remove a computer, or restart the NLB service, connections are redistributed among the array members. When this occurs, secondary connections and repeated connections may fail. For examples, see Connection Failure Scenarios.

Solution: Reestablish the primary or initial connection (start a new client session).

Connection Failure Scenarios

Examples of connection failure:

• Example 1. The primary connection for a File Transfer Protocol (FTP) request may be handled by one array member, but the secondary connection may be passed to another member when the connections are redistributed, causing the secondary connection to fail.
• Example 2. When a client connects to Microsoft Office Outlook Web Access for Exchange Server 2003 through an NLB-enabled ISA Server array, one array member stores a cookie for that client on one of the array members, and the repeated requests and responses between the client and the Outlook Web Access server are handled by the same array member. When the connections are redistributed, the client requests may be handled by an array member that does not have the cookie, and the request will be rejected.

Firewall Client Connections Fail

Issue: When you add a computer to the ISA Server array, remove a computer, or restart the NLB service, connections are redistributed among the array members. When this occurs, connections from a Firewall client may fail.

Cause: The Firewall client control channel is established with a specific ISA Server array member, and all Firewall client communication must be handled by the same array member to succeed. When connections are redistributed, the Firewall client communication is likely to be handled by an array member that does not handle the control channel, and the connection will fail.

Solution: If the connection is associated with a specific application, close and open the application. If it is associated with a service, restart that service.

Loss of Network Connectivity When Making NLB Configuration Changes

Issue: After enabling or disabling NLB, network connectivity to ISA Server was lost.

Cause: After configuring NLB integration for any network object, network connectivity to ISA Server may be lost. This may happen when you enable or disable NLB integration. Connectivity may also be lost when you change the virtual IP address of the NLB-enabled network. This network connectivity loss may be the result of NLB making a change to the MAC address of the network adapter, while the computers connecting to ISA Server have not been updated with this change.

Solution: Do one of the following:

• Wait approximately 10 minutes to allow the change to propagate.
• Alternatively, on the computer that has lost connectivity to the ISA Server computer, force an update of MAC addresses. To do this, type arp –d at a command prompt.

Firewall Service Will Not Start

Issue: After configuring NLB, the Microsoft Firewall service will not start.

Cause: The network on which NLB is configured, or one of the networks that communicates with it through ISA Server, is fragmented (is comprised of many IP address ranges, such as 200 to 300). Fragmentation is not only a function of the number of network ranges, but how many subnets are defined by ISA Server to include those IP addresses. For example, the IP address range

200.0.0.0 to 200.255.255.255

is represented as one subnet:

• 200.0.0.0/255.0.0.0

Whereas, the IP address range200.0.0.0 to 200.255.255.254is represented by these subnets

• 200.255.255.254/255.255.255.255
• 200.255.255.252/255.255.255.254
• 200.255.255.248/255.255.255.252
• 200.255.255.240/255.255.255.248
• 200.255.255.224/255.255.255.240
• 200.255.255.192/255.255.255.224
• 200.255.255.128/255.255.255.192
• 200.255.255.0/255.255.255.128
• 200.255.254.0/255.255.255.0
• 200.255.252.0/255.255.254.0
• 200.255.248.0/255.255.252.0
• 200.255.240.0/255.255.248.0
• 200.255.224.0/255.255.240.0
• 200.255.192.0/255.255.224.0
• 200.255.128.0/255.255.192.0
• 200.255.0.0/255.255.128.0
• 200.254.0.0/255.255.0.0
• 200.252.0.0/255.254.0.0
• 200.248.0.0/255.252.0.0
• 200.240.0.0/255.248.0.0
• 200.224.0.0/255.240.0.0
• 200.192.0.0/255.224.0.0
• 200.128.0.0/255.192.0.0
• 200.0.0.0/255.128.0.0

Solution: Reconfigure the network to reduce its fragmentation.

Inconsistent Handling of Traffic

Issue: NLB traffic is handled inconsistently. Traffic in both directions is not always handled by the same array member. When this occurs, the traffic will fail. You will receive an alert if this is due to an NLB misconfiguration.

Cause: NLB is not configured on all of the networks that are handling the traffic.

Solution: NLB must be enabled on source and destination IP address ranges.

Switch Flooding

Issue: Masking the cluster media access control (MAC) address on outgoing packets prevents the switch from associating the cluster MAC address with a single port. When a client request (which contains the cluster MAC address) enters the switch, the switch does not recognize the MAC address in the packet and so sends the packet to all ports. This is called switch flooding.

Cause: In unicast mode (in which ISA Server integrated NLB operates), NLB induces switch flooding by design, so that packets sent to the cluster's virtual IP address go to all the cluster hosts. Switch flooding is part of the NLB strategy of obtaining the best throughput for any specific load of client requests.

If, however, the cluster shares the switch with other (noncluster) computers or other clusters, switch flooding can add to the other computers' network overhead by including them in the flooding.

Solution: You can avoid flooding noncluster computers by putting a network hub between the switch and the NLB cluster hosts, and then disabling the MaskSourceMAC feature. The hub delivers each packet to every host, and the switch associates the cluster MAC address with a single port, satisfying the switch's requirement that each port be associated with a unique MAC address.

To disable the MaskSourceMAC feature, follow this procedure on each member of the ISA Server array, after you configure integrated NLB on the ISA Server computer. At a command prompt, type the following commands:

1. nlb registry masksrcmac off
2. nlb reload

IP Address Conflict

Issue: An IP address conflict occurs.

Cause: The virtual IP address you selected is in use.

Solution: Configure the virtual IP addresses only after fully configuring NLB for that network. Similarly, be sure to remove the virtual IP addresses before you subsequently disable NLB on that network. Otherwise, the IP addresses may conflict.

Additional virtual IP addresses should not be the same as any of the dedicated IP addresses or virtual IP addresses on the network adapter.

Host ID Conflict

Issue: An alert is issued in Event Viewer indicating that a conflict has occurred in host ID assignment.

Cause: During installation, ISA Server assigns a persistent, unique host ID (in the range of 2–32) to each array member. This host ID is used to uniquely identify the server for NLB configuration.

Solution: The host ID value should not be changed, unless an alert is issued. Modify the host ID for a server as follows:

1. In the console tree of ISA Server Management, expand Arrays, expand the specific array, expand Configuration, and then click Servers.

2. In the details pane, right-click the required server, and then click Properties.

3. On the General tab, in Host ID, click the drop-down arrow and select one of the unallocated host IDs. (Only unallocated host IDs are displayed.)

4. Click OK, and in the details pane of ISA Server Management, click Apply to apply the change.

   Note

   If the Microsoft Firewall service did not start, and you have modified the host ID to resolve a host ID conflict, manually restart the Firewall service after modifying the host ID and applying the change.
   If you run unattended setup, you can specify a HostID property. If no value is specified, a host ID is automatically assigned to the server. If an invalid host ID value is specified in the .ini file during unattended setup, or there is a conflict in the automatically allocated host ID, Setup will complete but the Microsoft Firewall service will not start, and an alert "This server has the same host ID as server %1, which is not a valid configuration" will be generated. For more information, check Event Viewer.
   In some export/import configurations, a host ID may be automatically generated during the export process. If this host ID causes a conflict during the import process, the import process will succeed but the Microsoft Firewall service will not start, and an alert will be generated. After the import, for more information, check Event Viewer.

Appendix A: Using RemoveAllNLBSettings.cmd

The RemoveAllNLBSettings.cmd script is provided at the Microsoft Download Center. This download provides the following tools:

• RemoveAllNLBSettings.cmdscript. This script stops the Firewall service, modifies selected registry keys, and restarts the Firewall service.
• UnbindNLB.exetool. This tool unbinds NLB from all network adapters.

To use the script, do the following:

1. Download the script and tool, and copy both files to any folder on the ISA Server computer.

2. Double-click RemoveallNLBSetting.cmd to run it.

   Note

   The script only runs on the English version of ISA Server 2004 Enterprise Edition.
   Run the script will all network adapters enabled. A disabled adapter with NLB enabled will not function properly when you enable it, until NLB is disabled on it. To do this, clear Network Load Balancing on the network adapter properties sheet, and click OK to close the dialog box.
   If ISA Server services are no longer installed on the computer, ignore any errors about failure to stop or start the Firewall service.
   You can run the script without UnbindNLB.exe present. In this case, you will be prompted about the missing tool, and will have to complete the steps described in the section "Unbind NLB from all network adapters after running RemoveAllNLBSettings.cmd", in the tool help.

Additional Information

Additional ISA Server 2004 documents are available at the ISA Server 2004 Guidance page.