Export (0) Print
Expand All

Configuration of the Network Load Balancing Clusters at Microsoft.com

Updated: September 1, 2003

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

This section describes the details of the hardware, software, and Network Load Balancing configurations of the clusters of Web servers that support the Microsoft.com Web site. Because the purpose of this document is to focus on the details of the clustering implementations, only limited information is provided regarding general network hardware configuration and other configuration settings that are not specifically related to Network Load Balancing. However, each Network Load Balancing setting is discussed in depth.

  • For information about hardware configuration, see Hardware Configuration for Network Load Balancing Hosts.

  • For information about the software configuration, see Software Configuration for Network Load Balancing Hosts.

  • For information about the Network Load Balancing configuration, see Network Load Balancing Configuration.

Hardware Configuration for Network Load Balancing Hosts

Each host in the Network Load Balancing clusters at Microsoft.com has the same hardware. The servers are Compaq ProLiant ML570s (7U) with the following standard hardware:

 

Processors

4 x Pentium III XEON 700 Mhz; 1-MB cache processors

Memory

2 GB, 100 Mhz SDRAM

Controller

Smart array controller 5302/64: PCI; 64-bit/66 MHz; 2-channel; U160; 64-MB cache

Hard disk drives

10 x 18.2 GB hot-plug; 15K RPM drives

2 x 36.4 GB hot-plug; 15K RPM drives

Network adapters

2 x NC3134 Fast Ethernet network adapters; 32/64-bit PCI; 33/66 MHz; dual-base 10/100

Power supply

Hot-plug redundant power supply

Explanation

This hardware provides the necessary processor capacity for unexpected spikes in Web-site traffic at Microsoft.com. At the time of this writing, these are currently some of the fastest four-processor computers available. With 2 gigabytes (GB) of memory, the computers can handle several IIS 6.0 processes at once, in addition to providing plentiful caching capacity.

Storage Configuration for Each Host

Each host has the following storage configuration:

  • 120-GB content capacity

  • 34-GB logging capacity

  • Virtual disk 1: RAID-1 mirrored system volume that contains the operating system and applications

  • Virtual disk 2: RAID-1 mirrored logging volume that contains IIS log files

  • Virtual disk 3: RAID-5 striped content volume that contains all Web content

3738ba94-5175-408c-a03b-368172fda6a2

Diagram 4

Diagram 4 shows the storage configuration for each host. Each host has 12 physical disks, which are configured into three virtual disks. Virtual disks 1 and 2 are RAID-1 configurations and the third is a RAID-5 configuration. Virtual disk 3 (RAID-5) is distributed across a two-channel RAID controller.

Explanation

Microsoft.com implemented this storage configuration for each host to achieve high capacity and redundancy. Windows Server 2003, Enterprise Edition, and Windows Server 2003, Datacenter Edition, support RAID-0, RAID-1, and RAID-5 configurations. The purpose of RAID-1 and RAID-5 configurations is to guard against the loss of data in the event of a catastrophic hard-disk failure. Microsoft.com used RAID-1 and RAID-5 configurations to provide the required fault tolerance at a minimum cost.

The virtual disks and hard disks were split across two channels because of the limitations of hard disk space. Although each RAID channel can only accommodate six hard disks, each server has space for 12. Therefore, these are split between two channels.

This design is a compromise between minimizing cost and optimizing performance. With only two channels and a single controller to use, Microsoft.com chose to put the Web content on a single channel and therefore, had to put the operating system and the IIS logs together on the other channel. The ideal high-availability solution, which includes storing the mirrored volumes on separate channels, requires additional hardware costs.

Software Configuration for Network Load Balancing Hosts

Each host in the Network Load Balancing clusters at Microsoft.com has the following software:

  • Windows Server 2003, Enterprise Edition

  • IIS 6.0

  • Microsoft Operations Manager (MOM)

Network Adapters

Each server has two network adapters installed. The cluster adapter is labeled FE (by Microsoft.com administrators) for front-end. The management adapter is labeled BE for back-end.

The network adapters have the following:

 

  "FE" network adapter "BE" network adapter

Protocols

Internet protocol (TCP/IP)

Network monitor driver

Internet protocol (TCP/IP)

Network monitor driver

Clients

Network Load Balancing

Client for Microsoft networks

File and printer sharing for Microsoft networks

IP address

Static virtual IP address (VIP)

Static dedicated IP address (DIP)

Static IP address

Explanation

The adapters are labeled for easy identification. The FE adapter is the cluster adapter, which is used for Internet traffic, while the BE adapter is the management adapter, which is used for content replication and administration. As a best practice, you should use two network adapters in all Network Load Balancing implementations. In this case, the two network adapters provide additional security by physically separating the Internet traffic from the perimeter network (also known as the screened subnet) traffic. Also, they address the restriction that a Network Load Balancing cluster that is operating in unicast mode cannot distinguish among single adapters on individual hosts. Therefore, to make communication among cluster hosts possible, each cluster host must have at least two network adapters.

TCP/IP Configuration

  • Each network adapter is given a static IP address and a static DNS server address. The static IP address on the FE adapter is known as the dedicated IP address (DIP).

  • In addition to the DIP, the FE adapter also has the virtual IP address (also known as the cluster IP address, or VIP), which is used by all Internet clients connecting to Microsoft.com.

  • NetBIOS is disabled on the FE adapter through the advanced TCP/IP Properties dialog box.

Explanation

Each adapter that is bound to Network Load Balancing (in this case the FE adapter on each host) has a dedicated IP address (DIP) in addition to the virtual IP address (see Diagram 5). The dedicated IP address must be a static IP address that is unique for each host. The DIP cannot be a DHCP address. The DIP is only used to allow remote connections to a specific host through the Network Load Balancing Adapter. Traffic addressed to the DIP is never load balanced.

Be aware that you should always enter the dedicated IP address first in TCP/IP properties, before the virtual IP address.

88314de1-b2c8-48b4-a311-67127366e340

Diagram 5

Network Load Balancing references the dedicated IP address (DIP) only when a single network adapter is used to handle both client-to-cluster traffic and other network traffic that must go specifically to the dedicated IP address; for example, Telnet access to a specific host within the cluster. At Microsoft.com, the DIP is primarily used by ClusterSentinel. All other administrative communication, such as content replication, occurs through the BE adapter.

Communication over the direct hosting port, as well as other ports that are considered unnecessary, is restricted by router access control lists (ACLs). The NetBIOS is disabled on the FE adapter in order to provide a secondary layer of protection to protect against malicious activity behind the ACLs. However, even with the NetBIOS disabled, direct hosting of Server Message Block (SMB) traffic is still active. For more information about direct hosting, see article 204279, Direct Hosting of SMB Over TCP/IP, in the Microsoft Knowledge Base (http://go.microsoft.com/fwlink/?LinkId=4441).

Network Load Balancing Configuration

This section describes the parameters that are configured for the Network Load Balancing clusters as a whole, the parameters that are specific to each host in the Network Load Balancing cluster, and the port rules that control how the cluster functions.

Cluster Parameters

These parameters appear on the Cluster Parameters tab that is available when you configure or view Network Load Balancing properties. These parameters must be set identically on each host of the cluster. The easiest way to achieve this is to use Network Load Balancing Manager when implementing and configuring the hosts.

 

Configuration setting Explanation

Network Load Balancing is enabled on the network adapter labeled FE.

Network Load balancing is only enabled on the FE adapter because the BE adapter is used exclusively for administrative network traffic and communication among the hosts. The BE adapters also handle content replication across the hosts. The BE adapter does not handle any cluster traffic from clients.

IP address is set to the IP address of the cluster (the VIP). In this document, these addresses are 0.0.0.x.* See Set of Network Load Balancing Clusters Running IIS 6.0 (Diagram 2) and the Network Load Balancing Host Diagram (Diagram 3), earlier in this document, for more information.

Subnet mask is set to the subnet mask of the cluster's IP address.

* These are not the actual Microsoft.com VIPs.

Note that each cluster has only one virtual IP address. Microsoft.com has not used the Virtual Cluster feature of Windows Server 2003, Enterprise Edition, which allows you to enable multiple virtual IP addresses for each cluster.

Unicast is selected.

Unicast mode is typically the preferred cluster operation mode. While multicast mode can sometimes be used as an alternative and does offer certain advantages, it is not supported by all routers and often presents additional complexity. Unicast mode with multiple adapters is a model that is suitable for a cluster such as the one described in this document, in which ordinary network communication among cluster hosts is necessary or desirable and a second management adapter is available for that communication.

Remote control is enabled.

Remote control is enabled to allow the administrators to use ClusterSentinel for controlling the cluster.

noteNote
Although used at Microsoft.com, the remote control option is generally discouraged because it presents significant security exposures including the possibility of data tampering, denial of service and information disclosure. We strongly recommend that sites use other remote management tools such as Network Load Balancing Manager or Windows Management Instrumentation (WMI) instead of remote control. If you choose to enable remote control, it is vital that you restrict access by specifying a strong remote control password. In addition to the password, it is also imperative that you use a firewall to protect the Network Load Balancing UDP control ports (the ports receiving remote control commands) to shield them from outside intrusion (see Diagram 6). By default, these are ports 1717 and 2504 at the virtual IP address. Microsoft.com has implemented these safeguards.

46030e8f-e2c5-4668-9138-0215bf538d45

Diagram 6:

Host Parameters

These parameters appear on the Host Parameters tab that is available when you configure Network Load Balancing. Some of these parameters are unique for each host, but in most cases are identical across all of the hosts in the cluster.

 

Configuration setting Explanation

Priority (Unique Host ID) is set to the host ID that the particular server represents in the cluster.

The Priority (Unique Host ID) determines which host handles certain cluster traffic. The host with the lowest numerical priority among the current members of the cluster handles all of the cluster's network traffic that is not covered by a port rule. Each host must have a different Priority value. If Network Load Balancing Manager is used to configure the hosts, Network Load Balancing Manager will ensure that each host has a unique priority.

Initial host state is set to stopped and the Remain suspended check box is not selected.

Initial host state is set to stopped to allow the Microsoft.com administrators to perform a variety of tests on the host once it is restarted after a shutdown, but prior to allowing it to rejoin the cluster. This allows the administrators to ensure that the host is operating properly before it is included in the Network Load Balancing rotation.

Dedicated IP address and Subnet mask are set to the IP address and subnet mask of the FE network adapter.

Once again, it is important to distinguish between the dedicated IP address (the DIP), which is configured on this tab, and the virtual IP address (VIP), which is the cluster's IP address and is configured on the Cluster Parameters tab. If you use Network Load Balancing Manager to configure your cluster, the dedicated IP address parameter will already contain the correct address. The addresses configured on this tab (and the Cluster Parameters tab) are assigned to the FE network adapter.

Port Rules

Port rules are set up to control how each port handles network traffic addressed to the cluster. The number and type of rules must be exactly the same for each host in the cluster. We recommend that you use Network Load Balancing Manager to configure port rules. If you are using Network Load Balancing Manager, when you add additional hosts, they will automatically inherit the cluster port rules from the initial host. If you are not using Network Load Balancing Manager and a host attempts to join the cluster with a different number of rules, or with different rules from the other hosts, it is not accepted as part of the cluster. The rest of the cluster then continues to handle the traffic as before. At the same time, a message is entered into the Windows event log.

Be aware that all network traffic that is addressed to the cluster and is not governed by port rules is handled by the host with the highest host priority among the current members of the cluster. This single host handles all of the cluster network traffic not governed by a port rule, with another host taking over the traffic only in the event that the highest priority host fails or goes offline. This default behavior ensures that Network Load Balancing does not affect cluster network traffic for ports that you do not specifically manage with the Network Load Balancing load-balancing mechanisms. It also provides high availability for handling your cluster network traffic.

Port Rules Configured at Microsoft.com

Microsoft.com has only two port rules configured:

  • Port range: 80 to 80

    • Virtual IPs: all

    • Protocols: TCP

    • Filtering mode: multiple hosts

    • Affinity: none

    • Load weight: equal

  • Port range: 443 to 443

    • Virtual IPs: all

    • Protocols: TCP

    • Filtering mode: multiple hosts

    • Affinity: none

    • Load weight: equal

Explanation

The port range that is specified in each of the rules includes only ports that are handled by the cluster's virtual IP address (VIP). Therefore, they are the only ports handling traffic managed by Network Load Balancing. Be aware, however, that this does not prevent network traffic from being sent to and from each host's dedicated IP address (DIP) or other ports on the VIP. Microsoft.com has used router access control lists (ACLs) to manage this kind of traffic. You can also disable the remaining ports by creating port rules that include all remaining ports, and then using the Disable setting in the Port Rules Configuration dialog box. While this configuration provides additional protection for the host, do not use port rules as a substitute for a firewall.

Microsoft.com does not make use of virtual clusters in this document, thus the port rules are configured as global port rules by choosing All for the virtual IP address.

TCP is the only type of traffic that this cluster handles.

Setting the filtering mode to multiple hosts allows all of the hosts in the cluster to handle network traffic for the associated port rule. This filtering mode provides scaled performance and fault tolerance by distributing the network load among multiple hosts. You can specify whether the load is equally distributed among the hosts or that each host handle a specified load weight. Microsoft.com has specified equal load weight for each host.

Affinity is set to None, which specifies that multiple connections from the same client IP address can be handled by different cluster hosts (that is, there is no client affinity).

noteNote
Although not used by Microsoft.com, Class C or single affinity is typically recommended for Secure Sockets Layer (SSL) transmissions on port 443 to make the most efficient use of server resources. SSL connections are associated with a particular session ID, which (within a specified time constraint) can be reused when communicating with a particular host. If the client computer presents the session ID to a different host, the session will be rejected and a new session will be negotiated. (This process is transparent to the user). The negotiation requires additional host resources and increases the server overhead. Therefore, if you use Class C or single affinity, you can minimize how often the session must be renegotiated.

This concludes the configuration section of this document. As described earlier, information provided here is meant to cover only those settings directly related to the Network Load Balancing implementation at Microsoft.com. Other software and hardware details that are necessary for a production Web site have not been described.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback

Community Additions

ADD
Show:
© 2014 Microsoft