Windows Server 2008 R2: Networking in Failover Clusters

When failure isn’t an option, configuring failover clusters in Windows Server can help ensure high availability.

John Marlin

The networking model in Windows Server 2008 and Windows Server 2008 R2 Failover Clustering provides more robust and reliable communication among all cluster nodes, which greatly improves the efficiency and dependability of failover clustering. There are also several new features, including:

• More reliable communication using TCP and UDP unicast
• Support for IPv6
• Support for locating cluster nodes on separate, routed subnets
• More fine-grained control over network failure detection

You’ll need to use network hardware marked as “Certified for Windows Server 2008.” Any other component of your failover cluster solution must also be similarly certified. If you use iSCSI, your network adapters need to be dedicated for either network communication or iSCSI—not both.

When designing the network infrastructure to connect your cluster nodes, it’s essential to avoid single points of failure. There are many ways you can accomplish this. You can connect your cluster nodes with multiple, distinct networks. You could also connect your cluster nodes with one network built with teamed network adapters, redundant switches, redundant routers or similar hardware that remove single points of failure. These architectural requirements differ from server clusters in Windows Server 2003, which required two distinct networks.

Cluster Communications

Windows Server 2008 Failover Clustering now uses a virtual network adapter called Microsoft Failover Cluster Virtual Adapter to communicate between nodes in the cluster. You’ll also see this in Device Manager under Network Adapters (select Show hidden devices). You’ll also see it when issuing the command IPCONFIG /ALL. This network adapter handles all packet routing over the proper networks for communication, joins and so on.

This adapter will have an APIPA address defined in the address block 169.254.0.0/16. In IPv6, they’re assigned with the fe80::/10 prefix. In some environments, when adapters have an APIPA address, those adapters are disabled. If you disable the Cluster Virtual Adapter, you’ll disable communication between the nodes.

The goal is to sustain TCP/IP connectivity between two or more systems, despite the failure of any component in the network path. So there has to be an alternate physical path. In other words, a network component failure (whether it’s an NIC, router, switch or hub) shouldn’t cause a communication breakdown.

Communication should continue in a timely manner. There might be a slower response, but communication will persist as long as there’s an alternate physical route or link. This really comes into play when you talk about having nodes in separate sites or subnets.

Another change in Windows Server 2008 Failover Clustering is the cluster heartbeat mechanism. While it still uses port 3343, it has transitioned from a UDP broadcast health-checking mechanism to a UDP unicast communication. It’s similar to a ping in that it uses a Request-Reply process, but it includes more sophisticated features such as security and sequence numbering.

The default behavior has also changed in terms of how many replies are needed before the node is considered unreachable, initiating a Regroup to obtain a new view of the cluster membership. The cluster heartbeats let all nodes know which is up and down. As a default, the settings for this are controlled by:

• SameSubnetDelay: heartbeat frequency for nodes in the same subnet
• SameSubnetThreshold: threshold of the delays for nodes in the same subnet
• CrossSubnetDelay: heartbeat frequency for nodes in different subnets
• CrossSubnetThreshold: threshold of the delays for nodes in different subnets

These settings, and the method for changing them, are defined on the “Configure Heartbeat and DNS Settings in a Multi-Site Failover Cluster” TechNet Library page. There’s a “heartbeat” sent across with a sequence number, say from Node1 to Node2. Node2 responds with the same sequence number. Node1 again sends the same sequence number to Node2, and Node2 returns it one last time.

Node1 would then determine this heartbeat sequence complete and start the process again with another sequence number. If any of the heartbeat sequences are dropped or not received, it’s considered a “missed” heartbeat. By default, if any five of these sequences are missed, the node is considered down or inactive.

You can change these settings to increase the delay or threshold, but you can only work around any network problems. If there are any network latency issues, this could get around it, but it won’t fix the problem. So keep in mind that making changes to the delay or threshold settings isn’t considered a troubleshooting technique.

The heartbeats, by default, are going to use IPv6, as it’s a faster protocol than IPv4. If IPv6 has been disabled, it will instead use IPv4. A failover cluster will not mix and match IPv6 and IPv4. It will use one or the other, but not both at the same time.

Cluster Creation

When you create a cluster in Windows Server 2008 and Windows Server 2008 R2, the cluster-networking driver detects and creates the networks based on whether a default gateway is on the adapter. If it detects a default gateway, that network is set to allow clients to connect and use it for cluster communications.

This lets cluster IP addresses and client access points (network names) use the network. It also gives it a metric value starting at 10,000. If a network doesn’t have a default gateway, it will be given a metric value starting at 1,000. Then it will only be selected for Cluster Communications. Each network it detects increases the metric increment by 100.

One thing about the way it works now is that there’s no more concept of a “public” and “private” network. Therefore, the old “Recommended Private ‘Heartbeat’ Configuration on a Cluster Server” article for Windows Server 2003 clustering is invalid. Cluster communications are still going to go through all networks.

In previous versions, you defined which network you wanted to use for cluster communications. As long as that network was available, the cluster would use only that network. Windows Server 2008 and Windows Server 2008 R2 use all networks. If there’s an issue with one network, it will automatically switch between networks.

There’s an internal metric the Cluster Network driver uses. It doesn’t use the general TCPIP metric value. You can see the metric values with the following Windows PowerShell command:

Get-ClusterNetwork | FT Name, Metric

The metric values really come into play when talking about having a cluster with highly available virtual machines (VMs) and using Cluster Shared Volumes.

For example, say you want to run this command with these networks configured:

When using Cluster Shared Volumes, it will use the lowest metric value network for any CSV traffic or redirected mode access. When using the live migration feature of failover clustering, it will use the second-lowest metric.

In the example, CSV traffic will go over the iSCSI Network and live migrations will go over the network used for backups. When taking a backup of the VMs, the Cluster Shared Volumes will go into a redirected mode access. This is going to interfere with the ISCSI connections and could lead to disk failures. A data backup on the local drive of Node1 and a Live Migration would interfere with each other.

You need to reconfigure the networks to get everything you need. For the Live Migration network, you can change this by bringing up the properties of one of the VMs. On the Live Migration tab, change it to the LM Cluster network. For this, you only need to do it on a single VM because this is a global setting for all VMs.

For the CSV network, you can only affect this change through Windows PowerShell. To order the networks from Low to High, use the following commands:

Get-ClusterNetwork "CSV Cluster" | %{$_.Metric=800}

Get-ClusterNetwork "LM Cluster" | %{$_.Metric=900}

Get-ClusterNetwork "Backup Network" | %{$_.Metric=1000}

Get-ClusterNetwork "ISCSI Storage Network" | %{$_.Metric=1100}

Running the command to see the metrics will now show:

The CSV cluster network is set for metric 800. Adding any new network that doesn’t have a default gateway would be higher. Now with properly configured metrics, you can take backups or live migrate VMs without any conflicts on the networks.

The last thing to mention is cluster validation. You can run some network validation tests to determine connectivity issues, network configurations and so on. You can run these tests at any time without affecting production.

The cluster validation tests include:

• Cluster Configuration
• List Cluster Network Information
• Network
• List Network Binding Order
• Validate Cluster Network Configuration
• Validate IP Configuration
• Validate Multiple Subnet Properties
• Validate Network Communication

You can find the details of the Cluster Validation tests on the “Understanding Cluster Validation Tests” TechNet Library page. This will show you exactly what the tests look for and what each test does.

John Marlin is a senior support escalation engineer in the Commercial Technical Support Group. He has been with Microsoft for more than 19 years, with the last 14 years focusing on Cluster Servers.

Related Content

• Data Protection: Backup Basics in Windows Server 2008 R2
• Windows Server 2008 R2: Best Practices for Server Core Deployments
• Windows Server 2008 R2: Failover Clustering Troubleshooting Scenarios