The Management of MS Cluster Server (MSCS) Computing Environments

By Richard R. Lee, President

Data Storage Technologies, Inc.

Author of the book Windows NT Microsoft Cluster Server

Overview:

Microsoft Cluster Server (MSCS) has quickly become the de-facto solution for high-availability clustering in the Windows NT space of the enterprise computing market. While certainly not the first to market in the growing segment of high-availability computing (Digital, Tandem, NCR, Veritas, Vinca and several other solutions were all available well in advance of MSCS), Microsoft has nonetheless captured the hearts and minds of most IT users based upon MSCS's tight integration with the Windows NT operating system (OS), along with the use of a formidable team of hardware partners to co-develop and extensively test the product prior to release. These so-called "Early Adopter partners" included IBM, Compaq, Tandem, Digital, NCR, Data General1, all leaders in clustering technology for the past 20 years and major contributors of intellectual property to MSCS.

Like many clustering solutions of the day, Microsoft Cluster Server (Version 1.0 is part of Windows NT Enterprise Edition which was released in the Fall of '97) can be both difficult to deploy and challenging to manage. It uses a centralized management console called the Cluster Administrator (CA) for setup, manipulation and monitoring of all Resources and Failover Groups, which can be most challenging when used with individual two-node clusters, and virtually impossible in multi-cluster environments2.

To create a more manageable MSCS environment, one that delivers on the clustering promises of TCO reductions and availability enhancements, NuView (a Houston, Texas start-up) has developed an MSCS cluster management solution called ClusterX. It is a cluster, application and NT service configuration/management solution for use by administrators in MSCS-based computing environments. Announced in 1998, ClusterX is now available in its second version (2.0) with many enhanced features and benefits for MSCS administrators to use in any size Microsoft Cluster Server-based computing environment. ClusterX has been especially tailored to manage distributed clusters in anyone's enterprise computing environment.3

Within the context of this white paper we will discuss the background of both MSCS and ClusterX. You will quickly see why ClusterX is becoming the de-facto Cluster Management Solution for MSCS, regardless of the size of the deployment (from a single cluster HA solution to an enterprise oriented, multi-cluster environment).

Microsoft Cluster Server (MSCS):

Microsoft Cluster Server (MSCS)4 today is a two-node high availability solution for use in application environments that require major enhancements to Availability5 with respect to stand-alone NT Server solutions.6 It is based upon the use of intelligent middleware that resides between the Windows NT Operating System (OS) and the Applications and Services that the user would like enhanced.

MSCS uses a "shared nothing" architectural model, where each node (server) has ownership of specific storage resources located on a shared-storage bus (SCSI, FC or SSA) except in times of Failover. During Failover, these storage resources are transferred in ownership (via SCSI lock/release commands) from the failed node to the surviving node. The surviving node has duplicate instances of those applications that we designated for Failover installed on it, which are then started once the Cluster Executive initiates transfer of storage resource ownership, along with all the other Resources (if any) contained in the Failover Group.7 The surviving node after this transfer and subsequent start-up of its copies of the failed application or service, then resumes operations that were interrupted at the time of the Failover, e.g., file and print sharing, web services, database transactions and queries (via roll-back restart to the last "committed" transaction). The surviving node also takes ownership of the Quorum Resource, a special disk or volume that contains the cluster database.8 This mode of

Figure 1: An architectural block diagram of MSCS deployed in an Active/Active mode

operation continues until such time as the failed node is revived and brought back on line9

One of the key differentiators found in MSCS with respect to other NT-based clustering solutions is the use of a "cluster-aware" API developed by Microsoft and its partners. This API is used to create cluster-aware applications and services that increase the granularity of Failover detection down to specific Applications and Services, rather than just node-wide, allowing failed applications and services to be failed over as opposed to the entire node. Most system failures or "hangs" are software-driven according to statistics found today, so this increase in granularity allows for software errors to be overcome without failing over an entire node and all of its applications and services.. At that time the failed node (or application/NT service) can then have its physical disk Resource (and others in the Failover Group) transferred back to its ownership on either an automatic or manual basis. It then resumes normal operations until such time as a failure on either node (or its applications/NT services) occurs again.

MSCS can be deployed on an application by application basis in a number of ways. There are five deployment models outlined by Microsoft. Each takes advantage of the specific attributes that MSCS has to offer in respect to enhancing Availability, Scalability and Manageability. These five basic models are as follows:

1. Active/Active: In Active/Active mode, both nodes are allowed to have varying workloads and applications residing on them. They each perform work independent of the other, with instances of those applications setup for Failover/Failback residing on both nodes.

2. Active/Standby: In Active/Standby mode, the active node is online and doing work, while the inactive node sits in a "hot standby" mode waiting for any type of failure to occur on the primary node.

3. Partial Cluster Solution: In this mode there are a mixture of applications/resources that can Failover/Failback, along with those that cannot (non-cluster-aware). The cluster-aware applications are set up in a normal manner under MSCS using shared storage resources, and those that aren't utilize local storage resources found on the node where they permanently reside.

4. Virtual Server Only: This model utilizes MSCS virtual server mode, without having formed an actual cluster. It can be deployed on any node that has cluster server running at the time.

5. Hybrid Solution: The hybrid model is a combination of all the others previously described.

These deployment models are intended to support several types of Failover scenarios. These scenarios are designed to be tailored to the specifics of each application type. Typical of these scenarios are the following;

1. Automatic Failover w/Automatic Failback: In this scenario, the application or service that fails or becomes unavailable, automatically fails over to its alternate node when there is a loss of heartbeat, or in the case of a cluster-aware application, when the "LooksAlive", "IsAlive" messages are not returned. When the failed node or application returns to its normal state and the heartbeat has been re-established , then the Failover Group's resources are all returned automatically to their original state and owner.

2. Automatic Failover/Manual Failback: In this scenario, the failed node or application is brought back online manually by the administrator. This allows for thorough testing and monitoring prior to the transfer – eliminating any potential for the Failover Group to fail again and potentially bringing down the entire cluster.

3. Manual Failover/Manual Failback: This scenario is used for facilitating rolling upgrades and routine maintenance of the cluster. Using full manual control of the cluster, the administrator can bring applications, services and the node itself offline gracefully, with full monitoring. All designated applications can then be re-started on the alternate node, while upgrades or routine maintenance operations are performed on the failed node. Once these are complete, this node can then be brought back online and its original workload and Failover groups can then be transferred back. At that time the alternate node can then be brought offline and its workload can be transferred to the alternate node, so that it can undergo routine maintenance or have applications upgraded.

These are just a few of the Failover scenarios that are supported by MSCS. The administrator has the option of creating many more depending upon his or her requirements on an application by application basis.

Cluster-Aware Applications:

One of MSCS's most significant differentiators with respect to competing clustering solutions is its software interface specification called the "Wolfpack API". This API (some 120+ pages in length) is designed to let software developers take full (or partial) advantage of the power of MSCS with the applications that they develop. It significantly increases the granularity of Failure detection (down to the service level), and provides mechanisms for more comprehensive application monitoring and reporting. In future versions of MSCS, it will be used to help provide unprecedented levels of linear scaling through inter-node communications (use of the MSCS private network and the VIA specification).

Figure 2: A sampling of commands from the Wolfpack API

Early usage of this API has been dominated by Microsoft in the form of Enterprise Editions of its familiar BackOffice products. These include:

• SQL Server 7.0 (and 6.5)

• Exchange 5.5

• Internet Information Service

• Message Queue Server

• Transaction Server

• SMB File Share

• Print Spooler

All take advantage of the Wolfpack API in one manner or another (LooksAlive vs. IsAlive), and can be easily configured as virtual servers (including symmetric).

These first generation cluster-aware applications provide significant availability enhancements over their non-cluster aware counterparts (with some scalability enhancements as well when you use virtual symmetric servers and partitioned data sets), but can be difficult to set up and administrate. These applications come bundled with setup wizards, but much of the work in setting them up is done via the Cluster administrator on a manual basis. This can prove tedious and confusing to the untrained administrator and can be a source of errors for even the well-trained administrator.

Figure 3: CA View of a Cluster-aware application and its Resources (Exchange 5.5 EE)

The MSCS Cluster Administrator:

Included with MSCS is a management and configuration tool known as the Cluster Administrator or CA. It is an explorer-type GUI for use in setting up, managing and monitoring your cluster. The Cluster Administrator must be run on an NT Workstation designated during setup and provides the interface to the cluster during every phase of its operation.10 The Cluster Administrator also contains a number of setup wizards for creating and installing applications and services on the cluster.

Although straightforward in approach, the CA can be very challenging and difficult to use in day-to-day administration of MSCS. Many of these challenges are driven by the lack of true visualization of the cluster and its resources during operation and setup. The administrator is left in many cases to fend for him or herself to determine the status of key services and hardware components not displayed by the CA.

Figure 4: The MSCS Cluster Administrator. Shown on the right are the Resources in the Cluster Group

All of the commands and functions found in the CA are duplicated under a command line version known as Cluster.exe. These commands can be setup under several scripting scenarios including Windows NT Scripting Host.

Management Challenges and Shortcomings:

The MSCS Cluster Administrator is sorely lacking in both functionality and flexibility. It is difficult to install and configure applications and Failover groups for even the most seasoned administrator. It has limited management functions, and cannot span across multiple clusters, much less perform rudimentary cluster tasks such as Load Balancing and Cluster Database protection and management. The CA was designed in a vacuum with little feedback from users in the field. The majority of sites that evaluated MSCS and the CA had never had a GUI-based management tool (most clusters are CLI or script-driven) to work with and did not learn until much later how limited the Cluster Administrator really was. This level of frustration, along with the inability to maximize the capabilities of MSCS with its own management tools prompted NuView to develop ClusterX.

ClusterX for Microsoft Cluster Server (MSCS):

ClusterX is generally described as a "one to many" management tool for use in MSCS-based clustering environments. It was designed to enhance the existing MSCS Cluster Administrator, in order to provide dramatic improvements in the scope of functionality options available to administrators, as well as to substantially increase the number of clusters that can be effectively managed from a single console. It was designed specifically for the Windows NT Server environment, and is based upon such Microsoft standards as:

• COM (common object model) and DCOM (distributed COM)

• ActiveX scripting technology (Jscript and Vbscript)

• The Microsoft Management Console (MMC "snap-in")

• ActiveX Containers (ClusterX can be used with IE 4.01 or later)

ClusterX was designed to be non- blocking to allow for multiple tasks to be performed on a parallel basis. It is also multi-threaded in design to allow multiple commands to be processed simultaneously, with prioritization support of these same commands.

ClusterX adds major functionality to any MSCS environment. The new capabilities that it provides are key to administrators achieving the full capabilities of their MSCS clusters. These new capabilities are:

The creation of a load balancing infrastructure across all cluster nodes.

• Administrators can now monitor workloads on each node on a real time basis. Information is then displayed in terms of percentage utilization of available resources (CPU, etc.), allowing administrator to move Failover Groups off of over-worked servers to lesser used ones, while continuing to monitor them no matter where they ultimately reside.

The addition of backup and restore functions to protect Cluster Configuration Data.

• ClusterX allows the system administrator to backup the entire cluster configuration information. This data can then be used to restore the cluster in the event of a catastrophic failure (cluster-wide or on a single node).

The support for duplicating Failover Groups across multiple clusters.

• A knowledge-based rules system to allow administrators to setup and manage complex Resources and Applications.

The ability to move Resources from one Failover Group to another.

• Using the Dependency View, administrators can simply "drag 'n drop" resources from one group to another.

The creation and manipulation of Failover Group Dependency trees.

• Failover Groups and their complex dependencies can be manipulated and viewed in their logical context.

The creation of comprehensive logs for all cluster activities, problems and changes.

• User actions, and all other cluster activity, is logged and displayed in a consolidated Audit Log View. Multiple levels of filtering can be applied to display log information in the most relevant manner.

Setup wizards for use with cluster-aware applications and services.

• Advisors and wizards are provided for the most popular cluster-aware applications and Windows NT services to allow for enhanced setup and configuration. These tools can configure MSCS as well as the application themselves.

All of these major functional areas (as well as some others not mentioned here) provide significant enhancements over what is available with the MSCS Cluster Administrator. In total, ClusterX and its expanded capabilities deliver on the promises that MSCS (and Microsoft) made during its development with respect to improvements in Availability, along with attendant reductions in Management burden and costs – both key components of everyone's efforts to reduce TCO, while increasing operational efficiencies.

Figure 5: A ClusterX "Enterprise-wide" view of Clusters, Failover Groups and Resources

Figure 6: A schematic view of two clusters managed by ClusterX

MSCS and ClusterX:

Microsoft Cluster Server is the cornerstone of Microsoft's efforts to move Windows NT into the Enterprise Computing space of the market. It solves major problems associated with availability, scalability and manageability that have plagued Windows NT since its earliest days. Given this significance, Microsoft and its hardware/software partners have invested heavily in MSCS with respect to core product development and testing. What they missed along the way in this process was creating a simple to use, but powerful management environment for these clusters. They also apparently envisioned that end-users would only deploy MSCS as autonomous clusters with little need to manage multiples of these in most circumstances. Both of these short-sided decisions left a gaping hole in Microsoft's first-generation cluster offering – MSCS.

Additionally, the growth in the number of cluster-aware applications available for MSCS exposed other shortcomings in the CA. Although many these applications came with setup wizards they did not set the application up correctly on the cluster itself, they merely set up the application and its single or multiple instances. The Systems Administrator was then required to intervene and either setup the applications' Failover Groups and their Resources (including dependencies) in advance of installing the application itself, or tweak it after the fact. Administrators were left to their own devices to fashion dependency trees and to determine how they were structured. This did little to make IT managers believe that MSCS was going to save time and reduce costs.

Responding to this growing level of frustration expressed by early adopters of this technology, NuView began to develop ClusterX as the enterprise-wide, comprehensive management and configuration tool that systems administrators were looking for.

ClusterX came to market at just the right time to meet the growing demands of MSCS deployers and administrators. ClusterX Version 1.0 filled in the most gaping holes found in the management and configuration of MSCS, while Version 2.0 has greatly expanded the functionality of MSCS overall. The capabilities of ClusterX have not gone unnoticed within the MSCS hardware camps in the computer industry. At this time, Data General, Unisys and Dell Computer are bundling versions of ClusterX with their MSCS solutions. It is anticipated that this list will grow eventually to include all providers of MSCS solutions as ClusterX becomes the de-facto standard for MSCS management and configuration.

Figure 7: Several ClusterX Dependency Trees on an MSCS Cluster

ClusterX Deployment:

The deployment of ClusterX is very straightforward. During installation, ClusterX assigns agents to each cluster node (server) that it manages. These agents provide feedback and logging to the ClusterX client workstation that is used for command and control. This data is then used to support policy-based management of each cluster, along with load balancing activities.

ClusterX is then configured for each cluster. This process is based on ClusterX "learning" the specifics of the cluster and its Failover Groups and Resources. It also examines each cluster's hardware configuration with respect to public and private network connections, CPU and memory information, physical disk devices, etc. All of this information is then displayed to the administrator graphically, as well as logged by ClusterX.

As shown in several of the figures in this white paper, ClusterX displays on a single console all information about the clusters its manages, with varying messages displayed as icons or popups.

With regards to interoperability with other systems used across the enterprise, ClusterX is easily integrated into the frameworks of such enterprise management applications as: Unicenter TNG, HP Open View, Tivoli TME, etc.

Currently, ClusterX has advisors and wizards to support the most popular applications and NT services being deployed in MSCS environments. These include:

• SMB File Shares

• Print Spoolers

• Microsoft Exchange 5.5 EE

• Microsoft SQL Server 6.5/7.0 EE

• Microsoft Internet Information Server 3.0/4.0

ClusterX has demonstrated itself as easy to install and familiarize ones self with – all within a short time from startup. It has easy to read messages and icons and gives Systems Administrators all the information and tools required to manage and support MSCS clusters – regardless of how widely they are disbursed or the complexity of the applications and NT services that they have clustered.

Summary and Recommendations:

Microsoft Cluster Server and NuView's ClusterX go hand-in-hand together. They work synergistically to create a Windows NT clustering offering that rivals long established high-availability solutions found in the Unix and proprietary OS spaces of the marketplace. They combine forces to meet the challenge of improving the Availability, Manageability and, to a lesser extent, Scalability of major Windows NT applications such as SQL Server, Exchange and IIS. In addition to improving these major shortcomings found in Windows NT, they reduce costs. These costs are not only in respect to TCO (total cost of ownership), but include business losses due to downtime. MSCS and ClusterX reduce downtime losses from hundreds of hours per year to hours per year in many cases. Additionally, with the current wave of service level agreement offerings available from many vendors in the market, ClusterX allows MSCS users to effectively monitor compliance with these agreements.

As MSCS evolves under the Windows 2000 umbrella, so will ClusterX. The need for ClusterX will not be eliminated by the introduction of this major upgrade of Windows NT – it will be enhanced. With the delivery of a multi-node clustering solution from Microsoft looming on the horizon, the requirements for a "best of breed" management and configuration tool will be paramount. Investments in ClusterX today will be protected long into the future, enhancing the ROI of this investment even further.

I recommend the use of ClusterX in any MSCS environment, regardless of size or complexity. It will go a long way in guaranteeing you the ROI increases and TCO reductions that clustering on Windows NT promised several years ago, along with making your application environment as bullet-proof and manageable as possible.

For further information on ClusterX and NuView see their web site at https://www.nuview.com

References:

1. Microsoft Cluster Server general information and whitepaper links – https://www.osborne.com

2. NuView web site for ClusterX information and to download a 30-day free trial copy – https://www.nuview.com

3. Microsoft Cluster Server – API Specification (now included in the Windows Platform Development Kit) – https://www.intelligententerprise.com