Failover Cluster Troubleshooting
This topic provides information about the following issues:
Basic troubleshooting steps.
Recovering from a failover cluster failure.
Resolving the most common failover clustering problems.
Using extended stored procedures and COM objects.
When working with SQL Server 2005 failover clustering, remember that the server cluster consists of a failover cluster instance that runs under Microsoft Cluster Services (MSCS). The SQL Server instance may be hosted by Microsoft MSCS-based nodes that provide the Microsoft Server Cluster.
If problems exist on the nodes that host the server cluster, those problems may manifest themselves as issues with your failover cluster instance. To investigate and resolve these issues, troubleshoot a SQL Server failover cluster in the following order:
Hardware: Review Microsoft Windows system event logs.
Operating system: Review Windows system and application event logs.
Network: Review Windows system and application event logs. Verify the current configuration against the Knowledge Base article, Recommend e d Private "Heartbeat" Configuration on a Cluster Server.
Security: Review Windows application and security event logs.
MSCS: Review Windows system, application event, and cluster logs.
SQL Server: Troubleshoot as normal after the hardware, operating system, network, security, and MSCS foundations are verified to be problem-free.
Usually, failover cluster failure is to the result of one of two causes:
Hardware failure in one node of a two-node cluster. This hardware failure could be caused by a failure in the SCSI card or in the operating system.
To recover from this failure, remove the failed node from the failover cluster using the SQL Server Setup program, address the hardware failure with the computer offline, bring the machine back up, and then add the repaired node back to the failover cluster instance.
For more information, see How to: Create a New SQL Server 2005 Failover Cluster (Setup) and How to: Recover from Failover Cluster Failure in Scenario 1.
Operating system failure. In this case, the node is offline, but is not irretrievably broken.
To recover from an operating system failure, recover the node and test failover. If the SQL Server instance does not fail over properly, you must use the SQL Server Setup program to remove SQL Server from the failover cluster, make necessary repairs, bring the computer back up, and then add the repaired node back to the failover cluster instance.
Recovering from operating system failure this way can take time. If the operating system failure can be recovered easily, avoid using this technique.
For more information, see How to: Create a New SQL Server 2005 Failover Cluster (Setup) and How to: Recover from Failover Cluster Failure in Scenario 2.
Also consider the following changes to failover clustering in SQL Server 2005:
In SQL Server 2005, log files are located in a different directory than in previous SQL Server releases. For more information, see How to: View SQL Server 2005 Setup Log Files.
In SQL Server 2005, instance-unaware components are installed on the primary node only; in SQL Server 2000, instance-unaware components were installed on all nodes. For more information, see Setting Up Windows Service Accounts.
In SQL Server 2005, directory and registry structures are different than in previous releases. For more information, see Setting Up Windows Service Accounts and File Locations for Default and Named Instances of SQL Server 2005.
The following list describes common usage issues and explains how to resolve them.
Problem: Incorrect use of command-prompt syntax to install SQL Server 2005
Issue 1: It is difficult to diagnose Setup issues when using the /qn switch from the command prompt, as the /qn switch suppresses all Setup dialog boxes and error messages. If the /qn switch is specified, all Setup messages, including error messages, are written to Setup log files. For more information about log files, see How to: View SQL Server 2005 Setup Log Files.
Resolution 1: Use the /qb switch instead of the /qn switch. If you use the /qb switch, the basic UI in each step will be displayed, including error messages.
Issue 2: Not following the format used in the template.ini file, located in the root directory of SQL Server 2005 installation media.
Resolution 2: Though unexpected characters may be ignored by the Setup program, include all required variables in your installation command.
There should be no space between a variable and its value in a command line. For example, there is no space in "ADDLOCAL=ALL". If "ADDLOCAL = ALL" is used, Setup fails. Another example is the IP and its value. "IP=www.xxx.yyy.zzz,Local Area Connection" is correct. If a space is inserted after ",", Setup fails.
Problem: SQL Server 2005 cannot log on to the network after it migrates to another node
Issue 1: SQL Server service accounts are unable to contact a domain controller.
Resolution 1: Check your event logs for signs of networking issues such as adapter failures or DNS problems. Verify that you can ping your domain controller.
Issue 2: SQL Server service account passwords are not identical on all cluster nodes, or the node does not restart a SQL Server service that has migrated from a failed node.
Resolution 2: Change the SQL Server service account passwords using SQL Server Configuration Manager. If you do not, and you change the SQL Server service account passwords on one node, you must also change the passwords on all other nodes. SQL Server Configuration Manager does this automatically.
Problem: SQL Server cannot access the cluster disks
Issue 1: Firmware or drivers are not updated on all nodes.
Resolution 1: Verify that all nodes are using correct firmware versions and same driver versions.
Issue 2: A node cannot recover cluster disks that have migrated from a failed node on a shared cluster disk with a different drive letter.
Resolution 2: Disk drive letters for the cluster disks must be the same on both servers. If they are not, review your original installation of the operating system and Microsoft Cluster Service (MSCS).
Problem: Failure of a SQL Server service causes failover
Resolution: To prevent the failure of specific services from causing the SQL Server group to fail over, configure those services using Cluster Administrator in Windows, as follows:
Clear the Affect the Group check box on the Advanced tab of the Full Text Properties dialog box. However, if SQL Server causes a failover, the full-text search service restarts.
Problem: SQL Server does not start automatically
Resolution: Use Cluster Administrator in MSCS to automatically start a failover cluster. The SQL Server service should be set to start manually; the Cluster Administrator should be configured in MSCS to start the SQL Server service.
Problem: The Network Name is offline and you cannot connect to SQL Server using TCP/IP
Issue 1: DNS is failing with cluster resource set to require DNS.
Resolution 1: Correct the DNS problems.
Issue 2: A duplicate name is on the network.
Resolution 2: Use NBTSTAT to find the duplicate name and then correct the issue.
Issue 3: SQL Server is not connecting using Named Pipes.
Resolution 3: To connect using Named Pipes, create an alias using the SQL Server Configuration Manager to connect to the appropriate computer. For example, if you have a cluster with two nodes (Node A and Node B), and a failover cluster instance (Virtsql) with a default instance, you can connect to the server that has the Network Name resource offline using the following steps:
Determine on which node the group containing the instance of SQL Server is running by using the Cluster Administrator. For this example, it is Node A.
Start the SQL Server service on that computer using net start. For more information about using net start, see Starting SQL Server Manually.
Start the SQL Server SQL Server Configuration Manager on Node A. View the pipe name on which the server is listening. It should be similar to \\.\$$\VIRTSQL\pipe\sql\query.
On the client computer, start the SQL Server Configuration Manager.
Create an alias SQLTEST1 to connect through Named Pipes to this pipe name. To do this, enter Node A as the server name and edit the pipe name to be \\.\pipe\$$\VIRTSQL\sql\query.
Connect to this instance using the alias SQLTEST1 as the server name.
Problem: SQL Server Setup fails on a cluster with error 1058
Issue: Disabling Task Scheduler Service on cluster nodes causes Setup to fail with Error 1058. The following entry is in core.log:
Error: SetTargetComputer on \\machinename failed with Unable to start service (1058) Error: RunRemoteProcess Received return code 1058 from STPCOMPAQ3790N2
This is the last line of the log:
<EndFunc Name='DwLaunchMsiExec' Return='1058' GetLastError='183'>
Resolution: Use cluster manager to enable Task Scheduler Service on all cluster nodes. For more information, see How to: Enable Windows Task Scheduler Service.
Problem: SQL Server Setup fails on a cluster with error 11001
Issue: An orphan registry key in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.X\Cluster]
Resolution: Make sure the MSSQL.X registry hive is not currently in use, and then delete the cluster key.
Problem: Cluster Setup Error: "The installer has insufficient privileges to access this directory: <drive>\Microsoft SQL Server. The installation cannot continue. Log on as an administrator or contact your system administrator"
Issue: This error is caused by a SCSI shared drive that is not partitioned properly.
Resolution: Re-create a single partition on the shared disk using the following steps:
Delete the disk resource from the cluster.
Delete all partitions on the disk.
Verify in the disk properties that the disk is a basic disk.
Create one partition on the shared disk, format the disk, and assign a drive letter to the disk.
Add the disk to the cluster using Cluster Administrator (cluadmin).
Run SQL Server Setup.
Problem: Applications fail to enlist SQL Server 2005 resources in a distributed transaction
Issue: Because the Microsoft Distributed Transaction Coordinator (MS DTC) is not completely configured in Windows, applications may fail to enlist SQL Server 2005 resources in a distributed transaction. This problem can affect linked servers, distributed queries, and remote stored procedures that use distributed transactions.
Resolution: To prevent such problems, you must fully enable MS DTC services on the server where SQL Server 2005 is installed.
To fully enable MS DTC, use the following steps:
In Control Panel, open Administrative Tools, and then open Computer Management.
In the left pane of Computer Management, expand Services and Applications, and then click Services.
In the right pane of Computer Management, right-click Distributed Transaction Coordinator, and select Properties.
In the Distributed Transaction Coordinator window, click the General tab, and then click Stop to stop the service.
In the Distributed Transaction Coordinator window, click the Logon tab, and set the logon account NT AUTHORITY\NetworkService.
Click Apply and OK to close the Distributed Transaction Coordinator window. Close the Computer Management window. Close the Administrative Tools window.
|For installations of SQL Server 2005 on computers participating in a failover cluster, MS DTC must be fully enabled and clustered before you run Setup. If MS DTC is not clustered, Setup fails. Before running Setup, use the Microsoft Cluster Administrator to ensure that MS DTC has been clustered.|
Problem: SQL Server 2005 failover cluster Setup might fail when installing from CD
Issue: When selecting a failover cluster component and a client tools component during SQL Server 2005 Setup, the installation might fail with a "Remote Setup Failed" error after you insert CD 2.
Resolution: Copy installation files from both CDs to the same directory on the active cluster node or network share. For example,
c: cd\ md SQLENT
Copy both CDs to the SQLENT directory:
|This is not an issue with DVD installations.|
When you use extended stored procedures with a failover clustering configuration, all extended stored procedures must be installed on a SQL Server-dependent cluster disk. Doing so ensures that when a node fails over, the extended stored procedures can still be used.
If the extended stored procedures use COM components, the administrator must register the COM components on each node of the cluster. The information for loading and executing COM components must be in the registry of the active node in order for the components to be created. Otherwise, the information remains in the registry of the computer on which the COM components were first registered.