Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)

Article
07/22/2014

Applies to: FAST Search Server 2010

If the farm deployment includes a backup indexer row, you must manually change the deployment configuration if a non-recoverable server error occurs on the backup indexer server. You must also synchronize the indexer servers after you have replaced the failed server.

Follow these steps to replace a backup indexer server if a non-recoverable hardware failure has occurred:

Step 1: Pause item processing and crawling
Step 2: Install and configure a new backup indexer server
Step 3: Resume item processing and crawling

Step 1: Pause item processing and crawling

You must pause item processing and crawling before you reconfigure the indexer servers.

To pause item processing and crawling

Pause crawling:
- If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about pausing a crawl.
- If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about suspending a crawl.
- If you use another FAST Search specific connector, make sure that all crawling is paused or stopped.
Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.
1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.
  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.
  2. On the Start menu, click All Programs.
  3. Click Microsoft FAST Search Server 2010 for SharePoint.
  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.
2. Inspect the Web Analyzer schedule status:
```
waadmin ShowStatus
```
  Note
  
  Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.
  
  In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, perform the following steps:
  1. Set the view to scheduled state:
```
waadmin enqueueview
```
  2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and verify that the Schedule status is set to running.
3. Stop all Web Analyzer processing:
```
waadmin AbortProcessing
spreladmin AbortProcessing
```
Wait until all item processing is finished. On the administration server, type the following command:
```
psctrl status
```
Note

All processor servers must report "idle" before you proceed to the next step.

Step 2: Install and configure a new backup indexer server

Follow these steps to create a new backup indexer server to replace the failed server.

To install and configure a new backup indexer server

Verify that all indexing is finished. On the administration server, type the following command:
```
indexerinfo status -a
```
Verify that all index partitions have partition status="idle". Repeat this step until all index partitions are idle.
On the administration server, open the deployment configuration file: <FASTSearchFolder>\etc\config_data\deployment\deployment.xml.

Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

Locate the host element that specifies the failed indexer. The column attribute of the searchengine element indicates the index column for this indexer.

The following example shows an extract of a deployment.xml file that has the host definition for the primary and backup indexer for index column 0. The first host element specifies the primary indexer server. The second host element specifies the failed backup indexer server. The searchcluster element specifies which server that is the primary indexer and which server that is the backup indexer, by using the row element. Typically, row 0 corresponds to the primary indexer server.
```
<host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  ...
  <host name="fs4sp5.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>
  ...
  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>
```
Stop the primary indexer server. On the primary indexer server in the index column with the failed backup indexer server, type the following command:
```
nctrl stop indexer
```
Install the new server and reconfigure the deployment to change the indexer server configuration:
1. Change the host element that specifies the failed backup indexer server for this index column. Change the name attribute of the host element to specify the host name of the new server that will replace the failed backup indexer server.
  
  Note
  
  If you use the same host name for the new server, you do not have to modify the content of the file. However, you must modify the file for the Windows PowerShell cmdlet Set-FASTSearchConfiguration to redeploy the configuration. You can modify the file by opening it in Notepad, add a space and save the file.
2. Install the new server and reconfigure the deployment to add a backup indexer:
  
  Important
  
  When you perform this step, do not restart the new server after you have installed and configured it. You restart the server later in this procedure.
  
  Follow the steps in Reconfigure the deployment including adding new servers, but do not restart the new server.
3. If you have changed the host name of the new server, you must update the connection information in the SharePoint Server 2010 farm. Perform the steps in Make the necessary changes on Microsoft SharePoint Server.
Copy all FIXML files from the primary indexer server to the new backup indexer server.

The indexer uses the FIXML files as input to the indexing process. The FIXML files are stored in the following directory on the indexer servers:

<FASTSearchFolder>\data\data_fixml\

Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.
1. On the new backup indexer server, create two folders for the index data that you will copy. At the Windows command prompt, type the following commands:
```
mkdir <FASTSearchFolder>\data
mkdir <FASTSearchFolder>\data\data_fixml
mkdir <FASTSearchFolder>\data\ftStorage
```
2. On the new backup indexer server, at the Windows command prompt, copy the data_fixml folder from the source server:
```
robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <source_path>\data\data_fixml <FASTSearchFolder>\data\data_fixml
copy <source_path>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage
```
  Where <source_path> is the network path of the FAST Search Server 2010 for SharePoint installation folder on the primary indexer server.
  
  Note
  
  The /LOG: folder (\incoming) does not exist by default and you must have write access to the folder to run the command.
Restart the new backup indexer server.
Start the primary indexer server. On the primary indexer server in the affected index column, type the following command:
```
nctrl start indexer
```
Verify that the new backup indexer server is registered properly with the primary indexer server. At the administration server, type the following command:
```
indexerinfo --row=<rowNumber> --column=<columnNumber> status
```
<rowNumber> is the row number for the primary indexer row, typically 0.

<columnNumber> is the index column number.

Verify that the following status information is correct:
- indexer element: status="running ok"
- indexer/column_role element: state="master" backups="1"
  
  backups="1" indicates that the primary indexer server has identified the new backup indexer server.

Step 3: Resume item processing and crawling

Follow these steps to resume item processing and start a full crawl.

To resume item processing and crawling

Resume processing for the Web Analyzer components. On the administration server, type the following commands:
```
waadmin EnqueueView
spreladmin Enqueue
```
Start a full crawl:
- If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.
- If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.
- If you use another FAST Search specific connector, make sure that a full new crawl is initiated.
Note

We recommend starting a new full crawl, as there is a risk that items may be lost at the time of the primary indexer server failure. This will include items that are received correctly by the old primary indexer server just before the failure, but that have not yet been copied to the backup indexer server.

Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)

Step 1: Pause item processing and crawling

Step 2: Install and configure a new backup indexer server

Step 3: Resume item processing and crawling

See Also

Concepts

Other Resources

Additional resources