FSA: Queries always fail on a specific query processing node (FAST Search Server 2010 for SharePoint)

 

Applies to: FAST Search Server 2010

Issue: Queries always fail on a specific query processing node.

Summary: If Microsoft FAST Search Server 2010 for SharePoint does not always work, determine whether queries are failing on one specific query processing node. If that is true, the cause might be that the FAST Search Server 2010 for SharePoint query processing service is not running on the given node, the FAST Search Server 2010 for SharePoint query processing node is not set up correctly to communicate with Microsoft SharePoint Server 2010, the FSA worker process is not up to date with configuration changes, or the QRProxy service cannot communicate with FSA. We recommend that you work through each possible resolution in the given order.

Symptom: Queries always fail on a specific query processing node.

Causes: This symptom can be the result of the following causes:

  • Cause 1: The FAST Search Server 2010 for SharePoint query processing services are not running

  • Cause 2: The FAST Search Server 2010 for SharePoint query processing node might not be set up correctly to communicate with SharePoint

  • Cause 3: The FSA worker process is not up to date with configuration changes

  • Cause 4: The QRProxy service cannot communicate with FSA

Cause 1: The FAST Search Server 2010 for SharePoint query processing service is not running

Resolution: Restart FSA

Verify that FAST Search Server 2010 for SharePoint is running:

  1. Log onto the server and open a FAST Search Server 2010 for SharePoint command prompt.

  2. Run the command: nctrl status.

  3. Verify that these processes are running: samworker, qrproxy, and qrserver.

If any of these processes are not running, restart FSA. For more information, refer to Restart FSA.

Cause 2: The FAST Search Server 2010 for SharePoint query processing node might not be set up correctly to communicate with SharePoint

To verify the setup of FAST Search Server 2010 for SharePoint, follow the guidance in FSA: The search request was unable to connect to the Search Service (FAST Search Server 2010 for SharePoint)

Cause 3: The FSA worker process is not up to date with configuration changes

Resolution: Ensure that all configuration changes have been propagated to all query processing nodes.

When configuration changes are made to FSA, the changes must be copied to all the query processing nodes.

Configuration changes occur when you run the Windows PowerShell cmdlets with the prefix verb-FASTSearchSecurity. For example, the Set-FASTSearchSecurityLogLevel cmdlet updates the configuration with the log level setting. When a FAST Search for SharePoint Sam Worker is restarted, it must wait for the FAST Search for SharePoint Sam Admin process to communicate with it and to verify that it has all the configuration changes. This process may take several minutes depending on how many query processing nodes are in the system and how many changes occurred while the node was down.

To verify that all the worker nodes have the most recent configuration changes, follow these steps:

  1. Open a FAST Search Server 2010 for SharePoint command prompt

  2. Run Get-FASTSearchSecurityWorkerNode

  3. Verify that the status for all nodes is active

If any node is not active, wait for several minutes and recheck. If it is still not active after 5 to 10 minutes, increase the log file levels and look in the logs for both the FAST Search for SharePoint Sam Worker and FAST Search for SharePoint Sam Admin.

Cause 4: The QRProxy service cannot communicate with FSA

Does the event viewer contain a message for the FAST Search QRProxy that says, “An unhandled exception occurred: Unable to connect to SAM”? You may receive this error message because the FAST Search for SharePoint Sam Worker is not started, because the FAST Search for SharePoint Sam Worker does not have the most recent configuration changes, or because a WCF error is preventing it from communicating. If the FAST Search for SharePoint Sam Worker is running and has the most recent configuration, restart QRProxy. If the problem still exists, verify that the FAST Search for SharePoint Sam Worker is ready to receive requests. If restarting the FSA worker and QRProxy does not resolve the problem, and the FAST Search for SharePoint Sam Worker is ready to receive requests, enable WCF tracing to diagnose the error.

Resolution 1: Restart QRProxy

  1. Open a FAST Search Server 2010 for SharePoint command prompt.

  2. Run the command: nctrl restart qrproxy.

  3. After the QRProxy service is started, retry the search.

Resolution 2: Ensure that the FAST Search for SharePoint Sam Worker is ready to receive requests

Follow these steps on the node where the QRProxy and FAST Search for SharePoint Sam Worker communication is not working.

  1. Open the FAST Search Server 2010 for SharePoint command prompt as an administrator and run the following commands:

    set-alias installutil $env:windir\Microsoft.NET\Framework64\v2.0.50727\installutil
    
    installutil Microsoft.SharePoint.Search.Extended.Security.Worker.PowerShell.Commands.dll
    
    add-pssnapin FASTSearchSecurityWorkerSnapIn
    
  2. Modify the configuration file base port value.

    There is a known issue which requires a minor change to the file at <FASTSearchFolder>\bin\Microsoft.SharePoint.Search.Extended.Security.Worker.Powershell.Commands.dll.config. This file includes a base_port XML element. Add one to the value and save the file. For example, if the value is 13000, change the value to 13001 and save the file.

  3. Run the cmdlet:

    Get-FASTSearchSecurityConfigurationStatus
    

If the return value from this cmdlet is false, increase the log levels and look at the FSA worker logs.

Resolution 3: Enable WCF tracing to diagnose the error

To enable WCF tracing between QRProxy and the FSA worker, follow these steps.

  • Modify <FASTSearchFolder>\bin\Microsoft.SharePoint.Extended.Security.WorkerService.exe.config and <FASTSearchFolder>\bin\QRProxyService.exe.config. Remove the comments <!-- and --> from around the <diagnostics> and <system.diagnostics> XML elements in both files.

  • Save the files.

  • Restart both the FSA worker and QRProxy. This will create two files in the <FASTSearchFolder>\bin with extension .svclog.

You can find more information about how to view WCF trace files by searching TechNet.