Performance and capacity tuning (FAST Search Server 2010 for SharePoint)

Applies to: FAST Search Server 2010

This article describes various methods of tuning your Microsoft FAST Search Server 2010 for SharePoint farm to achieve extended performance and capacity targets.

• Crawler capacity tuning

• Index capacity tuning

Crawler capacity tuning

Only tune the crawler capacity if your deployment:

• Doesn’t have a backup indexer row but more than 100 item processing component instances deployed per crawl component in the Content SSA

• Does have a backup indexer row and more than 50 item processing component instances deployed per crawl component in the Content SSA

• Does have more than 3 index columns per crawl component in the Content SSA

For large deployments, you must dimension the number of crawl components within the Content SSA to avoid network bottlenecks. Note that this scaling will often eliminate the need for additional configuration tuning. But for deployments with backup indexer rows, the batches submitted by the Content SSA typically exceed the number of item processing components. These "additional" batches are content that is processed but not yet been persisted in both indexer rows. By default, the Content SSA will throttle crawls in order to avoid more than 100 "batches submitted". For large installations, you should consider adjusting the throttling limits to allow for more batches to be processed at the same time.

You can adjust the throttling limits by using the MaxSubmittedBatches and MaxSubmittedPUDocs properties. The default value is 100 for MaxSubmittedBatches and 1000 for MaxSubmittedPUDocs. The throttling limits apply for each crawl component within the Content SSA. If you use two crawl components, the maximum total number of batches submitted will be two times the configured value. Use the following formula to calculate the new throttling limits:

For example, a scenario with a=1, b=48, c=3, s=2 will result in MaxSubmittedBatches = 54 and MaxSubmittedPUDocs = 5400. The default value for MaxSubmittedBatches is not altered in this case. If the crawling and indexing performance is limited by a high rate of ACL changes, you can increase MaxSubmittedPUDocs (the maximum number of items that have ACL changes submitted).

The MaxSubmittedBatches and MaxSubmittedPUDocs properties are configurable through the Microsoft SharePoint Server 2010 Management Shell on the farm hosting the Content SSA. The following commands set the default values:

$ssa = Get-SPEnterpriseSearchServiceApplication -Identity <nameOfContent SSA> 
$ssa.ExtendedConnectorProperties["MaxSubmittedBatches"] = 100 
$ssa.ExtendedConnectorProperties["MaxSubmittedPUDocs"] = 1000 
$ssa.Update()

where the <nameOfContentSSA> is the name of your Content SSA.

Index capacity tuning

Only tune the index capacity if your deployment:

• Has high content volumes where the number of content changes over time is low

• Has low query throughput requirements (maximum 5-10 queries per second, depending on CPU performance of the servers)

• Doesn’t require low indexing latency (there will be a longer delay from an item is updated until it is searchable)

FAST Search Server 2010 for SharePoint has a default configuration that is optimized for handling up to 15 million items per index column, with a hard limit of 30 million items. For scenarios with low content update rates and moderate query volume, you may want to index more items per column in order to save hardware cost. You can configure the FAST Search Server 2010 for SharePoint indexer to handle up to 40 million items per column. This is known as extended capacity configuration. The extended content capacity configuration has more index partitions within each server. In this manner, you can maintain low query latency at the expense of reduced maximum QPS.

There are some tradeoffs by extending the capacity: Query throughput (QPS) is reduced. Query latency (while not exceeding the throughput limitation) is less affected. You can compensate for query throughput reductions with multiple search rows, but then the reduction in server count is diminishing. Indexing will require more resources, and also more disk accesses. More items per column require more storage space per server. The total storage space across the farm is mainly the same. There are fewer servers for distributing item processing components. Initial crawl rate will be reduced, as the crawl rate is mainly dependent on the number of available CPU cores. Incremental crawls will also have lower throughput as each index column has more work. You can temporarily speed up initial pre-production bulk crawls by adding item processing components to eventual search rows or additional servers temporarily assigned to the cluster. Finally, you must have more hardware resources per server. We recommend that you do not use the extended settings on a server that has less than 16 CPU cores/threads (it should be 24 or more). We recommend 48 GB RAM and a high-performance storage subsystem.

Refer to Extend content capacity for the indexer (FAST Search Server 2010 for SharePoint) for the operational details.

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Performance and capacity testing (FAST Search Server 2010 for SharePoint)
Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)