Estimate performance and capacity requirements for SharePoint Foundation 2010 Search

 

Applies to: SharePoint Foundation 2010

This article discusses capacity planning information for a deployment of Microsoft SharePoint Foundation 2010 search.

In this article:

  • Introduction

  • Test farm characteristics

  • Health and performance data

  • Test results

  • Recommendations

  • Troubleshooting performance and scalability

Introduction

This article describes a specific deployment of SharePoint Foundation 2010 and includes the following specifications that were used during the performance and capacity testing of SharePoint Foundation 2010 :

  • Test environment specifications, such as hardware, farm topology, and configuration

  • The workload used for data generation that includes the number and class of users, and farm usage characteristics

  • Test farm dataset, which includes database contents, search indexes, and external data sources

  • Health and performance data that is specific to the tested environment

This article also contains common test data and recommendations for how to determine the hardware, topology, and configuration that you need to deploy a similar environment, and how to optimize the environment for appropriate capacity and performance characteristics.

SharePoint Foundation 2010 Search provides basic search functionality for local SharePoint Foundation content. If you need more advanced search functionality, such as indexing local shared folders and remote SharePoint Foundation or HTTP content, use Microsoft Search Server Express 2010 or Microsoft SharePoint Server 2010 search.

This article describes how to do the following:

  • Define performance and capacity targets for the environment.

  • Plan the hardware that is required to support the number of users and the features that you intend to deploy.

  • Design the physical and logical topology for optimal reliability and efficiency.

  • Test, validate, and scale up or scale out the environment to achieve performance and capacity targets. To increase the capacity and performance of one of the starting-point topologies, you can do one of two things. You can either scale up by increasing the capacity of the existing servers or scale out by adding additional servers to the topology.

  • Monitor the environment for key indicators.

Test farm characteristics

This section describes the scenario of the test; the dataset that was used during the testing; the workloads that were put on SharePoint Foundation during performance gathering; the hardware that was used during the testing; and the topology for how that hardware was deployed.

This SharePoint Foundation Search scenario describes a typical search deployment, with the following characteristics:

  • Search is provided for the SharePoint Foundation content in the farm. No external content can be added to the index (for example, intranet Web sites or file systems).

  • Search can be run on only one server in the farm at a time. Scaling out is not supported.

  • Limited system administration is available for search.

Dataset

This section describes the test farm dataset, which includes database contents and sizes, search indexes, and external data sources.

Search index size (number of items)

5677481

Size of search database

54 GB

Size of search log file

2 GB

Size of content database

161 GB

Size of content database log file

10.6 GB

Size of index partition

23.7 GB

Total number of search databases

8

Other databases that were used during performance and capacity testing of SharePoint Foundation 2010

SharePoint_Config; SharePoint_AdminContent; State_Service; Bdc_Service_db; WSS_UsageApplication; WSS_Content

Hardware settings and topology

This section describes the kinds of computer hardware that we used in our lab and the farm configuration topology that we used in our tests.

Lab hardware and topology

This section describes the hardware that was used for testing.

Important

Because the farm was running pre-release versions of SharePoint Foundation 2010, the hardware that was used has more capacity than is required under more typical circumstances.

Application server and content server

Server Combined

Processors

1px4c@ 3 GHz

RAM

16 GB

Operating system

Windows Server 2008 R2, 64-bit

Storage

6 x 450 GB serial attached SCSI (SAS)

Number of network adapters

2

Network adapter speed

1 gigabit

Authentication

NTLM

Load balancer type

None

Software version

SharePoint Foundation 2010 (pre-release version)

Services running locally

Microsoft SharePoint Foundation Workflow Timer Service; SharePoint Foundation Search

Object cache settings

Default (100-MB maximum); cache for 60 seconds; clmq multi: 3

Microsoft SQL Server computer

Server Combined

Processors

2px4c@ 3 GHz

RAM

16 GB

Operating system

Windows Server 2008 R2, 64-bit

Storage

6 x 450 GB SAS

Number of network adapters

2

Network adapter speed

1 gigabit

Authentication

NTLM

Load balancer type

None

Software version

SQL Server 2008 Enterprise

Services running locally

SQL Server Service

Usage database location

Shared drive with other databases

Topology

The test farm is a two-server farm. One server is the Web and application server, and the second server is the database server.

Topology of the test farm

Workload

This section describes the workload used for data generation. Workload characteristics include the number of users and farm usage characteristics.

Workload characteristics Value

High-level description of workload (collaboration, Web content management, social)

Search farm and content farm

Average queries per minute

Not applicable

Average concurrent users

Not applicable

Total number of queries per day

Not applicable

Timer job load

Timer jobs: Immediate Alerts, SharePoint Foundation Search Refresh, Health Analysis Job

Health and performance data

This section provides health and performance data that is specific to the test environment.

Query performance data

The following measurements were taken with 140,000 items in the search index. The columns give the measurements taken during the specific test. The results are at the bottom of the table. The CPU memory and IOPs were read directly from the Perfmon counters. The latency was measured by using a Microsoft Visual Studio Team System 2008 Database Edition project, which executed a set of approximately 2,000 queries on a team site.

Metric category Scorecard metric Query test

CPU

Average SQL Server CPU

21.83

Performance counters

Average front-end Web server, query server, indexer CPU

95.84

ASP.NET requests queued (average of all front-end Web servers)

Not applicable

SQL Server locks: average wait time (ms)

0.342

SQL Server locks: lock wait time (ms)

0.171

SQL Server locks: deadlocks/sec

0

SQL Server latches: average wait time (ms)

0.147

SQL Server compilations/sec

191.75

SQL Server statistics: SQL Server recompilations/sec

0.075

Average disk queue length (query server)

0.007

Disk queue length: reads (query server)

Not applicable

Test results

Disk queue length: writes (query server)

0.007

Disk reads/sec (query server)

0.188

Disk writes/sec (query server)

125.676

Average disk queue length (indexer, front-end Web server, query server)

0.006

Disk queue length: reads (indexer, front-end Web server, query server)

Not applicable

Disk queue length: writes (indexer, front-end Web server, query server)

0.003

Disk reads/sec (indexer, front-end Web server, query server)

0.706

Disk writes/sec (indexer, front-end Web server, query server)

11.79

Average memory used (indexer, front-end Web server, query server)

14.129

Maximum memory used (indexer, front-end Web server, query server)

14.331

Cache hit ratio (SQL Server)

100

Test results

Query UI latency (75th percentile) (sec)

0.128

Query UI latency (95th percentile) (sec)

0.151

Query throughput (80 user load) (requests/sec)

60.86

Crawl performance data

The following measurements were taken during full crawls of the given content source. (Content source size is given in millions of items.) The columns show the measurements taken during the specific crawl, and the results are at the bottom of the table.

Metric category Scorecard metric Crawl test to reach 140,000 index size

CPU

Average SQL Server CPU

9.48

Performance counters

Average front-end Web server, query server, indexer CPU

87.32

ASP.NET requests queued (average of all front-end Web servers)

Not applicable

SQL Server locks: average wait time (ms)

13.45

SQL Server locks: lock wait time (ms)

11.39

SQL Server locks: deadlocks/sec

0

SQL Server latches: average wait time (ms)

3.03

SQL Server compilations/sec

27.8

SQL Server statistics: SQL Server recompilations/sec

1.048

Average disk queue length (SQL Server)

4.524

Disk queue length: reads (SQL Server)

Not applicable

Test results

Disk queue length: writes (SQL Server)

3.918

Disk reads/sec (SQL Server)

71.76

Disk writes/sec (SQL Server)

349.485

Average disk queue length (indexer, front-end Web server, query server)

0.058

Disk queue length: reads (indexer, front-end Web server, query server)

Not applicable

Disk queue length: writes (indexer, front-end Web server, query server)

0.049

Disk reads/sec (indexer, front-end Web server, query server)

3.295

Disk writes/sec (indexer, front-end Web server, query server)

138.83

Average memory used (indexer, front-end Web server, query server)

15.28%

Maximum memory used (indexer, front-end Web server, query server)

15.84%

Cache hit ratio (SQL Server)

99.865

Test results

Total crawl speed (items/sec)

171.73

Number of successes

138590

Number of errors

12

Test results

This section provides test data that show how the farm performed under load at different index sizes.

Query latency

The following graph displays the query latency percentiles for this farm as user load increases. A query percentile of 95 percent means that 95 percent of the query latencies measured were less than that value.

Graph of query latency compared against number of

From this graph, you can see that increasing the size of the content crawled by this farm will affect query latency as the user load exceeds 20 concurrent users.

With approximately 5 million items in the index, the query latencies are very high, as shown in the following graph.

Graph of query latency as user load increases

Query throughput

The following graph displays the query throughput for this farm as user load increases.

Graph of query throughput as user load increases

This graph shows that increasing the size of the content crawled by this farm will affect query throughput significantly, increasing it as the load exceeds about 10 concurrent users.

With approximately 5 million items in index, query throughput may be severely decreased as shown in the following graph.

Graph of query throughput compared against number

Crawl rate

The following graph displays the crawl rate for this farm during the index acquisition stage at different existing index sizes. Notice that the crawl speed decreases as the index grows larger.

Graph of crawl rate during index acquisition

Recommendations

This section describes specific actions that you can take to optimize the environment for appropriate capacity and performance characteristics.

Hardware recommendations

For specific information about minimum and recommended system requirements, see Hardware and software requirements (SharePoint Foundation 2010).

Scaled-up and scaled-out topologies

The search subsystem for SharePoint Foundation 2010 cannot be scaled out. In general, we recommend that if there are more than 1 million items to be indexed, you should use Microsoft Search Server 2010 or SharePoint Server 2010 Search.

Some of the problems with low query throughput and high latency can be avoided by scaling up the CPU, RAM, and IOPS on the application servers and SQL Server computers.

Troubleshooting performance and scalability

To help you determine when you should scale up or scale out the system, use performance counters to monitor the health of the system.

Performance monitoring

Use the information in the following tables to determine which performance counters to monitor, and to which process the performance counters should be applied.

Web and application servers

The following table shows performance counters and processes to monitor for Web servers in the farm.

Performance counter Apply to object Notes

Processor time

Total

Shows the percentage of elapsed time that this thread used the processor to execute instructions.

Memory utilization

Application pool (W3wp.exe)

Shows the average utilization of system memory for the application pool. You must specify the correct application pool (instance of W3wp.exe) to monitor. The guideline is to determine peak memory utilization for a given Web application, and assign that number plus 10 to the associated application pool.

Database servers

The following table shows performance counters and processes to monitor for database servers in the farm.

Performance counter Apply to object Notes

Average disk queue length

Hard disk that contains SharedServices.mdf

Average values larger than 1.5 per spindle indicate that the write times for that hard disk are insufficient.

Average disc sec/transfer

Hard disk that contains SharedServices.mdf

The average time used to transfer data to and from the disc.

  • Less than 20 ms is excellent.

  • 20–40 ms is good.

  • More than 50 ms means that it needs optimization by adding IOPS.

Processor time

SQL Server process

Average values larger than 80% indicate that processor capacity on the database server is insufficient.

Memory utilization

Total

Shows the average utilization of system memory.

Buffer cache hit ratio

SQL Server Buffer Manager

Buffer cache hit ratio less than 96% indicates a memory bottleneck. Add more RAM to resolve the problem. This ratio should ideally be more than 98%.

About the author

Brion Stone is a Senior Program Manager for SharePoint Server search at Microsoft.