Capacity Model for Internet Transactions

Microsoft Corporation

	Abstract
	Introduction
	Conventional Capacity Planning Versus TCA
	Transaction Cost Analysis Requirements
	Transaction Cost Analysis Methodology
	Capacity Planning Using Transaction Cost Analysis
	Conclusion
	Appendix A - Recommended Reading

Abstract

The purpose of capacity planning for Internet services is to help you to deploy Internet sites that accomplish the following:

• Minimize the total dollar cost of ownership of the host platform
  
• Support transaction throughput targets
  
• Remain within acceptable response-time bounds
  

Conventional solutions to capacity planning often attempt to determine the cost of Internet services by extrapolating generic benchmark measurements. To better meet the stated objectives of capacity planning, a methodology based on Transaction Cost Analysis (TCA) has been developed at Microsoft for estimating system capacity requirements.

TCA methodology simulates client transactions on the host server by a load generation tool that supports standard network protocols. By varying the client load, transaction throughput is correlated with resource utilization over the linear operating regime. A usage profile is then defined based on the anticipated behavior of users, such as browsing habits. This usage profile determines the throughput target and other important transaction parameters from which resource utilization and capacity requirements are then calculated.

Top of page

Introduction

The TCA methodology is applicable to any client-server system; however, this paper focuses on applying TCA to Internet services.

The section, "Conventional Capacity Planning Versus TCA" outlines a conventional solution to capacity planning and compares it against a methodology based on transaction cost analysis (TCA). In the section, "Transaction Cost Analysis Methodology," the methodology of TCA is described in detail, including a discussion of the usage profile, instrumentation of probes, performance measurement, calculation of resource costs, and verification of the model. The section "Capacity Planning Using Transaction Cost

Top of page

Conventional Capacity Planning Versus TCA

Two possible approaches to capacity planning include the conventional approach and the TCA approach, as seen in Figure 1.

Conventional Approach

Conventional approaches to capacity planning often involve benchmarking the maximum workload of some specific combination of services on specific hardware, each with a particular usage profile. This process is incomplete and time-consuming, because it is not practical to iterate through all possible hardware, service, and usage profile permutations. A common response to this limitation is to determine these benchmarks for only a moderate number of representative deployment scenarios, and then attempt to estimate costs by extrapolating these benchmarks to fit each particular deployment scenario. This approach also increases the risk of over-provisioning hardware, because these benchmarks are designed to capture only the maximum workload attainable.

Transaction Cost Analysis Approach

Transaction Cost Analysis (TCA) attempts to reduce the uncertainty inherent in the cost analysis process by providing a framework for estimating each resource cost as a function of any usage profile, service mix, or hardware configuration. Hardware resources include, but are not necessarily limited to, CPU, cache, system bus, RAM, disk subsystem, and network. By using TCA methodology, the potential for over-provisioning hardware is also reduced, because the entire (linear) workload range is measured. Finally, unlike some code profiling techniques, TCA captures all costs present during run time, such as operating system overhead.

The TCA methodology is applicable to any client-server system, but this white paper focuses on its application to Internet services. See Appendix A for other recommended reading concerning capacity planning and performance.

Top of page

Transaction Cost Analysis Requirements

The requirements for performing Transaction Cost Analysis (TCA) include:

• Create the usage profile, the scalability limits, and the latency requirements as success criteria.
  
• Instrumentation must be built into the service being measured that enables it to be monitored.
  
• Performance monitoring tools for data collection and analysis, such as Windows® 2000 System Monitor.
  
• Load generation tools that support user simulation of the service, such as the Application Center Test (ACT) tool which is available as part of Microsoft® Visual Studio® .NET, or Mercury Interactive LoadRunner.
  
• Load generation and usage profile scripts and the hardware to simulate the load.
  

Hardware that is similar in class to the deployment hardware, which will be used as your production server, to run the service component being evaluated.

Top of page

Transaction Cost Analysis Methodology

Transaction Cost Analysis (TCA) can be used for capacity planning and can also detect performance bottlenecks in your Internet service. In many cases, TCA is used for both purposes simultaneously. However, this paper discusses using TCA for capacity planning only.

For simplicity, the methodology presented here makes the following assumptions:

• The service being analyzed has no code bottleneck.
  
• The service scales linearly (approximately) in implementation.
  
• The hardware bottleneck resides on the computer running the service.
  

These assumptions are necessary in order to accurately assess capacity related information.

Usage Profile

The effectiveness and flexibility of the capacity planning model depends on a careful assessment of the expected usage profile for each service being evaluated. This usage profile consists of both individual and aggregated user behavior, and site profile information. Analysis of the transaction logs of similar services provides a helpful starting point when defining a usage profile. Characteristics that are derived from this usage profile are then used to establish performance targets. For more information about creating a usage profile, see the white paper, Commerce Server 2002: Creating a Usage Profile for Site Capacity Planning, available from the same Web page as this white paper.

Site Profile Information

A representative site profile must be defined that specifies the following:

• Services deployed
  
• Number of concurrent users using each service
  
• Expected deployment configuration
  

Note: The expected deployment configuration should identify each service component, as well as the server on which each service component will reside.

The following table lists the notation used in TCA equations throughout this white paper:

Scenario

An Internet Service Provider is interested in deploying a Web hosting platform that supports browsing of subscriber Web pages across the Internet. This platform also provides subscribers with the use of a File Transfer Protocol (FTP) service to control file transfer of these Web pages to and from the host site.

The services required to support this scenario consist of Web (HTTP), FTP, directory (for example, LDAP), and database (for example, Microsoft® SQL Server™) services. The Web and FTP services are configured on front-end servers, while the directory and database services each reside on separate back-end servers. To simplify the subsequent discussion of this scenario we will analyze the Web and FTP services only.

User Behavior

Individual user behavior must be characterized to include the following:

The following table lists the notation used in TCA formulas throughout this white paper:

Scenario

Continuing with the previous scenario, it is anticipated that of all possible FTP transactions only "delete," "get," "open," and "put" will generate the most significant load in aggregate. Therefore, only these transactions need to be considered in the analysis. In particular, FTP "open" may stress a directory service such as LDAP and its database back-end if the connection requires an authentication sequence. FTP "get" and "put" may stress disk input/output (I/O) resources and saturate network capacity while FTP "delete may only stress disk I/O resources, and so on. Similar deductions are made for the HTTP "get" transaction. Therefore, the fundamental transactions for these analyses consist of "delete," "get," "open," and "put" for FTP, and "get" for HTTP.

Clearly, the size of files transferred or deleted using FTP, and the Web page size requested using HTTP are important transaction parameters to consider.

Performance Targets

The performance targets consist of the minimum required transaction throughput and maximum acceptable transaction latency. The minimum throughput (transaction rate) of transaction j for service i required to support this usage profile is given by Equation 1:

Equation 1

Tij = nij Ni / ti

Where:

• T_ij is the transaction rate of transaction j for service i.
• n_ij is the number of transactions (j) per user session for service i.
• N_i is the number of users concurrently using service i at the peak time.
• t_i is the user session time for service i.

Scenario

In this scenario, the total Web page audience is 2 million users with 1.0 percent of these users concurrently browsing Web pages at peak time. Furthermore, each web user performs 5 HTTP "get" operations over 20 minutes, and each FTP user performs 3 FTP "put" and 2 FTP "get" operations together over a 5-minute interval.

The following summarizes this data:

N_HTTP = 20,000 concurrent users = (2,000,000 * 1.0 percent)

t_HTTP = 1,200 seconds

n_HTTP,get = 5 HTTP gets per user

Use the following formula to calculate the total number of HTTP GET requests for the given time interval based on the information listed above:

n_HTTP,_get * N_HTTP = n_ij N_i = (5 *20,000) = 100,000 HTTP gets

Using the formula from Equation 1, you can now calculate the transaction rate (per second) of GET requests for the HTTP service:

T_ij=n_ij N_i / t_i = (100,000 / 1,200) = 83 HTTP gets per second

Now suppose the same Internet site has 1 million Web hosting subscribers, and that 0.1percent of them are concurrently using FTP at peak time. Each FTP user performs 3 FTP "put" and 2 FTP "get" operations together over a 5 minute interval. The following summarizes this data:

N_FTP = 1000 concurrent users = (1,000,000 * 0.1 percent)

t_FTP = 300 seconds (5 minutes)

n_FTP,put = 3 FTP puts per user

n_FTP,get = 2 FTP gets per user

Based on the information listed above you can calculate the total number of FTP "put" and FTP "get" requests over the given time interval.

Use the following formula to calculate the total FTP "put" requests:

N_FTP * n_{FTP, put=}n_ij N_i = (3 * 1,000) = 3,000 FTP puts

Use the following formula to calculate the total FTP "get" requests:

N_FTP * n_FTP, _get=n_ij N_i = (2 * 1,000) = 2,000 FTP gets

Using the formula from Equation 1, you can now calculate the transaction rate (per second) for the FTP service:

T_{FTP, put} = n_ij N_i / t_i = (3,000 / 300) _{= 10 FTP puts per second}

T_{FTP, get} =n_ij N_i / t_i = (2,000 / 300) _{= 6.7 FTP gets per second}

Instrumentation of Probes

Transaction Cost Analysis (TCA) requires correlating resource utilization with service run-time performance. Although adequate instrumentation to enable the measurement of resource utilization is often already built into the operating system, the metrics to measure the performance of service transaction characteristics must also be defined, and then built into the probing apparatus. Figure 2 illustrates the intended purpose of these probes in the context of capacity planning and performance management.

Windows 2000 System Monitor

Microsoft® Windows 2000® System Monitor is equipped with probes, also referred to as counters, that measure resource utilization and other system parameters as well as transaction flow, state, queue, and latency parameters for each service.

Note: The aggregate of Windows 2000 System Monitor and Performance Logs and Alerts replaces the functionality of the Windows NT version 4 Performance Monitor.

Some System Monitor counters commonly collected during TCA (and as part of regular performance monitoring) include:

• Windows 2000 Operating System Counters

  These counters include processor utilization, context switching rate, processor and disk queue lengths, disk read and write rates, available memory bytes, memory allocated by process, cache hit ratios, network utilization, network byte send and receive rates, and so on.

• Transaction Flow Counters

  These counters include transaction rates such as HTTP gets/sec, FTP puts/sec, LDAP searches/sec, and so on.

• ServiceState, Queue, and Latency Counters

  These counters include concurrent connections, cache hit ratios (for example, for LDAP and SQL Server transactions), queue length, and latency within each service component, and so on.

See the Microsoft Windows 2000 Sever Resource Kit (Microsoft Press, 2000) for more specific examples of Windows NT counters.

Performance Measurement

TCA methodology imposes strict performance measurement requirements related to configuration of the testing environment and generation of simulated user load. This section discusses both system configuration and load generation.

System Configuration

To leverage TCA measurements made with one hardware configuration into variable configuration contexts, it is advantageous to separate service components among servers as much as possible. An un-cached environment enables better isolation of resource costs for each transaction as a function of each component servicing the transaction.

The separation of resource costs of transactions by service component is illustrated in Figure 3, which shows the propagation of a single transaction (for example, LDAP "add") between client and servers and its accumulated resource costs. These resource costs are shown in Figure 3 as C1 on Server 1 (LDAP Service running here) and C2 on Server 2 (SQL Server running here).

If all of the components servicing this transaction reside on a single server for a particular deployment, then these separate costs may be added together as C1+C2 in order to roughly estimate the integrated cost. Of course, certain performance properties will change in this case. For example, certain costs disappear, such as the CPU overhead for managing the network connection. On the other hand, other costs may increase, such as disk Input/Output (I/O), because the capacity for caching transaction requests is diminished (which reduces throughput), and so on. When caching plays a particularly important role, variable configurations should be analyzed separately.

As illustrated in Figure 4, the load generation clients and performance-monitoring tools should reside on computers other than the servers under analysis. This minimizes the impact on the servers being monitored, and results in a more accurate analysis.

The service configuration should be deployed with the highest performance hardware available. This helps to ensure that resource requirements based on measurements on higher-performance hardware will scale down (at worst linearly) to lower performance hardware. However, the converse is not necessarily true.

For example, multi-processor scaling is often sub-linear, in which case total CPU costs on a four-way SMP server may be greater than on a two-way SMP server under the same load (due to additional context switching and bus management overhead). Suppose CPU costs are measured on a 500 MHz four-way SMP server and the actual deployment hardware consists of a 300 MHz two-way SMP server. Then if all other hardware characteristics remain unchanged, CPU requirements should increase by no more than a factor of 3.3 = (500 MHz/300 MHz)*(4 processors/2 processors). Thus, CPU requirements for a server with a given number of processors can be estimated based on measurements taken from a server with a greater number of processors.

Load Generation

Load generation scripts must be written to simulate each transaction separately, and have the following form:

do transaction

sleep rand timeInterval

The sleep is intended to reduce load on the client queue, and timeInterval should be chosen randomly over a representative range of the client-server transaction latencies. The load generation scripts must be distributed among enough clients to prevent any single client from becoming its own bottleneck.

For more information about choosing a random time interval, the following book is recommended: The Benchmark Handbook for Database and Transaction Processing Systems by Jim Gray, Morgan Kaufman Publishers, 1992

Before each test run, the system should be shut down to flush caches and restore the system to a consistent state for data collection consistency. Furthermore, each test run should continue for at least long enough to reach run-time "steady-state." This state is reached when, for example, initial transient (start-up) costs are absorbed, network connections are established, caches are populated as appropriate, and periodic behavior is fully captured. The measurements collected for each run should be time averaged over the length of the run in "steady-state."

In addition to collecting measurements of resource utilization and throughput, counters that help to isolate points of contention in the system should be monitored. These include queue lengths, context switching, out of memory paging, network utilization, and latencies. In particular:

• On multiple disk Redundant Array of Inexpensive Disks (RAID) arrays, the average disk queue length per array should not exceed the number of physical disks per array.
  
• On Ethernet networks, the layer-2 Carrier Sense Multi-Access/Collision Detection (CSMA/CD) protocol attempts to enforce network card utilization to less than 36 percent to deliver maximum shared network throughput. Network utilization in excess of 36 percent should be approached with caution.
  

For more information about performance considerations that are specific to Microsoft Windows 2000, the following book is recommended: Tuning Netfinity Servers for Performance: Getting the Most out of Windows 2000 and Windows NT 4.0 by David Watts,Chris Neophytou,Murat Gulver,Gregg McKnight,Peter Mitura, Prentice Hall PTR, July 2000.

The number of concurrent users should start small and increase incrementally up to N^max at which point the transaction throughput begins to decrease from its maximum at T^max. This decrease in throughput is due to factors such as high rates of context switching, long queues, and out of memory paging, and often corresponds to the point at which transaction latency becomes nonlinear.

These relationships are depicted in Figure 5. Each circle (O) and (X) in this figure represents a run and are points at which data is collected during the load generation. C_max^resource denotes the maximum resource capacity.

Cost Equations

This section describes how to calculate resource costs for both individual and total transactions. The following notation is used throughout this section to describe these calculations:

Resource Costs for each Transaction

Each measured transaction rate (T_ij) corresponds to a measured utilization of each resource denoted by C_ij^resource. These data point pairs are interpolated over the linear operating regime to construct an analytic relationship. The relationship expresses the resource utilization as a function of transaction rate. This relationship is shown in Figure 6.

For example, C_ij^CPU(T_ij ; other) denotes the number of CPU cycles consumed by transaction j for service i with transaction rate T_ij, where other is a placeholder for other transaction parameters. For example, file size for HTTP "get".

C_ij^resource is defined over the transaction rate range 0 <= T_ij <= T_ij^max, but if T_ij > T_ij^max, then the equation for C_ij^resource may still be applied with the interpretation that hardware needs to be scaled up linearly. In this case, it is especially important that the TCA assumptions listed at the beginning of the section "Transaction Cost Analysis Methodology" be satisfied for this interpretation to be valid.

Total Resource Costs

It is assumed that resource utilization adds linearly in a mixed environment in which multiple services are running and multiple transactions are being processed by each service. Equation 2 calculates the total cost for each resource and for all transactions:

Equation 2

C_total^resource = å_i=1^I å_j=₁^JC_ij^resource( T_ij; other)

For performance reasons and in order to account for unexpected surges in resource utilization, it is advantageous to operate each resource at less than full capacity. A threshold factor for each resource is introduced to reflect this caution in the provisioning of resources. This factor, also referred to as "head-room," represents the percentage of resource capacity that should not be exceeded.

The following table lists the notations used to describe these threshold factors:

Equation 3 calculates the total number of processors required to support this transaction mix without exceeding the CPU utilization threshold:

Equation 3

C_total^CPU / ( q^CPU C_max^CPU )

Where C_max^CPU denotes the peak clock speed per processor. q^CPU is typically chosen between 60 percent and 80 percent.

Similarly, Equation 4 calculates the total number of spindles required:

Equation 4

C_total^reads / (q^readsC_max^reads) + C_total^writes / (q^writesC_max^writes)

Where C_max^reads and C_max^writes denote the maximum number of disk reads/sec and disk writes/sec per spindle, respectively. Similar equations can be deduced for the other hardware resources.

It should be noted that the disk subsystem must be calibrated to determine C_max^reads and C_max^writes. The disk calibration process performs a large number of un-cached reads and writes to the disk to determine the maximum number of reads and writes the disk array can support. In hardware, C_max^reads and C_max^writes are most strongly a function of disk seek time and rotational latency.

In light of the layer-2 CSMA/CD protocol, q^network is often set to 36 percent of the total bandwidth of your Ethernet network.

Model Verification

The actual deployment scenario will consist of a mixed transaction environment as defined by the usage profile. The purpose of verification is to simulate this deployment scenario in order to (1) confirm that transactions do not negatively interfere with each other, and (2) verify that the resource costs estimated from the cost equations, (see the Total Resource Costs section above), accurately reflect the actual costs measured.

Verification Script

The usage profile is translated into a verification script for each service. (There is nothing in principle to prevent creating a single verification script to invoke all services.)

For a single service with multiple transactions, this script logic can be written in pseudo-code as:

count < 0
while ( count < ti )
{ if ( count mod ( ti / ni ,1 ) = 0 ) do transaction 1
if ( count mod ( ti / niJ ) = 0 ) do transaction J
sleep sleepIncrement
count < count + sleepIncrement
}
sleep rand smallNumber 

Here sleepIncrement = GCD( t_i / n_i1 ,…, t_i / n_iJ ) where GCD denotes the Greatest Common Divisor (GCD) and t_i / n_iJ is the length of time that must elapse between initiating each transaction j using protocol i. For each transaction j, this script initiates the correct number of transactions n_iJ uniformly over the length of the service session time t_i as defined by the usage profile. The statement sleep randsmallNumber is included to randomize the transaction arrival distribution.

For more information, about benchmarking transaction processing systems, the following book is recommended:The Benchmark Handbook for Database and Transaction Processing Systems by Jim Gray, Morgan Kaufman Publishers, 1991.

Load Generation Script

The load generation process is the same as that described in the section "Load Generation," except that only one test run is made for each simulated usage profile. During this run, the number of simulated users per service should equal (approximately) the number of concurrent users using that service as defined by the usage profile. Each load generator instance runs a single script (such as the script shown in the section "Verification Script" above) representing the behavior of multiple users using one service. Multiple instances of the load generation script are run to generate the correct aggregate user load.

Comparison with TCA Estimates

For each simulated usage profile, resource utilization measurements are compared against the utilization estimates, which were calculated using the cost equations in the topic "Cost Equations." The difference between the measured costs and estimated total costs represents the modeling error. Figure 7 illustrates the error between cost equation estimates and simulation measurements. If this error is large, then you should verify that TCA assumptions are satisfied. You should also consider repeating performance measurements.

Scenario

The usage profile from the scenario in the section "Usage Profile" indicates that the required throughput for Web page requests is 83 HTTP gets/sec and 16.7 FTP transactions/sec. These resource costs are then calculated using the equations developed in the section "Cost Equations."

In particular, suppose that the equations for CPU costs show that the combined CPU cost for the HTTP and FTP services is C_total^CPU = 2,000 MHz.

Additionally, suppose that the Internet site will be deployed using servers with 400 MHz processors and that the requirement is to run these servers at less than 70 percent CPU utilization. Then C_max^CPU = 400 MHz and q^CPU = 70 percent. Therefore, the total number of 400 MHz processors required is calculated as follows:

C_total^CPU / (q^CPU * C_max^{CPU )} = 2000 MHz / (0.7 * 400 MHz) = 7.1.

In this case, two four-processor servers will support the required load. Similar estimates are made for the other resources. Note that in this scenario you should have gathered your data on four-processor servers.

For verification, the system is then deployed with two four-processor servers (as indicated by the calculations above) and the appropriate load is generated using the verification scripts. This load should simulate 20,000 concurrent HTTP users and 1,000 concurrent FTP users, as indicated by the usage profile example. Suppose an average of 85 HTTP gets/sec, 10 FTP puts/sec, 7 FTP get/sec, and average CPU utilization of 1900 MHz are measured. Then the throughput requirements are satisfied and the CPU utilization estimates are in error by no more than 5 percent.

Top of page

Capacity Planning Using Transaction Cost Analysis

Once your site is deployed, you should periodically use the Transaction Cost Analysis (TCA) methodology to accurately recalculate your hardware resource costs. Your hardware resource costs can increase (or decrease) over time based on changes in usage. Increased usage of a site can be attributed to business growth, a change in average customer behavior, and so on.

After determining the resource costs for each transaction for every service, this data can be applied to capacity planning using the following procedure:

1. Define a new usage profile and set of throughput targets using Equation 1. (See the section "Performance Targets). This compensates for the growth of your business or change in actual user behavior.
2. Calculate total resource costs for the new usage profile using Equation 2. (See the section "Total Resource Costs.")
3. Calculate capacity requirements and perform "what if" analyses using Equations 3 and 4. (See the section "Total Resource Costs.")

For an example of capacity planning based on TCA methodology, see the white paper titled Microsoft Commerce Server 2000 SVT Site Performance Characteristics: Transaction Cost Analysis, available from the same Web page as this paper. A spreadsheet that incorporates a system of equations for performing ongoing analysis is included with this white paper. Use this white paper as a case study on how to perform actual capacity planning.

Top of page

Conclusion

This capacity model based on Transaction Cost Analysis (TCA) has been successfully applied to estimate hardware platform deployment requirements for Microsoft® Commercial Internet System (MCIS), Microsoft® Site Server products, Microsoft® Commerce Server products and Microsoft® Exchange products. It is recommended that you use this model to help you deploy an appropriate hardware configuration to run your services, while reducing the potential for over-provisioning hardware.

Top of page

Appendix A - Recommended Reading

The following references focus on capacity and performance modeling in general:

The Art of Computer Systems Performance Analysis Techniques for Experimental Design, Measurement, Simulation, and Modeling
By Raj Jain
Publisher: John Wiley & Sons, 1991

Capacity Planning and Performance Modeling: From Mainframes to Client-Server Systems
By Daniel Menasce, Virgilio Almeida, and Larry Dowdy
ISBN: 0137895461 paperback 1999
ISBN: 0130354945 hardcover 1994
Publisher: Prentice Hall, Incorporated

Capacity Planning for Web Performance: Metrics, Models, and Methods
By Brian Wong
Publisher: Prentice Hall, February 1997

Microsoft Windows 2000 Server Resource Kit
Publisher: Microsoft Press, 2000

Modeling Techniques and Tools for Computer Performance Evaluation
By Ramon Puigjaner and Dominique Potier
Plenum Press, 1989
The following reference focuses on capacity planning for Sun Solaris systems, but offers helpful general insight as well:

Configuration and Capacity Planning for Solaris Servers
By Brian Wong
ISBN: 0133499529
Prentice Hall, February 1997

For performance considerations specific to Microsoft Windows 2000, the following book is recommended:

Tuning Netfinity Servers for Performance: Getting the Most out of Windows 2000 and Windows NT 4.0
By David Watts,Chris Neophytou,Murat Gulver,Gregg McKnight,Peter Mitura / Paperback
Prentice Hall PTR
July 2000

For information about choosing a random time interval, the following book is recommended:

The Benchmark Handbook for Database and Transaction Processing Systems
By Jim Gray
Morgan Kaufman Publishers, 1992 and 1993

For more information about creating a usage profile see the white paper titled Commerce Server 2002: Creating a Usage Profile for Site Capacity Planning, available from the same Web page as this white paper.

Top of page