Using Transaction Cost Analysis for Site Capacity Planning

Microsoft Corporation

	Methodology
	Conclusion
	Appendix A - Recommended Reading
	Appendix B - Test Tools

Methodology

The purpose of capacity planning for Internet services is to support transaction throughput targets with acceptable response times, while minimizing the total dollar cost of ownership of the host platform. Conventional solutions often attempt to evaluate the cost of Internet services by extrapolating from generic benchmark measurements. However, a more effective methodology is based on Transaction Cost Analysis (TCA).

You can use TCA methodology to simulate client transactions on the host server using a load-generation tool that supports standard network protocols. You can determine your resource utilization (as supported by your hardware configuration) by varying the transaction rate of the client load. You can then design a usage profile to capture anticipated user behavior. You use this usage profile to determine the throughput target and other important transaction parameters from which you can then calculate resource utilization and capacity requirements.

TCA methodology can be used to measure the cost of individual shopper operations, such as registering on the site, browsing, searching, adding an item to the shopping cart, checking out, and so on. You determine the capacity of a site by dividing the costs of shopper operations into the total resources available from the server. To simplify the discussion of TCA methodology, this white paper focuses on determining the CPU capacity of the server only. For a discussion of calculating multiple resource costs on the server, see the Capacity Model for Internet Transactions white paper, available from the same Web page as this white paper.

After you understand the cost and relative frequency of each shopper operation, you can view the performance numbers for a site's pages to identify where the bottlenecks are, and identify where you can optimize performance to improve the site's capacity.

For example, TCA methodology can help you determine which operations are the most costly, which pages are accessed most frequently, and so on. You can use the performance data that you collect, using TCA methodology, to perform "what if" scenarios. This way you can see how performance improvements (or degradation) affect your site. Once you know the cost of each operation, you can improve the performance of your site by optimizing high cost or frequently used Web pages. If you need additional capacity, identifying and lowering the cost of these "expensive Web pages" will help. In addition, you will clearly see the impact of adding features to a Web page, and can plan to install additional hardware required to service the load generated by additional shoppers or new code releases.

A Transaction Cost Analysis is comprised of the following:

• Compile a User Profile
  
• Measure the Cost of Each Operation
  
• Estimate Site Capacity
  
• Calculate Site Capacity
  
• Verify Site Capacity
  

Compile a User Profile

The first step in running a Transaction Cost Analysis is to compile a user profile by analyzing customer Internet Information Server (IIS) logs. To obtain good usage averages, it is best to import IIS logs for at least a one-week period. You can create Usage Profile Reports from the Internet Information Server (IIS) log files, also referred to as Web log data, using the Analysis module in Microsoft Commerce Server Business Desk.

You can use the Usage Profile Reports to gather the following types of information:

• Number of visitors accessing your site
  
• Number of hits on each Web page
  
• Number of checkout transactions
  
• Rate at which transactions are taking place
  

For more information about Usage Profile Reports, see "Appendix A -, Recommended Reading."

The number of visits to each page is used to profile typical shopper operations for the site. This information comprises the user profile, which is used for the transaction site analysis. The following table shows a user profile for a sample site:

Table 1

Although the operations in Table 1 above account for only 10 percent of the Web pages on the site (most requested pages), these operations account for 90 percent of the traffic to the site. To simplify the example, other traffic numbers (including image and miscellaneous requests that generate a much lighter load) are not included in shopper operations.

Another important aspect of compiling a user profile is to determine the rate at which transactions take place. You use these numbers to create scripts that simulate a load on the sample site to verify the calculated capacity. For example, transaction rates at a different sample site are shown in Table 2:

Table 2

Measure the Cost of Each Operation

After you have established a user profile, you can create a load-monitoring script to simulate each of the identified shopper operations. A shopper operation is a particular action that a shopper performs on your site, such as adding an item to a shopping cart, performing a search, and so on.

The Application Center Test (ACT) tool, provided with Visual Studio .Net is recommended for creating the load-monitoring script. You can also use the Microsoft Web Application Stress (WAS) tool, which is available free of charge. See "Appendix B - Test Tools" for more information about these tools.

The objective of running a script exclusively for an individual operation is to load the IIS server with as many requests as possible, in order to achieve maximum ASP throughput per second. Maximum ASP throughput occurs when you measure a drop in ASP throughput with a higher shopper load, or when you measure a sudden increase in operation latency or ASP requests queued. You can calculate operation latency using the following formula:

Average ASP latency (ASP execution time + ASP wait time) * number of ASP pages

Since it is difficult to determine averages before running the scripts, use an average ASP latency of less than 2 seconds to account for additional download latency of modem users.

Use the Active Server Pages Object—ASP requests/second counter, and the Processor Object—% Total Processor Time counter of Microsoft® Windows® 2000 System Monitor to record CPU utilization, and determine when you have reached the maximum number of ASP requests/second.

You can increase the number of users running the script, in steps, to increase the load on the IIS server. Typically, ASP requests/second and CPU utilization grow with the number of users. However, if adding more users results in lower ASP requests/second throughput, the previous step achieved the maximum throughput. If the number of ASP requests/second continues to grow until CPU utilization reaches 100 percent, the number of ASP requests/second will then reach the maximum.

Another way to help you determine optimal throughput is to select a maximum percentage that you will utilize your CPU resources. This should also be the target maximum for health monitoring in your data center. In practice 85 percent as a peak level is recommended, but more conservative percentages are very common. Note that this peak should not be a sustained average. This peak should represent the busiest five minutes of the day.

The result of TCA measurement is the CPU cost for each shopper operation, calculated on the basis of Megacycles (Mcycles), also referred to as 1 MHz. For example, a dual processor 400 MHz Xeon Pentium II processor has a total capacity of 800 Mcycles. Using the maximum number of ASP requests/second, you can calculate the cost per ASP request as follows:

CPU utilization * number of CPUs * speed of CPU (in MHz) / ASP requests per second

Browse Operation Example

You use a server with two 400 MHz CPUs, and the browse operation results in 11.50 ASP requests/second with CPU utilization of 84.10 percent. The cost per ASP page is calculated as follows:

* 2 * 400 / 11.50 = 58.50 Mcycles

Note: It is recommended that you perform your TCA analysis on the highest performance (multi-processor) hardware available. This ensures that resource requirements that are based on the measurements of higher performance (multi-processor) servers will scale down (at worst linearly) to lower performance hardware. However, the converse is not necessarily true.

Before you can calculate the cost of an operation, you also need to know the total number of ASP page requests used in a shopper operation. For example, checkout operations typically involve several ASP pages (shopper information page, credit card page, shipping page, confirmation page, and so on). You calculate the cost of a shopper operation by multiplying the number of ASP pages involved by the cost per ASP page. Some ASP pages, such as action pages, are never seen, because they are usually posted to and redirected to a continuing page.

Note: Action pages are the Active Server Page (ASP) pages that constitute a module. Action pages can be used for pure-processing; that is, they are not responsible for the display of any user interface. These pages do appear in the IIS log as well as any Network Monitor, also called Netmon, trace of the interaction.

In the browse operation example above, only one ASP page is involved in the browse operation. Thus, the cost of the browse operation is 58.50 Mcycles. If there are two ASP pages, the cost is 58.50 * 2, or 117 Mcycles, and so on.

TCA measurement results in a set of CPU costs for each shopper operation. The following table summarizes cost per shopper operations based on a server using two 400 MHZ processors.

Table 3

Table 4

The following table shows how each column in Table 3 is calculated.

Estimate Capacity

Estimating capacity requires some assumptions about basing your user profile on typical shopper operations. These typical shopper operations are those expected of an average user. The behavior of actual shoppers can cause random peaks and valleys in performance; however, when analyzed over time these peaks and valleys statistically even out to average behavior. The user profile you create reflects expected average user behavior. Therefore, to estimate site capacity, you need to assume that an average user will be using your site.

In the following example, there are 11 shopper operations that occur in an 11-minute interval. Checkout transactions occur 0.1 times every 11 minutes.

To simplify the example, the performance team created the user profile script to perform the 11 shopper operations in 11 minutes. This generated a ratio of operations for use in capacity calculations and for a verification script.

Note: It is recommended that you run your profile script for at least one minute before you start monitoring the site to achieve a steady state measurement.

The performance team used 0.1 checkouts/user, to reflect a heavier load. This number is based on a 10:1 browse-to-buy ratio, instead of the actual 30:1 (or 40:1 when non-cookie shoppers are included) browse-to-buy ratio.

The performance team then calculated the number of shopper operations during the course of 11 minutes, and following that, the number of shopper operations per second. You need to know the number of shopper operations per second because the total shopper operation cost number is expressed in terms of processor clock speed per second (400 MHz CPU speed is 400 Mcycles per second). The cost, based on the example user profile, is seen in Table 5.

Table 5

Calculating the Values Shown in Table 5

The Ratio of Hits column is copied from Table 1.

The Number of Shopper Operations column is the ratio of hits, which was chosen to be 11 shopper operations in the user profile normalized over the requests per visit. In other words, for the View Cart operation, 2.53 percent of the 11 hits per visit is actually 0.3 operations per visit.

The Operations per Second column is calculated as follows:

Number of Shopper Operations / 660 seconds = Operations per Second

The Cost per Shopper Operation (Mcycles) column is copied from Table 3.

The Cost per Shopper Operation per Second column is calculated as follows:

Cost per Operation * Operations per Second

Note, in Table 5, that the Total for the cost per shopper operation per second number (1.4408), indicates that the cost of the total user profile is 1.4408 Mcycles per user. This number reflects the cost of an average user that is executing shopper operations described by the user profile. This number is then used to estimate the capacity of the site, based on the assumed user profile.

For example, the cost of 100 concurrent users is calculated as follows:

Concurrent users * Total cost per user = Total user cost = 100 * 1.4408 = 144 Mcycles

To improve readability, you can plot these numbers on a graph.

Also, note the cost of a checkout transaction (Add Item + Checkout), which is very high at 415.24 Mcycles. This translates to a server capacity of only two checkout transactions per second because the limit for this machine is 800 Mcycles. Although this seems like a very low capacity for this operation, the average frequency of this operation is also very low (0.000152 per second), so server capacity for concurrent shoppers is not overly affected. The cost per user operation per second shows a very low number at 0.0631 Mcycles (approximately 5 percent of the cost of the entire user profile).

Conversely, the browse operation has very low cost, at 58.50 Mcycles. However the frequency of the browse operation is very high, at 0.006061, so the browse operation places a heavy load on the site. The cost per user operation shows a high number of 0.3545 Mcycles (approximately 25 percent of the cost of the entire user profile).

The Search (Good) operation shows a moderate cost, at 150.12 Mcycles. However, the frequency of the search operation is also high, at 0.002424, so it results in a high cost per user operation per second of 0.3639 Mcycles (approximately 25 percent of the cost of the entire user profile).

These numbers suggest that the browse pages and search pages are high cost pages. To improve shopper capacity at the site, it is best to optimize the performance of these pages first.

Calculate Capacity

In Table 6 below, the maximum capacity of the site is at 550 concurrent shoppers, with a cost of 792.3 Mcycles. This is the maximum limit, because the two CPUs provide the site with a budget of 800 Mcycles of capacity. A more conservative approach is to use 85 percent CPU utilization. This results in a CPU budget of 680 Mcycles, with a calculated maximum capacity between 450 and 500 concurrent shoppers.

Table 6

Verify Capacity

You can verify site capacity by running a verification script that reflects the user profile with an increasing number of shoppers, and then compare the results against Table 6. The verification script is simply a collection of all individual TCA measurement scripts aggregated and run as a single script.

While running the verification script to simulate a load based on your user profile, you can use Microsoft Windows 2000 System Monitor to record CPU utilization and ASP requests/second to verify the cost of the load. You can then create a utilization chart, as shown in Table 7. For the sample customer site (dual-processor Pentium II Xeon 400 MHz CPUs). The table appears as follows:

Table 7

Table 7 shows that the verified maximum capacity (with an acceptable quality level) appears to be 300 concurrent shoppers. ASP request throughput is higher at a load level of 400 shoppers. However, at 400 shoppers, notice that operation latency (response time) jumps to 15.77 seconds, which is no longer an acceptable quality level. As mentioned in the section "Measure the Cost of Each Operation" above, a good target latency is below two seconds.

Note that the units for the operation latency in the table above are in milliseconds. If latency is not a consideration, the shopper capacity is 500 shoppers as ASP requests –per second continues to climb at this load level, perhaps up to 550 shoppers.

Keep in mind that the verification script runs at four times the transaction and request rate of the live customer site. The maximum number of shoppers observed here, translates to approximately 4 x 300, or 1,200 shoppers on a live e-commerce site.

Top of page

Conclusion

TCA methodology can be used to profile average site usage, which is used to calculate the hardware you need to support your site profile and also verify site capacity.

TCA methodology can be used to measure the cost of individual shopper operations, such as registering users on the site, browsing, searching, adding an item to the shopping cart, checking out, and so on. Once you understand the cost and relative frequency of each shopper operation, you can view the performance numbers for a site's pages to identify the bottlenecks in your site, and identify where you can optimize performance to improve the site's capacity.

Top of page

Appendix A - Recommended Reading

For information about how to create Usage Profile Reports, see the following topics in Commerce Server 2002 Help:

• Using Transaction Cost Analysis
  
• Before You Analyze Commerce Server Data
  
• Web Usage Reports
  
• Commerce Server 2002 Reports
  

For information about creating a usage profile, see "Creating a Site Usage Profile" (Chapter 18) in the Microsoft Commerce Server 2000 Resource Kit. For additional information, see the white paper, Commerce Server 2002 Creating a Usage Profile for Site Capacity Planning, available from the same Web page as this white paper.

Top of page

Appendix B - Test Tools

• The Microsoft Web Application Stress (WAS) tool can be downloaded from the Microsoft Download Center.
• The Application Center Test (ACT) tool is available as part of Visual Studio .NET.

Top of page