Operational Recommendations for Using Exchange Load Generator


Topic Last Modified: 2008-02-05

The following operational recommendations will help you achieve solid results from the Exchange Load Generator tests.

Exchange Load Generator simulates the cached mode of Microsoft Office Outlook 2003 and later versions, through a combination of tasks that are common to online mode. To simulate cached mode, use the Cached Mode client type as a starting point to increasing and decreasing task frequencies to match your user profiles.

By using stress mode, you can run a test with the Exchange Load Generator users performing their tasks as fast as they can. When in this mode, Exchange Load Generator does not spread out the tasks throughout the defined length of a simulation day, but runs them as fast as it can and will finish probably faster and unpredictably.

You should only use this mode when the objective is to stress the server at its mode and ignore the deterministic way (spreading a fixed load for a fixed number of hours) that Exchange Load Generator runs.

To use RPC over HTTP, your topology has to be configured to use one or more RPC proxy servers. Instructions for doing this using the configuration file are in the topic, Customizing the Exchange Load Generator Test. The following are two recommendations for user loads when you use RPC over HTTP.

With a reasonable number of users who are running on an Exchange Load Generator computer, inactive RPC/HTTP connections will be disconnected automatically by the Windows RPC runtime component at 90-second intervals. This means that with such a small level of load running through RPC/HTTP, you may find that the number of active connections is very low.

Because of this limitation, you should not use Exchange Load Generator to size a server for maximum HTTP connections. Instead, use the Exchange Server Stress and Performance tool to open generic Web connections to the server to determine the maximum number of HTTP connections that can be simultaneously serviced by a particular computer. You can download the 32-bit or 64-bit version of this tool at Tools for Exchange Server 2007.

The number of users who are supported by the client depends on the actions and frequencies specified in the configuration file. Generally, the number of users is limited by the connections and ports configured on the server. If you are running Outlook 2003 online simulation, there are two RPC connections to the mailbox store and one to the public folder for each user. Each connection generates two HTTP connections (data in and data out). Make sure that the server is configured correctly and there are no other applications on either side that consume more TCP port numbers than you expected. For a client computer that has a single processor (2 GHz) and 1 gigabyte of RAM, no major issues are presented in supporting 1200 to 1500 RPC over HTTP users driven by Exchange Load Generator. The command-generated LoadGenConfig.xml file would be used for setting up the configuration.

Make sure that you monitor the value of \Exchange Load Generator Engine\Task Queue Length to make sure that it is not increasing. The value of \Exchange Load Generator Engine\Task Completed/sec and \Exchange Load Generator Engine\Task Dispatched/sec should also match. This means that the client is catching up. Usually the out-of-memory problem or MAPI connection errors occur you hit the CPU bottleneck.

The following information about the Client Access server applies to Exchange 2007 only.

The number of RPC over HTTP users depends on the hardware profile of the Client Access server. For a 4-processor (1.9 GHz) computer that has 4 gigabytes of RAM, 4000 RPC over HTTP users use only 14% CPU on the Client Access server (most of the work is being done in the mailboxes) but the connections it opens are approximately 24,000.

In the Read and Process Message task, you can configure a percentage of the messages to be sent to distribution groups (also known as DLs) or query-based distribution groups. Although you can configure the average size of a distribution group in Exchange Load Generator, query-based distribution groups are generally larger than the distribution groups. This is because they can include all per database, per server, or for a whole topology.

Sending mail to query-based distribution groups can severely stress the server and global catalogs’ domain controller when the query-based distribution groups are being expanded. We recommend that you set up a dedicated expansion server for the distribution groups. You should also modify Exchange Load Generator’s distribution groups and query-based distribution groups accordingly. Additionally, consider using the “one per MDB” query-based distribution group that is typically smaller. For information about how to configure an expansion server for Exchange Server before Exchange 2007, see Microsoft Knowledge Base article 328791, "HOW TO: Use Expansion Servers in Exchange 2000" (http://go.microsoft.com/fwlink/?LinkId=3052&kbid=328791).

For Exchange 2007 information, use the Set-DynamicDistributionGroup and Set-DistributionGroup cmdlets to modify the settings. Both these cmdlets have an ExpansionServer parameter. Detailed information about cmdlets is in the Exchange 2007 Online Help at http://go.microsoft.com/fwlink/?linkid=76150.

Exchange Load Generator can be an effective tool for deployment verification. For example, you can use it to verify that you will be able to support the number of users who have been planned for your deployment. You can also examine how the system might react to stress loads that are heavier than what you would typically expect.

Although Exchange Load Generator can be used to verify your deployment in various ways, it should not be used in production environments.

We recommend that Exchange Load Generator be run for verification in separate topologies and then rolled out on production environments. Also, it is a good idea to run each Exchange Load Generator test multiple times. This will help make sure of the validity of your results.

Exchange Load Generator has profiles that enable you to verify deployments and that help in capacity planning. To model your deployment’s users, enable or disable tasks and change the settings and task frequencies of the Heavy profile. The Heavy profile has many features to better simulate real-life users.

We encourage you to use the well-known profiles such as "Heavy" that are provided by Exchange Load Generator. Avoid creating a custom profile unless you are very familiar with Exchange Load Generator parameters, and you understand what type of difference your customization will make in terms of the load that is generated on the server.

To control the stress load that Exchange Load Generator puts on the Exchange server, you can change the duration settings of the simulation in the Specify test settings page of the user interface, and enable or disable tasks. Exchange Load Generator calculates the stress from the frequencies of the tasks, and then spreads the load homogeneously across the configured length of the simulation day. For example, if Exchange Load Generator must run a task four times in an eight-hour daytime period, it will run a task every two hours. By increasing the task frequency to eight times a day and keeping the length of the simulation day setting the same, a task will be run every hour, thereby doubling the stress load. Therefore, if the length of the simulation day increased, the stress is actually diminished, because the same load is spread over a longer time. To increase the stress load, increase the task frequencies and/or reduce the simulation day length.

After you change the topology settings, you should re-create the topology to make sure that the physical topology matches your settings.

After you initialize users for a test run that will be changed and repeated over time, it is a good idea to back up the databases so that the initialization process does not have to be repeated. After each test run, restore the databases and re-start your tests without having to reinitialize all users.

Backing up and restoring databases to skip repeating the initialization phase makes sense only if the initialization and topology settings are unchanged between test runs for the server in question.

The typical goal of Exchange Load Generator tests is to analyze the load that the server can support and still provide sufficient response times. However, sometimes the goal of Exchange Load Generator tests is to determine what load the network can support while still providing sufficient response times. You should saturate the resource you are focusing on with the test, but not saturate other resources in the test. For example, you cannot effectively study the server behavior if the client computers or the network are saturated.

The following two methods can be used to monitor for saturation:

  • Monitor the demands on the clients, network, and server.

  • Do not let the clients become the bottleneck in an experiment.

Use System Monitor to make sure that mail delivery is working correctly. Specifically, monitor the Send Queue Size counter of the MSExchangeIS Mailbox performance object if you are using Exchange Server 2003. For Exchange Server 2007, the counter to monitor is MSExchangeIS Mailbox: Message Queued for Submission.

If there are several messages in the queue, this indicates a server bottleneck. Because the server must operate at a consistent baseline for sufficient benchmarking, a server bottleneck invalidates your test results. For example, if the queue is continually growing for the whole duration of the test, the server is probably overloaded. This indicates a bottleneck.