Understanding the Performance Impact of High Item Counts and Restricted Views

Understanding the Performance Impac...

Topic Last Modified: 2009-01-14

This topic will help you understand, diagnose, and resolve Microsoft Exchange Server 2007 performance issues related to high item counts in critical path folders and restricted view requests when you use Microsoft Office Outlook running in online mode, Outlook in cached mode with delegate access, and Outlook Web Access. The critical path folders are the Calendar, Contacts, Inbox, and Sent Items folders. Restricted views are data views that restrict information based on search criteria that result in views of only a subset of items in a folder. Performance issues related to these situations are frequently related and can become visible to end-users in the form of slow client access. It only takes a few users who have abnormally high item counts in their critical path folders to cause performance issues which are felt throughout your whole Exchange organization.

With Exchange Server 2003, the recommended maximum item count per folder was 5,000 items. In Exchange 2007, improvements in I/O, larger page size, and increased cache can help enable an increase in the recommended maximum item count. With properly architected hardware, an acceptable user experience can still be maintained with item counts as high as 20,000 items.

Note:
An acceptable user experience for common operations means 100 milliseconds response from click to action. For rarely used operations, such as creating a new sort order or selecting a folder for the first time, response times of up to one minute are acceptable.

This recommended maximum varies, depending on whether any third-party programs access user mailboxes. These third-party programs include the following and other similar programs:

Outlook add-ins
E-mail archiving
Antivirus
Mobile device
Voice mail
Other programs or Outlook add-ins that create additional views or search folders

In this scenario, the item count must be determined by reviewing the overall server load together with how the server processes each request from the particular third-party program.

This recommended maximum also depends on the performance capability of your Exchange environment. Your specific hardware choices may result in lower maximum numbers. Ideally, it is best to keep the Inbox and Sent Items folders less than 20,000 items, and the Contacts and Calendar item counts less than 5,000. Even when maintaining item counts that are at or under the recommended maximum values, there are some operations which may still take noticeable time (usually this is approximately one minute). These operations include new sort orders and selecting folders for the first time. First time views of a folder can take even more time to generate the view. High item counts in critical path folders have an adverse effect on server performance because they are the most frequently accessed folders in a user's mailbox. Other folders, especially custom folders that are created by end-users, can handle having larger numbers of items without having an adverse effect on the user experience because they are accessed less frequently. Be aware that, although the performance effect of having higher item counts in less frequently accessed folders is reduced, high item counts in these folders can still present intermittent performance issues as the number of folders like this increases, and the number of active users on the server increases.

Important:
First time views of a folder can take even more time to generate the view. Therefore, move mailbox operations will result in first view performance degradation for all folders that have high item counts. This can result in intermittent performance issues while users who have been moved to new servers access their folders and re-create the folder views. Following operational best practices for moving mailboxes by spreading the moves across multiple servers, and by decreasing the number of mailboxes moved at the same time to the same server, can help minimize this issue.

Important:

First time views of a folder can take even more time to generate the view. Therefore, move mailbox operations will result in first view performance degradation for all folders that have high item counts. This can result in intermittent performance issues while users who have been moved to new servers access their folders and re-create the folder views. Following operational best practices for moving mailboxes by spreading the moves across multiple servers, and by decreasing the number of mailboxes moved at the same time to the same server, can help minimize this issue.

When you consider the appropriate maximum item count for your organization, understand that there is no cliff at the maximum value, but instead a progressive degradation in performance. The stated limits are not hard-coded. They are numbers based on testing and code analysis. As the item count increases, performance may degrade to a user perceivable level. The level of performance that is acceptable for users in your environment will dictate the appropriate maximum item count for the environment.

Understand that there are no inherent size limits on individual mailboxes. The main factors that limit mailbox size are as follows: available disk space, backup and restore times, service level agreements, and Outlook performance. When we refer to Outlook performance, we are specifically referring to the latencies experienced by the end-user.

Understanding the Performance Impact of Hardware Choices

While it should be self-evident, it bears repeating: if your organization's hardware is improperly sized, you may experience poor performance at a lower number of items. In online mode, I/O requirements of a system will grow as the item count in the mailboxes increases. Disk latencies are one of the most important hardware considerations when you evaluate hardware performance around high item counts. For optimal user experience, make sure disk latencies are low (20 milliseconds or less), even during peak server usage.

To show how disk latency can add up and impact server performance, consider the following example. When requesting a view, the request for the data is performed in individual, serialized requests from the disk, not bulk operations. For example, if a plug-in is returning a view of 1000 items, then the Exchange store will probably make about 200 separate requests for data (assuming about 5 messages are retrieved per request). At 20 milliseconds, that is a guaranteed 4 second delay from the disk subsystem alone. Consider the growth of the performance impact as disk latency is increased to 50 milliseconds or 100 milliseconds. The issue is further compounded if multiple plug-ins are used making similar requests. Users may find that Outlook is frequently blocked in this situation.

For more information about server sizing recommendations to ensure good performance with Exchange 2007, see Planning Your Server and Storage Architecture.

Understanding Why Additional Processing Time May Be Required

When you consider the performance impact of high item counts and restricted view requests on your server's processing ability, you should be aware of the underlying processes that occur and how the high item counts and restricted view requests relate to those processes.

Folder contents are stored in a table in the information store database. As the number of items increases, there is a corresponding growth in storage complexity. The storage mechanism for the Exchange store is the Extensible Storage Engine (ESE). ESE uses B+ tree data structures to store records. As the number of records increases, the potential number of disk I/O requests that are required to locate the information and traverse the B+ tree also increases. For more information, see Extensible Storage Engine Architecture.

As the number of items increases, the possibility of any requested data being located on physically adjacent locations on the hard disk is greatly diminished. Therefore, more I/O requests are required.

Creating restricted views requires processing from the Exchange server.

There are two ways to filter messages in MAPI:

Search folders

A search folder resembles a typical MAPI folder. However, instead of messages, the search folder contains only links to messages in other folders. These messages meet a specified restriction.

Restrictions
A restriction is created by calling the IMAPITable::Restrict method on a MAPI table. The resultant table displays only the items that match the specified restriction. A restricted contents table displays only messages from a single folder. This kind of restriction may sometimes be confused with the MAPI-defined structure that describes the filter criteria for a search folder. This kind of MAPI-defined structure is also known as a restriction.

Search folders and restrictions require additional processing. For example, creating a search folder and populating the search folder with items that match the user's criteria. This process requires each item in the folder to be examined to determine whether each item should be put in the search folder. Therefore, when a folder contains many items, more time is required to create the view. These search folders are stored as a SearchFID under each folder where the search was originated. By default, after a search folder is populated, the folder may exist for up to 40 days.

The presence of restricted views affects how quickly modifications can be made to the items in a folder. When a search folder is associated with a folder, more processing must occur when items are added, deleted, or updated in the other folder. This behavior occurs because Exchange must determine whether the search folder must be updated. When you establish many search folders, each change must be evaluated against all search folders to determine whether the search folders must be updated.

When more non-standard properties are added to a folder view in Outlook, additional remote procedure calls (RPC) may be required to retrieve the non-standard properties from the Exchange store. When a folder contains many items, more round trips are required to retrieve the data from the Exchange server.

When you try to view a calendar folder, Outlook must locate all appointments in the specified date range. This requires at least two processing requests. The first request obtains all static appointments in the specified date range. The second request locates any recurring appointments that occur in the specified date range. The time that is required to process the second request is proportional to the number of items in the calendar folder. This behavior occurs because Outlook requests all recurring appointments. After the list of recurring appointments is received, each recurring appointment must be examined to determine whether the appointment occurs in the specified date range. When the calendar contains many items, more time is required to process these requests.

Understanding the Performance Impact of Item Count Versus Mailbox Size

Understand that most performance issues are not the result of large mailbox size (defined as a mailbox that is 2 GB or larger), but instead the number of items in the folder or folders that are being accessed on the server. Having many items in a folder adversely affects performance because operations in those folders will take longer. In particular, performance is largely influenced by the number of items in the critical path folders: Calendar, Contacts, Inbox, and Sent Items folder. For more information about how to plan for large mailboxes, see White Paper: Planning for Large Mailboxes with Exchange 2007.

Operations that depend on the number of items in the folder include adding a new column to the view, sorting on a new column, finds and searches. Many Outlook plug-ins perform sorts or searches as they are running, and these requests may overlap with other Outlook MAPI requests. This results in a poor user experience.

If you are running in cached-mode, (the default mode for Outlook 2003 and later versions), then client performance can be an issue as the number of items in a folder increases. One thing that you should do is keep your OST files (the local data cache) free of fragments. You can use the Windows SysInternals tool Contig for this purpose. To download the latest version of Contig, see Contig v1.55.

In addition to the number of items in critical path folders, there are other factors that affect the Outlook experience which are exacerbated by high item counts, such as the number of other MAPI applications or Outlook plug-ins running on the user’s computer. All MAPI requests require processing time with emsmdb32.dll. If a user has lots of plug-ins making requests, Outlook will run slower, and the impact on the server may increase. Additionally, the complexity of the requested action will have an impact. For example, marking all items in a folder as Read will take much longer than marking one item. Other actions that may take a long time include retrieving free-busy information for lots of users on a meeting request, or performing a search across multiple folders. If users are frequently performing complex actions, have lots of Outlook plug-ins, or have high use of the Contacts and Calendar folders, the probability that they will experience decreased performance because of high item counts is greatly increased.

Understanding Restricted Views (MAPI Restrictions)

MAPI represents data typically in the form of a table. There is a table that is presented to the client when it requests a list of providers, a table for folders, folder contents, attachments, and more. Each table consists of columns. Each column is a different MAPI property representing attributes such as the sender, subject, and delivery time. Each row represents a specific item. For a folder contents table, each row represents a message. Client actions on these tables are represented by activities like sorting data. The client can seek in the table to a specific row that matches specified criteria. This operation is known as a FindRow(). It can also request that only items fitting a certain criteria be included in the table. An example would be to only include items created on a specific day. This is what is known as a restriction. The resulting folder contents table would only have items in the table that meet the given criteria. Restrictions are used when it is expected the client will be requesting the same representation of data on a frequent basis.

How Restricted Views Impact Performance

To understand the performance ramifications of restrictions, you must consider that the Information Store actually stores the data a MAPI client requests and how it interprets requests such as FindRow and Restrict. Inside the storage schema of the store, there are various tables that collectively represent things such as mailboxes, folders, folder contents, and messages.

When a client requests a list of the contents of a folder, that request is mapped to a special folder known as a MessageFolder Table (MsgFolder). Each folder that is created in the system has a separate message folder table. The purpose of the MsgFolder table is to map a folder to its contents.

To handle the expectations of a Restrict call (frequent re-request for the same data) a new special folder (and corresponding MsgFolder table) is created that is known as a Restricted Search Folder. This folder is linked back to the original folder and logical relationship exists between these two folders. A condition is put on the search folder so that it should only include items that meet the criteria that is specified by the restriction. In this search folder, a backlink to the original row in the MsgFolder table exists for each message in the MsgFolder table that meets the criteria of the restriction.

The performance issue that is encountered with this behavior is the time that is required to manage the update to each of the search folders. When a change occurs on the original folder, the change is compared to each of the restricted search folders associated with the folder in question to determine whether they must be updated also. This has a bigger impact when lots of search folders exist for one or many folders. The second issue encountered is the creation of the restricted search folders. Creating a restricted search folder requires a full pass of the original folder to extract individual items that must be linked into the restricted search folder. If this process occurs as a direct result of an action of the client, the time delay may result in the user experiencing a hang or receiving the Outlook RPC pop-up box that indicates a request is taking a long time to process. The time that is required to create the restricted search folder is proportional to the number of items in the regular folder. As the number of items in a folder increases, the probability that those items will be located on adjacent disk sectors is greatly reduced. This results in a significant increase in random disk I/O when fulfilling restricted view requests on folders with high item counts. As the number of restricted view requests increases on folders that have high item counts, the increased I/O impact will be felt by all users who access resources on the server.

In Exchange 2000 Server, Exchange Server 2003, and Exchange Server 2007, there are a maximum number of restricted search folders that are allowed on a per-folder basis. The default maximum number is 11, and the default lifetime for the search folder is 40 days.

Restricted search folders are processed on a first-in first-out (FIFO) basis. If a folder already has 11 restricted search folders associated with it, and a new restriction request is made, the list of search folders is evaluated using the last time the restriction was actually used. This means that the least used restricted search folder is removed to allow the processing of the new request. As mentioned earlier, this requires a full pass of the regular MsgFolder table and if done as a direct action by the client, the client may perceive a performance issue while the table is being built and presented to the client. So, perhaps on a daily or weekly basis, more than 12 restricted search folders may be required. This results in a client incurring a restriction request that currently does not have a matching search folder. This in turn results in deleting and creating new restricted search folders. All these requests are processed serially. This means that if you have lots of users who have very high item counts in folders upon which restricted views are regularly requested, those requests may queue up. This will result in overall perceived slowness for all users who are accessing mailbox data that is homed on the server. Understand that this affects more than just users who are accessing their mailboxes. For example, users on other servers who access calendar data stored on the server will also perceive slowness.

Important:
Installing additional locales on the client side can create even more views for users of Outlook Web Access and Outlook in Online Mode.

Restricted Views and Shared Calendars

When you view someone else's Calendar, Contacts folder, and so on, there may be delays before the folder can be viewed. When the folder has been viewed, switching away and back is fairly quick. But after a time, accessing the folder will become slow. The delay will be especially long if the number of items in the calendar is over 5000.

When Outlook accesses someone else's folders, it applies a view which restricts the user from viewing private items.

The act of applying a view to a folder creates search folders in the store. When a search folder is created, it is cached for later use. If a user tries to create a search folder which already exists, the cached search folder is used. This allows future viewings to be fairly quick.

By default, Exchange does not cache all search folders indefinitely. Caching too many search folders would cause server-side delays associated with updating the search folders. Conversely, if a sufficient number of search folders are not cached, a similar problem will occur as search folders must be generated and updated.

To illustrate the issue, consider the scenario where an Exchange server has been configured to keep 11 search folders (views) per folder. Suppose Frank has a calendar folder that he shares out to 15 other users. Sally accesses the folder and sees a delay while her search folder is built. After it is built, access is quick. Then Sally does not view the folder for a day and 11 other users access the folder. A new search folder will be built for each of them. Because only 11 search folders are cached, when the twelfth user hits the folder, Exchange will delete the search folder that was built for Sally. Now, the next time Sally hits the folder, she'll have to wait while Exchange builds her search folder.

Suppose we configure Frank's Mailbox server to cache 20 views instead. Then Sally and the other 14 users can all hit Frank's calendar folder, and only 15 search folders will be created. Because 15 is less than 20, we never have to cycle out a view, so access is quick for everyone after the initial hit to create the search folders.

The default number of cached search folders is 11. This configuration is set at the database level. The configured value can be viewed by using ADSIEdit. Use ADSIEdit to view the Store object, and then examine the msExchMaxCachedViews attribute. The distinguished name is as follows:

CN=Database, CN=Storage Group,CN=InformationStore,CN=Server NAME,CN=Servers,CN=AG Name,CN=Administrative Groups,CN=Orgname,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Contoso,DC=com.

In some cases, modification to increase the value greater than 11 can be useful to tune the performance impact on an environment.

Important:
The msExchMaxCachedViews attribute value should never be set higher than 50.

Important:
If you have Outlook 2003 RTM in your environment, make sure that Service Pack 1 or a later version is deployed to address a known issue with slow performance when you are working with shared calendars.