Exchange Server 2003 Performance: Ten Things to Think About

  • Article

The performance of Microsoft® Exchange servers is a complicated topic that can be viewed from several perspectives. To help simplify the topic, I've come up with ten considerations for designing, building, and operating Exchange Server 2003 for top performance. I've also included links to other resources that can provide you with additional details. This list targets Exchange Server 2003, but there are basic principles that apply to Exchange 2000 Server and Exchange Server 5.5 as well.

Number 10: Exchange Infrastructure and Architecture

Having a solid architecture upon which to deploy Exchange servers is like building a solid foundation before putting a house on top of it. If the foundation is defective, it doesn’t matter how well the house is built.

As stated in the Planning an Exchange Server 2003 Messaging System guide, before you design your Exchange-based messaging system, you need to gather both business and technical information. Exchange depends on a variety of infrastructure technologies, such as TCP/IP networking, name resolution services (DNS and NetBIOS), Active Directory® directory service, and Internet Information Services (IIS). If one or more of these infrastructure components fails, becomes overloaded, or is compromised, Exchange performance can suffer, or worse, your end-users or customers may not be able to use e-mail.

It is particularly important to ensure that a global catalog server is available to your Exchange servers and Outlook clients and that you have sufficient network bandwidth available.

Global Catalog Servers

All Exchange servers and users should have fast access to a global catalog server. The proximity of the global catalog server to both the Exchange server and to the client computer directly affects the performance of the client. At least one global catalog server must be installed in each domain that contains Exchange servers; to achieve optimal performance in larger organizations, additional global catalog servers are required. In addition, if the Exchange server and the Microsoft Office Outlook® client are not in the same Active Directory site, you should consider using the "Closest GC" registry value on the Outlook client.

Per site, there should be a 4:1 ratio of Exchange processors to global catalog server processors, assuming that the processors are of similar models and speeds. However, increased global catalog server usage, large Active Directory implementations, increased application usage, or large distribution lists can necessitate more global catalog servers. Exchange 2003 can utilize global catalog servers that have up to eight processors installed.

Client/Server Communications

  • To achieve optimal performance, there are certain bandwidth requirements that must be met. For MAPI clients, to estimate the requirements for available bandwidth, you can multiply the number of concurrent users by 2.5 Kbps. For heavily active MAPI users, multiply by 3 Kbps instead. For optimal performance, you should ensure that the available network bandwidth is at least 20 percent greater than the amount required for client/server and server/server communications.
  • For optimal Outlook Web Access performance, you'll want to deploy Internet Explorer 6 with Service Pack 1. To ensure that the Outlook Web Access compression kicks in, you should enable HTTP 1.1 (including HTTP 1.1 through proxy connections in cases where proxy servers are used). For more information about improving performance for your Outlook Web Access users, see Microsoft Knowledge Base article 327675, HOW TO: Increase Outlook Web Access Performance.

Number 9: Exchange Server Sizing

A very common question asked by Exchange Server architects is How many users can a single Exchange 2003 server support without compromising performance? Or, as my friends ask me, I have X users; can’t you just tell me what server(s) I need? As Sam Khavari stated in his blog in May 2004, it’s difficult to get a straight answer to the server sizing question. The minimum system requirements for Exchange 2003 are the same as those for Exchange 2000; so, if you're moving from Exchange 2000 to Exchange 2003, you won't necessarily need to upgrade your hardware. However, you need to consider whether you will use your servers differently after you install Exchange 2003. For example, if you plan to use query based distribution groups, you may need more powerful hardware for the server(s) that expand your queries.

In almost all cases, you won't need more than 4 GB of physical memory in an Exchange server. Exchange can't use more than 3 GB of physical memory, and it does not support extended memory technologies, such as Physical Address Extensions. When 4 GB is installed in an Exchange server, 3 GB is typically used for Exchange, and the remaining 1 GB is used for Microsoft Windows® and ancillary applications (such as backup, anti-virus, management and other supportive software).

Ensure that you are using the proper hardware for Exchange 2003:

If you want more precise specifications, use the following hardware sizing tools:

Number 8: Exchange Server Optimization

The goal of optimizing an Exchange server is to decrease the server's response time while increasing the number of supported users. Optimal performance is achieved by properly tuning and configuring each Exchange server.

When optimizing a server, start with the hardware and ask two questions:

  • Are there any BIOS, firmware, or driver updates that need to be applied? Check with your hardware manufacturer to see if updates are available for your server's BIOS, controller BIOS and firmware, network interface card firmware, tape drive firmware, backplane firmware, and so forth. Many firmware updates include driver updates that must be applied first, so be sure to grab those as well. Before applying any updates to your systems, test them in a lab environment that mimics your production environment. If a lab environment is not available, consider testing in a virtual lab environment using technology such as Microsoft Virtual PC 2004 or Microsoft Virtual Server 2005.
  • Are there any hardware components that can be adjusted or turned off to conserve resources or reduce the attack surface (or the Relative Attack Surface Quotient [RASQ]) of the system? There may be one or more enabled hardware components in your server(s) that aren't really needed. COM ports, parallel ports, and USB ports are some examples. Granted, these ports are useful when performing advanced debugging or running with in a lights-out environment; but if you aren't engaging in these activities, you don't need the ports. Turning them off can also increase physical security because disabling these ports results in fewer I/O channels into the system.

Are there ways in which you can tune Windows to increase performance? Sure. Begin by stopping all unnecessary services. For a list of the default services and their startup values, see Default settings for services in Windows Server 2003 product documentation. Also, for a list of the Windows services required for Exchange Server 2003, see Chapter 2, "Exchange Server 2003 Service Dependencies" of the Exchange Server 2003 Technical Reference Guide, as well as the Exchange Server 2003 Security Hardening Guide. These unnecessary services typically include:

  • Computer Browser
  • Distributed File System
  • Distributed Link Tracking Client
  • Distributed Transaction Coordinator (except in an Exchange cluster)
  • Print Spooler (except on an Exchange Server that is also used as a Terminal Server)
  • Task Scheduler
  • Wireless Configuration

You'll also want to tune the File and Printer Sharing for Microsoft Networks (also known as memory management) setting based on the Exchange server's role and the number of mailboxes present on the system. On front-end servers, bridgehead servers, and mailbox or public folder servers that serve fewer than 500 clients, tune memory management for file caching. On mailbox and public folder servers that server 500 or more clients, tune memory management for network applications.

If Exchange is already running on some or all of your messaging servers, the easiest way to verify that Exchange and Windows are tuned properly is to download and use the Microsoft Exchange Server Best Practices Analyzer Tool (ExBPA).

Number 7: Performance Tips for Developers

Exchange 2003 includes support for XML, WebDAV, and Active Server Pages, as well as data access technologies such as OLE DB and ActiveX Data Objects (ADO). You can find a complete list of the available Exchange development technologies in the Exchange Server 2003 Software Development Kit . In general, the more applications you have manipulating the data, the more I/Os per user will be required (because of increased indices, search folders, content conversion, as well as an increase in the number of requests per user).

Regardless of the technology or language you use, consider the following in regard to performance:

  • Deep Traversals   Because deep traversals can potentially examine every folder in the hierarchy, they should be used sparingly to reduce server workload. Generally, you should avoid using deep traversals to search across large public folders and large public folder hierarchies.
  • Message tracking searches using Windows Management Instrumentation (WMI)   The message tracking features in Exchange 2003 enable administrators to track messages passing through Exchange transports. Among other things, message tracking enables you to diagnose message transfer problems and to verify that the Exchange routing configuration is functioning properly. Exchange administrators can use WMI to access the Exchange_MessageTrackingEntry Class programmatically. When doing so, it is recommended that you use date and time restrictions. Without such restrictions, all message tracking entries are searched and overall server performance will be degraded.
  • Registering store event sinks   If you have to register an event sink for all mailboxes in a store, you should register the event sink for store-wide events. When Exchange starts up, it looks for and loads all registered event sinks, which takes some time. Using many individual event sink registrations instead of a single store-wide event registration item significantly increases server startup time.
  • Using store event sinks   Always use care when running store event sinks, especially when using synchronous event sinks. A synchronous event sink is triggered after a data request to the Exchange store but before the item is committed to the store. The synchronous event sink has exclusive control over the item while it is processing. Therefore, a complex or poorly written sink can lock a resource for some time, or even bring an Exchange server to its knees.
  • Use Search Folders for repeated search queries   If you repeatedly use the same search query, it is recommended that you use a Search Folder. When creating a Search Folder, you specify a SQL query that the Exchange store uses to populate the folder with items. After the folder has been populated, links to all items that match your request are placed in the folder. The SQL query is run in the background and periodically updates the folder, which is a more efficient use of system resources. That said, don't use any more Search Folders than you need. If possible, use small scopes for your searches.
  • MAPI Object Limits   Before developing MAPI-based applications against Exchange 2003, take some time to familiarize yourself with the MaxObjsPerMapiSession limits. Each object requested via MAPI requires resources (specifically, memory in the store's address space) on the server. If an application doesn't adhere to the configured limits, it will use a disproportionate amount of resources and degrade performance.

Number 6: Application Compatibility

It is not uncommon to find applications running on an Exchange server that work in tandem with Exchange. Examples of these include anti-virus, anti-spam, content and attachment filtering, backup, and workflow and collaboration applications. You should ensure that any third-party software installed on your Exchange servers is compatible with Exchange 2003 and that you are using the most recent versions (including service packs and hotfixes). Ideally, to ensure compatibility, you should test all software interactions in a lab prior to applying the software in a production environment.

Issues with anti-virus, content filtering, backup software, and so forth can be difficult to troubleshoot because they can manifest themselves as problems with Exchange or Windows. I've seen several cases where anti-virus software caused the Microsoft Exchange Information Store process (Store.exe) to consume excessive amounts of CPU time. I've seen other software on an Exchange server causes the Windows local security authority process (Lsass.exe) to consume excessive amounts of memory and CPU time.

If you suspect that an application is degrading performance on your Exchange server, your first action is to temporarily disable the application and see if performance returns to the expected levels. In fact, if you call Product Support Services about a slow performance issue, one of the first things they will probably recommend is to disable third-party applications.

Number 5: Performance Monitoring for Exchange

Exchange, IIS, and Windows publish an extensive list of performance objects and counters that can be used to baseline, trend, and proactively monitor each server's four "food groups": processor, memory, disk, and network. Many Exchange-related counters you'll want to monitor are listed in Appendix B, "Performance Counter Definitions" of the Exchange Server 2003 Performance and Scalability Guide. For details about how to isolate performance degradations and how to use existing tools and products (such as Performance Monitor, Load Simulator 2003, Exchange Server Stress and Performance 2003, Network Monitor, and Filemon), see Troubleshooting Exchange Server 2003 Performance.

Knowing which counters to monitor is only half of the battle. You also need to know when to monitor these counters. When an Exchange server first comes online, it is in a transient state. The length of each Exchange server's transient state depends on a variety of factors, such as the number of processors, the amount of installed memory, and the load being handled by it. Generally speaking, the transient state lasts from two hours to one day.

After this period, Exchange enters a stationary state where it should exhibit a consistent load with a predictable variance. This state holds until the load changes or until some external event takes place, such as a nightly backup, data restoration, database maintenance, and so forth.

When you are ready to monitor your servers, Troubleshooting Exchange Server 2003 Performance includes several performance counters and their acceptable thresholds. There are different methods you can use to collect and monitor performance counter data:

  • The built-in System Monitor tool of Windows (also known as Performance Monitor and Performance Logs and Alerts).
  • The sample WMI scripts available for Exchange 2003.
  • The Performance Monitor Wizard. This is a relatively new tool that simplifies the process of gathering performance monitor logs. This tool can create logs for troubleshooting operating system or Exchange server performance issues.

There's also the Server Performance Advisor tool you can use to diagnose root causes of performance problems on a server running Windows Server 2003, including performance problems for IIS 6.0 and Active Directory.

If your organization uses Microsoft Operations Manager (MOM), you can get more comprehensive health monitoring of Exchange by using the Exchange Server 2003 Management Pack for MOM 2000. For details about the rules available in this management pack, see the following guides:

Number 4: Operations and Management

Unplanned downtime of an infrastructure component used by Exchange or Outlook (such as a global catalog server) can cause a substantial reduction in performance. One way to minimize these failures, which in turn can help minimize the overall amount of unplanned downtime, is to adopt and follow specific guidelines around operations, change management, and configuration management. For more than ten years, Microsoft has published and evolved this guidance in two complimentary and well-integrated frameworks: the Microsoft Operations Framework (MOF) and the Microsoft Solutions Framework (MSF).

Whether you are the only IT administrator in your organization, or part of an IT staff, ask yourself the following questions:

  • Does your organization routinely make a large number of changes to existing infrastructure (servers, workstations, and applications)?
  • Does your organization frequently deal with infrastructure or application issues in production?
  • Do you find it takes an extended period of time to restore service in your organization because the escalation paths are unclear?
  • Has your organization experienced poor Exchange performance or Exchange downtime due to a failure of people or process?

If you answered Yes to any of these questions, MOF and/or MSF may provide answers. For more information about MOF as it relates to Exchange and Exchange performance, see the Exchange Server 2003 Operations Guide.

Number 3: Troubleshooting Performance Problems

Resolving problems quickly (and in some cases permanently) can dramatically improve the performance of Exchange. Both Exchange and Windows include troubleshooting tools and error reporting facilities for diagnosing and resolving problems. And both products include documentation about how to troubleshoot issues. However, knowing which tool to use and when to use it can be difficult to determine, especially when you’re sitting in the hot seat trying to fix a problem.

When performing root cause analysis, it is important to scope the problem as quickly as possible. In addition to reducing the time it takes to resolve problems, quickly scoping the problem often reduces the number of resources used while increasing the satisfaction of the users whose problem is being solved. Chapter 3, "Isolating the Causes of the Degraded Performance" of Troubleshooting Exchange Server 2003 Performance provides information about how to isolate the cause of degraded performance. The performance counters listed in that guide are also useful when troubleshooting performance problems.

Number 2: Security

Secure your Exchange infrastructure or else! When dealing with Exchange, security comes in many forms: permissions and roles, anti-virus, anti-spam, hardening, secure messaging, and secure remote access to name a few. In addition to compromising the integrity of a system, destruction of data, and the theft of confidential information or intellectual property, incorrect security-related settings can cause a substantial reduction in performance.

To help your Exchange infrastructure exhibit top performance, check out the Top 4 Exchange Security Best Practices. As is the case with all of your Windows-based servers and workstations, you'll also want to make sure your systems have the latest security updates.

Because unwanted e-mail consumes resources and can slow performance, familiarize yourself with the anti-spam capabilities of Exchange 2003 and check out the Exchange Intelligent Message Filter. The less that unwanted e-mail is processed by your Exchange servers, the better performance you'll experience.

Finally, I recommend reviewing the following security-related documents:

Number 1: Storage

When it comes to Exchange performance, storage isn’t just your friend, it’s your best friend. Your storage solution can make or break performance. In fact, the number one cause of RPC-related communication problems between Outlook and Exchange is storage latency. You need to consider several factors when designing and implementing a high performance storage solution for your Exchange servers. These factors include:

  • User response times
  • System responsiveness
  • Message throughput
  • Uptime and availability
  • Backup windows and recovery service level agreements
  • Overall customer satisfaction

You probably noticed that capacity is not on this list. When it comes to Exchange storage solutions, an oft-repeated mistake is purchasing disks and disk-subsystems based solely on capacity. Creating a high performance Exchange storage solution requires more than knowing how much disk space is needed.

Before purchasing a storage solution for your Exchange server(s) I recommend reviewing the following:

Summary

Achieving optimal performance for your Exchange-based messaging system requires consideration and analysis of a variety of infrastructure components. Exchange Server 2003 integrates with and depends on many external factors, and as a result, Exchange performance depends in part on the design and health of these external factors.

For More Information