Having a solid architecture upon which to deploy Exchange servers is like building a solid foundation before putting a house on top of it. If the foundation is defective, it doesn’t matter how well the house is built.

As stated in the Planning an Exchange Server 2003 Messaging System guide, before you design your Exchange-based messaging system, you need to gather both business and technical information. Exchange depends on a variety of infrastructure technologies, such as TCP/IP networking, name resolution services (DNS and NetBIOS), Active Directory® directory service, and Internet Information Services (IIS). If one or more of these infrastructure components fails, becomes overloaded, or is compromised, Exchange performance can suffer, or worse, your end-users or customers may not be able to use e-mail.

It is particularly important to ensure that a global catalog server is available to your Exchange servers and Outlook clients and that you have sufficient network bandwidth available.

Global Catalog Servers

Client/Server Communications

A very common question asked by Exchange Server architects is How many users can a single Exchange 2003 server support without compromising performance? Or, as my friends ask me, I have X users; can’t you just tell me what server(s) I need? As Sam Khavari stated in his blog in May 2004, it’s difficult to get a straight answer to the server sizing question. The minimum system requirements for Exchange 2003 are the same as those for Exchange 2000; so, if you're moving from Exchange 2000 to Exchange 2003, you won't necessarily need to upgrade your hardware. However, you need to consider whether you will use your servers differently after you install Exchange 2003. For example, if you plan to use query based distribution groups, you may need more powerful hardware for the server(s) that expand your queries.

In almost all cases, you won't need more than 4 GB of physical memory in an Exchange server. Exchange can't use more than 3 GB of physical memory, and it does not support extended memory technologies, such as Physical Address Extensions. When 4 GB is installed in an Exchange server, 3 GB is typically used for Exchange, and the remaining 1 GB is used for Microsoft Windows® and ancillary applications (such as backup, anti-virus, management and other supportive software).

Ensure that you are using the proper hardware for Exchange 2003:

• For Windows 2000 systems, use hardware listed on the Windows 2000 Hardware Compatibility List
  
• For Windows Server™ 2003 systems, use hardware listed on the Windows Server Catalog.
  

If you want more precise specifications, use the following hardware sizing tools:

• Exchange Server 2003 Jetstress
  

The goal of optimizing an Exchange server is to decrease the server's response time while increasing the number of supported users. Optimal performance is achieved by properly tuning and configuring each Exchange server.

When optimizing a server, start with the hardware and ask two questions:

• Are there any BIOS, firmware, or driver updates that need to be applied? Check with your hardware manufacturer to see if updates are available for your server's BIOS, controller BIOS and firmware, network interface card firmware, tape drive firmware, backplane firmware, and so forth. Many firmware updates include driver updates that must be applied first, so be sure to grab those as well. Before applying any updates to your systems, test them in a lab environment that mimics your production environment. If a lab environment is not available, consider testing in a virtual lab environment using technology such as Microsoft Virtual PC 2004 or Microsoft Virtual Server 2005.
  
• Are there any hardware components that can be adjusted or turned off to conserve resources or reduce the attack surface (or the Relative Attack Surface Quotient [RASQ]) of the system? There may be one or more enabled hardware components in your server(s) that aren't really needed. COM ports, parallel ports, and USB ports are some examples. Granted, these ports are useful when performing advanced debugging or running with in a lights-out environment; but if you aren't engaging in these activities, you don't need the ports. Turning them off can also increase physical security because disabling these ports results in fewer I/O channels into the system.
  

Are there ways in which you can tune Windows to increase performance? Sure. Begin by stopping all unnecessary services. For a list of the default services and their startup values, see Default settings for services in Windows Server 2003 product documentation. Also, for a list of the Windows services required for Exchange Server 2003, see Chapter 2, "Exchange Server 2003 Service Dependencies" of the Exchange Server 2003 Technical Reference Guide, as well as the Exchange Server 2003 Security Hardening Guide. These unnecessary services typically include:

• Computer Browser
  
• Distributed File System
  
• Distributed Link Tracking Client
  
• Distributed Transaction Coordinator (except in an Exchange cluster)
  
• Print Spooler (except on an Exchange Server that is also used as a Terminal Server)
  
• Task Scheduler
  
• Wireless Configuration
  

You'll also want to tune the File and Printer Sharing for Microsoft Networks (also known as memory management) setting based on the Exchange server's role and the number of mailboxes present on the system. On front-end servers, bridgehead servers, and mailbox or public folder servers that serve fewer than 500 clients, tune memory management for file caching. On mailbox and public folder servers that server 500 or more clients, tune memory management for network applications.

If Exchange is already running on some or all of your messaging servers, the easiest way to verify that Exchange and Windows are tuned properly is to download and use the Microsoft Exchange Server Best Practices Analyzer Tool (ExBPA).

Exchange 2003 includes support for XML, WebDAV, and Active Server Pages, as well as data access technologies such as OLE DB and ActiveX Data Objects (ADO). You can find a complete list of the available Exchange development technologies in the Exchange Server 2003 Software Development Kit . In general, the more applications you have manipulating the data, the more I/Os per user will be required (because of increased indices, search folders, content conversion, as well as an increase in the number of requests per user).

Regardless of the technology or language you use, consider the following in regard to performance:

• Deep Traversals   Because deep traversals can potentially examine every folder in the hierarchy, they should be used sparingly to reduce server workload. Generally, you should avoid using deep traversals to search across large public folders and large public folder hierarchies.
  
• Message tracking searches using Windows Management Instrumentation (WMI)   The message tracking features in Exchange 2003 enable administrators to track messages passing through Exchange transports. Among other things, message tracking enables you to diagnose message transfer problems and to verify that the Exchange routing configuration is functioning properly. Exchange administrators can use WMI to access the Exchange_MessageTrackingEntry Class programmatically. When doing so, it is recommended that you use date and time restrictions. Without such restrictions, all message tracking entries are searched and overall server performance will be degraded.
  
• Registering store event sinks   If you have to register an event sink for all mailboxes in a store, you should register the event sink for store-wide events. When Exchange starts up, it looks for and loads all registered event sinks, which takes some time. Using many individual event sink registrations instead of a single store-wide event registration item significantly increases server startup time.
  
• Using store event sinks   Always use care when running store event sinks, especially when using synchronous event sinks. A synchronous event sink is triggered after a data request to the Exchange store but before the item is committed to the store. The synchronous event sink has exclusive control over the item while it is processing. Therefore, a complex or poorly written sink can lock a resource for some time, or even bring an Exchange server to its knees.
  
• Use Search Folders for repeated search queries   If you repeatedly use the same search query, it is recommended that you use a Search Folder. When creating a Search Folder, you specify a SQL query that the Exchange store uses to populate the folder with items. After the folder has been populated, links to all items that match your request are placed in the folder. The SQL query is run in the background and periodically updates the folder, which is a more efficient use of system resources. That said, don't use any more Search Folders than you need. If possible, use small scopes for your searches.
  
• MAPI Object Limits   Before developing MAPI-based applications against Exchange 2003, take some time to familiarize yourself with the MaxObjsPerMapiSession limits. Each object requested via MAPI requires resources (specifically, memory in the store's address space) on the server. If an application doesn't adhere to the configured limits, it will use a disproportionate amount of resources and degrade performance.
  

It is not uncommon to find applications running on an Exchange server that work in tandem with Exchange. Examples of these include anti-virus, anti-spam, content and attachment filtering, backup, and workflow and collaboration applications. You should ensure that any third-party software installed on your Exchange servers is compatible with Exchange 2003 and that you are using the most recent versions (including service packs and hotfixes). Ideally, to ensure compatibility, you should test all software interactions in a lab prior to applying the software in a production environment.

Issues with anti-virus, content filtering, backup software, and so forth can be difficult to troubleshoot because they can manifest themselves as problems with Exchange or Windows. I've seen several cases where anti-virus software caused the Microsoft Exchange Information Store process (Store.exe) to consume excessive amounts of CPU time. I've seen other software on an Exchange server causes the Windows local security authority process (Lsass.exe) to consume excessive amounts of memory and CPU time.

If you suspect that an application is degrading performance on your Exchange server, your first action is to temporarily disable the application and see if performance returns to the expected levels. In fact, if you call Product Support Services about a slow performance issue, one of the first things they will probably recommend is to disable third-party applications.

Exchange, IIS, and Windows publish an extensive list of performance objects and counters that can be used to baseline, trend, and proactively monitor each server's four "food groups": processor, memory, disk, and network. Many Exchange-related counters you'll want to monitor are listed in Appendix B, "Performance Counter Definitions" of the Exchange Server 2003 Performance and Scalability Guide. For details about how to isolate performance degradations and how to use existing tools and products (such as Performance Monitor, Load Simulator 2003, Exchange Server Stress and Performance 2003, Network Monitor, and Filemon), see Troubleshooting Exchange Server 2003 Performance.

Knowing which counters to monitor is only half of the battle. You also need to know when to monitor these counters. When an Exchange server first comes online, it is in a transient state. The length of each Exchange server's transient state depends on a variety of factors, such as the number of processors, the amount of installed memory, and the load being handled by it. Generally speaking, the transient state lasts from two hours to one day.

After this period, Exchange enters a stationary state where it should exhibit a consistent load with a predictable variance. This state holds until the load changes or until some external event takes place, such as a nightly backup, data restoration, database maintenance, and so forth.

When you are ready to monitor your servers, Troubleshooting Exchange Server 2003 Performance includes several performance counters and their acceptable thresholds. There are different methods you can use to collect and monitor performance counter data:

• The built-in System Monitor tool of Windows (also known as Performance Monitor and Performance Logs and Alerts).
  
• The sample WMI scripts available for Exchange 2003.
  
• The Performance Monitor Wizard. This is a relatively new tool that simplifies the process of gathering performance monitor logs. This tool can create logs for troubleshooting operating system or Exchange server performance issues.
  

There's also the Server Performance Advisor tool you can use to diagnose root causes of performance problems on a server running Windows Server 2003, including performance problems for IIS 6.0 and Active Directory.

If your organization uses Microsoft Operations Manager (MOM), you can get more comprehensive health monitoring of Exchange by using the Exchange Server 2003 Management Pack for MOM 2000. For details about the rules available in this management pack, see the following guides:

• Exchange Server 2003 Management Pack Rules Technical Reference for MOM 2000 SP1
  
• Exchange Server 2003 Management Pack Guide for MOM 2000 SP1
  

Unplanned downtime of an infrastructure component used by Exchange or Outlook (such as a global catalog server) can cause a substantial reduction in performance. One way to minimize these failures, which in turn can help minimize the overall amount of unplanned downtime, is to adopt and follow specific guidelines around operations, change management, and configuration management. For more than ten years, Microsoft has published and evolved this guidance in two complimentary and well-integrated frameworks: the Microsoft Operations Framework (MOF) and the Microsoft Solutions Framework (MSF).

Whether you are the only IT administrator in your organization, or part of an IT staff, ask yourself the following questions:

• Does your organization routinely make a large number of changes to existing infrastructure (servers, workstations, and applications)?
  
• Does your organization frequently deal with infrastructure or application issues in production?
  
• Do you find it takes an extended period of time to restore service in your organization because the escalation paths are unclear?
  
• Has your organization experienced poor Exchange performance or Exchange downtime due to a failure of people or process?
  

If you answered Yes to any of these questions, MOF and/or MSF may provide answers. For more information about MOF as it relates to Exchange and Exchange performance, see the Exchange Server 2003 Operations Guide.

Resolving problems quickly (and in some cases permanently) can dramatically improve the performance of Exchange. Both Exchange and Windows include troubleshooting tools and error reporting facilities for diagnosing and resolving problems. And both products include documentation about how to troubleshoot issues. However, knowing which tool to use and when to use it can be difficult to determine, especially when you’re sitting in the hot seat trying to fix a problem.

When performing root cause analysis, it is important to scope the problem as quickly as possible. In addition to reducing the time it takes to resolve problems, quickly scoping the problem often reduces the number of resources used while increasing the satisfaction of the users whose problem is being solved. Chapter 3, "Isolating the Causes of the Degraded Performance" of Troubleshooting Exchange Server 2003 Performance provides information about how to isolate the cause of degraded performance. The performance counters listed in that guide are also useful when troubleshooting performance problems.

Secure your Exchange infrastructure or else! When dealing with Exchange, security comes in many forms: permissions and roles, anti-virus, anti-spam, hardening, secure messaging, and secure remote access to name a few. In addition to compromising the integrity of a system, destruction of data, and the theft of confidential information or intellectual property, incorrect security-related settings can cause a substantial reduction in performance.

To help your Exchange infrastructure exhibit top performance, check out the Top 4 Exchange Security Best Practices. As is the case with all of your Windows-based servers and workstations, you'll also want to make sure your systems have the latest security updates.

Because unwanted e-mail consumes resources and can slow performance, familiarize yourself with the anti-spam capabilities of Exchange 2003 and check out the Exchange Intelligent Message Filter. The less that unwanted e-mail is processed by your Exchange servers, the better performance you'll experience.

Finally, I recommend reviewing the following security-related documents:

• Securing Exchange Communications
  
• Exchange Server 2003 Security Hardening Guide
  
• Exchange Server 2003 Message Security Guide
  
• Slowing and Stopping E-Mail Viruses in an Exchange Server 2003 Environment.
  

When it comes to Exchange performance, storage isn’t just your friend, it’s your best friend. Your storage solution can make or break performance. In fact, the number one cause of RPC-related communication problems between Outlook and Exchange is storage latency. You need to consider several factors when designing and implementing a high performance storage solution for your Exchange servers. These factors include:

• User response times
  
• System responsiveness
  
• Message throughput
  
• Uptime and availability
  
• Backup windows and recovery service level agreements
  
• Overall customer satisfaction
  

You probably noticed that capacity is not on this list. When it comes to Exchange storage solutions, an oft-repeated mistake is purchasing disks and disk-subsystems based solely on capacity. Creating a high performance Exchange storage solution requires more than knowing how much disk space is needed.

Before purchasing a storage solution for your Exchange server(s) I recommend reviewing the following:

• Optimizing Storage for Exchange Server 2003
  
• Chapter 4, "Planning a Reliable Back-End Storage Solution" of the Exchange Server 2003 High Availability Guide
  
• Chapter 4, "Tuning Exchange Server 2003 Performance" of the Exchange Server 2003 Performance and Scalability Guide
  
• Nicole Allen’s Exchange team blog entries, A few basic concepts in disk sizing and Some more thoughts on disk IO and calculations.
  
• Evan Dodd’s blog entry on Exchange Disk Sizing
  
• Overview of the Microsoft third-party storage software solutions support policy
  

Achieving optimal performance for your Exchange-based messaging system requires consideration and analysis of a variety of infrastructure components. Exchange Server 2003 integrates with and depends on many external factors, and as a result, Exchange performance depends in part on the design and health of these external factors.

• How to optimize memory usage in Exchange Server 2003
  
• HOW TO: Increase Outlook Web Access Performance
  
• How to troubleshoot virtual memory fragmentation in Exchange Server 2003 and Exchange 2000 Server
  
• Using the / Userva Switch on Windows Server 2003-based computer that are running Exchange Server
  
• Optimizing Storage for Exchange Server 2003
  
• Exchange Server 2003 Performance and Scalability Guide
  
• Best Practice Active Directory Design for Exchange 2000
  
• Exchange Server 2003 Security Hardening Guide
  
• Exchange Server 2003 Message Security Guide
  
• Best Practice Guide for Securing Active Directory Installations