Exchange Server 2003 Processor and Memory Scalability


Topic Last Modified: 2011-01-24

By Nino Bilic

This article addresses questions about the use of different Boot.ini switches on servers that run Microsoft Exchange Server. Unless otherwise specified, the switches mentioned here are for use on servers that run Microsoft Windows Server 2003.

The processor usage on a server should be maintained at a load of about 60 percent during peak working hours. This percentage level allows for periods of extreme load. If the processor usage is consistently greater than 75 percent, processor performance is considered a bottleneck.

Exchange can fully use multiple processors and, in many cases, using servers with more processors improves performance. However, the relationship between the number of processors and performance is complex. If the server has too many processors, the overhead associated with context switching can be greater than the benefit from the additional processors. The optimum number of processors is partly determined by the role that a server has. For example, a back-end mailbox server hosting many MAPI connections may make efficient use of an eight-processor computer. By contrast, a server used to host Microsoft Outlook® Web Access users makes better use of a four-processor computer.

For more information about how different processors perform, see the Exchange Server 2003 Performance and Scalability Guide.

Exchange Server can use computers that have four physical processors to increase performance when hyper-threading is enabled in the computer basic input/output system (BIOS). When you enable hyper-threading on a Xeon MP-based computer that is running at 1.4 gigahertz (GHz) or faster, you can gain a 25 percent increase in processor efficiency when running Exchange Server. Computers that have four Xeon MP 1.4 GHz processors or faster, and that have a 400 megahertz (MHz) host bus, offer high levels of scalability relative to the cost of the computer. We recommend these computers for running Exchange Server 2003 or Exchange 2000 Server.

Four processor servers could be the best overall value when considering the price versus performance for Exchange servers.

Exchange Server can make full use of computers that have eight physical processors. These computers can offer as much as 75 percent more scalability compared to four-processor computers that have the same CPU speed. Typically, on computers that have eight processors, other system limitations are reached before CPU becomes the bottleneck. Because the other limitations are reached first, Exchange Server cannot reach full CPU utilization. We recommend that you use computers that have eight processors when CPU-intensive processes such as context indexing or antivirus scanning is running on the Exchange server.

Exchange Server scales well to eight physical processors on a computer. For computers that have more than eight physical processors, use hardware partitioning to partition the computer into multiple eight-processor computers or into multiple four-processor computers. As an alternative, set the process affinity of the Store.exe process to only eight processors.

For optimum memory usage and efficiency, several tuning options are available through switches in the Microsoft Windows® Boot.ini file. Note that all Boot.ini switch changes require a restart to take effect. Additionally, the features exposed by the use of Boot.ini switches are all Windows features and not Exchange features.

A typical back-end mailbox server will consume between 3 gigabytes (GB) and 4 GB of physical memory. Exchange Server does not utilize instancing, Physical Address Extension (PAE), or Address Windowing Extensions (AWE). Therefore, 4 GB of random access memory (RAM) is the maximum amount of memory that an Exchange Server computer can efficiently use.

Note that running with PAE enabled is supported. Although Exchange Server cannot take advantage of more than 4 GB of memory, other features may rely on the PAE kernel.

If your Exchange Server 2003 server has 1 GB or more of memory installed, and if the computer stores mailboxes or public folders, make sure that you add the /3GB switch to the Boot.ini file on the server. By default, the Windows Server 2003 and Windows 2000 Advanced Server operating systems reserve 2 GB of virtual address space for system usage and another 2 GB for the user address space. Each process is allocated this range of virtual addresses at startup. Typically, the actual memory usage (the working set) of a process is much less than the address space allocated to that process. On a computer that is running Exchange Server 2003 with 1 GB or more of memory, you should modify the Windows Server 2003 and Windows 2000 Advanced Server operating systems so that 3 GB of user address space is available to each process. You can do this by using the /3GB switch in the Boot.ini file.

Remember that while /3GB is beneficial for the smooth running of Exchange, kernel memory resources are squeezed into 1 GB of virtual address space. This means that the maximum paged pool and nonpaged pool sizes are significantly lower. In addition, free system page table entries (PTEs), which map virtual addresses to physical memory, reduce from approximately 200,000 to 15,000. If a server runs low on free system PTEs, instability, including stop errors, can occur. Both Windows Server 2003 and Windows 2000 Server offer mechanisms to increase the number of free system PTEs.

For a Windows 2000 Advanced Server-based computer, you should configure the SystemPages registry in the following registry subkey:

\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SystemPages

Set the SystemPages registry entry to a DWORD value of 0x7918, which is 31000 in decimal notation. In some articles, you may see a recommendation to use 0xFFFFFFFF. Although the latter is a valid setting and can be used to gain the maximum number of free system PTEs, you must remember that memory is taken from the paged pool to bolster the number of free system PTEs. In the scale-up environments that you see today, paged pool resources are also critical to the running of an Exchange server. Therefore, you need to carefully balance the resources of all kernel memory objects. This same registry change is not required, nor is it recommended, on Windows Server 2003-based servers, because this functionality is achieved by using the /USERVA switch that is outlined in a later section.

Do not set the /3GB switch if you are running Windows 2000 Server. This memory tuning switch is not supported on Windows 2000 Server. Although the server will not generate an error message if you do this, the result of setting this switch is that a false memory address space will exist. In cases where a process tries to access this higher address space, a stop error occurs, and the server stops responding.

The /USERVA switch is new to Windows Server 2003 and provides better granularity for splitting virtual address space between user mode and kernel mode. This behavior lets you scale the server to a greater number of users without the risk of exhausting kernel resources. By using /USERVA=3030, an additional 42 megabytes (MB) of memory is allocated to the kernel for PTEs. However, this value may need more tuning. You can monitor the PTE consumption by using Performance Monitor. Monitor the Free System Page Table Entries counter. This counter can be found under the Memory object. If Performance Monitor reports a value of 8,000 or less, the /USERVA value of 3,030 must be reduced to avoid system instability.

Microsoft Product Support Services strongly recommends using a range of memory that is within the range of 2,800 to 3,030 for the /USERVA switch. This range is wide enough to provide a large enough pool of system PTEs for all currently observed issues. Typically, a setting of /USERVA=2800 provides close to the maximum available number of system PTEs that are possible. Remember that when you reduce the value for /USERVA, the large address space benefit to the Exchange store process is reduced. For maximum performance of both Exchange and the underlying operating system, careful tuning and monitoring is necessary.

The following table shows an overview of how and when the /3GB and /USERVA switches should be used. This applies to Exchange Server 2003 with Service Pack 1 (SP1) and Windows Server 2003 SP1.


Exchange Server 2003 server role Physical memory config Additions made to Boot.ini


>1 GB**

/3GB /USERVA=3030

Public Folder

>1 GB**

/3GB /USERVA=3030

Front End (FE)

>1 GB**


SMTP Gateway/Bridgehead

>1 GB**


SMTP Gateway/Bridgehead (Envelope Journaling*)

>1 GB**

/3GB /USERVA=3030

MTA/X.400/3rd Party Connector Bridgehead*

>1 GB**

/3GB /USERVA=3030

* Envelope journaling, MTA, and third-party connectors send gateway and bridgehead e-mail through the Store.exe process so the system benefits from /3GB.

** /3GB and /USERVA=3030 switches are not used on Exchange servers with less than 1 GB RAM.

When working with the /3GB and /USERVA switches, consider the following:

PAE is the added ability of the IA32 processor to address more than 4 GB of physical memory. Microsoft Windows Server 2003, Enterprise Edition, Microsoft Windows Server 2003, Datacenter Edition, Microsoft Windows 2000 Advanced Server, and Microsoft Windows 2000 Datacenter Server, can use PAE to take advantage of physical memory beyond 4 GB. To enable PAE, use the /PAE switch in the Boot.ini file. Newer servers that support hot-add memory and run on the Windows Server 2003 operating system will automatically load the PAE kernel. Additionally, the PAE kernel will be loaded on servers that support hardware-enforced data execution prevention (DEP) and run on the Windows Server 2003 SP1 operating system. This has been a source of confusion in the past as servers were identified as using the PAE kernel even when /PAE was not set in the Boot.ini.

When more physical memory is used on the system, the process of paging memory to the disk increases dramatically, and performance may be negatively impacted. The Windows Server 2003 and Windows 2000 Server memory managers use PAE to provide more physical memory to the operating system. This reduces the need to swap the memory in and out of the page file and results in increased performance. The program itself is not aware of the actual memory size. All the memory management and allocation of the PAE memory is handled by the memory manager independently of the running programs.

The preceding information is valid for programs that run when the /3GB switch is used. A program that requests a large amount of memory is more likely to have more of its memory remain in physical memory rather than be paged out. This increases the performance of programs that are capable of using the /3GB switch. The exception is when the /3GB switch is used with the /PAE switch. In this case, the operating system does not use any memory in excess of 16 GB. This behavior is caused by kernel virtual memory space considerations. Thus, if the system restarts with the /3GB entry in the Boot.ini file, and the system has more than 16 GB of physical memory, the additional physical RAM is not used by the operating system. Restarting the computer without the /3GB switch enables the use of all the physical memory.

Windows Server 2003, Enterprise Edition and Windows Server 2003, Datacenter Edition automatically enable PAE only if the server is using hot-add memory devices. In this case, you do not have to use the /PAE switch on a system that is configured to use hot-add memory devices. In all other cases, you must use the /PAE switch in the Boot.ini file to take advantage of memory over 4 GB.

As you add more memory to a server, the requirements for kernel resources are increased. For example, system PTEs track virtual to physical memory address locations. The more physical memory you have, the more system PTEs will be used, thus resulting in less free system PTEs. Newer servers come preconfigured with 6 GB or 8 GB of memory. As a best practice, for servers dedicated to Exchange, you should consider removing memory over 4 GB or limiting the memory recognized by the operating system with the /MAXMEM switch in the Boot.ini file. If you do not do this, not only will the memory be wasted, but it could lead to further instability due to kernel memory resource constraints.

To determine whether the PAE kernel is loaded when the computer starts, you can view the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PhysicalAddressExtension

If the PhysicalAddressExtension DWORD value is set to 00000001, the PAE kernel is loaded when the computer starts. This key is set by the operating system at startup. Users cannot configure this parameter.

Until recently, it was a best practice to disable PAEs with the /NOPAE switch on Exchange servers where the /3GB switch was set. This recommendation existed because Exchange Server 2003 or Exchange 2000 Server had not been tested with the PAE kernel, nor was the PAE kernel necessary on an Exchange server. New hardware and software capabilities now require the PAE kernel to be loaded. For example, servers with Lindenhurst or Twincastle chipsets require the PAE kernel to be loaded, otherwise not all installed memory will be recognized. In another example, Windows Server 2003 SP1 supports hardware-enforced DEP. This functionality cannot be enabled if the PAE kernel is not loaded.

As a result of testing by the Exchange Server product team, it is now a best practice to allow the PAE kernel to load on an Exchange server. Therefore, the /NOPAE switch is no longer necessary unless you are troubleshooting a specific issue where the PAE kernel is suspected as a contributory factor. On some servers running Windows Server 2003 SP1, the /NOPAE switch may be ineffective because hardware-enforced DEP, which is enabled with the /NoExecute switch, will override this instruction to disable PAE. If required, the /Execute switch can be used to disable the PAE kernel. For a description, see Microsoft Knowledge Base article 900524, "How to prevent the PAE kernel from loading in Windows Server 2003 with Service Pack 1 and later versions of Windows Server 2003 or in Windows XP with Service Pack 2 and later versions of Windows XP."

There are several other settings that can be changed on Exchange Server 2003 servers to optimize memory utilization. Some of the settings are the following:

  • Maximizing virtual address space by changing the HeapDeCommitFreeBlockThreshold registry value. This registry setting is critical to the health and stability of back-end Exchange servers.

  • Changing the Store Database Cache size, also known as the Extensible Storage Engine (ESE) buffer size, by making Active Directory® directory service changes.

For more information about memory-related settings, see Microsoft Knowledge Base article 815372, "How to optimize memory usage in Exchange Server 2003."

Event 9665, which is a warning event, might be logged in the application log of an Exchange Server 2003 server. The event wording will be as follows:

Event Type: Warning

Event Source: MSExchangeIS

Event Category: General

Event ID: 9665


The memory settings for this server are not optimal for Exchange.

This is a part of new functionality built into Exchange Server 2003, which checks for several known memory configuration settings at startup.

If Event 9665 is logged, for more information about troubleshooting, see Microsoft Knowledge Base article 815372, "How to optimize memory usage in Exchange Server 2003."

Consider the following frequently asked questions (FAQ).

If your Exchange Server 2003 server has 1 gigabyte (GB) of memory or more installed, and if the computer stores mailboxes or public folders, make sure that you add the /3GB switch to the Boot.ini file on the server. For details about when to use this switch, see the /3GB and /USERVA Overview (Table View) section in this document.

An Exchange server cannot efficiently use more than 4 GB of RAM. Exchange Server does not utilize instancing, PAE, or AWE. Therefore, 4 GB of RAM is the maximum amount of memory that an Exchange server can efficiently use.

Exchange Server does not utilize instancing, PAE, or AWE. The /PAE switch is supported in Exchange Server 2003, but that Exchange server is not able to take advantage of more than 4 GB of RAM even with the /PAE switch in place.

We recommend and support running Exchange Server 2003 with the PAE kernel on Windows Server 2003. To ensure system stability, either install the latest hotfixes for Windows Server 2003 or install Service Pack 1. Running the PAE kernel outside of this configuration may result in store failures.

Windows Server 2003, Enterprise Edition and Windows Server 2003, Datacenter Edition automatically enable PAE if the server supports hot-add memory. In this case, you do not have to use the /PAE switch. Do not use the /NOPAE switch on Exchange servers because this switch may cause unintended side effects. Two issues that result from the use of the /NOPAE switch are:

  • Hardware enforced DEP is disabled on servers running Windows Server 2003 SP1.

  • Servers using Lindenhurst or Twincastle chipsets will not be able to recognize all physical RAM.

In addition, on servers with Windows Server 2003 SP1, the /NOPAE switch will have no effect if hardware-enforced DEP, through the /NoExecute switch, is enabled.

Use the Microsoft Exchange Server Best Practices Analyzer Tool. The tool has been updated to include the support policy change outlined in this article. For more information, see Microsoft Exchange Server Best Practices Analyzer Tool.

We support, but do not recommend, running Exchange on Windows 2000 Server with PAE. Hotfix 838647 is required for support. Windows 2000 Server does not have the security features that require PAE, so the only benefit of running PAE on Windows 2000 Server for Exchange is to enable all memory access on newer PCI-express-based servers. It is not a well tested scenario, so it is not recommended.

The memory growth in Store.exe that can be seen in Task Manager is not necessarily a problem. You need to determine if there are any performance problems on the server, otherwise it is not a problem. For more information, see Why is Exchange Store.exe so RAM hungry?