Exchange Server TechCenter

Microsoft Exchange Server 2007

Table of Contents

• Introduction
  
• Outlook Connections
  
• TCP Protocol Connection Limit
  
• Outlook Anywhere Path
  
• Port Exhaustion Causes Scalability Limitations
  
• Mitigating Port Exhaustion Scalability Limitations
  
• Windows Server 2008 and Multiple IP Addresses
  
• Refer Outlook Directly to Global Catalog Servers
  
• LoadGen Does Not Simulate DSProxy Connections
  
• Client Network Traffic
  
• Conclusion
  
• Additional Information
  

Outlook Anywhere scalability and client network traffic are two areas that generate many questions about the number of connections Outlook makes and sustains to an Exchange server. This area is frequently the subject of discussion when site consolidation is being discussed which also raises the issues of network costs and Transmission Control Protocol (TCP) connection limits. The TCP connection limitations are largely hit by hosting companies and large enterprise customers who force all MAPI connectivity through RPC over HTTP (RCP/HTTP). In the following sections we will cover each of these areas in detail to help show the behavior you can expect to see when using Outlook Anywhere in your Exchange 2007 deployment.

Because of many variables that can exist in an Exchange 2007 environment, it is difficult to provide a solid number of client connections for all possible variables. The actual number of connections that will be seen in a non-default Exchange 2007 environment can vary based on using ISA Server, public folders, Outlook Add-ins, and so on. Outlook connections can also vary based on the features or usage patterns of the client, including accessing shared calendars, public folders, or offline address books. Because of these variables, it is most useful to provide information about the connection values that will be seen in a default Exchange 2007 installation. The numbers provided in this topic were collected by running TCPView on a default installation of Exchange 2007. This includes a server that has the Mailbox server role installed, a server that has the Client Access server role installed, Windows Server 2003 Active Directory, and default Outlook 2003 and Outlook 2007 clients. For more information about TCPView, see TCPView for Windows v2.53.

By default, at initial logon through Outlook Anywhere, Outlook2003 or 2007 running in cached mode uses six directory TCP connections and four TCP store connections. After a minute, directory connections are closed, and the number of connections is reduced to two consistent connections. An idle Outlook client uses two TCP directory connections and four TCP store connections. If the client is in online mode, and connects to an address book, Outlook opens two additional TCP directory connections. Table 1 provides an illustration of these connections.

The TCP protocol has a requirement that each connection have a unique ordered list, also known as an n-tuple, which consists of source address, source port, destination address, and destination port. All incoming connections use the same destination address or port, so the number of incoming connections is limited by the non-paged pool size. Each outgoing connection consumes a port on an address. The TCP port is a 16-bit number, so there are at most 65,535 ports.

The change to 64-bit hardware in Exchange 2007 exposed this scalability limit. In Exchange 2003, the memory constraints in 32-bit hardware hide this limit and because of those memory constraints, memory availability would be exhausted before the TCP connection limit could be reached. Now, with 64-bit hardware and an almost endless amount of memory, Exchange is no longer limited in this area and can therefore hit the TCP connection limitation. Usually, this affects enterprise customers who are running at very high scale and who are trying to maximize as much scale-up from their hardware as possible.

Return to top

RPC/HTTP is a tunneling protocol where Exchange uses a pair of virtual channels to create a virtual connection from Outlook to Exchange. Each virtual channel is a single directional data stream that is transported over various real channels. The client to RPC Proxy channel is HTTP/HTTPS and the RPC Proxy to Exchange channel is TCP. The client then establishes four channels. Data flows on these as follows:

1. Client to Proxy
   
2. Proxy to Exchange
   
3. Exchange to Proxy
   
4. Proxy to Client
   

Once all four channels are established, RPC then treats this as a single full duplex tunneled connection from Outlook to Exchange. Each real channel can be replaced without interrupting the data flow over the virtual connection.

Exchange has two kinds of connections, mail and directory. Each of these connections will appear as a pair of virtual channels. Mail connections flow from Outlook to the RPC Proxy component on the Client Access server to the Mailbox server. In deployments were Internet Security and Acceleration (ISA) Server is used, ISA Server will proxy these connections to the Client Access server (RPC/HTTP Proxy). Because ISA Server is still a 32-bit application, it will be unable to scale the TCP connections to the physical connections limit before it runs out of available non-paged pool memory. Non-paged pool memory is used for managing the high number of connections established. This limit will be reached before any Exchange limits are reached. The testing documented here does not deal with this issue. However, it is an important consideration for any real-world deployment. Exchange then uses its data store to service the requests and replies to the client. The directory connections flow from Outlook to the RPC Proxy component on the Client Access server to the DS Proxy component on the Mailbox server to an Active Directory global catalog server. The RPC connection is processed on the DC (not on the Exchange server), with the DS Proxy component merely copying bytes from one TCP connection to the other. The large number of outbound connections from Exchange to the DC is a function of the DS Proxy component that tunnels connections.

The TCP connections limits discussed earlier in this topic exist in both Windows Server 2003 and Windows Server 2008 as consumers of the TCP protocol. One IP address is used as the source IP address when it opens a connection to a remote computer. Each Client Access server is bound by the Windows port limit of 65,535 available ports. The Client Access server depletes the available pool of ports as each client uses anywhere between 2 and 8 connections. The information store process has a hard limit of 60,000 RPC context handles which are associated with each RPC/HTTP virtual connection between Outlook and Exchange. Therefore, the store process is limited to 60,000 of these mail connections.

The following performance counters are helpful in determining whether a server is reaching this limit:

RPC/HTTP Proxy (Windows Server 2008 Only)

• Current number of incoming RPC/HTTP connections
  
• Current number of unique users
  
• RPC/HTTP requests per second
  
• Number of failed back-end connection attempts
  

MSExchangeIS

• RPC Averaged Latency
  
• RPC Requests
  
• RPC Operations/Sec
  

Web Service (Windows Server 2003 Only)

• Current Connections
  

  Note:
  These counters are only useful on servers where no other Web service is used.

Memory

• Non-paged pool
  
• Paged Pool
  

Process

• Private bytes / LSASS, W3WP and any Exchange-specific processes running
  

The current number of incoming RPC/HTTP connections and current number of unique users counters that are available with Windows Server 2008 will determine how many user connections there are and how many different NT user accounts are connected. The other counters will help determine potential causes of the denial of new user connections to a server and how the server is failing.

Return to top

The key discovery and conclusion is that a Mailbox server will deplete outbound source IP ports far more quickly than it will reach any inbound limit. This occurs because of the way that DSProxy operates. DSProxy opens a single outbound connection for every inbound connection it receives. For every inbound connection to DSProxy, the Mailbox server opens an equivalent number of outbound connections to a global catalog.

As stated previously, when a client is being used, the number of TCP connections used will change. By default, at initial logon through Outlook Anywhere, Outlook2003 or 2007 running in cached mode uses six directory TCP connections and four TCP store connections. After a minute, directory connections are closed, and the number of connections is reduced to two consistent connections. An idle Outlook client uses two directory and four store connections. If the client is in online mode, and connects to an address book, Outlook opens two additional directory connections. These numbers should be reduced by half for non-Outlook Anywhere scenarios. This is important to be aware of when you view Table 2 because startup connections are not accounted for in these numbers. Therefore, at that point in the client access process, more connections will be used than listed in the following table. This behavior is seen in both Outlook 2003 and Outlook 2007. It is impossible to predict exactly how many connections will be used because of the previously mentioned variables. But an increase of between 25 and 50 percent from startup has been regularly observed.

Other related observations are:

• Clients do not share connections. New connections are established for every new client connecting.
  
• Connections are removed as soon as a client logs off.
  
• If a client opens a mailbox on a Mailbox server other than the one hosting their mailbox, or views a calendar or folder on another Mailbox server, additional TCP connections are established from the Client Access server to that Mailbox server.
  
• If multiple mailboxes or calendars are viewed on a Mailbox server other than the one hosting their mailbox, no additional connections are created beyond those established for the first mailbox or calendar viewed.
  
• Because ISA Server is still a 32-bit application, it will be unable to scale the TCP connections to the physical connections limit before it runs out of available non-paged pool memory. Non-paged pool memory is used for managing the high number of connections established. This limit will be reached before any Exchange limits are reached. The testing documented here does not deal with this issue, but it is an important consideration for any real-world deployment.
  

Return to top

There are two possible ways to mitigate the issue of port exhaustion: by using Windows Server 2008 with multiple IP addresses, reverting to Exchange 2003 RTM behavior and scaling out the Client Access server deployment by adding additional Client Access servers to service client connections.

A registry key was created in Windows Server 2008 which allows the server to use 65,535 outbound connections for each IP assigned to the system, even for multiple IPs assigned to the same network adapter. This feature will allow some additional headroom but does not address the other limits, such as the store connection limit. It should also be noted that each RPC/HTTP virtual connection consumes 61 KB of RAM on the Client Access server so that a server that will be using many TCP connections must be configured with sufficient RAM to manage the connections and also the load Exchange puts on it. If you do not plan for this, you will be encountering issues related to memory pressure which can cause the server to thrash (continuously page). For information about implementing the registry change detailed here, see Microsoft Knowledge Base article, How to Enable Port Scalability for RPC Proxies.

Another method that is available for mitigating the issue of port exhaustion is to refer Outlook directly to global catalog servers and scale out your Client Access server deployment by adding additional Client Access servers to service the client connection load.

The following change must be made to enable this configuration for the Mailbox server:

The Do Not Refer HTTP to DSProxy key must be set as detailed in How to control the DSProxy process for RPC over HTTP connections in Exchange Server 2003 SP1.

The following changes must be made to enable this configuration for the Client Access server:

Use the following procedure to modify the ValidPorts setting in registry key HKLM\Software\Microsoft\RPC\RPCProxy so that the entries referring to 6004 point to every available global catalog rather than to a Mailbox server. These entries must exist for both the fully qualified domain name (FQDN) and the short NETBIOS name of every available global catalog.

Use Registry Editor to modify the ValidPortsGC value

1. On the Exchange server that has the Client Access server role installed, open a Registry Editor.

2. Browse to: HKLM\Software\Microsoft\RPC\RPCProxy

3. Right-click ValidPortsGC and then click Modify.

4. In the Value data field, enter the FQDN and NETBIOS name for every available global catalog in the following format: GC01:6004;gc01.contoso.com:6004;

5. Close Registry Editor.

6. Restart the server for changes to take effect.

Use the following procedure to modify the PeriodicPollingMinutes value in registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\ and configure a value of "0" to prevent the System Attendant (SA) from updating the ValidPorts key automatically.

Use Registry Editor to modify the PeriodicPollingMinutes value

1. On the Exchange server that has the Client Access server role installed, open Registry Editor.

2. Browse to: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\

3. Right-click PeriodicPollingMinutes and then click Modify.

4. In the Value data field, enter a value of "0" without quotation marks.

5. Close Registry Editor.

6. Restart the server for changes to take effect.

The following changes must be made to enable this configuration for the global catalog server:

Use the following procedure to create a Multi-String Value entry named NSPI protocol interface sequences in the registry key HKLM\System\CCS\Services\NTDS\Parameters\ on each global catalog server and set its value to ncacn_http:6004.

Use Registry Editor to create the NSPI protocol interface sequences Multi-String Value entry

1. On the Exchange server that has the Client Access server role installed, open Registry Editor.

2. Browse to: HKLM\System\CCS\Services\NTDS\Parameters\

3. Right-click in the action pane, select New\Multi-StringValue, and enter the name NSPI protocol interface sequences

4. Right-click NSPI protocol interface sequences and then click Modify.

5. In the Value data field, enter a value of "ncacn_http:6004" without quotation marks.

6. Close Registry Editor.

7. Restart the server for changes to take effect.

Return to top

The Microsoft Exchange Load Generator (LoadGen) tool does not simulate any DSProxy connections. The affect of this is not significant from a performance perspective, but it is significant from a scale testing perspective. Customers who use LoadGen to simulate Outlook Anywhere users will not hit the outbound ports scalability issue described earlier in this topic. This will result in a test where a much larger number of users will be able to connect to Exchange by using Outlook Anywhere than would be able to do this in a production environment. The load on the server is believed to be minimal, but the missing DSProxy connections will allow the server to support far more clients during LoadGen testing than it would allow in a production environment.

The LoadGen Tool team is investigating adding support for directory connections in a future release of the LoadGen tool. Until LoadGen is updated to reflect these connections, it is critical that scalability testing of Outlook Anywhere with LoadGen not be used exclusively to determine the maximum number of concurrent users that a server can support.

LoadGen connections (Store connections only)	RPC connections	TCP connections (Outlook Anywhere only)
Outlook 2003 online	1	2
Outlook 2003 cached	1	2
Outlook 2007 online	1	2
Outlook 2007 cached	2	4

As part of the Outlook Anywhere scalability testing, analysis was done to determine the network costs between enterprise e-mail clients and Exchange 2007 SP1. The values presented here may help an organization determine an estimated value for the network use requirements that are part of connecting end-users to the Exchange 2007 infrastructure. The testing performed in this analysis included the following scenarios: Outlook 2007 online mode; Outlook 2007 cached mode; Outlook 2007 cached mode through RPC/HTTP (Outlook Anywhere) and Outlook Web Access. No reporting on the network bytes passed between Exchange roles was performed. This analysis is limited to the bytes entering and leaving the datacenter. Outlook Anywhere and Outlook Web Access connect to the Exchange servers with the Client Access server role installed, while Outlook 2007 (in both online and cached mode) connects directly to the Exchange servers that have the Mailbox server role installed. The network traffic from previous Outlook versions can be estimated from the Exchange 2003 results that are published in the Client Network Traffic with Exchange Server 2003 white paper because there have not been fundamental changes in Exchange-Outlook communications in the 2007 releases.

The user profile started with the message send and delivery rates from the "light", "medium", "heavy" and "very heavy" knowledge worker profiles. The following assumptions were made for the purposes of these tests:

• An average message size of 50 KB
  
• Every message delivered was read
  
• Half of all incoming mail was deleted
  
• Web clients logged on and logged off two times per day
  
• Logon and logoff costs from the other client types were not evaluated because enterprise e-mail users generally stay logged on for days at a time.
  

Profile	Light	Medium	Heavy	Very heavy
Sent per day	5	10	20	30
Received per day	20	40	80	120
Avg. message size	50k	50k	50k	50k
Messages read per day	20	40	80	120
Messages deleted per day	10	20	40	60
Outlook Web Access logon and logoff per day	2	2	2	2

The network bytes transferred for each action is independent of mailbox size, so separate measurements for each profile were not performed, but measurement of the costs of the actions were made and totaled for each profile.

In the following table, all values are in kilobytes per day per user. The sending portion has been separated from the other actions, which are labeled as 'aggregate'.

Profile		Light	Medium	Heavy	Very heavy
	Sending	190	380	760	1,140
Outlook 2007 online mode	Aggregate	2,510	5,030	10,050	15,070
	Total	2,700	5,410	10,810	16,210
	Sending	260	520	1,040	1,560
Outlook 2007 cached mode	Aggregate	1,040	2,080	4,160	6,240
	Total	1,300	2,600	5,200	7,800
	Sending	310	620	1,230	1,850
Outlook Anywhere in Exchange 2007	Aggregate	1,230	2,470	4,940	7,400
	Total	1,540	3,090	6,170	9,250
	Sending	800	1,600	3,200	4,800
Outlook Web Access 2007	Aggregate	5,390	10,620	21,070	31,530
	Total	6,190	12,220	24,270	36,330

To better understand how these values can be used, consider the following example:

Suppose a datacenter has 10,000 "Medium" Outlook cached mode users. Further assume these users are in the same time zone and they perform most of their work during an 8-hour day. The graphic here predicts what the average network bytes per second would be.

or

Assuming a daily peak of twice the average value, the network coming into the datacenter would have to support approximately 15 megabits per second from these users alone.

If these users were running in online mode, per-user bandwidth consumption value would be replaced as shown in the following formula:

or

Assuming a daily peak of twice the average value, the network coming into the datacenter would have to support approximately 30 megabits per second from these users.

Return to top

By using the information that is provided here, you can start to evaluate how to properly size your Outlook Anywhere deployment and the network utilization requirements for your Exchange 2007 environment.

For the complete Exchange 2007 documentation set, see the following resources:

• Exchange Server 2007 Help.
  
• Exchange Server TechCenter