Deploying a Worldwide Site Consolidation Solution for Exchange Server 2003 at Microsoft

Technical White Paper

Published: June 11, 2004 | Updated: August 9, 2004

Using Microsoft Exchange Server 2003 and Microsoft Office Outlook 2003 to consolidate more than 75 messaging sites worldwide into seven physical locations

Executive Summary

Prior to deploying its physical site consolidation solution, the Microsoft IT group used pre-release versions of Microsoft® Exchange Server 2003 and Microsoft Office System 2003 to upgrade its existing Exchange Server 2000 messaging platform. Microsoft IT then undertook the deployment of its physical site consolidation solution to reduce the number of physical locations running Exchange Server from 75 to seven physical sites (six regional data centers [RDCs] plus one Exchange site in Johannesburg, South Africa).

The Exchange Messaging team worked in conjunction with several teams from across Microsoft to redesign and deploy the Exchange Server 2003 site consolidation solution. This solution not only reduced the number of physical sites, Exchange servers, and other supporting servers, but also continued to deliver high employee satisfaction and productivity measurements and create opportunities for further server consolidation.

The Exchange Server 2003 physical site consolidation is a key component of the internal Model Enterprise (ME) initiative at Microsoft. The goal of the ME initiative is to model a continuously optimized enterprise infrastructure that satisfies the business needs of Microsoft by striking the right balance of availability, performance, flexibility, and cost – while using Microsoft products and solutions wherever possible. A key component of the ME initiative is the optimization of the overall enterprise total cost of ownership (TCO) for the Microsoft IT infrastructure taking into account Microsoft IT's unique role testing and deploying new versions of Microsoft software in a production enterprise environment.

The specific objectives of the Exchange Server 2003 physical site consolidation were to:

• Reduce the number of worldwide physical sites running Exchange Server 2003 servers from the current number of 75 to six RDCs (plus one remaining tail site) aligned with the Microsoft ME Initiative.
• Detailed analysis of the attendant reduction in annualized costs associated with the reduction in the number of Exchange Server physical sites.
• Accurately monitor, gather, analyze, and report on changes in network bandwidth utilization and latency directly related to the Exchange Server physical site consolidation.
• Measure any perceived changes in Microsoft employees' Microsoft Office Outlook® client user experience through surveys taken before and after the physical site consolidation.
• Document Microsoft IT's Exchange Server physical site consolidation experiences, best practices, and new lessons learned for the benefit of Microsoft customers.

To address these objectives, Microsoft IT moved toward a fully clustered mailbox server environment running on a smaller number of regionally located data centers. Each of these server clusters is connected to one or more Storage Area Network (SAN) enclosures for its data storage. The use of clustering technology has improved reliability and increased availability.

The planning, design, and deployment of the Exchange Server 2003-based physical site consolidation solution by the Microsoft IT Exchange Messaging team met these objectives and produced the following key business benefits:

• Reduced number of active mailbox servers by almost 70 percent and associated server operational costs, resulting in overall messaging cost reduction of at least four percent
• Maintained user satisfaction and productivity by reducing dependency on real-time network connectivity
• Created opportunity for further consolidation of distributed sites.

This white paper is for Microsoft customers who are running Microsoft Exchange Server in a distributed server environment who want to understand how they can benefit from consolidating the resources in their messaging infrastructure.

The focus of this white paper is on the Microsoft Information and Technology group's experience in consolidating a large number of Exchange Server 2003 physical sites into a much small number of RDCs. However, this white paper will also be of interest to Microsoft customers who are running Exchange Server 5.5 or Exchange Server 2000 in a distributed server environment and are looking to take advantage of the messaging infrastructure consolation benefits offered by Exchange Server 2003.

This white paper was specifically written for enterprise, business, and technical decision makers; IT architects; and operations managers who are considering an upgrade of their messaging infrastructure, or an implementation of a physical site or server consolidation strategy. It focuses on the needs of the Exchange messaging team who work in an organization with a physically distributed environment.

Note   For security reasons, the sample names of forests, domains, internal resources, organizations, and internally developed security file names used in this paper do not represent real resource names used within Microsoft and are for illustration purposes only.

Introduction

Customers frequently ask Microsoft about the methods employed and lessons learned when using Microsoft products and solutions in the company. This is where Microsoft IT serves an important purpose. Not only does Microsoft IT provide traditional IT functions for the company, but they also act as the company's first and best customer for each new server and business productivity software release. When an issue is encountered, Microsoft IT works closely with the Microsoft product development groups to address the issue.

Prior to the start of the Exchange Server 2003 physical site consolidation project, the following Exchange messaging team goals had already been achieved:

• Centralization of Exchange Server operations and Microsoft IT help desk functions during the migration from Exchange Server 5.5 to Exchange Server 2000. The centralization reduced the number of skilled Exchange administrators required at each local Exchange site. This was completed in 1999 and 2000.
• In-place upgrade of the individual Exchange Server 2000 servers to Exchange Server 2003. This was completed in the first quarter of 2002.

The pre-consolidation Exchange messaging environment at Microsoft is illustrated in Figure 1. Microsoft IT had 118 Exchange mailbox servers deployed into 75 locations. There were 94 additional Exchange public folder, Internet gateway, and other Exchange servers deployed to support these sites. Domain controllers, Global Catalog servers, and file and print servers were also deployed in these locations to support the local Microsoft employees. Prior to the start of the Exchange Server 2003 physical site consolidation project, all of Microsoft IT's Exchange servers had been upgraded in-place to Exchange Server 2003. This significantly reduced the complexity and risk for the physical site consolidation project and provided a more accurate baseline for comparing pre- and post-consolidation network utilization and latency as well as user's satisfaction with the messaging service provided by the RDCs.

If your browser does not support inline frames, click here to view on a separate page.

Figure 1   Pre-consolidation Exchange Server 2003 mailbox deployment

Pilot physical site consolidations began in the second quarter of 2002 and the worldwide production consolidation was completed in the first quarter of 2003 – approximately 12 months later.

The post-consolidation Exchange messaging environment is illustrated in Figure 2. Approximately 75 Exchange server physical sites were consolidated into seven physical locations. The number of Exchange mailbox servers was reduced by almost 70 percent from 118 to 36. The key enablers were the extensive use of Windows® Server 2003 and Exchange Server 2003 server clusters and SAN storage solutions coupled with the new consolidation features in Exchange Server 2003 and Outlook 2003; and a well-designed Wide Area Network (WAN) topology with capacity to handle increased messaging traffic.

If your browser does not support inline frames, click here to view on a separate page.

Figure 2   Post-consolidation Exchange Server 2003 deployment RDCs

A detailed description of the Microsoft pre-consolidation Exchange messaging environment, the project team organization, and its goals are provided in the Situation section. For those readers unfamiliar with Exchange Server 2003 and Outlook 2003 physical site consolidation features, a description of this feature set is also provided.

Related white papers document Microsoft IT's previous experiences with planning, implementing, and deploying Exchange Server 2003. An extensive list of white papers can be found at the end of this document. It is assumed that the reader has basic familiarity with Windows Server™ 2003, Exchange Server 2003, and Office System 2003 (particularly Outlook 2003).

Although this white paper provides recommendations based on what Microsoft experienced as an early adopter, it is not intended to serve as a procedural guide. Each enterprise environment is unique. The plans and lessons described in this white paper need to be adapted to the specific needs and requirements of each individual organization.

Situation

Microsoft began focusing on consolidating its IT infrastructure in 1999 with the deployment of Windows 2000 Server, the Microsoft Active Directory® directory service, and Exchange Server 2000.

Microsoft IT's recent efforts to consolidate Exchange Server 2003 physical sites represent a major initiative to further optimize the TCO of its enterprise IT infrastructure while promoting its ongoing strategy to create a model enterprise IT architecture with a global scope.

Previous Infrastructure Consolidation Efforts by Microsoft

In 1999, Microsoft began one of its largest ever server migrations when it upgraded its Windows NT® 4.0 server environment to Windows 2000 Server. At the same time, they deployed Windows 2000 Server Active Directory services in preparation for the upgrade from Exchange Server 5.5 to Exchange Server 2000.

In total, Microsoft has identified six different approaches for reducing costs through IT infrastructure consolidation:

• Physical Site
• Server
• Database
• Application and Services
• Operations Management
• Operating Environment

More information on these six different approaches for reducing IT infrastructure costs through consolidation can be found in Appendix A – IT Infrastructure Consolidation Strategies Overview.

In 1999, a uniform, across-the-board upgrade from Windows NT Server 4.0 to Windows Server 2000 and from Exchange Server 5.5 to Exchange Server 2000 reduced infrastructure costs through consolidation of operations management and operating environments.

The upgrade to Exchange Server 2000 focused on reducing the two key Exchange Server cost components: help desk costs and operations management costs. In a typical Microsoft customer organization, these two cost factors together account for 52 percent of the TCO, according to a META Group TCO study of Exchange 2000 deployments. Figure 3 shows how these two costs compare with the other annualized cost components of an Exchange Server 2000 messaging solution in a typical Microsoft customer environment.

Figure 3   Typical Exchange Server 2000 cost factors

As part of the upgrade from its Exchange Server 5.5 messaging environment to Exchange Server 2000, Microsoft IT took the proactive step of centralizing its Exchange messaging operations as well as its internal help desk services to achieve its first major infrastructure consolidation cost savings.

In 2000 and 2001, Microsoft IT undertook a number of "mini-consolidations" of selected, small Exchange Server 2000 tail sites where server hardware was close to being out-of-warranty and there was sufficient WAN bandwidth to support the consolidation of the site into its nearest neighbor site.

In the second quarter of 2002, Microsoft began its second major Exchange Server messaging infrastructure consolidation based on the additional consolidation features available in Exchange Server 2003 and Outlook 2003. Following the in-place upgrade of the company's existing Exchange Server 2000 servers to Exchange Server 2003, the Microsoft IT Exchange Messaging team began its Exchange Server 2003 physical site consolidation project – the focus of this white paper.

Project Team Members

The deployment of Microsoft IT's Exchange Server 2003 physical site consolidation at Microsoft involved a number of different groups. The Microsoft IT Exchange Messaging team led the planning, design and execution of the worldwide physical site consolidation, working closely with the Exchange Server and Outlook product groups.

Microsoft Information Technology Group

Microsoft IT is responsible for driving global operations and delivering information technology services to the entire Microsoft organization. The group directs all activities related to running and maintaining Microsoft information systems worldwide: technology infrastructure; and corporate and marketing information systems including production, distribution, and other key internal systems. Microsoft IT works to provide a world-class utility and excellence in business operations through its leadership in the design and integration of company strategies, processes, and architecture.

Microsoft IT provides a full range of services including server- and end-user support, telecommunications management, network operations, and information security. This role includes managing connectivity for more than 300,000 personal computers worldwide. Microsoft IT ensures that more than 50,000 employees and 20,000 contractors and vendors in more than 400 Microsoft locations are able to access corporate network services and resources 24 hours a day, seven days a week, from around the world.

Because the primary business of Microsoft is software design, Microsoft IT has an additional responsibility that is unique among global providers. In addition to running the company's IT utility, Microsoft IT is an early adopter of Microsoft technologies. They are responsible for testing and deploying Microsoft products such as Windows Server 2003, Exchange Server 2003, and SharePoint® products and technologies before these products are released to customers. This process is known by those within Microsoft as "eating our own dogfood" or simply "dogfooding."

Dogfooding New Software Releases

In the Microsoft IT "dogfood" messaging environment, servers regularly receive software patches, Windows Server test releases and upgrades, Exchange server test releases and upgrades, and more. Each Exchange "dogfood" server is regularly upgraded: twice a month on average. The changes are implemented to test new scenarios, stress specific requirements, and continually run the latest application concepts through real world, enterprise-level production-environment testing. The rate of change in the Microsoft IT Exchange Messaging environment is very high.

"Dogfooding is a crucial part of the product release cycle. Its part of how we ensure that our customers get the most reliable, secure and powerful product we can produce. If we're asking customers to use our products, it's only right that we do so ourselves, first."

Chris Baker, Group Product Manager
Exchange Server Product Group
Microsoft Corporation

Microsoft IT Exchange Messaging Team

The Microsoft IT Exchange Messaging team comprises the groups listed in Table 1.

Table 1   Exchange Server 2003 Physical Site Consolidation Project Team

Site Consolidation Deployment Team

In total, there were 12 people from the Microsoft IT Exchange Messaging team directly involved in different stages of the Exchange Server physical site consolidation.

In addition to the Microsoft IT Exchange Messaging team, the Exchange Server product group, and the Outlook product group, other stakeholders in the design, deployment, and operation of the Exchange Server 2003 physical site consolidation included the following groups from Microsoft IT:

• Client Services (including the central help desk and approximately 200 regional IT personnel and account managers)
• Collaboration Services
• Network Connectivity
• Data Center Services
• Identity Management
• Enterprise Application Services
• Security
• Windows Infrastructure Services

Operational Environment

Microsoft employees use e-mail as a primary mode of communication and as a result place a significant load on the Microsoft IT messaging infrastructure. The average employee at Microsoft possesses three computers, all of which are typically used to synchronize with their Exchange server. In addition, a portion of that population also carries a Pocket PC or Smart phone device that also synchronizes with Exchange. At Microsoft, the average Remote Procedure Call (RPC) execution rate (measured in operations per second), is significantly higher than at any other company known to Microsoft IT. The workload managed by the Exchange servers at Microsoft is typically more than double the load measured at other companies. The following statistics help characterize the use of e-mail at Microsoft:

• Global e-mail flow of 11,800,000 messages per day including:
  • 8,800,000 average incoming Internet e-mail messages per day — 70 percent of which are filtered out as either unwanted spam e-mail, virus-infected, or requesting delivery to invalid e-mail addresses
  • Average size of a typical e-mail message is 44 KB
• Approximately 100,000 mailboxes
• The maximum mailbox size was increased from a 100 megabyte (MB) to 200 MB limit (as part of the physical site consolidation project)
• More than 85,500 distribution groups
• More than 230,000 unique public folders
• 20 databases per server, with 50 gigabyte (GB) maximum database size on new clustered deployments, and a maximum mailbox size of 200 MB
• Worldwide e-mail delivery in less than 90 seconds, 95 percent of the time
• Backup and restore operation SLA of less than one hour per database

The Microsoft IT server infrastructure includes:

• Clustered mailbox servers employing SAN configurations are scaled per server to support 2,700 user mailboxes in regional locations and 4,000 user mailboxes in the headquarters data center.
• Support organization, which supports all Exchange servers worldwide, consolidated into two centers: Redmond, WA (and Denver, CO for additional user support).
• In addition to the primary corporate Active Directory forest, three additional forests are used for dogfooding purposes and have Exchange mailbox servers:
  • A Level A Test forest, dedicated to running development and test code for Exchange, operating in a frequently changing Exchange server software environment.
  • A specialized Level B Test forest, serving as a limited-use production environment used by one product division that hosts a limited number of user mailboxes. Specialized hardware configurations and test scenarios can be run in this environment. Level B Test uses a two-node server cluster connected to a SAN scaled to support 5,000 user mailboxes.
  • A legacy test environment forest, used for testing the compatibility of the previous versions of the Windows Server operating system versions (specifically Windows 2000 Service Pack-specific testing) with Exchange.

"When talking to customers, it is most often not the software itself that determines the success of an enterprise infrastructure deployment or upgrade but the way IT organizations organize themselves and quality of the communication between the IT teams - particularly between the network operations and e-mail messaging teams. To be effective, they need to be joined at the hip".

Jed Dawson, Senior Manager
Microsoft IT Exchange Messaging Team
Microsoft Corporation

Microsoft Model Enterprise Initiative

Microsoft IT's initiative to optimize its global investment in IT infrastructure is known internally as the ME initiative. This initiative began in fiscal year 2002-03 with a Microsoft Operations Framework assessment conducted by an outside consulting firm.

Like many enterprise IT organizations, Microsoft IT's operational goals are focused on striking the right balance of availability, performance, flexibility, and cost. However, Microsoft IT is committed to dogfooding Microsoft products as fully deployed enterprise solutions before they are released to customers and to using its own global enterprise infrastructure as a model by which Microsoft customers may benefit.

The ME initiative has the following specific objectives:

• Maximizing the number of management tasks performed centrally (remotely from the geographically distributed servers and other devices)
• Decreasing the number of sites through the consolidation of the smaller locations into a smaller number of RDCs
• Reducing the total number of infrastructure and application servers
• Standardizing infrastructure and devices worldwide

Pre-Consolidation Exchange Server 2003 Deployment

When the ME initiative was started, Microsoft IT was supporting over 192 servers running Exchange Server 2003 on Windows Server 2003 Enterprise Edition, distributed over 75 locations worldwide. There were 118 dedicated mailbox servers hosting more than 100,000 mailboxes.

Microsoft IT's strategy has been to deploy infrastructure servers, such as messaging servers, in dedicated roles for efficient administration. Table 2 shows how Exchange servers were distributed prior to the Exchange Server 2003 physical site consolidation.

Table 2   Pre-Consolidation Exchange Server 2003 Messaging Servers

A detailed description of the post-consolidation Exchange messaging environment is provided in the Solution section of this whitepaper. For those readers unfamiliar with Exchange Server 2003 and Outlook 2003 physical site consolidation features, a description of this feature set is provided in the following section.

Understanding Exchange Server 2003 and Outlook 2003 Physical Site Consolidation Features

Microsoft Exchange Server 2003 represents an important, continuing investment in enterprise messaging solutions by the Microsoft Exchange product group and the Microsoft IT Exchange Messaging team. Exchange Server 2003 offers several key improvements required by enterprise messaging and collaboration customers – including those designing and implementing an infrastructure consolidation strategy.

To understand how Microsoft IT was able to deploy its Exchange messaging worldwide physical site consolidation solution, it is important to understand the new site and server consolidation features in Exchange Sever 2003 and Outlook 2003.

"IT managers have been asking us to include these new back-end features -- for clustering, backup, caching, performance, and more -- into Exchange Server. Exchange Server 2003 has them and we're betting our own business to ensure for our customers that these new features work easily and well in a very demanding production environment: ours."

Jerry Cochran, Group Program Manager
Microsoft IT Group

Why Consolidate Exchange Server Sites?

Implicit in most Exchange Server physical site consolidation projects is a reduction in the actual number of deployed Exchange servers. If the number of the Exchange Server 2003 servers in a large distributed messaging environment can be reduced by, for example, 40 servers, the cost savings can be substantial:

• 40 times the software licensing costs
• 40 times installation and operations costs per server for the base hardware and operating system
• 40 times annual maintenance contract per server on the hardware
• 40 times credit per server for hardware depreciation
• 40 times cost savings for implementation of Exchange Server–specific updates and service packs per server

In addition, lower TCO is possible because of:

• A reduction in the amount of spare servers and server parts that must be maintained to meet internal requirements due to the reduced number of servers deployed in the Exchange Server 2003 environment.
• The redundancy features of Exchange Server 2003 that helped the company meet more rigorous uptime service level agreements.
• A drop in the acceptable time for system outages because of the implementation of multiple storage groups and databases per server, which resulted in faster server recovery.
• Considerably shortened backup time when multiple storage groups and databases were deployed on each server and the ability to back them up in parallel.
• Simplified system monitoring, such as tracking log events and troubleshooting, was due to the reduction in the number of servers supported.

In addition to the reduced server-related costs and lower TCO, the following additional benefits can be realized:

• Single mailbox restore, made possible by using the Recovery Storage Group (RSG) feature in Exchange Server 2003.
• Near-instantaneous backup and restore without the need to take storage offline using the Windows Server 2003 Volume Shadow Copy Service (VSS) with Exchange Server 2003.
• Larger mailboxes per user, which are easier to support with Exchange Server 2003 multiple database architecture (and VSS backups).
• A key trade-off is the cost of the increased WAN bandwidth requirements, which can vary significantly from one geographic region to the next.

Exchange Server 2003 and Outlook 2003 Consolidation Features

The list of features in Exchange Server 2003 and Outlook 2003 that enable physical site consolidation and provide for a highly responsive user experience includes:

• Outlook 2003 Exchange Cached Mode
• Messaging Application Programming Interface (MAPI) compression
• RPC-over-Hypertext Transfer Protocol (HTTP) access
• Buffer packing
• Outlook performance monitoring
• Incremental change synchronization
• Smart change synchronization
• Skip bad items
• Pre-synchronization reporting
• Improved Exchange Server 2003 scalability, clustering and storage management features
• New Exchange Server 2003 SP1 migration and consolidation tools

Outlook 2003 Exchange Online vs. Exchange Cached Mode

Outlook 2003 Exchange Cached Mode ("cached mode") was originally developed as an Exchange Server 2003 and Outlook 2003 feature to provide mobile professionals with more effective offline access to e-mail and address book information as well as Exchange server access over slow or poor-quality network connections. It has become the key to providing users in a remote site accessing Exchange servers in a RDC with the same quality client experience for sending and receiving e-mail as they would expect from being directly connected to a local Exchange server.

Previous versions of Microsoft Outlook supported offline usage with basic capabilities for local storage of e-mail messages and the Exchange global address list. Typically, these features were implemented using real-time RPC transactions to access the user's local Exchange server. Performance over slow or unreliable links for remote users was poor.

Cached mode is a new feature of Microsoft Outlook 2003 that is supported by Microsoft Exchange Server 2003. Cached mode is targeted at providing a uniform, highly responsive Outlook user experience regardless of the type and quality of Exchange Server 2003 network connection. This includes high-speed local network connections, dial-up connections, General Packet Radio Service (GPRS), and cases when a user is completely disconnected from their Exchange Server.

To achieve this, cached mode supports the local caching of the contents of a user's Exchange mailbox as well as downloading of a local copy of the offline address list. Cached mode is the new default configuration setting used by Outlook 2003 when it connects to an Exchange server.

For enterprise users, the key benefit of cached mode is that it enables remote users to work with Outlook 2003 over WAN links connected to remote mailbox servers without suffering the effects of low network bandwidth or high network latency for most mailbox and offline address list –related operations. The implication for the network is that high utilization can occur:

• When a large number of people use Outlook 2003 to connect to their Exchange Server 2003 for the first time.
• Many users in a region simultaneously access their regional Exchange servers (for example, on Monday mornings, the day after national holidays and vacations, or immediately after large company meetings).
• Outlook 2003 downloads a full copy of the offline address list (due to a substantial number of new or updated address book records).

The latter can occur because of changes to the Active Directory, such as the additions of each person's public key infrastructure (PKI) certificate to their user account or bulk additions or changes due to mergers and acquisitions, telephone area-code changes, or office moves.

Outlook 2003 users using cached mode perform most of their e-mail –related tasks from the local mailbox store. This reduces the number of requests to the server for data and improves user experience for items found in the local mailbox store. While cached mode works with previous versions of Exchange Server, additional improvements in compression and performance between Outlook 2003 and Exchange Server 2003 make the user experience even better.

When Outlook 2003 is using cached mode (either by default, by explicitly configuring it manually, or by using group policy), mailbox items (for example, e-mail messages, and incoming meeting requests) are downloaded and cached in the Outlook Offline Store (OST). Other server-supported functions are executed as real-time transactions against the Exchange server. These include delegate access to another person's mailbox, querying free/busy information, and access to information stored in Exchange public folders.

In addition, when Outlook 2003 is configured to download the "no details" offline address list and an address book entry is opened for display, a real-time, non-cached access to the user's Global Catalog server is needed to display user profile information that is not found in the "no details" offline address list. When the full offline address list is downloaded, a rich set of user profile properties is downloaded. When Outlook 2003 is configured to download the "no details" offline address list, only basic name information and e-mail address is cached locally. Any other requests for user profile information require Outlook to locate and access the user's Global Catalog server. The specific list of Outlook 2003 address list fields available in the full and "no details" downloads of the offline address list can be found in Appendix B.

To determine which Global Catalog server to access, Outlook 2003 sends a request to the Exchange server. The Exchange server always returns the address of its local Global Catalog server. If the Exchange server is located in a remote data center, this may not be the best choice from a network bandwidth and latency perspective. To address this situation, the address of the closest global address server can be set explicitly using a registry setting.

MAPI RPC Compression

When using Outlook 2003 and Exchange Server 2003, by default all content is compressed before being sent from the Exchange server to the Outlook client and vice versa. This can significantly reduce network bandwidth consumption between the client and server, depending on the type of message and attachment(s) sent.

RPC-over-HTTP Access

When used with the Microsoft Windows Server 2003 RPC Proxy Service and Exchange Server 2003, Outlook 2003 clients can connect simply using HTTP or Secure Hypertext Transfer Protocol (HTTPS), thereby reducing the need for virtual private networks (VPNs) or dial-up remote access. If remote users only need to gain access to corporate messaging information, the IT department does not need to deploy a VPN infrastructure. VPN-less access reduces costs and provides for increased security by ensuring that remote Outlook users do not need access to the entire network. This unifies the connection methods also found in Outlook Web Access and Outlook Mobile Access.

Buffer Packing

After information is compressed, all information sent from servers running Exchange Server 2003 to Outlook 2003 clients is packaged in larger and more efficient buffer packets, thereby reducing the number of requests to and from the servers running Exchange.

Outlook Performance Monitoring

Outlook performance monitoring makes it easier for the Exchange systems administrator to monitor and troubleshoot performance and network connectivity problems. Outlook collects latency and error information from Outlook clients and sends it to Exchange Server 2003. This data is held in the Exchange store as well as being recorded in the event log and performance counters where the data can be accessed by Microsoft Operations Manager (MOM) 2000. To understand when remote clients are experiencing performance issues related to poor bandwidth or poor connectivity, interpretation of the Exchange event log may be performed using the event viewer Microsoft Management Console (MMC) snap-in.

Incremental Change Synchronization

In earlier versions of Microsoft Outlook, when interruptions occurred during the offline mailbox synchronization process, the entire process had to start over from the beginning. Incremental change synchronization in Outlook 2003 and Exchange Server 2003 enables the mailbox synchronization process to resume where the outage occurred instead of starting the entire synchronization process over.

Smart Change Synchronization

In earlier versions of Outlook, the entire message and body was sent to the server. In Outlook 2003, when items are marked read, unread, flagged, or slightly modified in other ways, only the header that lists the change is sent back to the server.

Skip Bad Items

During synchronization, items marked as bad or conflicting are now moved to the Sync Items folder, enabling the synchronization to continue.

Pre-synchronization Reporting

The synchronization progress meter (found in the lower right corner of the Outlook 2003 user interface) shows detailed synchronization information such as new e-mail headers, total size left to synchronize, and whether the folder is up-to-date.

Improved Exchange Server 2003 Scalability, Clustering and Storage

Table 3 summarizes the key Exchange Server 2003 features that enable a larger number of users with larger maximum mailbox sizes to be hosted on an equivalently configured Windows Server 2003 server.

Table 3   Exchange Server 2003 Scalability Improvements

Recovery Storage Groups

The RSG is a specialized storage group that can exist with regular storage groups. Although an RSG is similar to a regular storage group, the primary way RSGs differ from regular storage groups is that all protocols, except MAPI, are disabled. This means that e-mail cannot be sent to or received from a mailbox store that is in an RSG. However, use of the Exmerge tool allows access to mailboxes to recover data.

In previous versions of Exchange Server (prior to Exchange Server 2003), it was necessary to configure a separate Active Directory forest on a recovery server if another copy or a different version of a production Exchange database was to be mounted. With the RSG feature in Exchange Server 2003, a separate recovery computer is not required when data is needed to be recovered from a mailbox store.

The RSG feature is not intended for use in disaster recovery operations that involve multiple servers or multiple storage groups. Its intended use is to recover data from a single mailbox, a single database, or a group of databases that are in a single storage group. For example, an RSG can be used to recover items that were deleted and purged from a user's mailbox, or a RSG can be used to restore or repair a copy of an alternative database while another copy of the database remains in production.

Improved Cluster Support

Windows Server 2003 provides up to eight-node clustering — an improvement over the earlier two- and four-node clustering. This new standard allows for better scalability and availability in a consolidated environment.

Improved Backup and Restore for Very Large Exchange Servers

The other key concerns for IT administrators have been the time, cost, and trouble of backing up their increasingly large Exchange servers, which can contain up to 500 GB of data on a single server. Exchange Server 2003 supports the VSS in Windows Server 2003 that allows administrators to mirror a disk in real time, making it fast and easy to backup.

New Exchange Server 2003 SP1 Migration and Consolidation Tools

The original Exchange Server 2003 release includes an enhanced version of the Move Mailbox tool that enables administrators to schedule mailbox moves to occur automatically at a particular time on a particular day. When used with the Skip Bad Items feature, this enables Exchange administrators to schedule the automatic migration of selected groups of users overnight or on weekends without having to be physically present. In addition, the Move Mailbox tool is now multithreaded, significantly improving performance.

Exchange Server 2003 Service Pack 1 includes additional new migration and consolidation tools that are of specific use to Exchange Server 5.5 administrators who want to migrate from a pure Exchange Server 5.5 environment or an Exchange mixed-mode environment (a mix of Exchange Server 5.5 and Exchange Server 2000 servers) to an Exchange Server 2003 native mode environment.

Exchange Cached Mode Network Usage

Perhaps the most important aspect of planning for an Exchange Server 2003 physical site consolidation is understanding the differences in network usage that result when Outlook 2003 changes from locally accessing an Exchange Server in the pre-consolidation environment to remotely accessing an Exchange Server deployed in a remote data center in the post-consolidation environment. The two scenarios are depicted in Figure 4, Figure 5 and Figure 6.

If your browser does not support inline frames, click here to view on a separate page.

Figure 4   Typical pre-consolidation tail site

In the pre-consolidation scenario, all e-mail traffic sent from users at this tail site to all users outside this tail site must be delivered over the WAN connection to the Exchange mailbox and public folders servers. All remote public folder access into or outside this tail site will also flow over the WAN connection. A moderate level of Global Catalog synchronization traffic is present on the WAN connection depending on the size of the Active Directory and the frequency and scope of directory entry changes.

Outlook users access their e-mail and public folders locally across the local area network (LAN) connection and the user experience is very responsive. In this scenario, cached mode is only an advantage to mobile users who require a high-quality offline experience when not connected to their Exchange server. An exception to this is external users who access their mailbox via Internet VPN connections located in each RDC.

Figure 5   Typical post-consolidation configuration

If your browser does not support inline frames, click here to view on a separate page.

Figure 6   Typical post-consolidation local office configuration

In this post-consolidation scenario, all Outlook e-mail and public folder traffic travels over the WAN connection between Outlook users at the local office (former Exchange tail site) and the Exchange servers running in the remote RDC. In this scenario, Outlook 2003 is assumed to be configured with cached mode enabled. The local OST is used to store mailbox and six files are used to store offline address list (five files for the "no details" option). Again, the exception is external users who access their mailboxes via the Internet VPN connections located in each RDC.

All e-mail traffic destined for users in the local office still needs to be delivered over the WAN connection, although in this scenario a copy of each message will be downloaded and cached into each user's local OST file. This helps ensure that Outlook 2003 users connecting to an Exchange server in a remote data center will still experience a high-quality user experience because all of the mailbox and address book information is stored locally. Real-time transactions across the WAN connection will still occur for functions such as free/busy information queries, delegate access to another person's mailbox, and public folder access.

Summarizing the above scenarios, there were five types of network traffic that Microsoft IT needed to consider:

• Outlook client to/from Exchange mailbox server. Either real-time transactions or cached mode downloads against a local or remote Exchange server. Real-time transactions are used for delegate access to another person's mailbox.
• Outlook client to/from Exchange public folder server. Access to information stored in public folders (including calendar free/busy information) is performed using real-time transactions against the local or remote Exchange public folder server.
• Outlook client to/from Global Catalog server. When access to the Global Catalog server is required, the Exchange mailbox server will return the network address of its local Global Catalog server to Outlook unless the address has been explicitly set (overridden) in the user's registry. The choice of whether to download the full or "no details" offline address list is dependent upon whether the additional items provided by full details address list, listed in Appendix B, are required by users when offline.
• Exchange server to/from Global Catalog server. The local or remote Exchange server makes regular requests of the Global Catalog server that is local relative to the Exchange server making the request.
• Exchange server to/from Exchange server. With fewer servers concentrated into a smaller number of physical sites after consolidation, it is possible that the amount of inter-server traffic will be reduced. In addition, the amount of inter-site WAN traffic might be reduced depending on the inter-site e-mail traffic pattern.
• Global Catalog to domain controller synchronization. With the removal of the Exchange mailbox and public folder servers from smaller sites ("tail sites"), a Global Catalog server may not be required and an additional level of secondary server consolidation (and corresponding reduction in WAN traffic) may occur.

When the primary consolidation of the Exchange servers in a tail site and the secondary consolidation of the Global Catalog servers and possibly additional servers occurs, an organization can also begin to simplify its network infrastructure measured in terms of the number of dedicated circuits, routers, and switches. However, it is likely that additional network bandwidth will be required for some links. Depending on the number of local employees and the bandwidth, cost and quality of ISP-provisioned Internet access, an organization can choose to further reduce its internal network costs and complexity by making the local office an Internet VPN connected office.

Given the complexity of the factors that affect network bandwidth utilization and latency, it is difficult to accurately predict the effects that a large physical site consolidation will have on the network as well as each user's experience. This implies that it is difficult to model this environment using simple formulas.

Solution

Having already completed the in-place upgrade of the company's Exchange Server 2000 servers to Exchange Server 2003, the Microsoft IT Exchange Messaging team set out to identify the strategies and key risks that needed to be considered as part of the Microsoft worldwide physical site consolidation plan. Three key focus areas emerged for the Exchange Messaging team:

1. Proactive, detailed monitoring and analysis of WAN bandwidth utilization and latency – before and after consolidating each group of physical sites
2. Need for an effective but flexible approach to project planning, scheduling, and cross-group coordination
3. Coordination and control of deployment of successive pre-release versions of Office System 2003 (including Outlook 2003) to the more than 70,000 employee, vendor, and contractor staff at Microsoft.

Planning

When planning the Exchange Server 2003 physical site consolidation project, the key parts of the project planning activities included physical site consolidation strategy, RDC design, Exchange topology design, planning WAN monitoring and measurement, bandwidth management, and scheduling.

Physical Site Consolidation Strategy

The first prerequisite of undertaking an Exchange Server physical site consolidation project is that an organization has two or more deployments of Exchange Server that are separated by slow communications links. In the case of the Exchange messaging infrastructure at Microsoft, more than 75 sites were candidates for physical site consolidation.

The second key consideration is where to locate the central and RDCs. This is often predetermined by the existing physical WAN infrastructure, or in the case of a planned network upgrade, by new planned or redesigned physical WAN infrastructures. In the case of the Microsoft Exchange Server 2003 physical site consolidation project, the location of the RDCs was determined by the overall analysis of IT infrastructure at Microsoft as well as current and future business needs. This was driven as part of the Microsoft IT ME initiative.

The third consideration is TCO: do the projected cost savings (excluding WAN cost increases) outweigh the additional WAN costs?

Lastly, the quality of the network connections and network management processes to support the consolidated physical site architecture must be in place and network SLAs sufficient to support the organization's agreed-upon messaging service levels.

Regional Data Center Design

Considerable engineering effort went into designing, building, and testing the Exchange Server 2003 environment deployed into each of the six RDCs. Each RDC Exchange Server configuration included:

• One clustered Exchange Server 2003 mailbox server using a SAN for mailbox storage
• Two Exchange Server 2003 front-end servers supporting Outlook Web Access, Outlook Mobile Access, RPC-over-HTTP and Exchange ActiveSync® access to the back-end mailbox servers
• Two Exchange Server 2003 public folder servers for user public folder as well as system folders for storing the offline address list and free/busy calendaring information
• One or two Exchange Server 2003 Simple Mail Transport Protocol (SMTP) gateway servers

When this Exchange server configuration was being deployed into the RDCs, existing Exchange routing servers that were no longer needed were removed from tail sites. In the RDCs, routing groups were consolidated and larger routing servers were deployed to handle the consolidated message traffic. Based on the volume of messages, dedicated routing servers were deployed in some RDCs while in the rest, RDC public folder servers acted as routing servers.

Exchange Server 2003 gateway servers were upgraded in the RDC locations. All incoming Internet e-mail continued to be routed through the Redmond data center were each message was processed by virus defense and junk mail scanners. Exchange Server 2003 gateway servers were deployed in the RDCs for outgoing Internet e-mail enabling faster delivery of e-mail to local customers and partners.

Similarly, Outlook Web Access and Outlook Mobile Access needed to be upgraded in each RDC location to handle the larger number of Exchange users being served by each RDC site.

Multiple public folder servers were used for user public folder storage due to:

• Volume of storage required. The Redmond, WA data center required 400-500 GB of user public folder storage and there was not enough free storage available on the existing Exchange SAN.
• User load on a single server would have been too great. Some RDC public folder servers (which are also used to store free/busy information as well as offline address book files) are accessed by as many as 15,000 to 20,000 users per day.

The Exchange Server 2003 server and storage hardware to be used in the RDCs was evaluated and selected by the Microsoft IT Exchange System Engineering team. The hardware was purchased centrally and shipped directly to each RDC where it was installed and configured based on the RDC design created by the System Engineering group.

Each RDC location needed to have more formal and elaborate planning for the power, HVAC (heating, ventilation, and air conditioning), racking, network cabling and floor space required by the large RDC server farm and SAN environments.

Exchange System Engineering was also responsible for creating the operating system and Exchange Server 2003 software image to be installed on each of the different Exchange Server 2003 servers. Additional staff was required for extended periods to install, configure, and test the RDC data center environments. Microsoft IT's largest SAN solutions have required as long as three months from receipt of the hardware components until they are available for full production use.

More information on the architecture and design of the RDC server hardware and software configuration can be found in the Microsoft IT Showcase whitepaper "Exchange Server 2003 Design and Architecture at Microsoft".

Exchange Topology Design

Worldwide, Microsoft has more than 400 offices. All locations are connected by the Microsoft corporate network, providing each employee location with high-speed access to services such the Exchange messaging service and the many resources hosted in the headquarters corporate data center in the Seattle/Puget Sound region.

From a network perspective, the Microsoft IT Network Connectivity team actively monitors and manages the WAN, which is constructed using a hub-and-spoke topology with cross-links for redundancy. Fifty percent of the links have a capacity of 2 megabits per second (Mbps) or greater, with most of the circuits operating at either T1 or E1 speeds. On the low-bandwidth circuits, especially those outside of the United States, circuit quality, and reliability can vary significantly, depending on the region and the carrier.

At Microsoft, the percentage of average network bandwidth devoted to e-mail and related messaging traffic is approximately 10 percent. This relatively low percentage is due to the disproportionately large amount of non-messaging WAN traffic created by the Microsoft product development teams.

Virtually all WAN technologies are represented in the current network design including direct connections, Internet-connected VPNs, Asynchronous Transfer Mode (ATM), frame relay, point-to-point, and Clear Channel.

As a general policy, Microsoft tries to deliberately ensure that the WAN backbone links are provisioned to exceed capacity requirements while exercising tight management on the more expensive bandwidth required to provision tail sites.

On its backbone circuits, Microsoft has worked with its carriers to negotiate the best possible cost per Mbps in exchange for purchasing larger than normal capacity. On these backbone circuits, the network traffic created by the product development groups represents the largest component of the network utilization – not the traffic generated by Exchange servers or Outlook clients.

In addition to negotiating for a best price, the Microsoft IT Network Connectivity team also negotiates for specific service levels whose results are reported on the monthly Microsoft CIO scorecard and regularly reviewed with the WAN carriers. The Network Connectivity team actively monitors and analyzes WAN performance on each of its links comparing service levels against a historical database. Packet loss, latency, and bandwidth utilization for each link are recorded and Microsoft works actively with its carrier partners to resolve any issues.

Planning WAN Monitoring and Measurement

The Microsoft IT Exchange Messaging team, working with the Exchange Server product group, wanted to develop a deep understanding of the effects that the Exchange Server 2003 physical site consolidation project would have on the Microsoft WAN. Given the complex variety of demands placed on the WAN by the Exchange Server messaging and related services, the changes in bandwidth utilization and latency due to consolidation is not something that can be modeled by a simple formula or deduced intuitively. The Microsoft IT Exchange Messaging and Network Connectivity teams developed a network monitoring and analysis strategy designed specifically to measure the effects of the Exchange Server 2003 physical site consolidation.

The goal of the Exchange Server 2003 physical site consolidation WAN monitoring project was to accurately measure the changes in WAN bandwidth utilization that resulted from consolidating the Exchange Server 2003 tail sites throughout Microsoft into the six RDCs. For each group of regional tail sites, this included:

• Pre-consolidation tail site preparation
• Pre-consolidation WAN link monitoring (two weeks)
• Migration of mailboxes from a site to the local RDC (one weekend to two-to-three weeks)
• Post-consolidation tail site preparation
• Post-consolidation WAN link monitoring (two weeks)

Managing Bandwidth Used for Physical Site Consolidation

The rate at which the consolidation of a physical tail site can proceed was limited by the desire not to affect regular business activities during the business day and the amount of available bandwidth during the evenings and on weekends. Hence, initial mailbox migrations were scheduled during the early morning or early evening when network utilization was low and Exchange Client Services professionals were available to monitor these mailbox migrations and address any systemic issues. The bulk of the mailbox migrations from the tail site to the RDC were then scheduled for the evenings and weekends.

Project Planning and Scheduling

Given that Microsoft IT is most often working with pre-release versions of most Microsoft software products, one of the most important inputs into a Microsoft IT project schedule is the product groups' planned internal release dates for the milestone versions of their products. In the case of the Exchange Server 2003 physical site consolidation project, the schedule was driven by the release schedules for Office System 2003, Outlook 2003, and Exchange Server 2003. The product groups were interested in testing these releases with specific types of users, network connection speeds, and other scenarios.

The next most important set of inputs into the project schedule was the company's business requirements such as upgrade blackout periods due to month-end and quarter-end processing coupled with employee vacation periods. Annual vacations, extended national and statutory holidays in some countries, as well as weekends, would normally be optimum times for scheduling mailbox migrations from tail sites to the RDCs. However, it was important for the pre- and post-consolidation network monitoring that most employees would be at work with normal workloads for two weeks leading up to the migration and the two-week period following the migration. The mailbox migration for a particular group of tail sites would take anywhere from one weekend to three weeks depending on the number of users and the size of their mailboxes.

Other factors that affected the scheduling of individual tail site consolidations included:

• Network bandwidth required for mailbox migration. These were usually performed after business hours or on weekends
• Regional time zones
• Logistics involved in acquiring, configuring, shipping and installing the Exchange server hardware and software components to each RDC

The largest number of tail sites that Microsoft IT migrated at the same time was 22.

A central project manager coordinated the activities of all the teams involved in the planning and execution of the physical site consolidation project.

Deployment

The deployment phase of the physical site consolidation followed the project plan and schedule initially created during the planning phase. The schedule was updated weekly based on the previous week's progress and the status of any product issues that had previously surfaced. The primary deployment activities included coordinated deployment of the appropriate version of Office System 2003 (including Outlook 2003), RDC build-out, tail site preparation, the actual movement of mailbox servers to the RDC Exchange servers, and the decommissioning of the tail site Exchange servers.

Client Deployment

As part of the dogfooding process of testing early pre-release versions of Office System 2003 and Outlook 2003 (with Exchange Server 2003), Microsoft employees were asked to install each major milestone version of these products. They were first sent an e-mail with instructions on where and how to install each version from the local product distribution server in their region. Microsoft employees feel naturally motivated to install each new version of a product as part of the extended testing process. Employees know that problems found in their pre-release versions of a product can be fixed before the product is made generally available to customers.

For this project, those employees who had not upgraded to the general release of Office System 2003 and Outlook 2003 by a prescribed date were forced to upgrade through Microsoft Systems Management Server (SMS), when they next logged on to their systems. Because SMS is not used on all of the product group development, consulting services, and technology specialist computers, users of down-level versions of Outlook 2003 were ultimately locked out from accessing their Exchange server using a feature built into Exchange Server 2003. This lockout feature disables MAPI client access to the Exchange Server 2003 server based on the minimum version of the Emsmdb32 dynamic link library (DLL). Emsmdb32.dll is part of the Exchange transport service and provides the services that are required for accessing public folders and your mailbox. Emsmdb32.dll is also involved if you are working with an offline folder store.

For the bulk of the physical site consolidations, the final release to manufacturing (RTM) version of Office System 2003 was available and was the prerequisite version that each Microsoft employee needed to have installed. Having a common version of Outlook 2003 installed and used by all employees was important from the point of view of gathering consistent and valid data from the pre-consolidation and post-consolidation network monitoring tests.

Tail Site Preparation

The goals for pre-consolidation tail site preparation were directly related to the pre- consolidation and post-consolidation network monitoring goals.

The purpose of the network monitoring activities was to measure network bandwidth utilization and latency related to a user's direct needs in terms of the Outlook functionality they use. Hence, the goals of the tail site preparation were to move specific tail site Exchange services to the RDC prior to the pre-consolidation network monitoring. The specific Exchange services to be moved were those services, other than mailbox services, that would not be required in the tail site following the consolidation of the Exchange services into the RDC. These included, for example, Exchange SMTP gateway servers, Outlook Web Access servers, and Exchange routing servers.

As part of the tail site preparation, local user public folders were replicated to the public folder servers in the RDC to enable the tail site public folder servers to be quickly decommissioned following the mailbox moves to the RDC and the decommissioning of the mailbox servers but before the start of the post-consolidation WAN monitoring. In one scenario, 100 GB of public folder data needed to be replicated, requiring several weeks for full replication to take place. To reduce the total amount of public folder replication required, Microsoft IT, working with its local regional IT representatives, chose to purge public folder data that was more than three years old.

From a client communications perspective, the Microsoft IT Services Management team had regular conference calls with the representatives from Client Services regional IT teams. The regional IT teams in turn would communicate upcoming physical site consolidation activities to the employees and contractors working in the local region.

Moving User Mailboxes from Tail Sites to Regional Data Centers

The physical mailbox moves were undertaken by the Exchange Client Support team within the Microsoft IT Exchange Messaging team.

Mailbox Move Support in Exchange Server

Starting with the release of Exchange Server 2000, native-mode Exchange environment mailboxes can be moved within the same forest without having to backup, delete, and recreate a user's mailbox. The Active Directory Users and Computers MMC snap-in is used to move user mailboxes from one server to another in the same site, or in a different site.

For the physical site consolidation project, the Microsoft IT Exchange Messaging team created a simple, internal bulk mailbox move tool that reads a list of users and Exchange servers from a file and creates five parallel threads to move as many as five users' mailboxes at a time. The Messaging team used the bulk mailbox move tool in addition to the Users and Computers MMC snap-in.

With Exchange Server 2000, if a mailbox has a corrupted item, the mailbox cannot be moved until it was fixed with the Exchange Server Exmerge tool. This reduced the productivity of the Microsoft IT Exchange Client Support staff and represented a potentially huge problem when a large number of mailboxes needed to be moved.

With Exchange Server 2003, the Active Directory Users and Computers MMC snap-in supports a feature called Skip Corrupted Items. This feature makes it possible to manually move a mailbox even if it contains corrupt items. When the bulk mailbox move tool failed to move a mailbox, the incident would be logged as a service request and was most often resolved using the Active Directory Users and Computers MMC snap-in with Skip Corrupted Items enabled.

This overall approach greatly improved the productivity of the Microsoft IT Exchange Client Support staff by allowing them to plan to move 100 and up to 200 mailboxes per day from each tail site to an RDC. They were able to do this without having to continuously monitor the process for failures due to corrupted mailbox items.

As part of the mailbox move process, the Exchange Client Support team also needed to update each employee's unified messaging configuration, which allows incoming voice mails to be received as e-mails in the employee's personal mailbox.

Physical Site Consolidation Project Mailbox Moves

The overall communications plan to the employees was managed by the Service Management team. Coordination across all of the teams was handled by the program management and service management teams in the Microsoft IT Messaging group.

The actual mailbox moves were scheduled to be executed in bulk using Microsoft IT's internal bulk mailbox move tool. The bulk mailbox move tool read a file containing a list of accounts to be moved and calls the same Exchange Server mailbox move Application Programming Interface (API) used by the Active Directory Users and Computers MMC snap-in. Approximately 100 to 200 tail site mailboxes were migrated per day from each tail site. The mailbox moves were scheduled to run in the evening based on the tail site's local time.

All problems that occurred during the mailbox moves were logged and addressed by the Microsoft IT Exchange Client Support team who would triage, troubleshoot, and resolve each issue as required. In many cases, the problem was resolved using by manually moving the mailbox using Active Directory Users and Computers MMC snap-in with Skip Corrupted Items enabled.

Special attention was given to tail site mailbox moves where there was a high failure rate (affecting greater than 10 to 15 percent of the tail site's mailboxes) indicating a more serious issue (such as a problem with one of the mailbox server storage groups). Server-level issues required the involvement of other Exchange Messaging teams within Microsoft IT. These teams were coordinated by the Exchange Client Services project manager.

Approximately 10,000 mailboxes were moved from tail site Exchange servers to Exchange servers in the RDCs. The remaining 45,000 employee mailboxes were not affected by the Exchange Server 2003 physical site consolidation project because these mailboxes were on Exchange mailbox servers that were already part of one of the six RDCs.

Decommissioning Exchange Site Servers

Once the most critical step of moving users' mailboxes from the mailbox server in a tail site to the mailbox server in the RDC was complete, the next important step for Microsoft IT was the orderly decommissioning of the Exchange servers in each tail site. In general, there were two common tail site configurations that needed to be addressed as shown in Figure 7:

1. Single Exchange server tail sites that used a single server to provide both mailbox and public folder services (including free/busy information); and
2. Dual Exchange server tail sites where separate servers were used for mailbox services and public folder (and free/busy) services.

If your browser does not support inline frames, click here to view on a separate page.

Figure 7   Common Microsoft IT tail site configurations

To decommission each of the two tail site configurations, there were two alternatives: 1) decommission the mailbox servers first, or 2) decommission the public folder servers first (including the offline address list and free/busy server). Microsoft IT chose to begin the decommissioning the public folder servers before site consolidation to eliminate as much tail site WAN traffic as possible and enable more accurate pre-consolidation monitoring and measurement of network bandwidth utilization and latency.

To prepare for the decommissioning of the public folder server in a tail site, Microsoft IT made sure that all user public folders had replicas on an RDC public folder server (or that the appropriate replicas were created where necessary). This enables the tail site public folder servers to be decommissioned immediately after the tail site mailbox moves were completed and before post-consolidation WAN monitoring was started. Full replication of a large public folder structure may take several hours, days, or even weeks depending on the amount of information in the public folders and the saturation of the WAN connection between the tail site and the RDC. This migration was scheduled to occur outside regular business hours.

After a mailbox is moved from a tail site to an RDC mailbox server cluster, Exchange Server automatically updates a user's Outlook profile to point to the new mailbox server the first time Outlook tries to connect to its old mailbox server following the mailbox move. For this reason, it is helpful to leave the tail site mailbox running as long as is reasonable after the mailboxes have been moved to the RDC to accommodate users who may be away on vacation. Otherwise, some users will have to manually reconfigure their profiles to point to the new RDC mailbox server. In the case of Microsoft IT's physical site consolidation project, the tail site mailbox servers were decommissioned as soon as the mailboxes had been successfully migrated to the RDC mailbox servers to allow accurate post-consolidation monitoring to occur as soon as possible, without any extraneous tail-site Exchange server traffic interfering with the network traffic results.

While not formally part of this project, secondary consolidation of selected Global Catalog servers and other servers was also made possible.

Operational Considerations

Once the physical site consolidation had commenced (and was subsequently completed), the operational focus was shifted to managing the six RDCs and continuing to monitor and gather network bandwidth utilization and latency statistics on each WAN segment used to connect the RDCs.

Planning for RDC Upgrades

Prior to the Exchange site consolidation project, the Exchange servers in the local tail sites require periodic upgrades as new employees were hired. It was a relatively straightforward, incremental process: a "small" server would be upgraded to a "medium" server, and a "medium" server to a "large" server. Once upgraded, the server would have enough capacity to support the hiring of several dozens to several hundreds of new employees.

With the smaller number of RDC-based large Exchange server farms, Microsoft IT has found that upgrading planning has become a more demanding process. Some RDCs are growing annually by more 1000 users (including employees and contractors). These types of growth rates require more planning effort and different levels of approvals given that these are not conventional single server upgrades. SAN storage may have to be increased; Outlook Web Access and Exchange Server 2003 gateway servers may have to be upgraded. Planning for online backup services (provided by a different organization within Microsoft IT) is similarly affected.

Microsoft IT has begun planning for the eventual time when maintenance and warranty programs on its large SANs will expire and replacement systems will need to be purchased. SANs are significantly more complex that the single servers removed from the tail sites. More pre-sales configuration and planning is required; longer delivery times are required.

Upgrades to consolidated servers and sites tend to be in larger increments than those experienced with tail site upgrades; requiring more engineering, budgeting and deployment planning.

Resolution of Outstanding Issues

As part of its dogfooding effort, Microsoft IT worked with the Exchange Server product group to reduce the network bandwidth usage for downloading the full offline address list. Frequent changes occur at Microsoft due to personnel relocations (requiring cross-forest mailbox moves) and external factors such as area code changes that cause bulk changes that affect many directory entries. When a sufficiently large number of changes occur, this triggers the download of the full offline address list over the network to each Microsoft employee's personal computer(s). Given the large number of entries in the Microsoft IT enterprise Active Directory, the compressed size of the downloadable offline address list was 50-60 MB in size.

This is an example of the dogfooding process at work within Microsoft IT and the Microsoft product groups to test and resolve these types of issues as early as possible in a live, worldwide production enterprise environment.

Conclusions

Through consolidation, Microsoft IT was able to reduce the number of Exchange Server 2003 physical sites from 75 to seven physical sites and the number of physical mailbox servers from 118 to 36. The results of its successful Exchange Server 2003 physical site consolidation project are shown in Table 4.

Table 4   Reduction in Microsoft IT Exchange Servers

In total, approximately 10,000 mailboxes were moved from 75 tail sites to six RDCs, and a significant network bandwidth issue related to offline address list full downloads was identified. It recommended that customers read the following Knowledge Base article http://support.microsoft.com/default.aspx?scid=kb;en-us;839826 for more details on how to address this issue.

Microsoft IT was able to leverage the new storage, scalability, and recoverability features in Exchange Server 2003 and Windows Server 2003 to increase the number of mailboxes deployed per Exchange mailbox server by 33 percent to approximately 4,000 mailboxes per server. At the same, they increased the server mailbox size from 100 MB under Exchange Server 2000 to 200 MB under Exchange Server 2003, as shown in Table 5.

Table 5   Microsoft IT Exchange Messaging Statistics

With significant increases in the total amount of storage deployed per mailbox server, the ability to restore an Exchange mailbox in a disaster recovery scenario becomes critically important. In any of the RDCs, Microsoft IT is able to restore an Exchange Server 2003 database in approximately 25 minutes, less than half the time it took with the Microsoft Exchange 2000-based messaging environment.

Business Benefits

Microsoft was able to realize two key business benefits from this project:

• Four percent overall direct cost savings from the Exchange Server 2003 physical site consolidation
• The Exchange physical site consolidation project was a key enabler and integral part of the Microsoft ME initiative, which through fiscal year 2003 has produced $23.2 million US in overall consolidation savings including $18.3 million in additional datacenter consolidation savings.

The overall cost savings were driven by reductions in messaging operations staff. Prior to the Exchange Server 2003 deployment, Microsoft IT's messaging operations were already highly centralized, with central support representing 94 percent of the staff costs. Operating efficiencies from the Exchange Server 2003 deployment included a further 14 percent overall staff reduction by eliminating 33 percent of the remaining regional staff support and reducing central staff by 11 percent. Allocated bandwidth costs increased seven percent due to increased WAN utilization, but because messaging traffic represents a small percentage of overall Microsoft bandwidth usage, no actual increased system capacity was required specifically for the Exchange Server 2003 deployment. The overall cost savings do not include any hardware savings because Microsoft opted to increase e-mail messaging service levels (for example, increased mailbox size and system reliability) as part of the Exchange Server 2003 physical site consolidation project.

IT Benefits

Microsoft IT was able to derive the following benefits from the Exchange Server 2003 physical site consolidation:

• Improved server utilization by maximizing every server's storage capacity, processor, and memory resources.
• Improved server management by reducing the number of servers which in turn reduced the time spent on maintaining operating systems and applications, backing up data, and deploying new applications; and by reducing the need for highly skilled IT staff in different geographical areas.
• Strengthened security by removing points of vulnerability and streamlining security management.
• Increased reliability by enabling tighter management practices required for high availability and reducing exposure to theft, accidental damage, and natural disasters.

Best Practices and Lessons Learned

Since Microsoft IT is a leading-edge implementer of Microsoft technologies and products, the organization brings a unique set of requirements as well as innovative approaches to meeting the needs of its internal customers.

Executive Sponsorship

As is the case for many large infrastructure projects, executive sponsorship at the CIO level was critically important from both a funding perspective; but also to ensure that the common vision, strategy and set of priorities represented by the Model Enterprise initiative were followed.

Budget Planning

Large Hardware Upgrade Considerations

For this Exchange site consolidation project, Microsoft acquired all of the upgraded servers and SANs for the RDC in the same fiscal year. The downstream implication is that three years from the initial hardware deployment, the hardware will need to be refreshed or the warranty programs extended for an additional one to two years. The ability to stagger large hardware refreshes over multiple years is one strategy Microsoft IT is considering.

Hardware Warranty Considerations

Extended warranty costs was an important factor in determining the pace for consolidating and decommissioning of tail site hardware; especially in regions where 24-hour daily support is required and where additional costs are incurred for this level of service.

Team Organization

Cross-Organization and Cross-Functional Teams Involved from the Beginning of the Project

The Microsoft IT Exchange Messaging team established a cross-organization team that represented the needs of all aspects of the Exchange Server 2003 physical site consolidation; including teams from outside the Exchange Messaging team itself. Representatives were involved in the first planning stages of the project, even if their actual involvement was not required until much later in the project. Says one team member, "With no representative, you don't have a vote."

Service-Oriented IT Organization Structure

Look at the integrated, end-to-end delivery of business services. Avoid adopting an organization structure that is oriented around technology services, such a file services, printer services, and directory services. For example, during the previous upgrade from Exchange 5.5 to Exchange 2000, Microsoft IT, like all Windows 2000 Server and Exchange 2000 Server customers, needed to design and deploy Active Directory services. This group, which initially focused on directory services as a necessary technology, is now responsible for all aspects of identity management at Microsoft, including secure smart card employee badges.

Communications

Looking back on the overall Exchange Messaging team experience since 1999, it was important to combine or otherwise ensure direct and open communication between the directory (identity management) and e-mail messaging teams in the early stages of upgrading from Windows NT Server 4.0, Active Directory services, and Exchange Server 5.5 to later versions of Windows Server.

Once Active Directory services and later versions of Exchange Server had been fully deployed, the Microsoft identity and directory services management team was established as a separate entity to address the growing needs within Microsoft for identity and directory services management.

Project Planning, Scheduling, and Coordination

Project Management Office

Coupled with best practice of creating cross-organizational and cross-functional project teams is the need for a project manager or project management office that acts as a common point of scheduling and coordination for the whole project team.

Similarly, a prerequisite for a successful organizational and cross-functional project team is a common, integrated project plan that is accessible to all members of the team.

Deployment Scheduling

A critical factor for the Microsoft IT Exchange Messaging team's success with the Exchange Server 2003 physical site consolidation was the creation of a detailed yet flexible schedule for building, testing, and releasing the Exchange Server environments in the RDCs, and coordinated scheduling of each element of the tail site migration to the RDC. At a high level, this included the following steps:

Pre-consolidation tail site preparation:

• Pre-consolidation WAN link monitoring (two weeks)
• Migration of mailboxes from a site to the local RDC (one weekend to two-to-three weeks)
• Post-consolidation tail site preparation
• Post-consolidation WAN link monitoring (two weeks).

Other best practices included: creating schedule buffers to allow the migration teams enough time to clean up loose ends and to prepare for the next phase of tail site migrations; being very much aware of the challenges (and opportunities) represented by end-user vacations, instead of focusing only on migration team vacations; national and statutory holidays; and upgrade blackout periods due to business restrictions and similar events.

Outlook 2003 Client Deployment

Use an approach that works best for your organization and your employee culture. Where possible, use an automated desktop management and application deployment solution like SMS. In addition, keep the following lessons in mind:

• Take into account effective geographic distribution of installation file servers for Outlook 2003 global deployments.
• Plan for the effect of propagating a large package.
• Locate distribution file servers for most efficient use of network links between users and these servers.

Network Planning

Given that Microsoft IT had no previous experience with the new site and server consolidation features in Exchange Server 2003 and Outlook 2003, extensive WAN monitoring was undertaken, both before and after the consolidation of each tail site into the appropriate regional RDC. The lessons learned by the network monitoring team included:

• Recognize that there are no simple formulas for predicting the changes in WAN bandwidth utilization when consolidating the servers and services in a global enterprise messaging environment.
• Ensure there is reasonable bandwidth available to support mailbox moves (and potential public folder synchronization for migrating tail site public folders to an RDC).
• Ensure there is sufficient bandwidth to support the aggregate requirements of tail site Outlook users after the physical site consolidation is complete.
• Beware of the bandwidth requirements needed to transfer the network monitoring data to a central site for analysis.
• Gather and maintain a historical base line of WAN utilization and latency data as well as Exchange related statistics such message deliver times between pairs of representative global office locations.
• Gather WAN utilization and latency data for as long as is feasible before and after each tail site consolidation to account for people on vacations and unexpected periods of low network usage as well as portion of Exchange vs. non-Exchange network traffic.
• It was important to have specific network monitoring data available for the times when some users questioned the responsiveness of accessing their Exchange services after the services have been consolidated into an RDC.
• Beware of large infrastructure changes such as Active Directory changes (area code changes and addition of cell phone numbers for a large number of employees) through effective cross-team communications.

Summary

Prior to deploying its physical site consolidation solution, Microsoft IT used pre-release versions of Microsoft Exchange Server 2003 and Office System 2003 to upgrade its existing Exchange Server 2000 messaging platform. Microsoft IT then undertook the deployment of its physical site consolidation solution to reduce the number of physical locations running Exchange Server from 75 to seven physical sites (six RDCs plus one Exchange site in Johannesburg, South Africa).

The objectives achieved by the Exchange Server 2003 physical site consolidation included:

• Consolidation of the number physical locations running Exchange Server from 75 to seven physical sites (six RDCs plus one Exchange site in Johannesburg, South Africa).
• Accurate monitoring, gathering, analysis, and reporting of changes in network bandwidth utilization and latency.
• Measurement any perceived changes in Microsoft employees' Outlook client user experience.
• Detailed analysis of the attendant reduction in annualized costs associated with the reduction in the number of Exchange Server physical sites.
• Documentation Microsoft IT's Exchange Server physical site consolidation experiences, best practices, and new lessons learned for the benefit of Microsoft customers.

The Exchange Server 2003-based physical site consolidation solution by the Microsoft IT produced the following key business benefits:

• Reduced number of active mailbox servers by almost 70 percent and associated server operational costs, resulting in overall messaging cost reduction of at least four percent.
• Maintained user satisfaction and productivity by reducing dependency on real-time network connectivity.
• Opportunities for additional consolidation of distributed sites.

For More Information

For more information, please refer to the following resources.

Microsoft IT Showcase White Papers and Case Studies

• Exchange Server 2003 Design and Architecture at Microsoft technical white paper
• Exchange Server 2003 Site Consolidation: Preliminary Results technical case study
• Messaging Backup and Restore at Microsoft technical case study
• Monitoring Messaging at Microsoft technical solutions brief

Microsoft Exchange Server 2003

• Kay Unkroth, Server Consolidation Using Exchange Server 2003, Microsoft Corporation, February 2004.
• Microsoft Exchange Product Group, Business Value of Microsoft Exchange Server 2003, Microsoft Corporation, March 2004.
• Microsoft Services, Beyond Cost Reduction to Higher Business Value: How consolidating servers can reduce the total cost of ownership, Microsoft Corporation, 2003.
• Joey Masterson, Exchange Server 2003 RPC over HTTP Deployment Scenarios, Microsoft Corporation, January 2004.
• Exchange Server 2003 ROI Evaluation Report, Nucleus Research Inc., 2003.
• Matt Cain, E-Mail: at What Cost, parts 1 and 2, META Group, 17 Jan 2002.
• Exchange Server 2003 Service Pack 1, Microsoft Corporation, June 2004.

Microsoft Office Outlook 2003

• Enabling a Superior Client Experience with Outlook 2003, Microsoft Corporation, November 2003.
• Microsoft Office 2003 Editions Resource Kit, Microsoft Corporation, September 2003.

Microsoft Knowledge Base Articles

• "How to configure how the Offline Address Book is downloaded when you use Outlook 2003 in Exchange Cached Mode" (KB Article 823580), Microsoft Corporation, January 2004.
• "High network usage occurs while Outlook clients download the Offline Address Book from Exchange Server 2003 at the same time" (KB Article 839826), Microsoft Corporation, April 2004.
• "Modify Remote Procedure Call Compression in Exchange Server 2003" (KB Article 825371), Microsoft Corporation, August 2003.
• "Feature to Disable MAPI Client Access" (KB Article 288894), Microsoft Corporation, November 2003.

Microsoft Sales Information Center

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/services/microsoftservices/howmsdoesIT.mspx

http://www.microsoft.com/technet/itsolutions/msit/default.mspx

http://www.microsoft.com/exchange/default.mspx

For any questions, comments, or suggestions on this document, or to obtain additional information about Microsoft IT Showcase, please send e-mail to:

showcase@microsoft.com

Appendix A - IT Infrastructure Consolidation Strategies Overview

To fully understand Microsoft IT's deployment of its Exchange Server 2003 physical site consolidation solution, it is important to understand the different approaches to IT infrastructure consolidation.

There are six basic options available to organizations that wish to consolidate a highly distributed enterprise computing infrastructure. Implementing these infrastructure consolidation options can create significant opportunities for long-term infrastructure improvements and increased operational flexibility. These six infrastructure consolidation options include:

• Physical site
• Server
• Data
• Application and services
• Operations management
• Operating environment.

Implementation of each of these consolidation options can return significant, measurable increases in efficiency, productivity, and cost benefits in organizations that have deployed a highly distributed server infrastructure by reducing server hardware and software costs; reducing the number of systems administration, monitoring, and maintenance staff; and increasing reliability, availability, security, and performance. Additional information can be found on the Microsoft Business and Technology Center web site.

Physical Site Consolidation

Physical site consolidation refers to reducing the physical number of locations where server resources reside. The benefits of physical site consolidation include:

• Improvement of both physical and system security because there are fewer components to setup, configure, and manage and fewer locations where this occurs.
• Reduced backup and storage costs because it is more efficient and less costly to backup a smaller number of larger capacity servers.
• Reduced systems management and update costs because there are a smaller number of individual servers and site locations that have to be managed.
• Enabling of a service infrastructure that can support fluctuations in staff, services, and business processes that is made possible by the larger economies of scale inherent in larger regional sites.

An example of physical site consolidation is the migration of Exchange messaging services from a highly distributed, branch office-based deployment to a significantly smaller number of RDCs. To fully realize the benefits of physical site consolidation, the set of applications and services need to support consolidation at the server level.

Server Consolidation

Server consolidation is the result of reducing the total number of individual servers for a particular application in a single physical site or across multiple physical sites (due to site consolidation). The resulting benefits include:

• Improved utilization of disk storage, backup, and processing capacity
• Increased effectiveness of high-availability and capacity management solutions
• Simplified service management and maintenance costs
• Reduced fees for server licenses
• Easier integration of applications and services with other data center services

A key requirement for server consolidation is that the application and services software platform needs to support a significantly increased number of users while maintaining both an acceptable user experience and service levels related to processing levels, backup and restore, and other data management tasks.

Data Consolidation

Combining data from multiple databases into a single repository is another approach to consolidating infrastructure resources and reducing the total cost of ownership. The benefits of data consolidation include:

• Reduced storage management costs
• Simplified backup and disaster recovery management
• Minimized duplicate data and improved access to business data
• Improved data integrity through more reliable storage systems and more rigorous data maintenance procedures

A practical example of data consolidation is the consolidation of the number of data storage servers in a front-end/back-end or multi-tier application.

Application and Service Consolidation

When possible, consolidating multiple different services and applications on fewer servers can:

• Increase utilization of existing servers
• Improve the scalability of applications

This option is applicable when there is a set of compatible applications and services that can be configured to run together at the same, or possibly larger, server.

Operations Management Consolidation

In some highly distributed server environments, many organizations often deploy additional IT operations staff to provide local operations and help desk support. Supporting a large distributed environment with a large distributed team of skilled operations staff is often difficult. The benefits include:

• Reduced number of remote server operations and help desk personnel
• More consistent service levels for remote sites

Remote administration features are the key prerequisites for centralizing operations management functions and staff, and reducing the number of operations staff required in remote locations. Without an effective way to remotely manage a set of applications and services, local operations staff will continue to be required.

Operating Environment Consolidation

Standardizing on fewer versions of the same operating systems can:

• Simplify service management and maintenance costs
• Increase utilization of existing servers

The larger the number of server operating systems and operating system versions that are in concurrent use, the greater the costs will be to support the overall environment. More importantly, a diverse operating environment is a key obstacle to effective application and service, data, server, and physical site consolidation.

Overall Infrastructure Consolidation Benefits

TCO is still the most compelling reason to consolidate. By some accounts, server management and support costs can amount to as much as 80 percent of IT spending, fueling the need for a well-conceived server consolidation project and the enormous business value that accompanies it. Overall infrastructure benefits include:

• Improved server utilization by maximizing every server's storage capacity, processor, and memory resources.
• Improved server management by reducing the number of servers which in turn reduced the time spent on maintaining operating systems and applications, backing up data, and deploying new applications; and by reducing the need for highly skilled IT staff in different geographical areas.
• Strengthened security by removing points of vulnerability and streamlining security management.
• Increased reliability by enabling tighter management practices required for high availability and reducing exposure to theft, accidental damage, and natural disasters.

Appendix B – Offline Address List Fields: Full Download vs. No Detail Download

The following fields are downloaded with the "no details" offline address list:

The following additional fields are downloaded with the Full offline address list:

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Microsoft grants you the right to reproduce this White Paper, in whole or in part, specifically and solely for the purpose of personal education.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2004 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Outlook, SharePoint, Exchange, Windows, Windows NT, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.