Contents

	Executive Summary
	Introduction
	Reasons for Microsoft IT to Upgrade
	Environment Prior to Exchange Server 2007
	Planning and Design Process
	Architecture and Design Decisions
	Deployment Planning
	Best Practices
	Conclusion

Executive Summary

Microsoft Information Technology (Microsoft IT) maintains a complex Microsoft® Exchange Server environment consisting of several geographic locations and multiple Active Directory® forests. There are 16 data centers, four of which host Exchange Mailbox servers, to support more than 515 office locations in 102 countries with 121,000 users, including managers, employees, contractors, business partners, and vendors. Site and server consolidation conducted with Microsoft Exchange Server 2003 and new deployment features available in Microsoft Exchange Server 2007 in combination with proven planning, design, and deployment methodologies enabled Microsoft IT to transition this environment to Exchange Server 2007 in less than eight months. Microsoft IT decommissioned the last Mailbox servers running Exchange Server 2003 in the corporate Active Directory forest shortly after Microsoft released the new Exchange Server release to manufacturing (RTM) version on December 7, 2006.

This technical white paper discusses the Exchange Server 2007 architectures, designs, and technologies that Microsoft IT chose for the corporate environment and the strategies, procedures, successes, and practical experiences that Microsoft IT gained during the planning and design phase. In addition to common planning and design tasks typical for many Exchange Server deployment projects, such as server design, high-availability implementation, and capacity planning, transitioning a complex messaging environment to run on Exchange Server 2007 also entails specific planning considerations regarding directory integration, routing topology, Internet connectivity, client access technologies, and unified messaging (UM).

The most important benefits Microsoft IT achieved with the production rollout of Exchange Server 2007 included a substantial reduction of hardware, storage, and backup costs while maintaining 99.99 percent availability of messaging services. New features, such as cluster continuous replication (CCR), enabled Microsoft IT to reduce single points of failure in the messaging environment, increase Mailbox server resilience from storage failures, and eliminate tape backups to reduce costs. Exchange Server 2007 also enabled Microsoft IT to overcome the scalability limitations of the 32-bit platform and increase mailbox quotas in the corporate production environment from 200 megabytes (MB) to 500 MB and 2 gigabytes (GB). Microsoft IT also lowered maintenance overhead and associated costs by replacing third-party unified messaging systems with Exchange Server 2007–based Unified Messaging servers. Other important benefits included increased fault tolerance and simplified server maintenance through load-balanced server configurations for middle-tier services such as Client Access, Hub Transport, and Unified Messaging server roles. Microsoft IT also increased messaging protection through Microsoft Forefront™ Security for Exchange Server installed on all Hub Transport and Edge Transport servers.

This paper contains information for business and technical decision makers who are planning to deploy Exchange Server 2007. This paper assumes the audience is already familiar with the concepts of the Windows Server® 2003 operating system, Active Directory, and previous versions of Exchange Server. A high-level understanding of the new features and technologies included in Exchange Server 2007 is also helpful. Detailed product information is available in the Microsoft Exchange Server 2007 Technical Library at http://technet.microsoft.com/en-us/library/bb124558.aspx.

Introduction

From the earliest days, e-mail messaging has been an important communication tool for Microsoft. Microsoft established the first company-wide messaging environment in July 1982 based on Microsoft XENIX (a UNIX version for the Intel 8088 platform). This environment evolved over more than a decade into a large and distributed infrastructure that was increasingly difficult to manage. By migrating to Microsoft Exchange Server version 4.0 in 1996, and subsequently upgrading to Microsoft Exchange Server version 5.0 and Microsoft Exchange Server version 5.5 in 1997, Microsoft IT achieved significant improvements in terms of functionality, maintainability, and reliability. At the beginning of the new millennium, Microsoft IT operated a messaging environment that included approximately 200 Exchange 5.5 servers in more than 100 server locations with approximately 59,000 users.

Many things changed for Microsoft IT with the upgrade to Microsoft Exchange 2000 Server, released in October 2000. Exchange 2000 Server so tightly integrated with the TCP/IP infrastructure, the Windows® operating system, and Active Directory that the Exchange Messaging team no longer could manage the messaging environment as an isolated infrastructure. A fundamental organizational change was necessary, which manifested itself in a new approach that viewed Microsoft IT as a provider of essential business services. Keeping Exchange servers running was no longer sufficient. The Exchange Messaging team now owned messaging as a service, which included all components upon which Exchange 2000 Server depended, such as Active Directory.

The shift of Microsoft IT toward a service-focused IT organization was also noticeable in the designs and service level agreements (SLAs) that Microsoft IT established with the rollout of Exchange Server 2003, released in October 2003. New technologies, such as support for multi-node server clusters and cached Exchange mode available in Microsoft Office Outlook® 2003, enabled Microsoft IT to concentrate Mailbox servers in four data centers and reduce the number of Exchange servers, including those for special purposes, from approximately 200 to roughly 100. The number of Mailbox servers dropped from 118 to 36. Individual Mailbox servers hosted up to 4,000 mailboxes per active cluster node with quotas of 200 MB per mailbox. Consolidating the corporate messaging environment yielded overall cost savings of $20 million USD in the fiscal year 2003 alone.

Microsoft IT designed the Exchange Server 2003 environment for the scalability and availability requirements of a fast-growing company. The consolidation included upgrades to the network infrastructure and the deployment of large Mailbox servers to support 85,000 mailboxes in total. Business-driven SLAs demanded 99.99 percent availability, including both unplanned outages and planned downtime for maintenance, patching, and so forth. To comply with the SLAs, Microsoft IT deployed almost 100 percent of the Mailbox servers in server clusters by using Windows Clustering and a highly available, shared storage subsystem based on storage area network (SAN) technology.

In 2006, prior to Exchange Server 2007, the environment had grown to include 130,000 mailboxes that handled 6 million internal messages and received 1 million legitimate message submissions from the Internet daily. On average, every user sent and received approximately 100 messages daily, amounting to an average e-mail volume per user per day of 25 MB. As the demand for greater mailbox limits increased, new technologies and cost-efficient storage solutions such as direct access storage (DAS) were necessary to increase the level of messaging services in the corporate environment.

"Our mission is to deliver value by enabling people with innovative and reliable information technology solutions that seamlessly integrate with, and improve how people work."

Jim DuBois

General Manager, MSIT

Microsoft Corporation

Reasons for Microsoft IT to Upgrade

A global survey of 1,400 chief information officers (CIOs) conducted by Gartner Executive Programs in 2006 indicated the focus of IT is increasingly shifting from cost-cutting to improving productivity, performance, and competitiveness. Following a decade of unrestrained growth in IT and approximately five years of consolidation and cutbacks thereafter, IT departments are now in a better position to refocus their strategies on future-oriented business goals.

Microsoft is a good example. After a period of cost-cutting through server and site consolidation, Microsoft IT used the deployment of Exchange Server 2007 as an opportunity to shift gears and focus on implementing solutions that help improve productivity and competitiveness, streamline system administration, and increase messaging protection beyond the levels already possible with Exchange Server 2003 Service Pack 2.

For the deployment of Exchange Server 2007, Microsoft IT defined the following key objectives:

• Increase employee productivity   This included increasing mailbox quotas by as much as 1,000 percent, from 200 MB to 500 MB and 2 GB, and deploying new productivity features available in Exchange Server 2007, such as unified messaging, which enables users to receive all messages in their mailboxes, including e-mail, voice mail, and fax messages. In addition to desktop and portable clients, users can use standard telephones to access these messages. Increasing employee productivity also included deploying Microsoft Office Outlook 2007 as the primary messaging client so users can benefit from new and advanced information management features such as instant search, managed folders, and more.
• Increase operational efficiency   This included reducing administrative overhead associated with maintaining the messaging environment through features that are directly available in Exchange Server 2007, such as Exchange Management Shell. Based on the Microsoft .NET Framework, Exchange Management Shell enables Microsoft IT to create custom scripts that facilitate mundane deployment tasks, such as applying a consistent set of configuration settings per server role across multiple servers.
• Decrease security risks   This included deploying Edge Transport servers and Forefront Security for Exchange Server in the perimeter network to increase security and messaging protection and to reduce the number of legitimate messages incorrectly identified as spam (false positives). This also included encrypting all internal server-to-server message traffic using Transport Layer Security (TLS) to help protect confidentiality for messages in transit.
• Decrease costs   This included redesigning server architectures and backup solutions for high availability to meet challenging SLAs. In redesigning server architectures, Microsoft IT heavily focused on incorporating the features directly available in Exchange Server 2007, replacing expensive SAN storage with a more cost-efficient DAS solution for CCR-based Mailbox server clusters, and eliminating tape backups. All of these considerations resulted in significant cost savings. From backup changes alone, Microsoft IT realized a cost savings of approximately 5 million per year.

Environment Prior to Exchange Server 2007

The Microsoft IT deployment options and design decisions for the transition to Exchange Server 2007 heavily depended on the characteristics of the existing network, Active Directory, and the messaging environment. Among other things, it was important to perform the transition from Exchange Server 2003 to Exchange Server 2007 without service interruptions or data loss. Like any organization operating a large messaging environment, Microsoft IT also had to take a phase of coexistence into account, because it was not possible to transition the entire environment in one gigantic step. To understand the Microsoft IT design decisions in detail, it is necessary to review the environment in which Microsoft IT performed the transition.

Figure 1 illustrates the locations of the data centers that contain Mailbox servers in the corporate production environment and the overall wide area network (WAN) connections between them. Concerning the WAN backbone, it is important to note Microsoft IT deliberately designed the network links to exceed capacity requirements. Only 10 percent of the theoretically available network bandwidth is dedicated to messaging traffic. The vast majority of the bandwidth is for non-messaging purposes to support the Microsoft product development teams.

Figure 1. The Exchange Server 2003 environment at Microsoft on March 31, 2006

Network Infrastructure

Each Microsoft data center is responsible for a different region defined along geographical boundaries. Within each region, network connectivity between offices and the data center varies widely. For example, a high-speed metropolitan area network (MAN) based on Gigabit Ethernet and Synchronous Optical Network (SONET) connects more than 70 buildings to the Redmond data center. These are the office buildings on and off the main campus in the Puget Sound area. In other regions, such as Asia and the South Pacific, Internet-connected offices, or ICOs for short, are more dominant. Broadband connectivity solutions, such as digital subscriber line (DSL) or cable modems, provide significant cost savings over leased lines as long as the decrease in performance and maintainability is acceptable. Microsoft IT uses this type of connectivity primarily for regional sales and marketing offices.

Figure 2 summarizes the typical regional connectivity scenarios at Microsoft. It is important to note there are no Mailbox servers outside the data center, whereas Active Directory domain controllers may exist in high-availability buildings and medium-sized offices for local handling of user authentication, authorization, and application requests.

Figure 2. Regional connectivity scenarios at Microsoft

For regional connectivity, Microsoft IT relies on a mix of Internet-based and privately owned/leased connections, as follows:

• Regional data centers and main campus   The main campus and regional data centers are connected together in a privately owned WAN based on frame relay, asynchronous transfer mode (ATM), clear channel ATM links, and SONET links.
• Office buildings with standard or high availability requirements   Office buildings connect to regional data centers over fiber-optic network links with up to eight wavelength division multiplexing (WDM) channels per fiber pair.
• Regional offices with up to 150 employees   Regional offices use a persistent broadband connection or leased line to a local Internet service provider (ISP) and then access their regional data centers through a transparent virtual private network over Internet Protocol security (VPN/IPSec) tunnels.
• Mobile users   These use a dial-up or non-persistent broadband connection to a local ISP, and then access their mailboxes through VPN/IPsec tunnels, or by using Microsoft Exchange ActiveSync®, remote procedure call (RPC) over Hypertext Transfer Protocol (RPC over HTTP, also known as Outlook Anywhere), or Microsoft Office Outlook Web Access over secure HTTP (HTTPS) connections.

Directory Infrastructure

Like many IT organizations that must keep business units strictly separated for legal or other reasons, Microsoft IT has implemented an Active Directory environment with multiple forests. Each forest provides a clear separation of resources and establishes strict boundaries for effective security isolation. At Microsoft, some forests exist for legal reasons, others correspond to business divisions within the company, and yet others are dedicated to product development groups for early pre-release deployments and testing without interfering with the rest of the corporate environment. For example, by maintaining separate product development forests, Microsoft IT can prevent uncontrolled Active Directory schema changes in the Corporate Forest.

The most important forests at Microsoft have the following purposes:

• Corporate   Seventy percent of the resources used at Microsoft reside in this forest. The Corporate Forest includes approximately 140 domain controllers. The Active Directory database is 11 GB in size, with over 1 million directory objects, including users, groups, organizational units, workstations, servers, domain controller accounts, and printer objects.
• Corporate Staging   Microsoft IT uses this forest to stage software images, gather performance metrics, and create deployment documentation.
• Exchange Development   Microsoft uses this forest for running pre-release Exchange Server versions in a limited production environment. Users within this forest use beta or pre-beta versions in their daily work to help identify issues prior to the release of the product. Microsoft IT manages and monitors this forest, while the Exchange Server development group hosts the mailboxes in this forest to validate productivity scenarios.
• Extranet   Microsoft IT has implemented this forest to provide business partners with access to corporate resources. There are approximately 30,000 user accounts in this forest.
• MSN®    MSN is an online content provider through Internet portals such as msn.com. Microsoft IT manages this forest jointly with the MSN technology team.
• MSNBC   MSNBC is a news service and a joint venture between Microsoft and NBC Universal News. Legal reasons require Microsoft to maintain a separate forest for MSNBC. Microsoft IT manages this forest jointly with the MSNBC technology team.
• Test Extranet   This forest enables the Extranet Technology team to test new solutions for partner collaboration without interfering with the Extranet Forest. Microsoft IT manages this forest jointly with the Extranet Technology team.
• Windows Deployment   Microsoft IT created this forest to launch pilot projects during the Windows Server 2003 deployment phase as a pre-staging environment prior to deployment and feature configuration in the Corporate Forest. It is a limited production forest. Users within this forest use beta or pre-beta software in their daily work to help product development groups identify and eliminate design flaws and other issues.
• Windows Legacy   This forest is used as a test environment for compatibility testing of previous Windows Server versions with Exchange Server (specifically Microsoft Windows 2000 service pack testing).

Domains in the Corporate Forest

Microsoft IT implemented nine domains in the Corporate Forest, separated into geographic regions. At the time of the production rollout, all domains in the Corporate Forest operated at the Windows Server 2003 functional level and contained between seven and 30 domain controllers. The domain controllers are 64-bit multi-processor systems with 16 GB of random access memory (RAM). The Microsoft IT Active Directory database in the Corporate Forest is approximately 11 GB in size. With 16 GB of RAM, domain controllers can load the entire 11 GB Active Directory database into memory, which provides good performance for Exchange Server and other applications that extensively perform directory lookups by using Lightweight Directory Access Protocol (LDAP).

Active Directory Sites

Overall, the corporate production environment (that is, the Corporate Forest) includes 202 Active Directory sites in a hub-and-spoke topology that closely mirrors the network infrastructure. The authoritative source of IP address and subnet information necessary for the Active Directory site definitions is an infrastructure database that the Microsoft IT network team maintains. Using MIIS 2003, Microsoft IT provisions site and subnet objects in Active Directory based on the data from the IP address and subnet infrastructure database and ensures in this way an accurate Active Directory site topology that mirrors the network layout. The MIIS solution automatically calculates all site links during the import into Active Directory. Based on this information, Knowledge Consistency Checker (KCC) updates the replication topology for the forest. Microsoft IT does not maintain the Active Directory replication topology manually.

Site Topology and Exchange Server 2003

A highly granular site and subnet topology provides advantages in terms of Exchange Server communication. Although Exchange Server 2003 relies primarily on routing groups to describe the physical network topology, communication with Active Directory and client referral logic is site-aware. The Directory Service Access (DSAccess) component prefers to communicate with domain controllers and global catalog servers in the local Active Directory site. Through an Active Directory site topology that closely mirrors the physical network infrastructure, Microsoft IT confines DSAccess communication to local network segments.

Figure 3 shows the Active Directory sites in the Corporate Forest that are relevant for the Exchange Server organization.

Figure 3. Exchange Server 2003 Active Directory sites and site links at Microsoft

As shown in Figure 3, there are four sites with Mailbox servers and a fifth site with Internet mail gateway servers running Exchange Server 2003. The remaining Active Directory sites, ADSITE_REDMOND and ADSITE_NORTH CAROLINA, contain infrastructure servers and domain controllers to handle authentication requests from client workstations and LOB applications, but no Exchange servers. Although these sites are not very relevant for Exchange Server 2003, they influence the future Exchange Server 2007 design, as explained later in this white paper. With respect to the Exchange Server 2007 design, it is important to note that all site links are bidirectional and IP-based.

Dedicated Exchange Site Design

It is also worth mentioning that the Active Directory site named ADSITE_REDMOND-EXCHANGE contains only Exchange servers and domain controllers configured as global catalog servers. Microsoft created this dedicated site design during the Exchange 2000 Server time frame to provide its Exchange 2000 and  2003 servers with exclusive access to highly available Active Directory servers, shielded from client authentication and other application traffic. For detailed information about the implementation of a dedicated Active Directory site for Exchange Server 2003, see the Microsoft IT Showcase Note on IT "Creating an Active Directory Site for Exchange Server," available for download at http://www.microsoft.com/downloads/details.aspx?FamilyID=6b263452-7a61-4253-9c9e-b337cb80d460&DisplayLang=en.

Microsoft IT continues to use the dedicated Exchange site in its Exchange Server 2007 environment for the following reasons:

• Performance assessments   Exclusive Active Directory servers provide an opportunity to gather targeted performance data. Based on this data, Microsoft IT and developers can assess the impact of Exchange Server versions and service packs on domain controllers in a genuine large-scale production environment.
• Windows Server 2008 domain controllers   Microsoft IT maintained Windows Server 2008 Beta domain controllers in the corporate production environment. Microsoft IT decided to separate the Windows Server 2008 deployment from the messaging environment by using Active Directory sites without Exchange servers, such as ADSITE_REDMOND.

Messaging Infrastructure

The design of an Exchange Server 2003 organization relies on two topologies: a flat arrangement of administrative groups and a topology of routing groups and routing group connectors. Figure 4 shows the administrative and routing group topology of Microsoft IT's Exchange Server 2003 organization in the Corporate Forest and connectivity to other production forests and Exchange organizations.

Figure 4. Administrative and routing groups in the Corporate Forest just prior to the start of the transition to Exchange Server 2007 (March 31, 2006)

Administrative Topology

Microsoft IT uses a centralized administration model for Exchange servers. Although there are four administrative groups (North America, Dublin, Singapore, and Sao Paulo) in the legacy Exchange Server topology, Microsoft IT did not use custom permissions based on administrative groups. All Exchange Server administrators are located in Redmond and perform system administration and remote monitoring. The regional data centers are only responsible for hardware maintenance.

Routing Topology

The Exchange Server 2003 routing topology followed a hub/spoke architecture that corresponded to the WAN links depicted in Figure 1 earlier in this white paper. The data centers in Redmond, Dublin, Singapore, and Sao Paulo corresponded to routing groups with Mailbox servers. Two additional routing groups existed with gateway servers dedicated to external communication: RG_REDMOND PERIMETER and RG_SILICON VALLEY PERIMETER. Silicon Valley provided Internet mail redundancy for the messaging environment.

Between the routing groups, Microsoft IT used routing group connectors (RGCs) with the default option Any local server can send mail over this connector. Accordingly, all Exchange servers were able to transfer messages to adjacent routing groups directly. Using only one physical path to each routing group and all Exchange servers as local bridgeheads eliminated the need to communicate link state changes within and across routing groups. It also enabled Microsoft IT to keep the number of dedicated bridgehead servers at a minimum.

To implement the hub/spoke topology, Microsoft IT deployed four central bridgehead servers in North America, which Microsoft IT specified as remote bridgehead servers in the RGC configuration for the regions. Exchange servers in RG_DUBLIN, RG_SINGAPORE, and RG_SAO PAULO that wanted to transfer messages to other routing groups could do so through one of these bridgehead servers. In this way, the bridgehead servers created a bifurcation point for messages addressed to recipients in multiple locations. By splitting messages into multiple copies (bifurcation) at the latest possible point (the bridgeheads), Microsoft IT preserved network bandwidth on WAN links.

The four central bridgehead servers also performed message routing for communication with external entities, such as Internet destinations, partner domains, and Exchange organizations in other forests. For communication with Exchange organizations in other forests, Microsoft IT configured Simple Mail Transfer Protocol (SMTP) connectors directly on the bridgeheads (see Figure 4). Messages addressed to partners or Internet recipients, on the other hand, reached their destinations through further bridgehead servers (for antivirus scanning) and Internet mail gateways.

The Internet mail gateways in RG_DUBLIN and RG_SINGAPORE only handled outbound Internet mail for their local routing groups, whereas the Internet mail gateways in RG_REDMOND PERIMETER and RG_SILICON VALLEY PERIMETER were responsible for outbound and inbound Internet mail transfer. All inbound Internet mail messages reached Microsoft through two redundant locations in North America, which was the most efficient configuration for Microsoft because it has the flat e-mail domain namespace (@microsoft.com) and the majority of mailboxes are located in its Redmond location. The Internet mail gateways performed a series of anti-spam and other filtering checks (for example, block-list, sender, and recipient filtering), before routing the messages to internal bridgehead servers for virus scanning. Microsoft IT opted not to co-deploy antivirus solutions on the 32-bit Internet mail gateway servers in order to be able to apply anti-spam filtering first and avoid the overhead associated with virus scanning.

Server Roles per Location

Microsoft IT assigned a dedicated role to most servers in the Exchange Server 2003 organization, which allowed a more precise hardware configuration for each server to reflect high performance and scalability demands. An exception to this rule was the Mailbox server in Sao Paulo. Due to moderate workload, Microsoft IT was able to consolidate server roles in this location. The server in Sao Paulo acted as a Mailbox server, public-folder server, and bridgehead server. Moderate workload also enabled Microsoft IT to assign multiple roles to the public-folder servers in Dublin and Singapore. These servers assumed the responsibilities of public-folder servers and bridgehead servers for the regions.

Table 1 lists the number of servers per role that Microsoft IT deployed in each routing group (March 31, 2006).

Table 1. Servers per Role per Routing Group Prior to Exchange Server 2007 Transition

* For hosting public-folder data, free-and-busy information, and offline address books

** To support messaging needs of internal LOB applications, etc.

Planning and Design Process

The Microsoft IT planning and design process is unique in the way that messaging engineers start their work early in the product development cycle and collaborate very closely with the Microsoft Exchange Server Product group to clarify how exactly the new Exchange Server version should address concrete business requirements, system requirements, operational requirements, and user requirements. Through an assessment of the Exchange Server 2003 environment and in discussions with partner and customer IT organizations, Microsoft IT identified general issues, such as high storage costs and server scalability issues on the 32-bit platform, and communicated the findings as opportunities for improvements to the developers. In this way, the planning and design process at Microsoft IT actually influenced the product itself, addressing not only the requirements of Microsoft but also the needs of partner and customer IT organizations.

Figure 5 illustrates how Microsoft IT aligned the Exchange Server 2007 design and deployment processes from assessment and scoping through full production rollout. The individual activities correspond to the phases and milestones outlined in the Microsoft Solutions Framework (MSF) Process Model.

Figure 5. Microsoft IT planning, design, and deployment processes

The next sections discuss key activities that helped Microsoft IT to determine an optimal Exchange Server 2007 architecture and design for the corporate production environment.

Assessment and Scoping

In extensive planning sessions, product managers, service managers, the Exchange Systems Management team, Tier 2 Support team, Helpdesk, and Messaging Engineering team collaborated in virtual project teams to identify business and technical requirements and translated these requirements into proposals to the Exchange Server Product group. The Exchange Server Product group reviewed and incorporated these proposals into its product development plans. The results were commitments and shared goals between the developers and Microsoft IT to drive deployment actions and investments intended to improve IT services.

Deployment Planning Exercises

Within Microsoft IT, the Messaging Engineering team is responsible for creating the architectures and designs of all Exchange-related technologies. At a stage when the actual product was not available, messaging engineers began their work with planning exercises based on product development plans. The objective of these exercises was to decide how to deploy the new Exchange Server version in the future. The messaging engineers based their design decisions on specific productivity scenarios, the scalability and availability needs of the company, and other requirements defined during the assessment and scoping phase. For example, the Exchange Messaging team decided to use CCR to eliminate single points of failure in the Mailbox server configuration and DAS to drive down storage costs while at the same time increasing mailbox quotas up to a factor of 10. The deployment planning exercises helped to identify required hardware and storage technologies that Microsoft IT needed to invest in to be able to achieve the desired improvements.

Engineering Lab

The Messaging Engineering team maintains a lab environment that simulates the corporate production environment in terms of technologies, topology, product versions, and interoperability scenarios, but without production users. The Engineering Lab includes examples of the same hardware and storage platforms that Microsoft IT uses in the corporate production environment. It provides the analysis and testing ground for the messaging engineers to validate, benchmark, and optimize designs for specific scenarios, test components, verify deployment approaches, and estimate operational readiness. Testing in the Engineering Lab enables the messaging engineers to ensure that the conceptual and functional designs scale to the requirements and scope of the deployment project. For example, code instabilities or missing features of beta products might require Microsoft IT to alter designs and integration plans. Messaging engineers can verify the capabilities of chosen platforms and work with the product teams and hardware vendors to make sure the deployed systems function as expected even when running pre-release versions.

Pre-Release Production Deployments

Microsoft IT maintains a pre-release infrastructure, which is a limited production environment for running pre-release versions of server products. Pre-release production deployments begin prior to the alpha phase and continue through the beta and release candidate (RC) stages until Microsoft releases the product to manufacturing. During the pre-alpha stage, pre-release production deployments are a developer effort. Additional employees from within Microsoft join the campaign during the beta stages as early adopters.

Pre-release production deployments enable the developers to determine the enterprise readiness of the software, identify issues that might otherwise not be found prior to RTM, and collect valuable user feedback. For example, Exchange Server 2007 pre-release verification started in February 2005, more than 22 months before the Exchange Server Product group shipped the product. In comparison, the Microsoft Exchange Server 2003 pre-release deployment period was only six months.

Technology Adoption Program

The Exchange Server 2007 Technology Adoption Program (TAP) started during the pre-alpha stage in April 2005. TAP is a special Microsoft initiative, available by invitation only, to obtain real-world feedback from Microsoft partners and customers. More than 90 Microsoft partners and customers participated in the Exchange Server 2007 TAP. The Messaging Engineering team was also actively involved by providing early adopters with presentations outlining the Microsoft IT design process based on the then-current state of the product.

Microsoft runs several types of TAP programs for partners and customers to obtain real-world feedback on Microsoft pre-release products. For more information, see the TAP early-adopter information on MSDN at http://msdn2.microsoft.com/en-us/isv/bb190413.aspx.

Pilot Projects

An important task of the Messaging Engineering team is to document all designs, which the messaging engineers pass as specifications to the technical leads in the Systems Management team for acceptance and implementation. The messaging engineers also assist the technical leads during pilot projects and server installations and help develop a set of build documents and checklists to provide operators with detailed deployment instructions for the full-scale production rollout.

Production Rollout

The server designs that the Messaging Engineering team creates include detailed hardware specifications for each server type, which the Infrastructure Management team and the Data Center Operations group at Microsoft IT use to coordinate the procurement and installation of server hardware in the data centers. The Data Center Operations group builds the servers, installs the operating systems, and joins the new servers to the appropriate forest before the Exchange Systems Management team takes over for the deployment of Exchange Server 2007 and related components. To achieve a rapid deployment, Microsoft IT automated most of the Exchange Server deployment steps by means of Exchange Management Shell scripts.

Architecture and Design Decisions

"Microsoft IT is our first and best customer. Almost two years prior to RTM, Microsoft IT began with pre-release production deployments to help us build an excellent product. The close relationship with Microsoft IT is so vital to our culture of quality and customer satisfaction that we do not ship products or service packs until Microsoft IT signs off on the enterprise readiness. We shipped Exchange Server 2007 on December 7, 2006, with the confidence and proof in hand that the product delivers on its potential to help customers build reliable enterprise-class messaging environments while reducing total cost of ownership."

Terry Myerson

General Manager

Exchange Server Product Group

Microsoft Corporation

One of the most important objectives that Microsoft defined for the Exchange Server 2007 production rollout was to finish the transition no later than the official RTM date of the product. Microsoft IT committed to perform the rollout at full scale by using the Beta 2 release to demonstrate the enterprise readiness of Exchange Server 2007 to Microsoft customers. However, the Beta 2 release was not perfect, and Microsoft IT did not know all of the product's performance parameters yet. For these reasons, Microsoft IT deliberately created initial designs for the Client Access, Hub Transport, Edge Transport, Mailbox, and Unified Messaging server roles that exceeded actual production requirements. To maximize the ROI of server hardware and storage technology, Microsoft IT began to optimize the designs after the completion of the initial rollout. The following sections discuss these updated designs.

Administration and Permissions Model

Like any public company in the United States, Microsoft must safeguard the corporate IT infrastructure to comply with legal and regulatory requirements, such as the Health Insurance Portability and Accountability Act of 1996 and the Sarbanes-Oxley Act of 2002. These regulations also apply to the Exchange Server environment. To prevent fraud, protect assets, and ensure that messaging resources are used as intended, Microsoft IT implemented a strict administrative design according to the principle of fewest privileges as well as formal approval processes for granting administrative rights.

Security Principles and Guidelines

The Messaging Engineering team used the following principles and guidelines in the administrative design for Exchange Server 2007:

• Always use groups to assign rights   Delegating administrative permissions through security groups is uncomplicated, easy to understand, and a best practice in enterprise IT environments. Among other things, users can determine their effective rights by analyzing their group memberships. Moreover, security groups eliminate the need to configure individual access control lists (ACLs) for resources, which helps to reduce complexities caused by possibly conflicting rights granted through multiple direct or inherited access control entries (ACEs) and helps to keep the size of ACLs under control. Granting rights through group membership is also more efficient than granting access permissions through individual assignments, because a security group's access permissions do not change when members are added or removed from the group.
• Use the default permissions model   The default permissions model of Exchange Server 2007 covers most of the Microsoft IT requirements. With the exception of specific needs, such as regulatory requirements, Microsoft IT does not deviate from the default model to maintain the fewest necessary number of ACLs on Exchange Server resources.
• Grant the least permission necessary   The principle of fewest privileges is a generally recognized security approach and a Microsoft IT best practice. Microsoft IT never grants full Domain or Enterprise Admin rights unless there is a compelling business or technical reason. For example, administrators who need to modify recipient attributes in Active Directory do not need and do not receive access permissions to resources in the Exchange Server organization.
• Implement approval processes   Each security group that Microsoft IT uses for granting rights must be controlled based on approval processes that include the IT team responsible for maintaining the service or data.

Exclusive Microsoft IT Management

The administrative model of Exchange Server 2007 relies on Active Directory forests to define security boundaries. Within a single forest, there is no security isolation, because forest owners and enterprise administrators can always gain access to all resources in any domain. Accordingly, Microsoft IT grants enterprise administrator and top-level domain administrator rights in the Corporate Forest only on a temporary basis and enforces very strict approval processes.

Very strict approval processes imply that developers from the Exchange Server Product group and employees who are not responsible for operating the corporate production environment are usually not granted administrator rights in the Corporate Forest. If individuals outside Microsoft IT require administrative access to Exchange Server resources for testing or other purposes, the corresponding resources cannot be part of the Corporate Forest. To accommodate these situations, Microsoft IT maintains separate forests, such as an Exchange Development Forest that the Exchange Server Product group can use to test pre-release versions of Exchange Server.

Only in rare situations does Microsoft IT grant developers temporary administrative access to production servers, such as when troubleshooting critical server failures. To accommodate these situations, Microsoft IT implemented an access model based on Group Policy objects (GPOs) and Restricted Groups policies. User accounts that are removed from the membership list in a Restricted Groups policy are removed from the corresponding security group during the next GPO refresh, which occurs every 90 minutes on a member server and every five minutes on a domain controller and also every 16 hours, whether or not there are any changes. In this way, Microsoft IT enforces the removal of temporary administrative access and maintains the principle of fewest privileges in the corporate production environment.

Centralized System Administration

Exchange Server 2007 supports centralized and decentralized administration by means of four administrator roles:

• Exchange Organization Administrators   As the name implies, the Exchange Organization Administrators role gives administrators organization-wide, full access to all Exchange properties, objects, and servers. All Microsoft IT operators within the Exchange Messaging team are Exchange Organization Administrators with the rights to read and change all Exchange configuration data in the Corporate Forest worldwide. This strictly centralized model greatly simplifies administrative complexities.
• Exchange Server Administrators   These administrators only have permissions to control the configuration of a particular server or group of selected servers and cannot perform global, organization-wide administration tasks. Because Microsoft IT manages all Exchange Server resources centrally from headquarters in Redmond, it was not necessary for Microsoft IT to delegate the Exchange Server Administrators role for subsets of Exchange servers to separate user accounts or security groups.
• Exchange Recipient Administrators   Users assigned the Exchange Recipient Administrators role take care of recipient-related tasks, such as mailbox-enabling user accounts, mail-enabling contacts, distribution groups, and other types of recipient objects in Active Directory, and configuring Client Access and Unified Messaging mailbox settings. It is important to note that the Exchange Messaging team does not perform recipient administration. Other teams, such as the HR department, maintain the user accounts and related Exchange settings.
• Exchange View-Only Administrators   These are users without operational tasks such as Helpdesk leads and messaging engineers who have a business need to examine the parameters and current state of the messaging environment.

Default Permissions Model

The Exchange Server 2007 permissions model is straightforward and flexible because it promotes the use of universal security groups for permissions management that closely correspond to the administrative roles listed in the previous section. The Exchange Setup /PrepareAD process creates these universal security groups in the root domain with a forest-wide scope. The groups are located in the Microsoft Exchange Security Groups container. They are globally available for permission assignments to Exchange Server resources in any domain, and they can contain users and groups from any domain in the forest.

Microsoft IT based its centralized administration model for Exchange Server 2007 on the following default universal security groups:

• Exchange View-Only Administrators   Have read-only access to Exchange recipient attributes on Active Directory objects and read-only access to the Exchange configuration data in Active Directory.
• Exchange Recipient Administrators   Have full control over Exchange recipient attributes on Active Directory objects. In addition, this group has read-only access to the Exchange configuration data in Active Directory because this group is a member of the Exchange View-Only Administrators group.
• Exchange Organization Administrators   Have full access to the Exchange configuration data in Active Directory. In addition, this group has full control over Exchange recipient attributes on Active Directory objects because this group is a member of the Exchange Recipient Administrators group.

One important feature of the default configuration is that there is no Exchange Server Administrators group because that is not how Microsoft IT applies permissions for the Exchange Server Administrator role. Instead, Microsoft IT applies the permissions directly on the object in the configuration partition. Another important fact is that Exchange Organization Administrators have permissions to modify recipient attributes through membership in the Exchange Recipient Administrators group. Microsoft IT considered removing the Exchange Organization Administrators from the Exchange Recipient Administrators group but found no compelling reason to deviate from the default model during the initial deployment. The default permissions do not grant Exchange Organization Administrators rights to create, modify, or delete objects within the Active Directory domain-naming context, which would require at least Account Operators privileges. The authoritative source of user account information is a separate LOB application, maintained by the HR department and synchronized with Active Directory through MIIS 2003.

Formal Approval Processes

The domain topology that Microsoft IT implemented in the Corporate Forest features an empty root domain to support strict management processes for schema updates and for granting forest-wide administrative rights and privileges. Only a very limited number of IT managers have administrative permissions in the root domain. Because the Exchange Setup /PrepareAD process creates the universal security groups of Exchange Server 2007 in the root domain, Microsoft IT implicitly gained the ability to enforce formal approval processes for Exchange Server administration as well.

Restricting access to the universal security groups of Exchange Server 2007 in the root domain does not prevent Microsoft IT from delegating approval processes to individual teams and groups that are ultimately responsible for the corporate and Exchange Server environment, such as the Legal and Corporate Affairs (LCA) team and the Exchange Messaging team. To delegate approval processes, Microsoft IT added security groups from child domains to the universal security groups in the root domain. Team managers with permissions to change group membership information in child domains can use these security groups to delegate administrative rights.

Figure 6 shows the principle of delegating approval processes to individual teams and groups through security groups. As a best practice, Microsoft IT always uses security groups (in child domains) as opposed to individual user accounts to assign administrative permissions. The membership information of security groups in the root domain rarely changes.

Figure 6. Delegating access control and approval processes

Permissions Review

One interesting aspect of the Exchange Server 2007 default permissions model is that when running setup with the /PrepareLegacyExchangePermissions option, it begins the automatic process of changing the permissions model of the Exchange Server 2003 environment for coexistence with Exchange Server 2007. Microsoft IT completes this process by running the setup with the /PrepareAD option, which finishes the changes to property sets and security groups. While applying new Exchange Server 2007 permissions, Microsoft IT took the opportunity to review and clean up permission assignments from the Exchange Server 2003 environment.

For example, Microsoft IT used security groups to grant administrative permissions to Exchange Server 2003 resources at the organization and administrative group levels. Yet, some members of these groups no longer need these permissions. By using new security groups in the child domains, added to the Exchange Organization Administrators group in the root domain, Microsoft IT effectively devised the following strategy to reevaluate all existing Exchange Server administrators:

• Current Exchange Server administrators   Individuals who need administrative rights to manage Exchange Server 2007 resources must request them from the Systems Management team. Following each individual approval, the Systems Management team adds the corresponding user account to the new security groups to grant Exchange Organization Administrators permissions.
• Former Exchange Server administrators   Individuals who no longer need administrative permissions automatically lose these rights as Microsoft IT decommissions Exchange Server 2003 resources and corresponding administrator groups.

Message Routing Topology

Agile companies that heavily rely on e-mail in practically all areas of the business, such as Microsoft, cannot tolerate messages that take hours to reach their final destinations. Information must travel fast, reliably, predictably, and securely. For Microsoft, this means 99 percent of all messages within the corporate production environment must reach their final destination in 90 seconds worldwide. Of course, this SLA does not apply to messages that leave the Microsoft environment, because Microsoft IT cannot guarantee message delivery across external systems. Microsoft established this mail-delivery SLA during the Exchange Server 2003 time frame, and it was equally important during the design of Exchange Server 2007. Another important design goal was to increase security in the messaging backbone by means of access restrictions to messaging connectors, data encryption through TLS, and messaging antivirus protection based on Forefront Security for Exchange Server.

Microsoft IT had to consider the following new Exchange Server 2007 routing features in the design of the message routing topology:

• Active Directory sites   Comparable to routing groups in Exchange Server 2003, Active Directory sites define a boundary for Hub Transport servers to deliver messages directly to Mailbox servers, distribution group expansion servers, connector servers, and Edge Transport servers subscribed to the local site. For destinations in remote Active Directory sites, the Hub Transport server in the local site must relay the messages to a Hub Transport server in the remote site for further delivery. At least one Hub Transport server must exist in every Active Directory site with mailbox servers.
• IP site links   Comparable to routing group connectors in Exchange Server 2003, IP site links define logical paths between Active Directory sites, which Hub Transport servers use to perform message routing calculations. It is important to note that Active Directory supports IP and SMTP site links, whereas Exchange Server 2007 ignores SMTP site links in the routing topology. If a Hub Transport server resides in a site connected only by an SMTP site link, routing errors will occur. It is necessary to replace the SMTP site link with an IP-based site link.
• Least-cost routing paths   Each IP site link is associated with a cost value that Exchange Server 2007 uses to calculate the message transfer paths across the routing topology. If multiple IP site links exist between two Active Directory sites with Hub Transport servers, Exchange Server 2007 transfers the message along the path with the least combined cost to the ultimate destination.
• Next hop selection   If the least-cost routing path to the ultimate destination includes multiple Active Directory sites and IP site links, Exchange Server 2007 attempts to relay the messages to an Active Directory site that is as close as possible to the ultimate destination when delivery to that destination fails. For example, Hub Transport servers might select the ultimate destination site as the next hop to deliver messages directly, skipping all intermediary sites that exist along the least-cost routing path.
• Queue at the point of failure and backoff   If message delivery to the next hop fails (for example, because there was no Hub Transport server available in the target site), Exchange Server 2007 attempts to deliver the messages to an interim site along the least-cost routing path. This backoff mechanism starts with the Active Directory site that is directly adjacent to the unavailable next hop, and then backs off, site by site, along the least-cost routing path until a connection is established or no further site exists. In this way, Exchange Server 2007 queues the messages at the point in the routing path where communication failed, which facilitates the troubleshooting of transfer issues.
• Delayed fan-out   If a message has multiple recipients in destinations that share part of or the entire least-cost routing path, Exchange Server 2007 transfers a single message copy to the point in the routing path where a fork occurs. At the fork, Exchange Server 2007 splits the message into separate copies. Again, each message copy might still have multiple recipients in destinations that share part of or the entire remaining least-cost routing path, and so forth. The bifurcation at the latest possible point to preserve network bandwidth is called delayed fan-out.

Network Infrastructure and Site Consolidation

An important aspect for planning message routing in an Exchange Server 2007 environment is the physical network topology. The physical network determines the IP routing topology, which directly influences the Active Directory site topology, which in turn determines the message routing topology of Exchange Server 2007. By using the existing Active Directory site infrastructure for message routing purposes, Exchange Server 2007 takes advantage of an optimal network configuration with little need for adjustments.

For example, Microsoft IT did not need to perform any network optimization to accommodate Exchange Server 2007, although the Exchange Server 2003 site and server consolidation that Microsoft IT conducted at a global scale three years ago benefited Microsoft IT in the Exchange Server 2007 design. Microsoft IT only had four main Active Directory sites with Exchange Mailbox servers and reliable WAN links with sufficient net-available bandwidth to take into consideration for Exchange Server 2007 topology and routing planning. The Exchange Server 2003 site consolidation greatly reduced the complexity of the transition exercise for Microsoft IT.

The fact that Microsoft IT only had to consider four main locations and Active Directory sites that already mapped to an optimized network infrastructure, as discussed earlier in this white paper, led to the following benefits:

• Uncomplicated messaging topology   No additional configuration was necessary to establish a functional Exchange Server 2007 routing topology—although opportunities to increase the efficiency of message transfer still existed. However, managing only four sites that house Mailbox servers and optimizing message flow by means of Exchange Server–specific IP site links was a straightforward undertaking for Microsoft IT.
• Best possible Hub Transport server utilization   Mailbox servers submit messages for routing and transport to a Hub Transport server in the local Active Directory site. Depending on the location of the recipients, the Hub Transport server then transfers the messages to a Mailbox server within the local site, to a Hub Transport server in another Active Directory site, or through a messaging connector to another external destination. The result is that message transfer cannot work if there is no Hub Transport server in the local site of the Mailbox server. In addition, this implies that a small number of Active Directory sites with a large number of Mailbox servers provide a better Hub Transport server utilization than a large number of Active Directory sites with a small number of Mailbox servers. For example, Microsoft IT deployed three Hub Transport servers for 15 Mailbox servers in ADSITE_DUBLIN to perform load balancing and provide fault tolerance. If the Mailbox servers in Dublin resided in two Active Directory sites instead, Microsoft IT would have had to deploy one additional Hub Transport server, because each site would have required a minimum of two Hub Transport servers to achieve load balancing and fault tolerance.

• Reduced chance of server communication issues   For purposes of message routing, client access, and unified messaging, it is important to deploy Hub Transport servers, Client Access servers, and Unified Messaging servers in each Active Directory site that contains Mailbox servers. Active Directory sites correspond to one or more IP subnets that represent areas with reliable, high-speed IP connectivity. Yet, it is possible to leave gaps or specify overlapping IP subnets in the Active Directory site topology, which can cause communication issues if Exchange Server 2007 cannot correctly determine its site membership. Keeping the Active Directory site topology straightforward helps Microsoft IT avoid these types of issues.

Dedicated Exchange Sites in the Active Directory Topology

Although dedicated Active Directory sites for Exchange Server may be used for domain controller/global catalog isolation and dedication of that infrastructure to Exchange servers for reliability or performance reasons, in the context of Exchange Server 2007 transport, dedicated Exchange sites can complicate the routing topology. They deviate from the classic definition of an Active Directory site as areas with reliable, high-speed, low-latency IP connectivity. Because dedicated Exchange sites generally do not correspond to the physical network layout, they can lead to a message routing topology that does not use the physical network links as efficiently as possible.

For example, Microsoft IT maintains a dedicated Exchange Active Directory site in addition to a central hub Active Directory site in Redmond, as indicated in Figure 3 earlier in this white paper. In the Active Directory replication topology, this dedicated Exchange site is a tail site. ADSITE_REDMOND is the Active Directory hub site that is used as the Active Directory replication focal point, yet this site does not contain any Hub Transport servers. Accordingly, at this point Exchange Server 2007 cannot use ADSITE_REDMOND as a hub site for message routing purposes and by default interprets the Microsoft IT Exchange organization as a full-mesh topology where Exchange 2007 Hub servers in each region connect to each other via a single SMTP hop. This does not match the Active Directory replication and network topology. Exchange Server 2007 uses the full-mesh topology in this concrete scenario, because along the IP site links all message transfer paths appear to be direct.

Figure 7 illustrates this situation by placing the Active Directory site topology and default message routing topology on top of each other.

Figure 7. Full-mesh message routing in a hub-and-spoke network topology

The routing topology depicted in Figure 7 works because Exchange Server 2007 can transfer messages directly between the sites with Hub Transport servers (such as ADSITE_SINGAPORE to ADSITE_DUBLIN), yet all messages travel through the Redmond location according to the physical network layout. In this topology, all bifurcation of messages, sent to recipients in multiple sites, takes place at the source sites and not at the latest possible point along the physical path, which would be Redmond. For example, if a user in Sao Paulo sends a single message to recipients in the sites ADSITE_REDMOND-EXCHANGE, ADSITE_DUBLIN, and ADSITE_SINGAPORE, the source Hub Transport server in Sao Paulo establishes three separate SMTP connections, one SMTP connection to Hub Transport servers in each remote site, to transfer three message copies. Hence, the same message travels three times over the network from Sao Paulo to Redmond. Microsoft IT could avoid this by eliminating the dedicated Exchange site ADSITE_REDMOND-EXCHANGE and moving all Exchange servers to ADSITE_REDMOND. The Hub Transport servers in ADSITE_REDMOND would then be in the transfer path between ADSITE_SAO PAULO, ADSITE_DUBLIN, and ADSITE_SINGAPORE, which would mean that Exchange Server 2007 could delay bifurcation until messages to recipients in multiple sites reach ADSITE_REDMOND. In this situation, the source server would only need to transfer one message copy to Redmond, the message routing topology would follow the physical network layout, and Microsoft IT would not have to take any extra configuration or optimization steps.

Based on such considerations, it is a logical conclusion that the design of an ideal Exchange Server 2007 environment takes the implications of dedicated Exchange Active Directory sites into account. On one hand, it is beneficial to keep the message routing topology straightforward and the complexities associated with maintaining and troubleshooting message transfer minimal. On the other hand, Microsoft IT had to weigh the benefits of eliminating the dedicated Redmond Exchange site ADSITE_REDMOND-EXCHANGE against the impact of such an undertaking on the overall deployment project in terms of costs, resources, and timelines. Among other things, Microsoft IT maintains ADSITE_REDMOND-EXCHANGE to shield Exchange Server 2007 from Windows Server 2008 domain controllers that exist in the corporate production environment. As mentioned earlier in this paper, Exchange Server 2007 support for Windows Server 2008 was not available at the time Microsoft IT transitioned the corporate production environment. Eliminating ADSITE_REDMOND-EXCHANGE would have required Microsoft IT to remove all Windows Server 2008 domain controllers from ADSITE_REDMOND, which was not an option. Furthermore, Microsoft IT takes advantage of the dedicated Active Directory site to measure the footprint of Exchange Server 2007 on domain controller/global catalog servers and provide this information as feedback to the product teams. Correspondingly, Microsoft IT decided to leave ADSITE_REDMOND-EXCHANGE in place. Instead, the Exchange Messaging team collaborated with the Active Directory team to adjust the Active Directory site topology by using alternative methods to optimize message transfer without affecting the established Active Directory replication architecture and topology.

Optimized Message Transfer Between Hub Transport Servers

Although Exchange Server 2007 generated a functioning message routing topology without any extra design work, Microsoft IT decided to review the routing topology based on business and technical requirements to drive further optimizations. Key factors that influenced the optimization decision included the "90 seconds, 99 percent of the time" mail-delivery SLA and the desire to save network bandwidth on WAN links by increasing the efficiency of message transfer.

Important reasons that compelled Microsoft IT to optimize the Exchange Server 2007 message transfer topology include the following:

• Efficient message flow   At Microsoft, 99 percent of the messages must reach their recipients within 90 seconds or less. Although optimized message flow is not a strict requirement and it is possible to meet mail-delivery SLAs in a full-mesh topology, optimized message flow can help to accelerate message delivery.
• Preserved WAN bandwidth   The corporate production environment handles more than 6 million internal messages daily. Although most message traffic stays in the local site or has Redmond headquarters as the destination, optimized message flow can help to preserve WAN bandwidth for all messages with recipients in multiple remote Active Directory sites.

Having made the decision to optimize message routing, Microsoft IT augmented the Active Directory site link topology in order to take advantage of the Exchange Hub Transport servers in ADSITE_REDMOND-EXCHANGE. To achieve efficient message flow and preserve WAN bandwidth, it was necessary to place ADSITE_REDMOND-EXCHANGE in the routing path between ADSITE_DUBLIN, ADSITE_SAO PAULO, and ADSITE_SINGAPORE by creating additional Active Directory site IP links. This approach ensured that Exchange Server 2007 could bifurcate messages traveling between regions closer to their destination. By configuring the ExchangeCost attribute on Active Directory site links, which Exchange Server 2007 adds to the Active Directory site link definition, Microsoft IT was able to perform the message flow optimization without affecting the Active Directory replication topology. The ExchangeCost attribute is only relevant for Exchange Server 2007 message routing decisions between sites, not Active Directory replication.

Microsoft IT performed the following steps to optimize message routing in the corporate production environment:

1. To establish a hub/spoke topology between all sites with Exchange servers, Microsoft IT created three additional Active Directory IP site links (see Figure 8).
2. Microsoft IT specified a Cost value of 999 (highest across the Active Directory topology) for these new IP site links so that Active Directory does not use these site links for directory replication.
3. Using the Set-AdSiteLink cmdlet, Microsoft IT assigned an ExchangeCost value of 10 to the new Exchange-specific site links. This value is significantly lower than the Cost value of all other Active Directory site links, so that Exchange Server 2007 uses the Exchange-specific site links for message routing path discovery.

Figure 8 illustrates how the Exchange-specific site links change the message routing topology. The dedicated Exchange site ADSITE_REDMOND-EXCHANGE in North America now acts as a fork in the routing path to the sites in ADSITE_DUBLIN, ADSITE_SAO PAULO, and ADSITE_SINGAPORE.

Figure 8. Optimized message routing topology in the Corporate Forest

Based on the Exchange-specific Active Directory/IP site link topology, Exchange Server 2007 routes messages in the Corporate Forest as follows:

• Messages to a single destination   The source Hub Transport server selects the final destination as the next hop and sends the messages directly to a Hub Transport server in that site. For example, in the Dublin to Singapore mail routing scenario, the network connection passes through Redmond, but the Hub Transport servers in ADSITE_REDMOND-EXCHANGE do not participate in the message transfer.
• Messages to an unavailable destination   If the source Hub Transport server is unable to establish a direct connection to a destination site, the Hub Transport server backs off along the least-cost routing path until a connection to a Hub Transport server in an Active Directory site is established. This would be a Hub Transport server in ADSITE_REDMOND-EXCHANGE, which queues the messages for transmission to the final destination upon restoration of network connectivity.
• Messages to recipients in multiple sites   Exchange Server 2007 delays message bifurcation if possible. In the optimized topology, this means that Hub Transport servers transfer all messages with recipients in multiple sites first to a Hub Transport server in ADSITE_REDMOND-EXCHANGE. The Hub Transport server in ADSITE_REDMOND-EXCHANGE then performs the bifurcation and transfers a separate copy of the message to each destination. Figure 8 illustrates this scenario. Exchange Server 2007 transfers a single message copy from ADSITE_SAO PAULO to ADSITE_REDMOND-EXCHANGE, where bifurcation takes place, before transferring individual message copies to each destination site. Again, Exchange Server 2007 transfers only a single copy per destination site. Within each site, the receiving Hub Transport servers may bifurcate the message further as necessary for delivery to individual recipients.

Connectivity to Remote SMTP Domains

For destinations outside the Corporate Forest, Microsoft IT distinguishes between external and internal remote locations. For external remote locations, Microsoft IT relays all messages over Edge Transport servers deployed in perimeter networks, as explained in the section "Internet Mail Connectivity" later in this white paper. For internal remote locations, Microsoft IT uses messaging connectors directly on the Hub Transport servers in ADSITE_REDMOND-EXCHANGE. This design mirrors the Exchange Server 2003 topology.

Increased Message Routing Security

To ensure compliance with legal and regulatory requirements, Microsoft IT encrypts most messaging traffic in the corporate production environment. The only exceptions are internal destinations without user mailboxes, such as lab and test environments.

Microsoft IT uses the following encryption technologies to prevent unauthorized access to information during message transmission:

• IP security   IPSec encrypts data communication and prevents unauthorized access to resources in the corporate production environment. Because IPSec works at the IP layer, it can help secure communication between servers without relying on the application to support the encryption natively. Microsoft IT extensively used IPSec encryption in its Exchange Server 2003 environment to help secure internal SMTP transactions and continues its use today in scenarios where SMTP servers and applications do not support native TLS-based SMTP encryption. Although IPSec technology offers strong encryption controls, managing custom IPSec policies can be quite cumbersome. With the transition to Exchange Server 2007, Microsoft IT was able to accomplish most of the transport encryption by using native Exchange Server product features.
• Transport Layer Security   Exchange Server 2007 supports TLS right out of the box. Hub Transport servers use TLS to encrypt all message traffic within the Exchange Server 2007 environment and rely on opportunistic TLS encryption for communication with remote destinations, such as Hub Transport servers in other Microsoft IT–managed forests. Edge Transport servers also support TLS a