Export (0) Print
Expand All

Deploying a Hybrid Messaging Infrastructure Using Office 365: Exchange Online

Enterprise Messaging Combining On-Premises and Cloud-Based Technologies

Technical White Paper

Published: June 2012

The following content may no longer reflect Microsoft’s current position or infrastructure. This content should be viewed as reference documentation only, to inform IT business decisions within your own company or organization.

A hybrid messaging deployment offers the possibility to have some mailboxes located on-premises while others reside in the cloud. Microsoft IT shares its planning and deployment experiences of a hybrid Microsoft Office 365 hybrid messaging solution.

Download

Download Technical White Paper, 1.29MB, Microsoft Word file

Situation

Solution

Benefits

Products & Technologies

Office 365: Exchange Online, offers Microsoft IT the opportunity to add flexibility to the messaging infrastructure and cut costs by deploying and operating a hybrid environment

As part of a long-term strategy, Microsoft IT onboarded 36,000 mailboxes to Exchange Online with the goal to migrate 80% of all mailboxes by 2015. This hybrid deployment offers the best features and benefits of both on-premises and cloud-based approaches.

  • Seamless user experience using Outlook Web App and Outlook 2010 client
  • Cost savings associated with using cloud-based messaging that Exchange Online provides
  • Flexibility to accommodate business growth through Exchange Online without expanding the on-premises infrastructure
  • SPAM and virus protection through Forefront Online Protection for Exchange
  • Synchronized address book and single sign-on
  • Office 365: Exchange Online
  • Exchange Server 2010
  • Active Directory
  • Windows Server 2008 R2

Contents

JJ218963.arrow_px_down(en-us,TechNet.10).gif Executive Summary

JJ218963.arrow_px_down(en-us,TechNet.10).gif Hybrid Advantages

JJ218963.arrow_px_down(en-us,TechNet.10).gif Microsoft Messaging Infrastructure At-A-Glance

JJ218963.arrow_px_down(en-us,TechNet.10).gif Designing for Hybrid Messaging

JJ218963.arrow_px_down(en-us,TechNet.10).gif Migrating Mailboxes to Exchange Online

JJ218963.arrow_px_down(en-us,TechNet.10).gif Supporting Users

JJ218963.arrow_px_down(en-us,TechNet.10).gif Lessons Learned and Best Practices

JJ218963.arrow_px_down(en-us,TechNet.10).gif For More Information

Executive Summary

Although hosted solutions for e-mail messaging have been available for many years, recent improvements have made it possible to deploy and operate a hybrid environment that makes the most of both on-premises and hosted services. Microsoft began offering Exchange Online as its multi-tenant enterprise messaging service in the cloud to customers starting at the end of 2008 based on Exchange Server 2007 technology with the goal of helping customers and its own workers realize the benefits of cloud computing. After onboarding millions of mailboxes from companies of all sizes, building out a scalable and highly available infrastructure, and upgrading Exchange Online to run Exchange Server 2010, Microsoft IT pursued an initiative to transition from operating its own on-premises Exchange environment to operating a hybrid environment. With a hybrid approach, Microsoft IT benefits from continuing to use previous investments in the existing on-premises infrastructure, with ability to accommodate business growth by using Exchange Online.

To overcome the engineering and business challenges in transitioning to a hybrid environment, Microsoft IT focused on ensuring user satisfaction by engaging all teams involved in the deployment effort. One key objective was to provide users with a seamless transition and automatic Outlook profile update to Exchange Online yet retain the same features and functionality of the on-premises service. To ensure the best user experience, the hybrid architecture incorporates design elements that include the following:

  • Single sign-on (SSO) using existing Active Directory credentials and Active Directory Federation Services (ADFS)
  • Shared address book for a unified global address list (GAL)
  • One microsoft.com domain namespace for both on-premises and Exchange Online
  • Centralized administration of mailboxes and mail flow
  • Synchronized calendar and free/busy scheduling
  • Landing page to inform Exchange Online users who log in to the Outlook Web App on-premises URL about the appropriate Exchange Online Outlook Web App URL

The Exchange Server architecture enables Microsoft IT to deploy messaging in a hybrid environment according to the needs of the business and desired project schedule. Microsoft IT moved mailboxes to Exchange Online after preparing the environmental dependencies such as identity management, security, and synchronization. In this way, Microsoft IT controls accounts, retention, e-discovery and other features in a unified way to ensure a centrally managed, homogenous environment.

This white paper contains information for business and technical decision makers who operate an on-premises messaging solution and are evaluating possibilities of transitioning to a hybrid environment that incorporates Exchange Online. The paper assumes basic familiarity with concepts relevant to messaging technologies, such as Active Directory, Exchange Server, TCP/IP, and DNS. A high-level understanding of the capabilities of Exchange Online and Office 365 is also helpful. For more information about Exchange Online, see http://www.microsoft.com/en-us/office365/exchange-online.aspx.

Note: For security reasons, the sample names of forests, domains, internal resources, organizations, and internally developed security file names used in this paper do not represent real resource names used within Microsoft and are for illustration purposes only.

Hybrid Advantages

The hybrid architecture results in many benefits for Microsoft IT not only in overall cost savings, but also in greater flexibility to accommodate business growth while saving time and money by not having to do capacity planning, update software, maintain servers, or manage hardware.

Cost Savings Due to Cloud Efficiencies

As a cloud service, Exchange Online provides the opportunities to reduce costs by eliminating the typical on-premises requirements of purchasing, deploying, and managing servers. These savings are possible due to Exchange Online features such as the following:

  • Large 25 GB mailbox size With Exchange Server 2010, Microsoft IT eliminated backups and relies on a cost-effective, Just a Bunch of Disks (JBOD)-based storage. This solution offers cost savings over the previous Storage Area Network (SAN) approach, yet it is a high expense to deploy and operate the storage subsystem. Exchange Online frees Microsoft IT from the need to manage any storage hardware.
    Quota management. During the initial phases of using Exchange Online in a hybrid environment, it is important to manage quotas in case mailboxes need to move back to on-premises. Microsoft IT uses the same quotas for both environments to prevent the possibility of having to increase on-premises quotas for specific users, or asking them to reduce their mailbox size.
  • Included technical support Exchange Online includes 24/7 phone support for the internal Microsoft IT support team, which helps to ensure timely responses and reliability.
  • Automatic failover Similar to the on-premises solution, Exchange Online also provides automatic failover for resiliency.
  • Highly available design Exchange Online includes mailbox resiliency technology, such as the ability to switch between database copies when disks fail, and automatic, database-level recovery from failures through database availability groups.
  • Flexible growth and expansion As Microsoft grows and changes, Exchange Online makes it straightforward to add mailboxes by simply buying additional licenses. This requires no capacity planning, server purchasing, or deployment.

Flexible Deployment and Management

Exchange Online and on-premises overlap in terms of management functionality. Both use Role-Based Access Control (RBAC) for task delegation and administration via the Exchange Control Panel web-based console or through Windows PowerShell using the Exchange Management Shell. Microsoft IT uses the remote PowerShell capability for managing Exchange Online from within the on-premises network. Exchange Online gives Microsoft IT management capabilities relevant to messaging-as-a-service, including recipient policies and groups, whereas Exchange on-premises provides all management capabilities.

A hybrid approach achieves the best of both worlds by enabling Microsoft IT to accomplish the following:

  • Deployment on Microsoft IT’s terms A hybrid approach offers Microsoft the flexibility to migrate mailboxes as needed to and from Exchange Online. As a way of validating the hybrid approach, Microsoft IT started migrating the mailboxes of a small number of volunteer early adopters, and sets the pace of migration according to its needs .
  • Infrastructure ownership and control Approaching messaging as a hosted service and an on-premises service gives Microsoft IT the flexibility to own the entire messaging continuum for the ultimate degree in infrastructure flexibility. Due to business needs, some mailboxes may remain on-premises, and others may be migrated to Exchange Online. If requirements change, Microsoft IT may move mailboxes from one environment to the other without affecting users.
  • Centralized management Both Exchange Online and on-premises share a unified approach to managing mailboxes, policies, recipients, and other Exchange objects. In the hybrid implementation, Microsoft IT manages all messaging details in a unified and centralized way.
  • Customer validation and dogfooding Validating hybrid performance and functionality as part of dogfooding efforts is one of Microsoft IT’s key goals. Part of the design and deployment entailed working through many types of possible scenarios to work out any issues and fine-tune best practices. This goal went beyond implementing quick fixes and resolving bugs, to validating administrative and support paths to ensure the hybrid architecture was suitable for enterprise needs.
  • Single namespace and unified experience Microsoft IT's hybrid design relies on auth headers in Exchange data, making communication appear internal to both on-premises and Exchange Online. As a result, Exchange features such as MailTips, and out-of-office (OOF) messages function and appear as expected to users and recipients.

Microsoft Messaging Infrastructure At-A-Glance

The Microsoft internal messaging infrastructure supports more than 200,000 mailboxes for employees, contractors, and business partners across three core divisions involving hundreds of products and services. As a company, Microsoft operates in more than 100 countries, with the majority of employees working in its Redmond, Washington headquarters.

To support the corporate messaging environment, Microsoft IT manages multiple regional data centers connected by high-speed WAN links. The network dependencies have been refined and improved over time to where routing, DNS infrastructure, bandwidth, and other similar considerations are stable with high levels of redundancy and availability. You can find out more about the Microsoft Exchange Server 2010 architecture at http://technet.microsoft.com/library/ff829232.aspx.

Although the technological capabilities of Exchange Server 2010 have enabled Microsoft IT to reduce costs and increase efficiencies by taking advantage of server consolidation and more flexible and larger storage, additional opportunities exist with a hybrid approach that incorporates Exchange Online.

On-Premises Messaging Architecture

The Exchange Server 2010 topology and architecture continues a tradition of following best practices, incorporating product group recommendations, and meeting business needs based on real-world performance data. Figure 1 shows a high-level overview of the Microsoft on-premises architecture before implementing a hybrid infrastructure.

Figure 1. On-premises messaging infrastructure
Figure 1. On-premises messaging infrastructure

The Exchange Server roles facilitate and separate the necessary functions of e-mail into servers that handle message filtering, transport, client access, mailbox storage, and unified messaging. As a best practice, Microsoft IT suggests deploying multi-role Exchange servers to support a hybrid infrastructure. For more information, including capacity planning, see http://technet.microsoft.com/library/dd298121.aspx.

Hybrid Messaging Architecture

As a cloud-based offering, Exchange Online provides messaging-as-a-service with an architecture that abstracts dependencies such as message filtering into additional services. The Exchange Online architecture uses a similar role-based approach as on-premises, but driven by the following services instead of roles:

  • Forefront Protection for Exchange (FOPE):   as a first-tier message handler for Exchange Online, FOPE provides protection from viruses and SPAM. Microsoft IT has used FOPE as a service since 2007 as its message filtering solution.
  • Office 365 directory   Exchange Online uses its own directory service for user data. To handle authentication, the directory service relies on Microsoft Online ID.
  • Exchange Online messaging   As the core service that handles messaging, Exchange Online includes transport and storage functionality to house mailboxes and facilitate mail flow.

Figure 2 shows Exchange Online in a hybrid architecture with Exchange on-premises.

Figure 2. Hybrid architecture
Figure 2. Hybrid architecture

In a hybrid infrastructure, Exchange Online relies on the following additional services to enable cross-premises mail flow, synchronization, and unified management.

  • Microsoft Federation Gateway   As an intermediary between Office 365 and on-premises services, the Microsoft Federation Gateway provides an identity service that connects users to the hosted services they want to use. For more information about the Microsoft Federation Gateway, see http://technet.microsoft.com/library/gg638824.aspx.
  • ADFS   To enable single sign-on and communicate with the Microsoft Federation Gateway, Microsoft IT relies on ADFS.
  • Microsoft Online Services Directory Synchronization tool To synchronize mailboxes, the global address list, and other data, Microsoft IT relies on the Directory Synchronization Tool.

Designing for Hybrid Messaging

The successful deployment of a hybrid messaging infrastructure for Microsoft IT requires that all the dependent on-premises services such as ADFS operate reliably and meet business needs. These services perform the intermediary data handling between on-premises and Exchange Online, that make account and messaging synchronization possible, as well as enabling workers to continue using the Outlook client, Outlook Web App, and mobile devices.

At a high level, Microsoft IT fulfilled the following requirements in the hybrid design:

  • Service domain to facilitate single domain namespace To forward e-mail from on-premises to Exchange Online, Microsoft IT configured a new DNS service domain for coexistence named messaging.microsoft.com. Upon sign up, new companies are automatically given a customizable coexistence domain with the format .mail.onmicrosoft.com.
  • On-premises federation through ADFS The ADFS infrastructure is the on-premises service that provides a trust relationship between on-premises and Exchange Online to make single sign-on possible.
  • Exchange federation through Microsoft Federation Gateway The Microsoft Federation Gateway is the trust broker that enabled Microsoft it to establish a federation trust from Exchange Online to the on-premises Exchange environment. This enables synchronization and sharing of Exchange information, such as free/busy data. For more information, see http://technet.microsoft.com/library/dd335047.aspx.

The Microsoft IT environment is specifically designed for Microsoft business needs, yet the technical requirements and steps for deploying a hybrid environment are the same for all companies. For a guided lab that shows the steps of configuring on-premises and Exchange Online components, see http://technet.microsoft.com/office365/hh744605.

Identity Management

To make the experience seamless for administrators and workers, the messaging environment must support a single authoritative source of user identity, with associated authentication, authorization, and permissions management. In a hybrid approach, the technical solution for a single authoritative source is to populate the Exchange Online directory with on-premises users, and then keep the two directories synchronized.

As shown in Figure 3, there are three technologies Microsoft IT uses for synchronization to take place:

  • ADFS 2.0 To communicate between the on-premises Active Directory environment and Exchange Online, Microsoft IT relied on the established ADFS infrastructure and created a relying party trust relationship between the ADFS federation server farm and Exchange Online. This relying party trust is a conduit for authentication tokens to facilitate single sign-on.
  • Microsoft Federation Gateway As an intermediary between Office 365 and on-premises services, Microsoft provides an identity service that connects users to the hosted services they want to use.
  • Directory synchronization tool Exchange Online begins using the on-premises identities the first time that the directory synchronization tool is run. The directory synchronization tool synchronizes key data every three hours, including mail-enabled contacts and groups, global address list (GAL), on-premises-based safe and blocked senders, and delegation details.

JJ218963.image004(en-us,TechNet.10).pngFigure 3. Identity, synchronization, and single sign-on technologies
Figure 3. Identity, synchronization, and single sign-on technologies

ADFS Architecture

The ADFS infrastructure at Microsoft supports single sign-on for over 300 line-of-business applications hosted on the cloud or by partners and vendors outside of the internal corporate network. ADFS handles claim requests to verify identities and returns tokens to the requesting party to enable applications to verify the identity of a user with Microsoft Active Directory credentials. ADFS relies on federation servers that authenticate users against Active Directory and issue claims, as well as federation proxy servers that reside in the perimeter network in front of the federation servers. Clustered SQL servers store configuration data, as shown in Figure 4.

Figure 4. DFS architecture
Figure 4. DFS architecture

To accommodate the additional traffic to the ADFS infrastructure due to Office 365, Microsoft IT more than doubled the size of its ADFS infrastructure. In July 2011, when mailbox migration first began to Office 365, Microsoft IT operated 12 proxy servers and 12 federation servers. As onboarding accelerated, Microsoft IT added more servers. In March 2012, after increasing server numbers, Microsoft IT operated 56 servers, divided evenly between proxy and federation roles.

The key metrics Microsoft IT uses to determine capacity planning come from the following product group recommendations shown in Table 1.

Table 1. ADFS performance metrics

Description

Threshold

Details

Authentication token requests/sec

Below 60

The number of client requests a proxy server handles per second during peak load times

CPU load

Below 50%

Average CPU load

The authentication requests per second is the major threshold. Microsoft IT tries to keep this at an even load of 10-12. During March 2012, Microsoft IT migrated over 14,000 mailboxes, with a plan to monitor performance of the existing ADFS environment, and then add additional capacity. Figure 5 shows an average of requests per second for March.

Figure 5. Average requests per second
Figure 5. Average requests per second

As Figure 5 shows, after deployment, the average requests decreased by half from about 20 per second to 14 per second. The number of auth requests per user depends on the location of the user when making the requests to Exchange Online. Microsoft IT modeled three types of users, as shown in Table 2 to understand projected server load and plan for ADFS capacity.

Table 2. User patterns for messaging-related ADFS load considerations

Type of user

Average daily auths

Peak daily auths

Min daily auths

Details

Light:

150

150-200

100-120

Low usage worker who accesses only from within the corporate network. Exits Outlook at end-of-day.

Moderate:

220

350-400

150-200

Even mix of access from within the corporate network and travel/mobile.

Power:

290

400-600

150-200

Majority of Microsoft users. Usage patterns have many more peaks due to travel, access from multiple devices. More remote access from home.

These models served as a starting point to determine how many more servers to add in order to accommodate the additional traffic expected from migrating mailboxes to Exchange Online.

For more information about designing and capacity planning for ADFS proxy and federation servers, see http://technet.microsoft.com/library/gg749899(v=WS.10).aspx and http://technet.microsoft.com/library/gg749917(v=ws.10).aspx.

Usage Patterns and Bandwidth

An important aspect for the hybrid design is modeling user patterns and user behaviors to understand how they affect the bandwidth requirements and user experience. The models Microsoft IT used for ADFS capacity planning do not necessarily address bandwidth and client experience needs related to messaging, calendaring and other Exchange traffic. To address these needs, Microsoft IT abstracted several user types, as shown in Table 3.

Table 3. Microsoft usage models for sizing considerations

Activity Per Day

Light

Medium

Heavy

Power User

Messages sent

5

10

20

30

Messages received

20

40

80

120

Average message size

50 KB

50 KB

50 KB

50 KB

Messages read

20

40

80

120

Messages deletes

10

20

40

60

Outlook Web App log offs/logons

2

2

2

2

E-mail Client

Outlook 2010

1,300 KB/ day

2,600 KB/ day

5,200 KB/ day

7,800 KB/ day

Outlook Web App

6,190 KB/ day

12,220 KB/ day

24,270 KB/ day

36,330 KB/ day

Microsoft IT's considerations for bandwidth requirements based on the user models followed established best practices of evaluating the connectivity at each gateway and monitoring performance. As migrations increase, Microsoft IT continues to monitor latency, jitter, collisions, utilization, and other network metrics to spot gateways and locations that need improvement. For more information about bandwidth planning, see http://blogs.technet.com/b/uspartner_ts2team/archive/2011/01/10/bpos-or-office-365-bandwidth-needs-determination-a-refresher.aspx.

One more performance consideration is the location of users relative to the Exchange Online data center, and the latency and bandwidth available between users and the data center. This is relevant both for the initial onboarding migration, due to the gigabytes of data transferred, as well as for ongoing needs, especially as Microsoft workers increasingly rely on mobile devices and work from home and while on the road.

Because Exchange Online relies on Internet infrastructure for mail traffic between office locations and the Exchange Online data center, performance and SLAs cannot be guaranteed. It is important to gather performance statistics from your environment. Two tools Microsoft IT uses for validating connectivity are https://www.testexchangeconnectivity.com/Default.aspx and http://speedtest.microsoftonline.com/.

Client Performance

Microsoft users are accustomed to high performance levels with messaging, expecting all message delivery to complete less than 90 seconds, maintain 99.99% or higher availability, as well as deliver fast e-mail operations to read and manage schedules and e-mail items. In an on-premises deployment, Microsoft IT controls the messaging infrastructure and its dependencies because all traffic flows internally within the corporate network, or between users accessing internal Exchange servers over the Internet. A hybrid deployment introduces additional variables that affect performance because users accessing Exchange Online from within the corporate production environment do so over the Internet, same as mobile and remote workers.

The differences among gateways, client devices, and connectivity in Microsoft locations mean that user experience at times may not be consistent among all sites. Microsoft IT looks at two factors when considering client performance: the MAPI RPC latency and the overall client system indicators, such as CPU, disk, and file fragmentation.

RPC latency includes round-trip latency to the mailbox server and server-side RPC processing. A helpful tool for determining these values is the connection status dialog accessible by holding down the CTRL key, right-clicking the Outlook icon, and selecting Connection Status from the Outlook context menu. Microsoft IT uses the following thresholds when analyzing latency:

  • Max Avg Proc Time (Exchange RPC Latency) = 25ms
  • Max Network RTT Time (Network Ping Time) = 300ms
  • Max Avg Resp Time (Exchange RPC Latency + Network Latency) = 325ms

For more information about client performance, see http://blogs.technet.com/b/neiljohn/archive/2012/01/23/outlook-performance-troubleshooting-including-office-365.aspx

Service Dependencies

At its core, Exchange Server has always and continues to deliver e-mail messaging and calendaring capabilities. Yet, Exchange Server 2010 integrates with other services and applications such as SharePoint, the Office suite, and Lync Server, both on-premises and through Exchange Online. This integration along with ADFS and directory synchronization helps to facilitate the following hybrid Exchange capabilities.

  • Delegate permissions for administrators To maintain the delegate permissions that administrators need to support managers and executives, Microsoft IT migrates manager and delegate mailboxes together. Delegate permissions do not persist in Exchange Online unless all affected mailboxes are migrated at the same time.
  • Free/busy sharing and synchronized calendaring As part of federated delegation, free/busy information is shared between on-premises and cloud-based users. After Microsoft IT establishes a trust through the Microsoft Federation Gateway and configures a sharing relationship between on-premises and Exchange Online, it is possible to share free/busy data. The user experience is transparent because the Outlook client communicates with the local CAS server, which requests a delegation token from the Microsoft Federation Gateway, impersonates the user, and makes free/busy requests on each user’s behalf.
  • Public folders Exchange Online does not support public folders. This is not an issue for Microsoft IT because users whose mailboxes are identified for migration do not rely on public folder functionality. For more information about public folder best practices in a hybrid deployment, see http://www.microsoft.com/download/en/details.aspx?id=27582.
  • Unified messaging Exchange Online supports unified messaging features for Exchange, including voicemail, automated attendant, Outlook Voice Access, speech-to-text voicemail preview in seven languages, and inline playback.
  • Outlook Web App redirection In the initial hybrid implementation, Microsoft IT created a landing page for users who access the on-premises Outlook Web App URL that directs users to the Exchange Online URL. If a user accesses from within the corporate network, only one login is required, whereas from the Internet, users see the need to authenticate twice. While working through the challenges, Microsoft IT collaborated with the Exchange Server product group to suggest improvements to streamline the experience. Exchange Server 2010 SP2 incorporates the latest changes with improvements to the Outlook Web App experience for hybrid deployments. For more information, see http://blogs.technet.com/b/exchange/archive/2011/12/12/owa-cross-site-silent-redirection-in-exchange-2010-sp2.aspx.

Mail Flow

Over the course of planning for and deploying the hybrid environment, Microsoft IT validated possible mail flow scenarios and developed best practices to streamline hybrid deployment for clients. Many of these configuration options are included in the Exchange Server Deployment Assistant and as improvements in Exchange Server 2010 SP2 on-premises.

The routing configuration in a hybrid deployment is relatively straightforward. It comes down to having on-premises or Exchange Online be the authoritative environment, and then relaying e-mails to the secondary environment. In a hybrid configuration, both the on-premises and the Exchange Online environment see each other as an internal, trusted environment. Figure 6 illustrates the configuration and mail flow.

Figure 6. Message flow overview
Figure 6. Message flow overview

To enable mail flow, Microsoft IT configured a dedicated send connector on Hub Transport servers secured by Transport Layer Security (TLS). That traffic traverses the Internet and enjoys the following protection measures:

  • Channel privacy   Exchange 2010 forces TLS encryption for all messages by requiring that a SAN or fully qualified domain name (FQDN) on the associated Secure Sockets Layer (SSL) certificate for the sending server is configured as authorized on the receiving server.
  • Receiver and sender authentication   To protect against impersonation, Exchange Server 2010 uses an encrypted auth header and domain validation, including validating the certificate of the receiving server against a revocation list with the certification authority (CA).

Exchange Server appends the auth header to messages to mark internal messages as trusted and authenticated, making messages and MailTips appear as internal in both Exchange Online and on-premises. The header works together with the certificates and send connector to ensure mail flow happens smoothly between Exchange Online and on-premises. Figure 7 illustrates the role of the auth header. Because Exchange Server appends the auth header to all internal communication, features such as OOF notifications and MailTips work seamlessly for users.

Figure 7. Auth header
Figure 7. Auth header

The auth header is relevant in the following mail flow scenarios for Microsoft IT:

  • E-mail flow between Exchange Online and on-premises   When an on-premises user sends an e-mail to a user whose mailbox resides in Exchange Online, the on-premises Hub Transport server verifies that the SAN or FQDN of the SSL certificate matches the configured value.  If the certificate subject is valid, then Exchange appends internal header to the e-mail and sends it to Exchange Online. The message bypasses the Edge server on premises. The reverse direction follows a similar path where the DNS and SSL configuration along with the send connector on the Hub Transport server enable encrypted mail to flow. The built-in features of Exchange Server give Microsoft IT the functionality needed to configure mail flow.
  • E-mails between Exchange and Internet hosts   For other e-mail communication to and from Internet hosts, Exchange Online and on-premises use the standard Simple Mail Transfer Protocol (SMTP) mail flow as detailed in http://technet.microsoft.com/en-us/library/ff645372.aspx.

Forefront Online Protection for Exchange (FOPE)

In the classic on-premises architecture, an Edge server running in a perimeter network provides initial mail filtering for anti-virus and antispam protection as well as SMTP relaying. For Exchange Online, FOPE provides a similar service. FOPE includes high accuracy SPAM filtering with over 98% of SPAM filtered, and 100% of viruses filtered by using multiple virus-scanning engines. FOPE also gives Microsoft IT a control center for advanced policy rules and reporting. Although it is possible to use an Edge server on-premises for mail filtering and SMTP relay in a hybrid architecture, Microsoft IT uses FOPE. The first contact point of handling e-mail messages is very important in the overall architecture, especially in the dependencies required when not using FOPE. The Exchange Deployment Assistant addresses this importance in the guidance it provides and accommodates both scenarios for initial mail handling. For more information about FOPE, see http://www.microsoft.com/exchange/en-us/forefront-online-protection-for-exchange.aspx.

Migrating Mailboxes to Exchange Online

The exact process that Microsoft IT followed in migrating to Exchange Online entailed many months of planning, validating scenarios, and working to improve the client and user experience. Because its mission includes real-world validation and early adoption of Microsoft technologies, the deployment process did not follow a more typical path. These efforts support customer needs. For example, before the changes introduced in on-premises Exchange Server 2010 SP2, the configuration requirements entailed over 50 distinct steps, which SP2 reduces to just six. As Microsoft IT migrates more mailboxes to Exchange Online, the migration velocity can increase from 5,000 to 15,000 mailboxes per month.

At a high level, the deployment entailed making the following changes:

  • Configure single sign-on As a recommended prerequisite to a hybrid Exchange deployment, the on-premises credentials and user data should be used to authenticate with Exchange Online. Microsoft IT already operated an ADFS infrastructure, and configured it to support Exchange Online. On the Exchange Online side, after signing up for Exchange Online and verifying domain ownership, Microsoft IT configured the Microsoft Federation Gateway to work with its ADFS infrastructure through a trust relationship.
  • Synchronize directories and data In order to onboard user mailboxes, users must exist in Exchange Online. Microsoft IT configures directory synchronization to populate Exchange Online with users from the Active Directory environment.
  • Configure DNS and certificates Exchange relies on DNS entries for autodiscover, which is necessary for a seamless online migration with no user interruptions. After migration, Outlook uses autodiscover to detect the mailbox move, and upon restart uses the Exchange Online service. Microsoft IT configured the MX records to point to FOPE.
  • Deploy/configure necessary on-premises Exchange dependencies To enable the full range of Exchange features and services, such as mailbox search, Outlook Web App redirection, MailTips, free/busy sharing, message tracking, and archiving, Microsoft IT made the necessary on-premises configuration updates to work with Exchange Online. The Exchange Hybrid Configuration Wizard in Exchange 2010 SP2 automates many of the configuration steps.
  • Verify mail flow The auth header is crucial to bypass filters and mark internal messages as originating from trusted sources. Microsoft IT configured and verified mail flow between Exchange Online and on-premises, as well as Internet hosts.

For deployment steps and instructions to deploy a hybrid environment, the best practice is to use the Exchange Deployment Assistant, which includes the latest steps. To access the Exchange Deployment Assistant, see http://technet.microsoft.com/exdeploy2010/default.aspx.

Migration Approach and Process

One of the advantages of a hybrid infrastructure is that it enables Microsoft IT to move mailboxes to and from Exchange Online without affecting availability, performance, or the user experience. The same core messaging and calendaring functionality remains available to users during the move without service interruption. The migrations are made as online mailbox moves, so users do not need to synchronize data after migration. In practical terms, this means Microsoft IT may schedule mailbox moves at any time if all the dependencies and prerequisites are met for preparing and configuring settings, as well as informing users. The overall process is as follows:

  • Test usage scenarios. Before Microsoft IT migrates mailboxes, it performs end-to-end system testing that includes all possible usage scenarios. This testing helps to discover and remedy system configuration and integration issues. Although the engineering staff audits configurations, sometimes real-world issues arise, especially with new changes. Thorough testing also enables Microsoft IT to better understand the environment and build a decision matrix to identify the users who can move their mailboxes to Exchange Online.
  • Scope mailbox migration Microsoft IT creates a list of potential users to be moved, gathers statistics about their mailboxes, and makes decisions about the migration order based on a decision matrix. This decision matrix depends on the business and IT needs. For example, Microsoft IT made the decision to simplify infrastructure and operational support by adopting the default configuration and reduce customization as much as possible. This may mean not introducing some features and functionalities. One of the examples is not migrating any mailboxes who are using a legacy telephone system and only migrating mailboxes with Lync 2010 Enterprise Voice to Exchange Online. This decision saves Microsoft IT third-party gateway costs and associated support overhead. It also simplifies the Exchange Unified Messaging configuration, and enables Microsoft IT to focus its efforts in driving Lync 2010 Enterprise Voice as the default telephony and collaboration platform. Microsoft IT is working to transition majority of mailboxes to Exchange Online to reduce costs and still offer users the best experience.
  • Verify configuration This includes ensuring that Exchange Online is prepared with the appropriate objects, directory synchronization functions, and mail flows between Exchange on-premises and Exchange Online. This step also serves as a safeguard to verify that there are no schedules service windows or current outages with dependent services.
  • Update user computer To ensure that users have the latest Outlook client version and required software such as Microsoft Online Services Sign-in Assistant, Microsoft IT uses System Center Configuration Manager (SCCM) to package the required software and deploy it on user computers.
  • Migrate mailboxes After notifying users of the migration schedule, Microsoft IT migrates mailboxes and sends notices upon successful completion.

For more information about determining how many mailboxes to migrate, the anticipated migration timeframe, and other migration performance details, see the migration performance guide at http://community.office365.com/en-us/b/office_365_technical_blog/archive/2012/03/29/new-exchange-online-migration-performance-guide.aspx.

Phases

The rate at which Microsoft IT migrates mailboxes is closely tied to the rate that improvements and change requests from previous phases are implemented as features. Between the phases, Microsoft IT allowed for a period of one to two weeks to implement changes and constantly improve the user experience and migration process. The phases were as follows:

  • Phase 1: Environmental validation The purpose of this phase is to discover and fix any system configuration errors and integration issues by creating test accounts and performing usage scenarios.
  • Phase 2: Early adopter validation The early adopter volunteers troubleshoot, gather logs, and provide constructive feedback to the project teams. In this phase, Microsoft IT migrates 10 to 20 mailboxes per week, stopping at approximately 100 mailboxes.
  • Phase 3: Expanded early adoption During the expanded early adoption phase, Microsoft IT migrated the accounts of 1,000 additional volunteers who are eager to explore new options in technology. The migration proceeded in phases, stopping when major issues are discovered and resuming upon resolution.
  • Phase 4: Executive opt-in To stress-test the approaches developed, Microsoft IT reached out to executives to migrate entire teams and reach the number of mailboxes necessary to perform larger scale performance testing. In this phase, Microsoft IT also introduced a stabilization period of 21 days where no changes are made, and statistics gathered to gauge availability and stability.
  • Phase 5: Company-wide signup Following team migration, Microsoft IT opened up signups to volunteers company-wide, having resolved any underlying high-severity issues.
  • Phase 6: Company-wide adoption Once the hybrid infrastructure meets the shared goals of Microsoft IT, product developers, and other infrastructure team members, Microsoft IT plans to migrate all mailboxes to Exchange Online, unless there is a business need to remain on-premises.

Supporting Users

During the transition to a hybrid infrastructure, Microsoft IT minimizes support tickets by informing users and designing architecture with the goal of least user impact. The typical process for any Microsoft IT improvement project includes a focus on user education. This entails a broad, multimedia approach of making help available to users on their own terms, including the following:

  • Online help Microsoft IT developed online help to answer frequently asked questions, provide user self-help capabilities, and inform users about working with Exchange Online by suggesting best practices.
  • E-mails detailing project schedule and status As a best practice, Microsoft IT informs users personally when a scheduled task affects them, and follows up after completing the task with status details.
  • Updated knowledge for front-line operators The support and escalation path remains the same for users due to the centralized controls that a hybrid infrastructure offers. However, as part of preparing for mailbox migration, Microsoft IT collects incident details and transfers the resolution specifics to internal front-line operators as well as the support team for Exchange Online to aid in issue resolution. To help facilitate this knowledge sharing, Microsoft IT established a supportability team to do deep analysis of each ticket and identify trends in order to support and prioritize change requests made to the Exchange product group.
  • Validation team Due to the need to validate many possible customer scenarios and features for all the scenarios, Microsoft IT created a dedicated validation team. This team has oversight to validate possible customer configurations, record findings, recommend improvements, and create best practices. Exchange Server 2010 SP2 on-premises incorporates some of the findings of this team as product improvements to simplify customer hybrid deployments. This team also validates features and functionality for Microsoft users to ensure a smooth transition process.
  • Feedback loop When Microsoft IT migrated the earliest mailboxes, this was done with the intention to obtain migration and usability feedback. The early volunteer users relied on a feedback portal to give real-time feedback as a smile, frown, improvement idea, or issue. This feedback loop complemented the one-week and one-month post-migration survey users filled out to help Microsoft IT gauge overall user experience such as migration experience and usage performance. This helped Microsoft IT to identify improvement areas for infrastructure, configuration and product design changes.
  • Self-help tool. Microsoft IT treats both on-premises and Exchange Online as a single service, and the helpdesk supports both groups of users. It is important to be able to identify the environment that hosts the mailbox, therefore Microsoft IT created a Web portal that provides information about the mailbox location, Outlook Web App link, ActiveSync, and other information pertaining to that user.

Lessons Learned and Best Practices

Over the course of designing, deploying, and operating a hybrid Exchange infrastructure, Microsoft IT learned many lessons for the best approaches in running a hybrid environment. Although some of these are applicable to the Microsoft production environment specifically, the following best practices apply to hybrid Exchange deployments in general:

  • Use available migration tools and wizards Many of the findings that engineers, architects, and implementers made are implemented in the configuration wizard and supporting tools that Microsoft makes available to anyone using Office 365. Whenever an easier solution of configuration step may be automated or implemented as a product change, Microsoft IT worked to transfer their knowledge into a standard for all customers.
  • Focus on core architectural elements At first glance, a hybrid infrastructure takes a potentially complex Exchange architecture and topology, and introduces additional configuration requirements, and management overhead. Once the underlying dependencies, such as ADFS, Internet ingress and egress, and network latency are established and configured with adequate performance, Microsoft IT found that a hybrid deployment still maintains centralized administration, and introduces little architectural complexity while preserving a unified user experience.
  • Adopt a services-based perspective With Exchange Online, every aspect (SPAM/virus protection, Microsoft Federation Gateway, messaging, and so on) is provided as a service and not as a feature or component. It is helpful in understanding a hybrid architecture to consider some aspects of on-premises as service counterparts, to abstract the architectural elements and understand their dependencies and relationships. For example, as a counterpart to the Office 365 directory, there is Active Directory. In any overlap that happens between services, it is important to remember that there must be a way to achieve a single, synchronized version that is transparent to the user. Enabling technologies such as ADFS and the Microsoft Federation Gateway facilitate this seamless integration.
  • Audit common sources of misconfiguration Microsoft IT investigates upstream and downstream possible causes to isolate root causes and remedy them. When troubleshooting typical on-premises components, there is not always a corresponding cloud counterpart, which makes it challenging to do direct comparisons and remedy issues. Microsoft IT proactively audits the most common possible issues as a preventative measure to reduce the troubleshooting necessary. One useful tool already mentioned to help with common troubleshooting and auditing tasks is the remote connectivity analyzer located at https://www.testexchangeconnectivity.com.
  • Identify send as relationships Whereas users may specify delegate rights, administrators assign send as rights to grant someone control over a specific mailbox. These send as rights do not synchronize automatically as you migrate mailboxes. Microsoft IT determines on-premises send as permissions through PowerShell before migrating mailboxes, and uses PowerShell scripting to apply the same permissions after migrating mailboxes to Exchange Online,
  • Engage infrastructure team early Mailbox migration to Exchange Online results in e-mail traffic traversing the Internet across provider backbone routers instead of internal WAN networks and internal routers. This change may require increasing capacity and sizing of the Internet proxy egress infrastructure, ADFS, bandwidth, gateway IP configuration if doing Network Address Translation (NAT), Exchange on-premises hub transport configuration, and so on. It is important to engage the various teams responsible for all these services early and carry out capacity re-engineering to ensure project success. For example, Microsoft IT discovered that a cause of intermittent client connectivity to be caused by a flood threshold of the proxy array, and quickly reached out to the associated team to engineer and implement a new solution.
  • Support the support department With a new service, support personnel must be trained on possible issues, and how to isolate and troubleshoot root causes. Having tools that identify mailboxes as on-premises or in the cloud helps when isolating root causes.
  • Practice change management. With new technology adoption, users generally want to start using the new and exciting features. Yet with messaging, there is a high expectation that the service needs to be reliable with high service availability, which may not be possible at very early deployment stages. Microsoft IT mitigates this by ensuring users have all possible collaboration tools so that when one service is not available, workers may continue to carry out their tasks. For example, when e-mail service is unavailable, users can continue to collaborate with colleagues through Lync 2010 via instant messaging or voice call. They may also work on documents via SharePoint or send documents via Lync. At Microsoft, many early innovators and adopters are very keen to be early adopters because service outages do not severely affect their ability to work. After Microsoft IT achieves stability with a new service, it migrates the rest of the company. This methodology satisfies all user needs, creates high satisfaction, gives Microsoft IT the ability to support the developers in testing, and create a better product.
  • Communicate with users Active communication to users via Web portal, newsletter and e-mails keep users excited about the program and informs them about new features or issues. Microsoft IT rewards and recognizes users who provides the most constructive feedback and support, which maintains user motivation and commitment to dogfooding additional products and services.
  • Audit gateway configuration Microsoft IT audited two configuration details for gateways: flood thresholds for TMG gateways, and Outlook client port exhaustion when using NAT. For more information about TMG configuration, see http://community.office365.com/en-us/w/exchange/office-365-move-mailbox-fails-with-transient-exception.aspx.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

 The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

 Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

 Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

 Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2012 Microsoft Corporation. All rights reserved.

 Microsoft, Active Directory, Forefront, Lync, Outlook, SharePoint, SQL Server, Windows, Windows PowerShell, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

All other trademarks are property of their respective owners.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft