Export (0) Print
Expand All
2 out of 2 rated this helpful - Rate this topic

IT Showcase On: Microsoft Exchange Server 2010 Deployment

Quick Reference Guide

Microsoft IT Uses Low-Cost Storage to Boost Mailbox Size and Significantly Reduce Costs

Download

Download Quick Reference Guide, 163 KB, Microsoft Word file

Executive Overview

Situation: The relationship between Microsoft IT (MSIT) and the Exchange product group has strengthened with each release of Exchange. For the Exchange Server 2010 deployment, MSIT deployed the first build of Exchange 2010 in their Exchange Dogfood forest and deployed Exchange 2010 to the Corporate forest in the third milestone phase. MSIT upgraded all users to Exchange 2010 before RTM. During the deployment phase, MSIT found and fixed 2000+ unique bugs, greatly improving the product for Microsoft's customers.

Why You Should Care:

  1. MSIT improved user productivity by increasing mailbox quota limits from 1 and 2 GB to 5 GB. At the same time, MSIT adopted a thin-provisioning model to ensure cost-effective use of hardware resources. MSIT also improved productivity by using online mailbox moves (no disruption to users) to increase migration velocity.
  2. MSIT reduced overall architectural costs by continuing the investment in direct-attached, low-cost storage with SAS/SATA technology to drive up capacity utilization to over 60% and took advantage of lower I/Os in Exchange 2010 to balance capacity and efficiency.
  3. MSIT maintained the same level of service availability (99.99%) since Exchange 2003 (includes scheduled and unscheduled outages) and improved on it with database-level failover.
  4. MSIT saved $6M in a single year by leveraging multiple database copies and Single Item Recovery to do away with backups.
  5. MSIT utilized database-level failover (as opposed to server-level failover) to minimize disruptions due to issues with single databases.

Pre-Consolidation Environment Vs. Today

In the pre-consolidation environment (2000/2003) timeframe, MSIT had a significant distributed environment with 74 sites that contained Exchange Servers. The following table contrasts that environment to today's environment that has just four sites with Exchange Servers. Most of the consolidation happened in the 2007 timeframe.

Number of

2000/2003

Today

PCs

150,000

270,000

Mailboxes

90,000

180,000

Public folders

230,000

30,000

Distribution lists (DLs)

85,000

386,000

Sites with Exchange servers

74

4

Exchange servers

215

258

Mailbox servers

118

77

Forests

4

5

MSIT Exchange 2010 Environment Highlights

Forests. MSIT operates five forests with Exchange in them: Corporate, Dogfood, Windeploy, WinSE, and Extranet.

User Profiles. Most full-time employees receive 150 to 200 messages per day. Contingent staff receive 50 to 75 messages per day. MSIT is continuing with a 10-MB message limit, but expects to increase to 20 MB or higher to support increased use of attachments.

Mobile Usage. Mobile usage has increased significantly with each Exchange release. Today, MSIT has about 90,000 unique Outlook® Web App (OWA) users, 60,000 unique Microsoft Exchange ActiveSync® users, and 80,000 Outlook Anywhere users per month.

Internet Messages. Internet messages have steadily increased from 6 million messages per day to around 25 million per day prior to filtering for spam or viruses.

MSIT Exchange 2010 Hardware Configurations

  • At the time of initial deployment during the RTM timeframe, all hardware configurations were 2x4 cores with 16 GB of memory except Sao Paulo. Sao Paulo uses 4 cores because it has only 2000 users. For the large Database Availability Groups (DAGs), MSIT increased the memory to 32 GB to help reduce the I/O footprint. Recent capacity expansions are based on newer hardware platforms.
  • MSIT still has a dedicated server role for each of the core servers: Hub Transport, Client Access, Unified Messaging (UM), and Mailbox and has not invested heavily in the multi-role platform, but expects to increase investment in that area.
  • MSIT designed each server role to take full advantage of the hardware platform, so MSIT does not invest in virtualization technology for its messaging deployment.

The following table shows the hardware configurations at the time of initial deployment.

Role

Configuration

Hub Transport

2x4 cores, 16 GB RAM

Client Access

2x4 cores, 16 GB RAM

Unified Messaging (UM)

2x4 cores, 16 GB RAM

3-node DAG (Redmond)

2x4 cores, 16 GB RAM

4-node DAG (Sao Paulo)

1x4 cores, 16 GB RAM

Large DAGs (Redmond, Dublin, Singapore)

2x4 cores, 32 GB RAM

Multi-Role (Hub Transport, UM, Client Access)

2x4 cores, 16 GB RAM

Load Balancing And Fault Tolerance

  • The Mailbox server is a fault-tolerant solution through High Availability (HA) using DAGs.
  • Hub Transport servers leverage their own internal load-balancing logic.
  • With Exchange 2007, MSIT moved to having Microsoft Exchange Hosted Services (EHS) be the ingress point in delivering to Edge Transport servers. With Exchange 2010, MSIT is removing Edge Transport servers completely and leveraging EHS and Forefront® Online Protection for Exchange (FOPE) to provide anti-virus and anti-spam. EHS is also the egress point now.
  • MSIT also moved away from Windows Network Load Balancing (NLB) to hardware-based load balancing (HLB) to provide resiliency against server and service failures and to provide scalability.
  • Current SLA is to deliver 99% of messages within 90 seconds.

Mailbox Architecture Dag Deployment Numbers

  • MSIT fully embraced the Exchange 2010 DAG architecture in all regions, which enables much lower-cost storage design.
  • At the time of the Exchange 2010 deployment, MSIT deployed 16-node DAGs in Dublin and Singapore and three small DAGs in Redmond. Since then, the capacity has been expanded to accommodate increased use of messaging services and growth of mailbox size.

The following table shows the initial RTM deployment.

DAG Model

Location

# DAGs

# DBs per DAG

# DB Copies per DAG

Total DB Copies per DAG

3-node

Redmond

1

20

3

60

3-node

Redmond

1

21

3

63

4-node

Sao Paulo

1

16

3

48

11-node

Redmond

2

96

4

384

10-node

Redmond

1

87

4

348

16-node

Dublin & Singapore

2

128

4

512

MSIT Approach For Exchange 2010 Backup

  • MSIT leverages multiple database copies for fast recovery from hardware or software failures and as the data redundancy model for resiliency.
  • MSIT can recover data through the dumpster via Single Item Recovery (data maintained for 30 days). The increase in quota limits provides long-term retention. MSIT is moving away from personal folder files (PSTs), which are a potential litigation and management issue.
  • MSIT hasn't utilized lagged database copies since Single Item Recovery was implemented.
  • MSIT has not deployed site-resilient copies, but is evaluating Dublin for site resiliency.

Mailbox Storage Provisioning

  • For MSIT, Exchange 2010 enabled a reduction in disk I/O requirements from .8 I/Os per user to .33 I/Os. This enabled MSIT to put more users on lower-cost drives.
  • The initial allocation of 3 GB per user (2 GB plus 1 GB for overhead) for 5 GB quotas results in 300 mailboxes per database vs. 85 mailboxes in Exchange 2007.
  • MSIT targets 60% storage capacity utilization before expanding capacity of its mailbox servers. I/Os are currently pushing up to 100 per disk.
  • The end result is that MSIT is deploying much larger mailboxes and making users happy with 5-GB quotas, which shows how administrators can do large, low-cost mailboxes and maintain high density per database.

Client Access

  • MSIT uses a common URL namespace per region for all messaging mobility scenarios.
  • In the Corporate forest there is a CAS array per AD site with Exchange services, which means one CAS array for each of the four key data centers.
  • The Fully Qualified Domain Name (FQDN) of the internal RPC end point in each CAS array leverages the AD DNS zone. All of the external client access records using HTTPS leverage the Microsoft.com DNS zone.
  • OWA and Exchange Control Panel (ECP) use cookie-based affinity at the load balancer level.
  • Other Internet protocols, like Exchange ActiveSync, Outlook Anywhere, and Exchange Web Services (EWS), use protocol-specific affinity for load-balancing.

Mailbox Migration Velocity

Online mailbox moves in Exchange 2010 enabled MSIT to greatly increase mailbox migration velocity compared to previous versions of Exchange. The following table shows the number of mailbox moves at different milestones.

Early Beta

Beta 2

At Peak

200–300 per night

1500–2000 per night

Over 4000 per day

Data transfer rates: single mailbox, 5–7 GB/hour; multiple mailboxes, 20 GB/hour (default Message Replication Services throttling policy)

Key Learnings

  • Train users ahead of time on productivity features. Users are very excited about 5-GB mailbox quotas and no downtime for online mailbox moves.
  • MSIT did a lot of work with Role-Based Access Controls to make sure that IT Admins had appropriate levels of access control.
  • Use PowerShell® scripting to automate the upgrade process and create a consistent server environment. MSIT also scripts the LUN creation and database copy addition process.
  • MSIT is currently using dedicated server roles, but Nehalem processors provide significant improvements in scalability. Multi-role deployments can leverage this to reduce the physical server architecture.
  • Having large mailboxes leads to a thin-provisioning model. But operational maturity is key to thin provisioning since administrators need to be able to adapt and react quickly.
  • Internet mail management on premises can be a huge undertaking. MSIT moved this task to the cloud. EHS and FOPE take care of anti-virus and anti-spam.
  • Customers on Exchange 2007 who are using old backup software will have to ensure that their backup infrastructure utilizes VSS-based snapshots.
  • WebDav has been deprecated in Exchange 2010, so giving application owners sufficient lead time to transition their applications is key. MSIT moved many applications to EWS to take advantage of this API.

Education Resources

Did you find this helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft. All rights reserved.