How Microsoft IT Reduced Operating Expenses Using Virtualization

The following content may no longer reflect Microsoft’s current position or infrastructure. This content should be viewed as reference documentation only, to inform IT business decisions within your own company or organization.

Learn how Microsoft utilizes virtualization in their data centers to reduce costs associated with deployment, management, and operations while at the same time improving client satisfaction and service levels.

Download

DownloadBusiness Case Study, 180 KB, Microsoft Word file

Customer Profile Situation Solution Benefits
As the worldwide leader in software for business and personal computing, the vision of Microsoft Corporation is to enable people and businesses throughout the world to realize their full potential. Employing more than 98,000 people, Microsoft had revenues exceeding $58.44 billion for the year ending June 2009. Ever increasing server deployments were causing Microsoft data centers to reach capacity. Servers deployed were underutilized, deployment times were increasing, and support costs were rising. Microsoft IT adopted a very aggressive approach for implementing virtualization on the Windows Server 2008 operating system with Hyper-V technology, changing the way Microsoft IT deploys, manages, and supports its current IT environment.
  • Virtual servers cost approximately 35 to 40 percent less per month to support than physical server.
  • End-of-life servers replaced with virtual machines deployed on blade servers can reduce data center space requirements by 98.4 percent and power requirements by 94 percent.
  • Virtualization increases operational efficiency, reducing build times, patching of servers, and errors.
  • Virtualization increases customer satisfaction by providing better availability and agility.
  • Virtualization could potentially save Microsoft $16M in all-up operating costs.

Overview

There are many articles and papers published about the benefits of virtualization and how virtualization can save you money. Most articles and papers look at the technology behind virtualization and do not look at how virtualization can improve operational efficiencies and reduce operational costs. Microsoft Information Technology (Microsoft IT) has adopted a very aggressive approach for implementing virtualization on the Windows Server® 2008 operating system with Hyper-V™ technology. This changed the way Microsoft IT deploys, manages, and supports its current IT environment.

Situation

Microsoft IT is responsible for providing infrastructure and managed services to internal groups within Microsoft. The early deployment of technology and continual growth at Microsoft result in a highly dynamic IT environment. The environment that supports internal Microsoft groups houses more than 21,000 servers that provide essential services, including line-of-business applications, email servers, Microsoft SharePoint® servers, IIS Web servers, servers running SQL Server®, and Microsoft IT–supported lab servers.

Like most large enterprises, the number of servers that Microsoft deploys in its primary data centers has grown rapidly, while the utilization of many of those servers has remained low in relation to the hardware capabilities. As early as September 2005, Microsoft IT calculated that the average CPU utilization for servers in data centers and managed lab environments was less than 10 percent, and continuing to decrease.

Microsoft IT also did not have a consistent capacity-planning mechanism in place within business groups. In 2005, approximately 40 business groups could purchase servers and place them in corporate data centers. Many of these business groups did not have a consistent planning process for purchasing hardware. Additionally, each business unit operated autonomously, with no planning for hardware capacity across business groups. The constant demand for servers was increasing deployment times and support costs.

Microsoft IT recognized that the uncontrolled and constant growth in physical server deployments was causing data center resources to be overused and sometimes depleted, especially in the area of physical space and power consumption. They needed to find an approach that would allow them to reduce physical space and power requirements, improve efficiency and optimize operational tasks, increase customer satisfaction, and reduce IT operating expenses.

Solution

Microsoft IT identified server virtualization as a prime strategy to address server underutilization and data center capacity. Their approach to data center virtualization was straightforward:

  • Deploy high-density racks of blades (64 blade servers per rack).

  • Deploy virtual machines rather than physical computers to decrease the total number of servers in the data centers.

  • Deploy multiple virtual machines (each replacing a physical server of two to four rack units [RUs]) on one blade server to decrease the space used in the data center.

  • Deploy an infrastructure that enables rapid movement of virtual components across hardware platforms, rapid provisioning of new servers and applications, and automated, policy-based management.

Utility Services Strategy

Microsoft IT developed an IT utility services strategy to deliver IT services to internal customers. This approach is based on the idea of creating large, shared, centralized services so that Microsoft IT customers do not have to worry about adding infrastructure capacity.

Two internal programs, the IT Cloud (originally called Compute Utility) strategy and the RightSizing initiative, were the first attempts at encouraging business groups at Microsoft to adopt virtualization. The IT Cloud strategy aims at changing the way business groups think about computing services. The goal of the RightSizing initiative is to identify the servers that are good virtualization candidates and encourage business groups to move forward with replacing those physical servers with virtual machines.

IT Cloud Strategy

Microsoft IT implemented the IT Cloud strategy to remove the concept of server ownership for business groups and replace it with a concept of purchasing computing capacity. With this strategy, Microsoft business groups define their computing capacity requirements for the applications that they need to run their business, and then Microsoft IT focuses on meeting the computing capacity requirements.

If a physical server is the only way that a group can meet business requirements, Microsoft IT provides one. However, in most cases, a virtual machine more than meets business requirements. The IT Cloud strategy looks to create a level of abstraction for the business group, which now purchases computing capacity and space on a storage area network (SAN) rather than a physical server.

RightSizing Initiative

The RightSizing initiative is one component of a broader IT Cloud strategy and has two goals. The first goal is to identify servers that would make good candidates for virtualization, and to encourage business groups to replace those physical servers with virtual machines. The second goal is to ensure that if a physical or virtual server is necessary to meet business group requirements, it is sized appropriately.

The RightSizing initiative collects information on all of the physical servers running in the Microsoft data centers and identifies those servers that are virtualization candidates. The initiative has a scorecard system for tracking and promoting server virtualization.

The IT Cloud strategy and RightSizing initiative have promoted the virtualization concept at Microsoft. Additionally, upper management has taken a very aggressive approach toward virtualization, essentially directing that all server deployments use virtual machines and that physical servers are deployed only for exceptional circumstances.

Microsoft IT Cloud Platform

The IT Cloud group defines the hardware standards and implements all virtual machines running in the Microsoft data centers and IT-managed labs.

Microsoft IT finances its services by charging business groups for providing services. An additional benefit of a virtual machine is that the business groups do not pay for the cost of the underlying physical machine; the monthly hosting fee includes that cost. As a result, the business group does not need to worry about replacing the physical hardware when purchasing a virtual machine, which increases savings significantly over the server hardware’s life. In turn, with multiple virtual machines per physical host, Microsoft IT sees an immediate recovery of server costs and is able to apply hosting revenue to fund future hardware refreshes.

The platform that Microsoft IT is using in their data centers is based on Scale Units (SUs). A SU is a high-density pool of compute, storage, and network resources that can be deployed in bundles that allow both extensibility and reuse/reallocation. An SU consists of:

  • Compute – Blade servers, deployed by one or more racks at a time

  • Storage – Enterprise SAN, with disk capacity to match compute

  • Network – New access and distribution designs

SUs enable Microsoft IT to utilize the same compute, storage, and networking elements across Microsoft IT data center environments and customer sets. Microsoft IT procures new capacity to support customer requirements.

Evolution of Virtualization at Microsoft

Microsoft IT has been working to improve utilization through server virtualization since 2004, starting with the Microsoft Virtual Server 2005 product. Microsoft IT determined that 30 percent of Microsoft IT servers could be virtualized. The proof of concept started in 2004, which led to the Virtual Server Utility (now IT Cloud) in 2005. Virtual machines were positioned as a lower-cost alternative to two-socket physical machines. The virtual machines were deployed on commodity rack mount servers. At its peak, approximately 3,500 virtual machines were deployed.

The Windows Server 2008 operating system and Hyper-V technology provided the next generation of virtualization. This next milestone saw approximately 3,400 virtual machines and approximately 560 hosts deployed mostly on stand-alone hardware.

Today, the Microsoft IT data center space requirements have stopped growing. In March 2010, Microsoft IT was at 34 percent virtualization with 1,554 total hosts supporting 7,224 virtual machines.

Microsoft is targeting 50 percent virtualization in calendar year 2010 and greater than 80 percent in the 2011–2012 timeframe. The customer on-boarding process steers appropriate candidates into virtual machines for all new growth and hardware refresh/migration requirements.

As Microsoft moves towards Windows Server 2008 R2 Hyper-V, clustering will provide for live migration for host maintenance and capacity management, further increasing the availability and agility of the virtual platform. Density of virtual machines to blades will also increase. With Windows Server 2008 Hyper-V, the average virtual machine per blade ratio was 6:1 on production servers and an average of 12:1 on lab servers. With Windows Server 2008 R2 Hyper-V, three offerings will be available, small/medium/large (8:1, 16:1, and 32:1 virtual machines per blade respectively), with the average blade hosting 16 virtual machines.

Benefits

Microsoft IT uses virtualization technology in various ways. In the data centers, Microsoft IT uses virtualization to consolidate infrastructure, applications, and server workloads. Windows Server 2008, Windows Server 2008 R2, and Hyper-V technology have provided a software platform to make the consolidation efforts possible. The move to virtualization has helped Microsoft IT reduce their operating cost through a variety of means:

  • Infrastructure consolidation provides a reduction in number of servers resulting in a reduction in space and power requirements.

  • Virtualization and new blade server technology has reduced the physical cable needs as well as labor for cabling the data center.

  • Virtualization has reduced the labor required to deploy and support servers.

Data center virtualization has the potential to achieve at least a 50 percent reduction in Microsoft IT's hard costs for data center space and power. It also offers the advantage of increased flexibility and agility, making it a winning scenario Microsoft IT and Microsoft business groups. Data center virtualization enables Microsoft IT to reduce costs while improving services.

Infrastructure Consolidation

Virtualization efforts have had a significant impact on controlling the growth and the cost of running the data centers at Microsoft. Server technology improvements have provided a more robust platform to host multiple virtual machines while the size of servers continues to shrink — providing similar or greater processing power at a much-reduced footprint.

Data center utilization and end-of-life (EOL) hardware initiatives across all Microsoft IT production data centers is designed to reduce overall hardware footprint and costs. EOL servers are replaced with new blade servers or virtual machines providing a much-reduced footprint and a significant reduction in power costs.

The average EOL server takes up 2.5 RUs, but can range between 2–7 RUs. The power required by these servers is approximately 330 Volt Amps (VA) (660 VA with cooling factored in). The new blade servers take 0.67 RU and run at 313 VA (626 VA with cooling factored in). With virtualization, Microsoft IT estimates being able to take 16 of the 2.5 RU servers and hosting them on a single blade with the next generation of IT Cloud services and the advances in Windows Server 2008 R2 Hyper-V. This provides:

  • An estimated space savings of 98.4 percent.

  • An estimated power savings of 94 percent.

Blade technology enables Microsoft IT to consolidate network ports, requiring approximately 90 percent fewer cables. It also provides flexibility, as a blade can be a physical system or used as a virtual server host, and can be changed more quickly than classic horizontal chassis systems.

The virtualization platform enables Microsoft IT to respond more quickly to changing business needs without expanding the overall footprint and power use in the data center. The following two virtualization projects provide examples of how power and space are being utilized in two Microsoft IT data centers.

Table 1 documents one of the Microsoft IT data center virtualization projects. The left side of the table shows the planned capacity for the data center, while the right side of the table reflects the current state of servers migrated to the new platform. The virtualized environment provides high-power utilization efficiency. At approximately 66 percent complete on server migrations, the server to power ratio is 2.1:1; once the data center is deployed to capacity, those numbers increase to 3.1:1.

Table 1. Data Center Migration Project

Planned Capacity Buildout

Current Migration State

# of Racks

Rack Footprint

Maximum # of Servers*

Power Utilization (kW)

Migrated Servers

Power Utilization (kW) for Migrated Servers

1 storage array

12

15

51.4

51.4

Virtual machines

2

2

640

33.2

350

33.2

Discrete blades

1

1

64

16.6

24

6.2

Physical servers

14

14

280

224

278

224

Subtotal

29

32

984

325.2

652

314.8

*Assumes an average of five virtual machines per blade.

Table 2 and table 3 document another Microsoft IT data center virtualization project. Table 2 is the planned capacity for the data center, while table 3 reflects existing servers running Enterprise Business Continuity and Disaster Recovery applications that will be migrated to the new platform.

Again, the virtualized environment provides high-power utilization efficiency. The existing environment has a server-to-power ratio of 1.8:1, whereas the planned environment will have a server-to-power ratio of 2.5:1 including the four storage arrays.

The overall footprint has increased by approximately 17 percent, but here again, we are including the four storage arrays in the calculations. The actual server rack footprint was reduced by approximately 37 percent. The net result is a slight increase in overall footprint for an environment with a flexible configuration and a much greater storage capacity.

Table 2. Planned Capacity Buildout for New Data Center Project

# of Racks

Rack Footprint

Power Utilization (kW)

Maximum # of Servers*

4 storage arrays

36

45

207

Virtual machines

2

2

33.2

640

Discrete blades

9

9

149.4

576

Physical servers

10

10

160

150

Subtotal

57

66

549.6

1,366

*Assumes an average of five virtual machines per blade.

Table 3. Servers to be Migrated to New Data Center

# of Racks

Rack Footprint

Power Utilization (kW)

Servers

4 storage arrays

Virtual machines

1.5

2

33.2

504

Discrete blades

Physical servers

54

54

864

1,083

Subtotal

55.5

56

897.2

1,587

Both scenarios in the preceding tables assume that each blade is hosting an average of five virtual machines. As Microsoft IT moves into their next generation of IT Cloud services, those numbers could increase to an average of 16 virtual machines per blade, significantly increasing the power utilization efficiency.

Operational Efficiencies

Even though Microsoft virtualization and management technologies are very capable, business and operational process changes are necessary to maximize savings potential. Adopting IT Cloud services, RightSizing, and centralized procurement and capacity management models works well for Microsoft IT to drive significant savings while improving services.

Microsoft IT uses new technologies such as Hyper-V to open new opportunities for standardization, eliminating or streamlining many of the manual steps needed to build an application environment. This is achieved by establishing Hyper-V guests built with a standard software configuration, including the operating system, appropriate business service, tools, and approved configurations.

The use of System Center Virtual Machine Manager (SCVMM) enables provisioning of virtual machines, migration of the workloads, and tuning of performance per instance. SCVMM enables Microsoft IT to replace the less efficient build processes with a standardized and automated deployment process.

  • Build times and errors are reduced with physical-to-virtual and virtual-to-virtual migration automation.

  • Patching of servers is simplified, and errors resulting from patches are reduced as the core operating system has a smaller exposed surface area.

Customer Satisfaction

Through virtualization and the use of SUs, customer satisfaction is improving due to improved manageability, increased business continuity, reduced hosting costs, and improved Service Level Agreements (SLAs).

Virtualization provides better availability and agility. Windows Server 2008 Server Core deployment mode has a small footprint and a reduced attack surface, minimizing points of failure. Virtualization enables quicker response to business needs for new servers and applications. Blade technologies coupled with virtualization allow for flexibility in configurations. Combined, these enable Microsoft IT to be more agile in support of the business, while controlling costs through improved utilization of system resources. This also helps enable Microsoft IT to maintain hardware supply in advance of business demand for hardware.

Virtualization benefits include business continuity and disaster recovery options because virtualized systems are easier to move around. The availability to freeze and thaw virtual machines brings additional flexibility as well, allowing new levels of efficiencies when working with multiple configurations or exercising rollback scenarios.

Customers see a reduction in costs for hosted services. Microsoft IT utilizes a chargeback model for IT Cloud services provided to its customers. Changes in hardware standards along with virtualization provide a cost saving benefits to customers. For example:

  • The onetime, non-recurring setup charge is reduced by approximately 40 percent, from $5,200 for a physical machine to approximately $2,070 for a virtual machine.

  • One year hosting charges for a 2 RU system is only slightly higher than purchasing a virtual machine plus one year of virtual machine hosting charges. For subsequent years, customers save approximately 50 percent for hosting charges when utilizing a virtual machine versus a 2 RU system.

Customers also see improved SLAs. For example, standard virtual machine deployments are now three days versus five weeks for physical server deployments.

Overall Cost Savings

Building, deploying, managing, and supporting virtual servers has increased Microsoft IT operational efficiencies and reduced overall operational costs. On average, virtual servers cost approximately 35 to 40 percent less per month to support than physical servers ($255 per month versus $385 per month).

The target goal for 50 percent virtualization translates into a server cost benefit of an annual $4.6M savings opportunity for Microsoft IT. With additional savings achieved through reduction in "all-up" support costs, 50 percent virtualization could potentially achieve a $16M savings.

In additional, by reducing the total physical servers, Microsoft IT will see an approximate $2.1M hard cost savings.

Best Practices

Products and Technologies

  • Windows Server 2008 and Windows Server 2008 R2

  • Windows Server 2008 Hyper-V and Windows Server 2008 R2 Hyper-V

  • System Center Virtual Machine Manager

Conclusion

In the past several years, Microsoft IT has replaced physical servers in data centers by deploying thousands of virtual machines on Hyper-V. During this process, Microsoft IT developed best practices for deploying and managing the host computers on which the virtual machines run. By using the latest server, network, and storage hardware in conjunction with Windows Server 2008 Hyper-V and System Center Virtual Machine Manager, Microsoft IT has deployed a virtualization infrastructure that provides a high service level to the Microsoft business units while reducing the overall space requirements for physical servers, markedly decreases the power required to run the servers, and reducing overall operating expenses.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

https://www.microsoft.com

https://www.microsoft.com/technet/itshowcase

© 2010 Microsoft Corporation. All rights reserved.

Microsoft, Hyper-V, SharePoint, SQL Server, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.