Improving Branch-Office Services Through Virtualization
Published: February 2010
To ensure that the services of Microsoft branch offices continue to meet changing business requirements in a flexible and cost-effective manner, Microsoft Information Technology (Microsoft IT) has developed a strategy based on the Windows Server 2008 R2 operating system with Hyper-V. Adopting this approach has enabled Microsoft to deploy a wider range of services, on a more cost-efficient and secure platform, to its branch offices around the world.
Article, 114 KB, Microsoft Word file
Products & Technologies
Microsoft IT supplies services to a user community in 106 countries and approximately 650 buildings worldwide. IT services supplied to staff within the organization typically have a service level agreement (SLA) that offers 99.99 percent availability, or higher, regardless of location, much like a bank or large retail chain that has centralized datacenters and many geo-diverse branch outlets.
To achieve this SLA in a cost-effective manner, Microsoft IT has adopted has a hybrid approach to its infrastructure. Core services such as Microsoft® Exchange and Microsoft SharePoint® Products and Technologies are delivered from a small number of regional data centers. Branch-office servers supply a targeted set of local services in locations where the branch-office size makes it effective to do so.
Until the introduction of Hyper-V in Windows Server 2008, the services that branch-office servers supplied were limited to technologies that could coexist on one physical server. A typical configuration was a server running Windows Server 2003 with Service Pack 2, with six core services:
- Data Distribution Services
- Windows® Deployment Services
- Microsoft Systems Management Server 2003
- Microsoft IntelliMirror® management technologies
The infrastructure model for services at Microsoft originated from a project that ran from 2002 to 2005, called the Model Enterprise Initiative (MEI). The aim of this project was to reduce growth of the number of servers (server sprawl), to remove the complexity around the infrastructure at Microsoft, and to reduce total cost of ownership (TCO).
The MEI project exceeded all its objectives, particularly in reducing cost and complexity, as well as producing a site taxonomy for infrastructure deployments that is still used today. MEI reduced global infrastructure servers by 27 percent and made it simpler to decide where different types of services were delivered from. For the most part, branch offices became single-server sites; a decision matrix determined which office locations had branch-office servers deployed and which offices were serverless.
Under the MEI model, Microsoft IT applied a set of criteria to all sites. These criteria took into account factors such as site size, site location, local business requirements, the connectivity options available, and the site's function in the organization. Additionally, Microsoft IT took into account the infrastructure's ability to meet the SLA for services and a new performance measurement called minimum performance levels (MPLs).
Unlike an SLA, which deals purely with service availability, an MPL regulates and measures client expectations regarding how a service should perform when it is available. Set performance measurements were negotiated with our supported business groups and tested as part of the MEI planning process. From this testing, Microsoft IT identified the six core services that were targeted to run on branch-office servers.
While the majority of services were delivered from data centers, branch-office servers provided a more efficient and customer centric solution for these six services. This enabled Microsoft IT to:
- Enhance wide area network (WAN) utilization and minimize replication.
- Achieve MPLs by hosting certain key services locally.
- Lower TCO by combining several services on one server.
Although the branch-office server model that MEI developed was useful, the range of local services that could reside under a single operating system was still limited. As more services were deployed on the same server, the likelihood increased that problems would occur in areas such as:
- Service overlap: Having multiple services installed on the one operating system created additional workload and complexity for service managers and regional IT staff. Any changes had to be coordinated across all the service managers who shared the same platform. This made any type of change, from product upgrades to regular patching, a complex exercise.
- Infrastructure optimization: The branch-office server design developed under MEI made it impossible for Microsoft IT to adapt to changing business requirements in a timely manner. The overhead of multiple services sharing a single platform meant that even though business requirements changed, adding more services onto the branch-office server was a slow and difficult process. This led to a basic approach to local services that did not meet businesses expectations of a truly customer-focused IT strategy.
- Security: Sharing a single operating system also had security implications. For example, the local resources that handled printer administration issues were directly logging on to a server that was delivering services such as document replication. Although rights could be managed based on roles, the potential to affect overall server performance or compromise confidential data was always a possibility without true service segregation.
- Cost: Consolidation opportunities were limited without some kind of virtualization on branch-office servers. Because of corporate security policies, security related services such as domain controllers were not allowed to share a server with other services. Stand-alone domain controllers that were on dedicated physical server hardware in strategic locations around the globe were running workloads well below that servers' capacity.
Microsoft IT recognized that virtualization would address the limitations that it had in branch offices. Deploying a virtualization solution for branch-office servers would enable new services to be delivered locally, without compromising existing services. It would also enable Microsoft IT to explore additional consolidation opportunities and to adopt a more dynamic approach in its branch-office network.
Since 2004, Microsoft IT has pursued aggressive virtualization targets for its infrastructure, first by using Microsoft Virtual Server 2005 and more recently by using Windows Server 2008 R2 with Hyper-V. These benefits range from cost-saving and hardware utilization benefits to a reduction in physical server sprawl.
Today, Microsoft IT manages around 9,000 servers in its infrastructure, including around 235 branch-office servers. By June 2009, virtual machines composed more than 30 percent of the total population of managed servers, with a target of 50 percent by 2010 and 80 percent by 2012.
Microsoft branch offices have followed much the same trends and targets as data centers: Microsoft IT developed a new branch office strategy leveraging our Virtual Branch Office Server (or VBOS) platform. The VBOS platform consists of a virtual server host physically located at each managed branch office that supports multiple guest virtual machines that support the various required branch office services. As existing branch office servers are reaching the end of their planned 3-year lifecycle, they are being refreshed with the new VBOS platform. Since its inception in mid-2008 through December 2009, 91 VBOS servers have been
A Windows Server 2008 Hyper-V solution enabled Microsoft to:
- Enhance security on the platform.
- Isolate services into separate virtual machines.
- Make additional services on the branch-office server a viable option.
- Improve WAN performance.
- Enable a simple, repeatable deployment process.
- Implement power management procedures as part of the Green IT environmental strategy.
Before architecting a solution, a small virtual team of field service experts was formed from IT managers based in the three Microsoft business regions: Asia Pacific Japan (APJ); Europe, Middle East, and Africa (EMEA); and the Americas. The purpose of this team was to develop a virtualization solution for branch offices in partnership with the service managers and the server engineering teams.
The approach was to mirror the services already deployed on existing servers onto a new virtual server design and plan for growth in locally supported services based on a road map of additional business requirements. These requirements might vary between locations, depending on the function of the branch. For example, software development sites would have different requirements than sales and marketing offices.
The project team then approached each of the existing service managers to get the requirements for their services based on current and known future developments. The project team used this information to develop server specifications for small, medium, and large sites.
Pilot and General Deployments
The VBOS project started in early 2008 with pilot sites targeted for all three Microsoft business regions. The pilot had several objectives, and the outcomes would determine what types of servers could be purchased in the refresh cycle for the next financial year. It would also have a significant impact on workloads, support, and training requirements for infrastructure staff, if successful.
The design was based on a host virtual machine running the Release to Manufacturing (RTM) version of a Server Core installation of Windows Server 2008, and four guest virtual machine servers. The VBOS project team wrote a simple Windows PowerShell™ script that could build the standard configuration on each new VBOS server, independent of whether it was a small, medium, or large site configuration. The pilot would also produce a documented process to transition services from the old to the new servers.
Overall, the pilot met all criteria in the original project charter. After the initial pilot phase finished, general deployment of virtualized branch-office servers began in mid-2008, based on the existing policy for server life-cycle management and replacement.
The pilot produced a simple and repeatable process to build and deploy a fully commissioned VBOS server with minimal training requirements. The VBOS project team used this process, with few modifications, for all 91 VBOS servers deployed in the first year. The following sections describe some key achievements.
Security on the Host
A Server Core installation was a practical choice for the host virtual machine in the Windows Server 2008 virtual environment. A thin hypervisor layer sits separately on the hardware, providing access to the physical server. A small Server Core host installation then sits on top of the hypervisor, acting as a management layer to provide functionality to the guest virtual machines. Both the hypervisor and the host are kept small to reduce the attack surface.
The server build process is much different from Virtual Server 2005 or Microsoft Virtual PC because the hypervisor layer does not reside in the Server Core host installation. Each of the layers is separated with its own security. The hypervisor, the host virtual machine, and the guest virtual machines all have strong levels of protection, giving the design robust defense-in-depth capabilities.
By choosing a Server Core installation of Windows Server 2008 as the host operating system, the number of updates and subsequent reboots also decreased significantly because of the small server footprint that this type of installation required. Initial estimates are that the updates and reboots in the branch-office server maintenance windows decreased by over 60 percent on the host. This was a significant improvement for local sites, which could now go for several months before they needed any reboots.
Security on Guest Virtual Machines
Segregating services on virtual servers is a more secure and efficient way of managing branch-office workloads. This was an important objective of the project, because changes to one service in the past had affected all services on the operating system, which in turn required extensive testing and coordination between teams.
Introducing separate virtual servers for each service meant that the service managers no longer had dependencies on anything but the server resources that their own guest virtual machine can access, much like in a single-purpose physical server environment. This drastically reduced the time and effort needed to test changes and gave the service managers complete control over their SLA.
Branch-office virtualization provided a range of cost benefits. Some of these benefits, like greater uptime, are difficult to quantify with exact measurements but are clearly beneficial to internal users and Microsoft IT. A more measurable benefit is the time saved in provisioning servers and adding more guest virtual machines.
Before the branch-office virtualization project, Microsoft IT required support staff to be physically on site to build, rack, provision, and deploy a new server. This was often a slow process, especially for sites that had no local IT staff available. In the new environment, Microsoft IT can do much of the deploying and troubleshooting via the Microsoft System Center Virtual Machine Manager console. This ability has reduced the average time for fully provisioning and deploying a server to three months.
Apart from the physical building and racking of the servers, which less skilled resources can perform, the steps to deploy the server can now occur remotely. Microsoft IT easily achieved this by using System Center Virtual Machine Manager and the range of the remote administration tools available to operations staff and IT managers.
Power Management Savings
For branch offices, Hyper-V virtualization provides power savings by consolidating multiple workloads onto a single server. Due to the MEI design that Microsoft IT had adopted in branch offices, this benefit was constrained by the fact that the six core services already sat on one physical server.
However, even without further server consolidation or more services being added on existing servers, Windows Server 2008 provided opportunities for savings. Tests within Microsoft found that Windows Server 2008 achieved power savings of up to 10 percent over Windows Server 2003 at comparable levels of throughput.
Although this was not specifically measured, when reviewing the 91 VBOS servers deployed in 2009, Microsoft IT believes that they have made a significant contribution to Microsoft power-saving targets.
First and Best
Another advantage of virtualization comes from the Microsoft IT role as the "first and best" customer of Microsoft. Before any software is released to market, it is tested extensively in the Microsoft organization. This testing also occurs for customer scenarios like branch-office virtual servers.
These efforts were not always practical with a server that had all services on one operating system. For example, if the old-type branch-office servers were going to be a part of a beta program and one of the branch services was not able to run with that software, branch offices could not be part of the testing program.
With virtualization, each service can participate in different testing efforts on a timetable that suits its individual requirements. This ability is much more useful to the Microsoft organization and allows regional IT to contribute in a timelier manner to business priorities. In turn, this helps Microsoft customers have confidence in deploying software scenarios that have been tested extensively in the global Microsoft production environment.
System Center Virtual Machine Manager provides centralized monitoring and management of the entire virtualized infrastructure environment. In data centers, features such as live migration, Cluster Shared Volumes, and rapid provisioning are used daily to help ensure high availability of services.
Within Microsoft, the System Center family of products is the most important tool for managing the entire infrastructure. Branch-office servers benefit from the support of a team of server and virtualization experts who can check on the health and performance of servers in remote locations as well as in data centers.
System Center resolves more than 90 percent of all issues that occur by using automatic procedures in its knowledge database. All incidents are recorded, so when problems arise, regional IT is informed of the issues that require intervention. Regional IT is also aware of the issues that were resolved automatically. All of this activity occurs through the incident tracking system, which is linked to System Center through the connector framework.
When local resources need to be engaged to help with incidents, they too can use the System Center Virtual Machine Manager console in conjunction with Remote Server Administration Tools for Windows 7 to access the servers that they are physically responsible for. Local resources can access their servers without having user rights to the whole infrastructure, which provides security for both the local resources and the organization as a whole.
Although internal Microsoft IT teams used Windows Server 2008 and Hyper-V extensively in production well before those technologies were released to market. In terms of technology, the key lessons that Microsoft IT learned were:
- Determine service specifications as early as possible and ensure that they meet all site type requirements.
- Plan for growth in the server specifications and identify additional services that will add the most value early on.
- Provide training for resources who will manage the new servers. Logging on to guest virtual machines and using multiple operating systems on one computer will involve a learning curve.
- Ensure that each server is in the correct organizational unit in Active Directory® Domain Services, and that it has been set up in the automatic update cycle.
In terms of planning, the key lessons that Microsoft IT learned were:
- Staff appropriately, because each service now has its own virtual server, and this has implications for day-to-day management and administration.
- Ensure that the pilot covers all scenarios, and use this phase of the project to update existing processes and documentation.
- Ensure that communications are frequent and reusable across all sites that are being changed.
- Avoid the temptation to mix services within a guest virtual machine, because this ties the future of those services to each other, which may cause issues in the long term.
- Baseline and measure the impact of changes.
The virtualization of branch offices has presented a range of future possibilities that would otherwise have not been available. While the VBOS deployments continue, several additional services are currently being considered for future use in branch offices. These include virtual lab servers to mimic customer environments, and streaming media services. However, from the list of possible options, the ones with the greatest potential return on investment are hosted branchcache and read-only domain controllers (RODCs).
Only a limited number of hosted branchcache servers were rolled out in the first year, because this was an additional service that Microsoft IT needed to test in order to monitor the benefits. After an initial pilot in 13 sites, Microsoft IT now considers this the most important additional service to deploy locally.
Branchcache in the Windows 7 and Windows Server 2008 R2 operating systems helps increase network responsiveness of centralized applications when users access those applications from remote offices. The performance of applications that use one of the following protocols significantly increases:
- Hypertext Transfer Protocol (HTTP) and HTTP over Secure Sockets Layer (HTTPS): The protocols used by Web browsers and many other applications (such as the Windows Internet Explorer® browser, Windows Media® technologies, and more).
- Server Message Block (including signed SMB traffic): The protocol used for shared folders.
The branchcache service is especially useful in the network area, because it clears some WAN bottlenecks that affect branches; especially regarding file downloads, without requiring expensive network upgrades. The local user experience significantly improved on sites that participated in the pilot.
Read-Only Domain Controllers
Domain controllers present an opportunity to further consolidate servers and improve physical security. Not all branches have domain controllers. However, where they do exist, this service resides on its own server, and this presents cost and security implications.
From a cost perspective, a single-purpose server in this scenario leads to underutilization of the physical server—something that virtualization specifically addresses. As domain controllers reach end of life or are required in new locations, Microsoft IT is now considering the option of using an RODC on the local VBOS server. This would use existing resources, avoid cost, and reduce risk to the organization in areas where deploying a full domain controller may present physical security risks.
For example, remote locations are often less physically secured than data centers. An RODC that is running on a branch server presents less risk to the organization, because it contains only a limited subset of domain controller information. No password or credential information is replicated.
The VBOS project enabled Microsoft IT to extend virtualization outside the data center and to showcase its benefits across a range of site types, from small sites to large and complex branch offices. This range mirrors everyday environments that Microsoft consultants see in their daily interaction with customers.
The VBOS project met, and in many cases exceeded, the expectations of the entire project team. Following the existing life-cycle replacement model for infrastructure replacement, all traditional branch-office servers are targeted for replacement with the VBOS servers within a three- to four-year time frame.
The project has enabled Microsoft IT to extend services, reduce costs, and manage the infrastructure more effectively in branches. It has opened more consolidation opportunities, enabled a more secure approach to service management, and helped Microsoft achieve its Green IT environmental strategy targets.
From this project, Microsoft IT has adopted a more dynamic and flexible approach to services supplied to branch offices. Microsoft IT has become a more valuable business resource as a result.
For More Information
For more information about Hyper-V in Windows Server 2008 R2, go to http://www.microsoft.com/windowsserver2008/en/us/hyperv-main.aspx.
For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Center at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:
© 2010 Microsoft Corporation. All rights reserved.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Active Directory, Hyper-V, Internet Explorer, SharePoint, Windows, Windows Media, Windows PowerShell, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.