Export (0) Print
Expand All

Microsoft IT Evolves its Network for Public Cloud Connectivity

Technical Case Study

Published April 2014

Microsoft IT realized network-related challenges as it moved its properties to the public cloud. To address these challenges, it undertook a series of projects aimed at improving network performance, including an analysis of network traffic flow changes, a redesign of the corporate network interface to the Internet, and the addition of a dedicated, private Microsoft Azure ExpressRoute connection.


Download Technical Case Study, 382 KB, Microsoft Word file




Products & Technologies

As Microsoft IT located more of its line-of-business (LOB) applications in Microsoft Azure, Microsoft Office 365, and other public cloud-based properties, it faced network infrastructure challenges. The user edge (public-facing Internet access point for network users) saw a significant change in the kind and volume of its network traffic. Network servers lacked secure and efficient traffic management to and from the growing number of application resources located in Microsoft Azure, and the process of migrating those applications interfered with other internet-bound traffic.

To resolve these issues, Microsoft IT performed the following activities:

  • Conducted a preliminary analysis of network traffic patterns to assess the existing and future impact of public cloud-based traffic to and from the corporate network.
  • Built and replicated a distributed user edge design that uses hardware-based firewall solutions to replace software-based proxies, and configured the new user edge to speed up the flow of traffic to trusted public cloud destinations.
  • Implemented ExpressRoute, a Microsoft Azure technology that bypasses the public Internet to create a private, dedicated connection between the corporate network and Microsoft Azure. This connection also assists with migrating applications to the cloud.
  • Reduced user edge hardware expense
  • Enterprise-level connectivity for applications
  • Increased availability of critical services
  • Higher bandwidth capacity
  • Improved predictability of performance
  • Improved security for outgoing requests
  • Microsoft Azure
  • Microsoft Azure ExpressRoute


The increasing shift of IT systems and workflows to the cloud has proven to benefit companies in multiple ways, from hardware and maintenance savings to scalability increases and ease of implementation. Many enterprises are eagerly planning their cloud-based strategy for achieving newer, better systems for long-term IT success, and Microsoft is no different. The productivity of its employees relies heavily on its widespread implementation of Office 365, and Microsoft IT is experiencing an ongoing discovery of the ways Microsoft Azure can benefit the implementation and operation of the LOB applications it develops and supports for its business customers.

One aspect of these changes that was not fully expected, however, was the impact on existing network infrastructure. Specifically, cloud migration significantly changed the volume and nature of traffic flows within and outside the corporate network, including both user-to-system and system-to-system traffic. The migration to Office 365 created a broad shift in user productivity traffic flows, resulting in new patterns that were incompatible with legacy network designs. The bandwidth requirements of moving on-premises applications to Microsoft Azure also altered the existing traffic flow models. Over a fifteen-month period, Microsoft IT noted an increase in its incoming Internet traffic from 3 GB to over 22 GB, with much of that increase attributed to public cloud activity. Overall, the existing IT networking infrastructure that was in place to deploy and support new cloud-based solutions at Microsoft proved to be insufficient in the following key aspects:

  • User edge demand issues. Traffic flow changes uncovered pent-up demand at the user edge—the access point for corporate network users to access the public Internet, and a critical connectivity piece for accessing Office 365 and other public cloud-based resources. User productivity was threatened if the traffic became bottlenecked at this edge, and the existing software-based methods for managing this capacity did not scale. The migration to Office 365 in particular put great and immediate strain on existing network traffic components, because instead of merely supporting users who need to view public websites, the user edge now needed to support all traffic going to and from Office 365.

  • Application migration slowdown. Many LOB applications moving from on-premises data center locations to Microsoft Azure required the migration of large virtual hard disks (VHD). Similar to the user edge challenges, these VHD migrations presented significant delays given the capacity required to transfer them over existing public internet-facing network resources.

  • Lack of secure, efficient system-to-system traffic flows. Microsoft IT considers many of its LOB applications to be hybrid in nature, either consisting of or depending on a mixture of public cloud-based and on-premises resources. Until all its LOB applications run in Microsoft Azure, Microsoft IT will require a secure design for system-to-system traffic flows between its corporate data centers and Microsoft Azure in order to support these hybrid applications. Depending on how a hybrid application is designed, providing it with system access to Microsoft Azure properties by using traditional Internet addresses introduces security considerations, and also subjects such traffic to changes in availability (such as slowdowns or outages) that are difficult to predict, adversely affecting application performance. To resolve these issues, in the case of Microsoft IT, a private, secure network connection was needed for these hybrid applications.

Figure 1. Some hybrid applications rely on the Internet  for system-to-system traffic with Microsoft Azure, leading to new security and  performance considerations.

Figure 1. Some hybrid applications rely on the Internet for system-to-system traffic with Microsoft Azure, leading to new security and performance considerations.

Note: In this paper, as in practice, Microsoft Azure is a separate business unit from Microsoft IT. Although both are Microsoft entities, all operations are separate, and Microsoft IT is functionally the same as any other Microsoft Azure enterprise customer.

As part of its strategy to address these issues, Microsoft IT performed a preliminary analysis of traffic flows to understand the specific changes. However, it was immediately clear that the issues were already affecting network availability and performance to a degree that in-depth analysis was not feasible. Among its recommendations, Microsoft IT strongly suggests that other enterprises charting a path to public cloud-based productivity resources perform a proactive, detailed analysis of traffic modeling change requirements before implementing the cloud-based solution so that issues can be addressed in a more deliberate way.

With a mandate to support more than 200,000 workers in more than 400 locations using more than 1,100 LOB applications running on more than 40,000 servers, and while working with business groups to develop roadmaps for moving these applications out of the seven Microsoft data centers and into the Microsoft Azure public cloud, Microsoft IT needed a way to address the bottlenecks in its network design while continuing to service the needs of Microsoft and its partners. It also needed to sustain the IT best practices of reducing costs and gaining agility for its own organization.


The key to addressing its network design challenges was for Microsoft IT to focus on improving traffic management between the Microsoft corporate network and the company's growing number of public cloud-based resources. Long-term, Microsoft IT knew that its network traffic would have less to do with data center connectivity and more to do with maintaining secure and efficient traffic patterns among its cloud-based properties. This realization led Microsoft IT to begin revising its network designs based on new cloud-based traffic flows, security models, and bandwidth priorities.

Using a dual approach, Microsoft IT deployed a technology called ExpressRoute to provide a dedicated private connection to its Microsoft Azure properties. It also redefined its traffic management policy along the existing user edge to more intelligently re-route various user and server requests based on their destinations.

As part of this approach, Microsoft IT also addressed bandwidth issues on the user edge by doing the following:

  • Validating bandwidth to ensure sufficient capacity for the network changes (traffic moving to public cloud-based properties in addition to moving to traditional data center properties).

  • Providing additional high-performance firewall devices at the user edge to provide sufficient capacity on the corporate network backbone to support all traffic if there is an outage.

Solution Design Principles

The new solution was derived from the following key design principles:

  • Designing the network for hybrid applications. While it is the stated mission of Microsoft IT to run all its applications in the cloud, this cloud is a mixture of both private cloud (on-premises) resources and public cloud technologies, such as Microsoft Azure, Office 365, SharePoint Online, Microsoft Dynamics Online, and Visual Studio Online. Like many other enterprises, Microsoft IT supports hybrid LOB applications whose resources are located in both cloud types—a necessary solution as long as cloud-based applications have dependencies on on-premises resources. Microsoft IT uses many hybrid applications, and therefore it included accommodation for these applications when redefining its network design.

  • Diverse access to Microsoft Azure for multiple traffic types. As a way to establish a more secure and faster method of operating its hybrid LOB applications, Microsoft IT needed to design a way for its system-to-system traffic to travel along a dedicated route to Microsoft Azure, bypassing the traditional gateway to the public Internet. The resulting technology, called ExpressRoute, creates a bypass over private network resources inside Microsoft Azure. In this way, Microsoft IT created a redundant network path that servers could use, further isolating and optimizing basic user access to Microsoft Azure, which occurs at the user edge.

  • Software-defined networking. Instead of relying only on network hardware to manage traffic routing for ExpressRoute, Microsoft IT applied software-defined networking principles to enable automatic router configuration changes by using a software portal. Developers and other users lacking network expertise can access the portal and request connectivity by describing the interaction of their Microsoft Azure virtual machines (VMs) with the corporate network. The software performs the required configuration changes on the routers, both at the Microsoft IT data center and Microsoft Azure locations.

  • Redundant networking to Azure. Microsoft IT increased the number and scale of user edges by implementing a new array of firewall devices at the existing user edge, and then duplicating this design in multiple locations worldwide. By doing this, it strengthened capacity for both the user edge and the corporate backbone, both for day-to-day use and also if there is an outage.

Solution Architecture

Distributed User Edge

To address the new volume of internet-bound traffic at the corporate network's user edge caused by users accessing resources in Office 365 and Microsoft Azure, Microsoft IT replaced existing network proxies with hardware-based firewalls to achieve an intelligently distributed configuration. By checking the destination of outgoing packets and diverting those bound for trusted cloud resources through less intensive scrutiny by corporate networking equipment, this new distributed user edge shows how a differential treatment of network activity by destination type can ease the strain on traditional network resources.

By implementing both the distributed user edge and ExpressRoute, Microsoft IT made significant gains in optimizing its network for cloud-based connectivity. The intelligent distributed re-routing of activity at the user edge created faster performance of internet-bound requests. Meanwhile, traffic volumes at the user edge were further diminished by the implementation of ExpressRoute, with a significant section of the server-to-server traffic that previously tested the network capacity being now offloaded to a private back-end connection.

Implementation Details

Understanding the value of implementing a distributed user edge depends on an acknowledgement that corporate network user traffic to and from the public Internet has changed a good deal with the advent of public cloud-based services. Whereas before, most user traffic was occasional and traveled to unknown destinations, now a high percentage of user traffic goes to and from these cloud properties, and almost all users are engaged in this traffic throughout their day.

Typically, all traffic passing between the Microsoft corporate network and the public Internet is subject to rigorous scrutiny by various technologies running in dedicated network devices. Data loss prevention (DLP) software inspects each outgoing packet to monitor, detect and block sensitive data leaving the corporate environment, while intrusion detection services (IDS) monitor incoming packets to identify and isolate any potential malicious activities or violations of IT policy.

For the fairly small percentage of traffic going to unknown locations outside the company (for example, users visiting external websites or receiving suspicious mail), this level of deep packet inspection—scrutinizing each traffic element for potential breach—is much needed and well worth the significant investment Microsoft IT makes in DLP and IDS technologies. This level of inspection, however, is overkill for the ever-increasing amount of internet-related traffic associated solely with trusted cloud properties, such as Office 365 and Microsoft Azure. Because IT knows to trust certain IP addresses, the software layer added to the distributed user edge saves lots of time by routing most packets to these requested destinations.

Figure 2. In the Microsoft IT design, the distributed  user edge design replaces existing proxies with firewall devices to identify  packets bound for trusted public cloud properties and subjects those packets to  a lower level of inspection, improving traffic.

Figure 2. In the Microsoft IT design, the distributed user edge design replaces existing proxies with firewall devices to identify packets bound for trusted public cloud properties and subjects those packets to a lower level of inspection, improving traffic.


Microsoft IT deployed ExpressRoute primarily to address the need for reliable, secure connections of on-premises hybrid LOB application components to Microsoft Azure. The ExpressRoute technology will be branded with various other names by Microsoft telecommunication provider partners, but the value is the same—helping enterprise IT organizations operate hybrid LOB applications effectively in the Microsoft Azure public cloud without sacrificing security or performance.

Figure 3. ExpressRoute establishes a private virtual  network inside Microsoft Azure.

Figure 3. ExpressRoute establishes a private virtual network inside Microsoft Azure.

ExpressRoute also plays a role in helping Microsoft IT migrate traditional (non-hybrid) LOB applications to Microsoft Azure. Migrating these applications requires moving large VHDs from their current data center locations to the cloud, a process that formerly consumed lots of bandwidth for each VHD and exposed the VHD to all the security vulnerabilities that come with public Internet data exchanges. By pushing these VHDs over ExpressRoute, the application is migrated over a private connection, making the migration faster, more secure, and less taxing on internet-facing corporate network resources. Post-migration, the VHD and other application resources located in Microsoft Azure manage their traffic over ExpressRoute as well, providing added speed and security over a dedicated, private virtual connection.

To date, Microsoft IT has migrated more than 100 of its traditional LOB applications using ExpressRoute, with a goal of migrating an additional 300 per month until all LOB applications—traditional and hybrid—are running in Microsoft Azure.

Telecommunications Scenarios

ExpressRoute implementation is designed to be coordinated with a Network Services Provider (for example, AT&T). The provider, a telecommunications service, delivers the required networking equipment, and the enterprise customer works with the provider to establish the virtual network and test access to the Microsoft Azure-based resources. However, enterprises also have the option of using an Exchange Provider (for example, Equinix) instead of a Network Services Provider. In this solution, the enterprise provides its own equipment and acts as its own telecommunications service, using the Exchange Provider of its choice.

Microsoft IT implemented the Exchange Provider model, using its own networking group and infrastructure to offer ExpressRoute as a service to its business customers. Similarly, large enterprises with robust telecommunication capabilities in data centers can add ExpressRoute to their networks and then make the newly secure Microsoft Azure connectivity available to business groups within their corporate organizations.

Implementation Details

Using a physical connection to provided by special telecommunications equipment, ExpressRoute creates a private, virtual network within Microsoft Azure by using corporate network IP addresses in Microsoft Azure to provide connectivity to incoming Microsoft Azure IP address requests from corporate network servers. This method helps secure and optimize traffic flows between Microsoft Azure and the corporate network by addressing Microsoft Azure resources using private addressing. Because the Microsoft Azure IP data is decoupled from the request, connection to the Microsoft Azure-based resources is kept private. It is also a dedicated connection, unencumbered by other network activities.

Figure 4. ExpressRoute translates Microsoft Azure IP  addresses to private, internal IP addresses before connecting incoming requests  to Microsoft Azure-based VM resources.

Figure 4. ExpressRoute translates Microsoft Azure IP addresses to private, internal IP addresses before connecting incoming requests to Microsoft Azure-based VM resources.


  • Reduction in user edge hardware expense. Although this benefit has not yet been achieved, Microsoft IT has identified that its capital expenditures for networking equipment will decrease over time, as its new software-defined networking model for automatically re-routing traffic bound for trusted public cloud properties reduces the reliance on deep-packet inspection.

  • Enterprise-level application connectivity. ExpressRoute provides the level of dedicated infrastructure needed for a large enterprise to provide secure, efficient connectivity of its LOB applications to the Microsoft Azure cloud.

  • Increased availability of critical services. As part of adding ExpressRoute to its network, Microsoft IT implemented redundant hardware throughout its upgrade solution, following its best practice of eliminating single points of failure as it increased the availability and scalability of LOB applications.

  • Faster time-to-productivity for Microsoft Azure projects. Robust connectivity and ease of migration to Microsoft Azure means that Microsoft IT can make better use of its cloud-based technologies and deliver value to its business teams sooner than was possible before.

  • Higher bandwidth capacity. The distributed user edge and ExpressRoute solutions reduce bottlenecks in user-to-system and system-to-system network traffic flows respectively, improving the user experience and expanding functionality options for bandwidth-intensive cloud-based applications.

  • Improved predictability of performance. Where application and network speed are required, ExpressRoute's private network design provides better predictability than previous scenarios by using a dedicated, private back-end connection to Microsoft Azure. This level of predictability makes it possible for Microsoft IT to offer service guarantees that meet the needs of its LOB application owners.

  • Improved security for outgoing corporate network requests. Because the newly distributed user edge routes eligible packets directly to trusted public cloud resources, and because ExpressRoute uses a virtual private network within Microsoft Azure to use corporate network IP addresses, these technologies help significantly improve the security of both user and server connections that would otherwise have been vulnerable to internet-based attacks.

  • Reduced reliance on servers, network equipment, and engineering. Microsoft IT has begun to realize cost savings from its gradual reduction in data center equipment and maintenance investments, and expects significant further savings as more of its LOB applications are situated in Office 365, Microsoft Azure, and other public clouds. A significant part of this savings will be the reduced need to deploy LOB applications in traditional data centers.

Best Practices

  • Analyze user edge traffic patterns for optimal cloud-based performance. Microsoft IT's changes to its network were the outcome of preliminary analysis performed by its Enterprise IT Architecture team. A key finding of this analysis was the abundance of traffic flow changes as more LOB applications migrate to the public cloud. With more time, a detailed analysis of traffic patterns might have provided additional insights into the new network design. Microsoft IT recommends that any enterprise planning to migrate its services to a public cloud make time to analyze and consider the impacts of this change to its traffic flows, and plan network changes accordingly.

  • Predict core network change requirements. Changes made to a company's network design require commensurate support on the corprate network backbone. As a company makes plans to increase the scale and performance of its user edge, Microsoft IT recommends taking the steps that are required to ensure that the company's network backbone adds sufficient capacity to support the changes.

  • Understand the impact of revised traffic flows on the user edge. Network changes that use new technologies can also affect user edge traffic. For example, many services use IP version 6 (IPv6 ) for Internet communications. When a user edge is upgraded to support IPv6, the abrupt change in traffic patterns can require additional network design consideration.

  • Consider multiple user edges. As Microsoft IT discovered, replicating its new distributed edge design in multiple locations helps scale out the volume of internet-bound traffic, and provide redundancy if there is an outage.

  • Understand application flow details before migration. When planning the migration of a hybrid application, it's important to examine the application architecture to understand which components will remain in the data center and which will be moved to the public cloud. Also, understanding how users access the application is important for understanding traffic flows, and using this understanding to validate the applicatio's performance will not decrease post-migration.

  • Include security teams in the application migration planning process. Successful migration of applications to the public cloud requires both network and security expertise. Research and analysis done by network teams can be audited by security experts to ensure holistic, secure migration planning and positive results.

  • For complex environments, make both a near-term and a long-term plan. Because the Microsoft IT migration of LOB applications is ongoing, not all the changes could be addressed by a single solution. Over time, however, hybrid applications, while currently found throughout the Microsoft IT environment, will gradually be engineered to have fewer dependencies on data center resources. The trend toward fully cloud-based applications will continue to change network traffic patterns until the dependency on traditional data center resources is minimized and most of the services are located in Microsoft Azure.


ExpressRoute home page

ExpressRoute technical overview

ExpressRoute pricing and telecommunications scenarios

Related videos

TechNet Radio: Delivering Results — How Microsoft IT Prepared its Network for the Cloud

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information online, go to:



© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office 365, SharePoint, and Visual Studio are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback
© 2015 Microsoft