Export (0) Print
Expand All

How Microsoft IT Leverages Internet Information Services 7.5 in Windows Server 2008 R2

Article

Published: April 2011

Microsoft.com is one of the busiest domains in the world with 301 million unique visitors each month. Learn how Microsoft manages Microsoft.com by leveraging the Internet Information Services (IIS) 7.5 enhancements in Windows Server 2008 R2. IIS 7.5 provides significant enhancements for performance, security, diagnostics, and administration.

Download

Intended Audience Products

Download Article, 252 KB, Microsoft Word file

Download Video

  • IT Professionals
  • Technical Decision Makers

  • IIS 6.0, 7.0, and 7.5
  • Windows Server 2008 R2
  • Hyper-V
  • PowerShell

Introduction

The microsoft.com Web site is the 14th busiest domain in the United States, which is a very large presence for a corporate portal. The site has 30K requests per second with 15K concurrent connections. As measured by Keynote™ Systems and its global monitoring system, the site maintains an availability rating of 99.9%.

The Microsoft IT Operations team is tasked with maintaining this high availability rating while at the same time introducing early versions ("dogfooding") of IIS, Windows Server, and the .NET Framework into the environment. This article discusses how the Operations team leverages IIS 7.5 and Windows Server 2008 R2 and how it architects microsoft.com to maintain high availability while fulfilling its dogfooding role.

The Microsoft.com Architecture

The microsoft.com infrastructure consists of 104 Web servers comprised of 24 Hyper-V™ virtual machines (VMs) and 80 physical machines that are divided up into six clusters. Microsoft IT uses hardware load balancing and three separately located geographic data centers. Figure 1 shows the architecture for each datacenter.

Figure 1. microsoft.com datacenter architecture
Figure 1. microsoft.com datacenter architecture

Each of the servers has:

  • The same configuration
  • Its own very large (330 GB) content set
  • More than 1700 applications
  • More than 3100 virtual directories

All servers run Windows Server 2008 R2 and IIS 7.5 with the exception of one or two servers that run the previous operating system or early versions of the next platform. The Operations team uses these servers for comparison purposes and to support dogfooding efforts. The server configuration uses a shared hosting model. Hundreds of developers and content providers publish content to microsoft.com from around the world 24 hours per day, which means that the Operations team does not necessarily control what code is published to the sites at any given time.

Deployment

The Operations team uses a "high touch with standard image" approach to deploy the platform, which means that the team spends quite a bit of time doing manual configurations. Since the platform is updated frequently with beta, RC, and RTM versions, this results in a lot of turnover.

The team uses tools in the Windows® Automated Installation Kit (AIK). They use an Unattended Setup to set up the operating system to begin with. Then they install the applications, generalize with Sysprep, and boot into the Windows Preinstallation Environment (Windows PE). The team uses Integrated Lights Out management technology to manage the servers remotely. They create and deploy the image with ImageX. As improvements are made to the operating system, imaging technologies, and deployment technologies, including the integration of System Center Configuration Manager and the Microsoft® Deployment Toolkit (MDT), the team would like to get to a point where they can deploy to the 104 servers with light or zero-touch, which would speed up deployments and offer more flexibility and agility.

Figure 2 shows the deployment cycle for each new build.

Figure 2. Deployment build cycle
Figure 2. Deployment build cycle

  1. The build is deployed to a single production server that is usually taking live traffic, at least initially. The server is pulled from the live rotation so that the team can configure it with the latest version of .NET, IIS, or the platform (Windows Server 2008 R2 in this case). The team then runs tests with tools such as the Web Capacity Analysis Tool (WCAT) and TinyGet. The team does a lot of log replay (using Log Parser to create the list of requests that WCAT and TinyGet will use) to replay live logs from another server and looks at the status codes to make sure that the server is responding as expected. The team looks at the logs very closely since there is no way to have hundreds of application owners and developers test their applications at this point in the process.
  2. The single server is placed back into live production rotation. As the server takes live traffic, the Operations team looks carefully at the logs to make sure that there are no problems.
  3. The server build is moved to a single cluster of 20 servers.
  4. The server build is moved to a pre-production and staging environment. The application owners can use these internal environments to take a look at their applications on the new platform to make sure that the applications are behaving as expected.
  5. If the team gets the go-ahead at this point, the new build is put into full production deployment.

Testing and Stressing with WCAT and TinyGet

To test and stress a server when there is a new platform, the team pulls a microsoft.com production server ("Server A" for example) from live rotation. They wrap a PowerShell® cmdlet around WCAT, which allows them to throw parameters at the server. Then they look at the 10 most recent logs and the 10 largest logs from the live Web server and play some of those logs back on Server A to make sure that everything is looking OK. The team uses Log Parser to parse the logs first, usually with a few passes. It is fairly straightforward for microsoft.com because it is an anonymous site (does not use Windows authentication). Most of the calls are GETs. The team does not want to POST as they do the stress test since they do not want to write data to the live databases.

The team has also had success with TinyGet, which is an HTTP client that does similar log replay. The team uses TinyGet to play back logs and look at status codes, not necessarily for stress as with WCAT, but to compare a live server to Server A to see which URLs are failing and which are succeeding. The team expects to see the exact same successes and failures on Server A as they run it.

When they do the stress testing, the team tries to make sure that they are as close to 100% CPU utilization as possible. They want to know the number of requests per second when the CPU is at 100% utilization, the number of 404 messages, whether there are any 500 messages, and so on. The WCAT output includes status codes, handshakes, bytes received, time to first byte, and the transaction request.

WCAT has support for performance counters. The team can also capture their own performance counters as they are running the server and look at performance in a number of different ways to ensure that the platform is comparable to previous platforms.

IIS 7.0 and IIS 7.5 Compatibility Layer

The move from IIS 6.0 to IIS 7.0 and IIS 7.5 was significant because the architecture changed significantly between IIS 6.0 and IIS 7.5. The team took advantage of the compatibility layer in IIS 7.0 and 7.5 (not on by default) to make use of existing legacy scripts. For example, the team was able to take advantage of adsutil.vbs, which is based on ADSI, and other legacy scripts that use ABO or the old IIS 6.0 WMI Provider. The compatibility layer enabled the team to use existing investments rather than write against the new APIs right away. Moving from IIS 6.0 to the new platform was very easy. There was one application that encountered a breaking change and there were some configuration changes for Integrated mode, but other than that, things just worked.

The Operations team took advantage of the compatibility layer for legacy scripts but also chose to start using the new management tools right away because of the power of the new tools, for example, the new PowerShell Provider. And although the compatibility layer provides excellent compatibility for legacy scripts, that compatibility is not 100%, and problems can occur.

URL Rewrite Module

The IIS 7.5 architecture, which provides an extensible and modular model, is significantly changed from the IIS 6.0/Windows Server 2003 era. Microsoft and others can write extensions that seamlessly integrate with the pipeline and with IIS Manager. The URL Rewrite module is a good example of this extensible model.

The URL Rewrite engine:

  • Is a rule-based URL-rewriting engine
  • Rewrites URLs based on HTTP headers and server variables
  • Has regular expression or wildcard pattern-matching (although there are performance hits when using very complicated regular expressions)
  • Can issue redirects, abort requests, and send custom status codes back to the client
  • Has support for IIS user and kernel mode output caching
  • Has support for Failed Request Tracing

The Operations team uses the URL Rewrite Module to delegate URL rewriting to application owners. Application owners can control their own distributed rewrite rules (in the application Web.config) for security and redirects. There are also global rules—the rules that the administrators retain control over. The Operations team uses the global rules for security purposes. For example, the team has had issues with certain traffic coming through and with the ability to match HTTP headers, they are able to send that attack traffic off to somewhere else. The Operations team also uses global rules for "vanity URLs"—URLs that are specific to some sort of event or a friendly name that people would intuitively type into a browser that would redirect to the actual application on microsoft.com. The URL Rewrite module is very valuable because it gives the team new architectural opportunities that it did not have before.

Application Request Routing

Application Request Routing (ARR) is another free, out-of-band IIS 7.5 module release. It is tightly coupled with the URL Rewrite module for rule-based routing. It is basically an application-level load balancer. The Operations team uses ARR with their hardware load balancers to manage server farms. It provides the opportunity to have applications in the same datacenter run on different platforms. ARR also supports Failed Request Tracing and has the ability to do proxying and edge caching, so it provides a lot of architectural possibilities.

Figure 3 shows how the Operations team uses ARR to create tiers of applications.

Figure 3. Using ARR to create tiers of applications in a single datacenter
Figure 3. Using ARR to create tiers of applications in a single datacenter

This figure, which is a representation of a single datacenter, is similar to Figure 1. In Figure 1, there was just a single tier of Web servers. These servers are monolithic architected clones that all have their own content set. They all have the same number of applications, which is a drawback for this type of architecture. ARR offers the potential to have different tiers. In this example, the Tier 1 servers are running ARR and Tier 1 applications that would be key high availability applications. These applications are not necessarily more important than other applications on microsoft.com but different applications have different performance needs and different availability needs. Tier 2 could include other applications—for example, legacy applications. These applications might not work well with the new versions of .NET or IIS, which would hamper Microsoft IT's dogfooding efforts. Having the applications run on different platforms gives the Operations team more flexibility and reduces memory and disk requirements. The team can provide a subset of the content on a set of servers instead of having a huge content set on all of the servers. Virtual machine density can also be increased because of the smaller footprint. The tradeoff for this flexibility is more complexity in the configuration and in the content publishing, but the Operations team is exploring this new scenario which is made possible by the architecture of IIS 7.5.

Virtualization

Virtualization is not specific to IIS 7.5 but it is an important part of Microsoft IT's Web architecture story. The Operations team uses Hyper-V in Windows Server 2008 R2 to increase the server density and consolidate a lot of the smaller Web sites and applications. The team does not see these benefits as much with microsoft.com but they can share some of the other sites and applications on a Blade server and realize better hardware utilization. Hardware utilization rates before virtualization were just 3-5% CPU across all of the servers in the enterprise. There were a lot of servers that were practically sleeping. Hyper-V enables the Operations team to stack some of those servers and mix large, small, and medium servers for better utilization, which results in huge cost savings, and provides improved flexibility, manageability, and agility.

Figure 4 shows some VM stacking examples.

Figure 4. Virtual Machine Stacking Examples
Figure 4. Virtual Machine Stacking Examples

With Windows Server 2008 R2, the Operations team uses Cluster Shared Volumes (CSV) to enable live migrations. CSV does not provide file content and sharing between hosts or VMs. The Operations team uses it specifically for the live migration scenario, moving VMs quickly between hosts without any perceived downtime. For example, if the team wants to patch a host, they can move VMs seamlessly to another host to patch them. This was also useful recently when the team encountered firmware issues. When the hardware for the host failed, the team moved the VMs quickly to a healthy host.

Performance

IIS 7.5 provides many features that can increase performance, such as dynamic and static compression and native output cache. For the move from IIS 6.0/Windows Server 2003 to IIS 7.0/Windows Server 2008, the Operations team saw a 10% improvement in efficiency in handling Web requests, which was pretty significant. With IIS 7.5/Windows Server 2008 R2, the Operations team is seeing the same benefits. They are not seeing any significant increase in performance over what they saw with IIS 7/Windows Server 2008, however.

Security

IIS 7.5 and Windows Server 2008 R2 provide increased security over and above IIS 7.0 and Windows Server 2008. The Application Pool identities in IIS 7.5 run the Application Pools under a unique account without having to create or manage domain accounts or local accounts, which are not quite as secure. The name of the account corresponds to the name of the Application Pool. So if an administrator creates an Application Pool called "DefaultAppPool", an identity is created under that name. The administrator can pull up the properties of the files and folder and can add "DefaultAppPool" as a user. So there is a very specific identity for the Application Pool that allows that partitioning and security layer from other applications that are running different Application Pools. For microsoft.com, the Operations team uses a slightly different architecture in that it is not a scenario where they have Brand A and Brand B on the same machine, and where Brand A cannot see Brand B. There are 12 Application Pools so there is a lot of sharing. Within a single Application Pool, there may be dozens of applications and it is not quite as critical for them to be isolated in their own security boundary. But as the team moves to a new architecture with ARR and is able to partition off high availability applications, some of those applications may need that kind of security. This is possible because that first tier can include fewer applications and fewer Application Pools.

Request Filtering, which was introduced in IIS 7.0, incorporates URLScan 3.0 functionality. The settings reside in applicationHost.config, which makes it very easy to manage. The errors are written to IIS logs with the new 404 sub-status codes so it makes it very easy to use a tool like Log Parser to find out where requests are being blocked when moving from URLScan Request Filtering. There is a Web Deploy urlScanConfig Provider that provides assistance for this task. But it is important to do some type of testing. Since the Operations team uses TinyGet and WCAT, the team parses the IIS logs to identify any blocked requests. If everything looks good, the team puts the server into live rotation and checks the logs again to make sure that everything that should be blocked is blocked and nothing is being blocked that is a valid request. This is a better approach than just putting the box into the live rotation. The team does not want to inadvertently block something that should not be blocked and cause a bad user experience.

Table 1 shows the Request Filtering status codes.

Status

Description

404.5

The Request Filtering Module rejected an URL sequence in the request.

404.6

The Request Filtering Module denied the HTTP verb of the request.

404.7

The Request Filtering module rejected the file extension of the request.

404.8

The Request Filtering module rejected a particular URL segment (characters between two slashes).

404.9

IIS rejected to serve a hidden file.

404.10

The Request Filtering module rejected a header that was too long.

404.11

The Request Filtering module rejected a request that was double escaped.

404.12

The Request Filtering module rejected a request that contained high bit characters.

404.13

The Request Filtering module rejected a request that was too long (request + entity body).

404.14

The Request Filtering module rejected a request with a URL that was too long.

404.15

The Request Filtering module rejected a request with a too long query string.

Table 1. Request Filtering status codes

All of the sub-statuses in the left column are for a specific type of rejection for the request. This makes it easy to use Log Parser to parse and sort these by status code to see why things might be blocked.

Diagnostics

IIS 7.5 includes the following diagnostic tools:

  • Failed Request Tracing
  • Detailed and actionable error messages
  • Run-time Status and Control APIs (RSCA) which offers the ability to look into an IIS 7.5 worker process and view the currently running request
  • Worker process performance counters (WAS_W3WP and W3SVC_W3WP)
  • Configuration logging

Failed Request Tracing enables server administrators to define error conditions that they want to monitor. This is a very valuable tool that the Operations team uses to diagnose a request at every step through the request pipeline. The Operations team is able to run Failed Request Tracing, even real time in production with very minimal performance impact. The trace events are buffered and it only writes to disk if it matches the defined criteria—for example, a specific status code or a specific URL. The Failed Request Tracing log provides a lot of detail. The log points the team in the right direction so they know where to look to find the problem in the request pipeline.

Configuration Logging was introduced in IIS 7.5. It is not enabled by default. There are four different logs with different auditing capabilities, which provide a lot of information. If an administrator runs into problems with configuration, this is a good place to start that was not available before.

Configuration

The configuration architecture itself has not changed significantly from IIS 7.0 to 7.5. It is a very simplified architecture where applicationHost.config is the root file of the IIS 7.5 configuration system. Distributed configuration in IIS 7.5 significantly reduces administrative overhead for the Operations team, however. The team receives dozens of requests from microsoft.com users for various things (for example, default documents or new redirects). With tools like the URL Rewrite module, the team can delegate these tasks to users so that they change the settings in their own application Web.config files. Configuration also allows for Xcopy deployment, which is a huge time saver for the Operations team. They can simply copy an applicationHost.config to another server to create an identical IIS configuration.

Administrative Tools

IIS 7.5 includes the following administrative tools in addition to the IIS 6.0 administrative tools:

  • AppCmd. AppCmd was introduced in IIS 7.0 and is also available in IIS 7.5.
  • PowerShell. There are more than 100 new PowerShell cmdlets in Windows Server 2008 R2 and at least 50 cmdlets in the IIS 7.5 Provider. PowerShell now also has the ability to do remote management.
  • IIS 7.5 WMI Provider. IIS 7.5 includes a new WMI Provider that enables WMI developers to take advantage of the new IIS 7.5 features. The WMI Provider allows for automating management tasks with scripts written in .NET or VBScript.
  • IIS 7.5 Manager. The new administration console, IIS 7.5 Manager, provides an extensible integrated view of managing Web and FTP sites on the Web server both locally and through remote administration.
  • Microsoft.Web.Adminstration API. This API allows developers to manipulate server configuration and access runtime information from managed code. This includes creating new applications, recycling applications, and listing currently executing requests.
  • Best Practices Analyzer (BPA). Like the BPA for Microsoft Exchange Server and other Microsoft products, the BPA for IIS 7.5 looks at settings to see if they fall within certain best practices and flags things that could possibly be improved.

Conclusion

The Microsoft IT Operations team is tasked with maintaining high availability ratings (99.9%) for the microsoft.com Web site at the same time that the team introduces early versions of IIS, Windows Server, and the .NET Framework into the environment. The Operations team developed a deployment process where they use tools such as WCAT and TinyGet (wrapped inside PowerShell) to test and stress servers that are pulled form live rotation.

The Windows Server 2008 R2 and IIS 7.5 platform has enabled the Operations team to move in a new direction in terms of architecture because of the platform's modularity and extensibility. For example, by using the URL Rewrite module, the team can distribute configuration to application owners and developers. The ARR module provides the ability to do application-level load balancing and have applications run on different platforms in the same datacenter.

IIS 7.5 also offers many performance, diagnostic, security, and administrative enhancements that the Operations team uses to help maintain high availability ratings for microsoft.com.

For More Information

The following table lists additional resources for IIS 7.0, IIS 7.5, Hyper-V, and Windows Server 2008 R2.

URL

Description

http://www.IIS.net

 

Community portal that provides a wealth of resources, including training, downloads of new extensions and modules (about 20 created by the IIS team including URLRewrite, AAR, and others) and forums.

http://technet.microsoft.com/mscomops/default.aspx

Inside Microsoft.com: The Microsoft.com Engineering Operations Team

http://forums.technet.microsoft.com/en-US/mscomops/threads/

Microsoft.com Engineering Operations: TechNet Forum

http://blogs.technet.com/mscom/

Microsoft.com Operations Blog

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032415084

How Microsoft is Architecting the Virtual Application Infrastructure

http://technet.microsoft.com/library/dd578346.aspx

Best Practices for Deploying Virtual Machines by Using Hyper-V Virtualization Technology

http://technet.microsoft.com/library/cc974012.aspx

How Microsoft Designs the Virtualization Host and Network Infrastructure

http://technet.microsoft.com/library/dd557540.aspx

SQL Server Consolidation at Microsoft

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Hyper-V, PowerShell, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft