Note on IT

Published: February 17, 2006

Microsoft.com is one of the busiest yet most available sites on the Internet. Standard server definitions and an automated server build and update process are critical factors in maximizing site availability and scalability.

Introduction

The Microsoft.com operations team operates one of the busiest Internet Information Services (IIS)-based Web sites on the Internet with daily activity of more than 70 million page views resulting in more than 10,000 Hypertext Transfer Protocol (HTTP) requests per second. That capacity is distributed over 80 identically configured Web servers that are grouped into 10 network load balanced (NLB) clusters of eight servers each, with a variety of supporting database servers. The IIS version 6.0 configuration of the Microsoft.com Web site includes more than 350 virtual roots and 190 Web applications across 12 IIS application pools.The operations team for Microsoft.com takes great advantage of the scripting capabilities of Microsoft® Windows Server™ 2003 to synchronize content and server and application configurations across the Web servers. All of the Web servers for Microsoft.com are designed to be identical. The database servers vary in configuration depending on specific database or application requirements, but the base operating system and general Microsoft SQL Server™ configurations are standardized as much as possible through the use of scripts. Scripting initial configurations and all subsequent updates to the servers provides a method to efficiently build and deploy new servers, and to roll out consistent changes across the entire population of servers.

This document describes the methods and provides samples of commands and scripts used at Microsft.com to perform these tasks. This document assumes that readers are engineers or administrators who use IIS or SQL Server and are already familiar with Windows Server 2003, Windows Script Host (WSH) and VBScript, and IIS 6.0 or Microsoft SQL Server 2005.The process of monitoring Microsoft.com servers after they have been deployed is discussed in more detail in the IT Showcase technical case study Monitoring and Troubleshooting Microsoft.com, available at http://www.microsoft.com/technet/itsolutions/msit/

An IT Showcase white paper titled Microsoft.com Moves to x64t Version of Windows includes a discussion on the operating system selection for Microsoft.com Web servers and is available at http://www.microsoft.com/technet/itsolutions/msit/.Note: For security reasons, the sample names of forests, domains, internal resources, organizations, and internally developed applications and files used in this document do not represent actual names used within Microsoft and are for illustration purposes only. In addition, the contents of this document describe how the Microsoft.com operations team runs its enterprise data center. The procedures and processes included in this document are not intended to be prescriptive guidance on how to run a generic data center and may not be supported by Microsoft Customer Support Services.

Note: For security reasons, the sample names of forests, domains, internal resources, organizations, and internally developed applications and files used in this document do not represent actual names used within Microsoft and are for illustration purposes only. In addition, the contents of this document describe how the Microsoft.com operations team runs its enterprise data center. The procedures and processes included in this document are not intended to be prescriptive guidance on how to run a generic data center and may not be supported by Microsoft Customer Support Services.

Standard Servers Definitions for the Data Center

Microsoft.com is hosted in data centers managed by Microsoft Network® (MSN.) The MSN engineering team has established a limited set of standard server definitions with a preloaded standard base operating system build for use within their data centers, including those used by Microsoft.com. As a customer of the standard server offerings, Microsoft.com uses servers that are ordered from those predefined configurations. MSN has adopted the use of predefined configurations, known as the cookie cutter approach, to reduce the amount of effort required to provide ongoing support for servers. This minimum number of server configurations takes into account the requirements of different server types, such as Web servers versus very large database servers. There are exceptions, however, most notably when a custom server configuration would provide specific agility to meet important business goals, but those exceptions are kept to a minimum.

Preloading a standard base operating system build is an excellent way to make sure that all servers have been hardened with consistent security configurations and include the administrative tools or applications common to all servers. When the standard build is kept current with the latest security updates, new servers are not more secure when they are first attached to the network.

For the same reasons that MSN has adopted a cookie cutter approach to acquiring and deploying servers companywide, the Microsoft.com operations team strives to further minimize the number of server configurations in its own environment.

In general, the Microsoft.com operations team has standardized on a single server model for most Web and database servers supporting the site. The current standard is an HP Proliant DL 585 configured with the following features:

• Four 2.2-gigahertz (GHz) AMD Opteron x64-based CPUs

• 16 gigabytes (GB) of RAM

It is important to note that all of the Web servers supporting www.microsoft.com have been upgraded to the x64 version of Windows, which requires 64-bit capable CPUs. The x64-based platform, which also has strong 32-bit capabilities, has enabled a seamless migration and greatly increases the operating system's ability to make use of larger amounts of RAM. Overall, the resource utilization on the Web servers decreased 50 percent by moving to 64-bit versions of Windows, even though the site still consists of 32-bit applications. Specific design features of the x64-based platform provide the versatility to run 32-bit versions of Windows with little or no performance degradation compared to 32-bit platforms. In fact, 32-bit applications running on 64-bit versions of Windows may actually perform significantly better because of the increased virtual memory allocations. For more information about why and how the Microsoft.com operations team moved to 64-bit versions of Windows, refer to the IT Showcase white paper titled Microsoft.com Moves to x64 Version of Windows at http://www.microsoft.com/technet/itsolutions/msit/.

Note: Intel also produces CPUs compatible with the x64-based version of Windows Server 2003. Intel refers to its CPU as EM64T.

The standard base operating system build that comes preloaded on the servers is available in the following variations that are most often used for servers in the Microsoft.com environment:

• 64-bit operating system (x64-based platform) with base IIS configuration (for Web servers)

• 32-bit operating system with base SQL Server configuration (for database servers)

• 64-bit operating system with base SQL Server configuration (for database servers)

Microsoft.com Modifications

After a server is received with the standard base configuration, the Microsoft.com operations team must make modifications and additions to the server to ready it for deployment to production. For the same reasons that the base configuration is a standardized definition and its deployment is automated to ensure consistency, the operations team has also developed standard definitions and automated processes to initially configure and subsequently update servers consistently with settings specific to Microsoft.com.

Scripting is at the heart of the automated processes for configuring and updating Microsoft.com servers. Scripting is used across the Microsoft.com family of Web sites to set baseline IIS configurations. These settings include IIS application pool properties, Web extensions, logging, and Internet Server API (ISAPI) filters to name a few. Where possible, the team packages the scripted tasks that set those values into a simple loop function to provide the ability to configure multiple servers simultaneously.

The baseline configuration is defined in an XML manifest. The information contained in the manifest provides the specific configuration parameters to the automated scripts. Updating the baseline configuration requires updating only the detail in the manifest, not the scripts themselves.

Changes required after the initial build are also made by the same scripting process. The scripting process can update multiple servers simultaneously with changes that were made to the XML manifest. By using the same scripting process and manifest for updates, the operations team ensures that updates are always incorporated into the build process for new servers.

There are two main definitions for servers supporting www.microsoft.com: Web servers and database servers. The Web server definition is a single standard because all Web servers within the site are intended to be identical by design. The detailed configurations for the database servers vary depending on specific application or database requirements, but the automated deployment process for basic configurations is the same for all database servers.

IIS 6.0 Web Server Configurations

Microsoft.com Web servers are identically configured to host 350 virtual roots and 190 IIS Web applications across 12 application pools. Before applying content and configurations specific to those applications, the specific baseline system level and global IIS level settings are applied to the standard server build for any Web server after the Microsoft.com operations team receives it. After the team configures baseline settings, it applies site-specific settings and content. When the server build process is complete, the settings listed in Table 1 will have been applied.

Table 1. IIS 6.0 Web Server Settings

The primary scripting tool used to automated IIS configurations is Adsutil.vbs. Adsutil.vbs is a useful tool that shipped with IIS 6.0 and uses VB Script in conjunction with Active Directory Scripting Interface (ADSI) to manipulate IIS configurations by modifying the IIS metabase. It can be used to automate many Web site administration tasks both locally and remotely. Adsutil.vbs requires CScript. CScript.exe can be started each time adsutil.exe is used, or the WSH can be configured to run CScript.exe as the default scripting host without a sign-in banner and in interactive mode, by issuing the following command.

c:\windows\system32\cscript //H:cscript //I //NoLogo //S

To get a full list of adsutil.vbs features and syntax, issue the following command:

adsutil.vbs HELP

Be careful when using adsutil.vbs because the IIS metabase is not backed up automatically. For security reasons, do not store the tool on the Web server itself. Instead, use the remote execution capabilities of the command.

The main commands for adsutil.vbs include:

• ENUM, which enumerates all parameters for given metabase path.

• GET, which displays a specific parameter.

• SET, which assigns a new value to a specific parameter.

• DELETE, which deletes a given path or parameter.

The following sections include specific examples of tasks that use these commands, which can be scripted as part of the automated configuration process.

Application Pool Settings

The initial IIS 6.0 application pool settings can be complex, and are frequently modified. Use Adsutil.vbs to create application pools in IIS, view their properties, and change application pool membership or recycle thresholds.

To remotely create an application pool, run the following command.

adsutil.vbs CREATE W3SVC/AppPools/<apppool_name> "IIsApplicationPool" -s:<server_name>

To remotely create an application pool on multiple servers, create a servers.txt file that lists server names and run the following command.

for /f %i in (c:\servers.txt) do adsutil.vbs CREATE 
    W3SVC/AppPools/<apppool_name> "IIsApplicationPool" -s:%i

Note: The -s switch instructs adsutil.vbs to execute the action on a remote server and is not found in adsutil.vbs HELP.

The resulting application pool structure can be programmatically viewed on a remote Web server by running the following command.

adsutil enum W3SVC/AppPools -s:<server_name>

To view a specific application pool on a remote Web server, run the following command.

adsutil enum W3SVC/AppPools/<apppool_name> -s:<server_name>

After the application pool structure is created on a Web server, specific Web applications can then be moved to it. Adsutil.vbs can be used to move a Web application to a new application pool by changing the metabase AppPoolId entry.

To remotely reassign an application's application pool membership, run the following command where 1 indicates the first Web site on the server, root indicates to start in the top level of site hierarchy.

adsutil.vbs SET W3SVC/1/root/<web_app>/AppPoolId "<apppool_name>"-
   s:<server_name>

After the applications are in the proper application pools, adsutil.vbs can be used to configure the application pool recycle settings. The default application pool recycling setting is based on elapsed time; however, the operations team prefers to modify the application pool recycling setting to be based on virtual memory limits due to the large volume of traffic and the characteristics of the code running in each of the application pools on Microsoft.com. For the current 64-bit Web servers with IIS configured to run 32-bit worker processes, the operations has configured a recycle to occur when an application pool reaches a virtual memory threshold of 4 GB. Keep in mind that these settings are tuned for a specific environment and they may not be appropriate for all environments. When the Microsoft.com site consisted of 32-bit Web servers, the application pool recycle limit was set at 2 GB. For more information about virtual memory differences on the x64-based platform, refer to the IT Showcase white paper Microsoft.com Moves to x64 Version of Windows at http://www.microsoft.com/technet/itsolutions/msit/ .

To remotely set the virtual memory recycle limit to 2 GB for all application pools on a 32-bit Web server, run the following command.

adsutil SET W3SVC/AppPools/PeriodicRestartMemory 2048000 -
   s:<server_name>

To remotely set the virtual memory recycle limit to 2 GB for a specific application pool on a 32-bit Web server, run the following command.

adsutil SET W3SVC/AppPools/<apppool_name>/PeriodicRestartMemory 
   2048000 -s:<server_name>

Other Scripted Tasks for Web Servers

Many administrators manage SSL certificates manually. The IIS 6.0 Resource Kit script IISCertDeploy.vbs can be used to programmatically import and bind an existing certificate to an IIS Web site.

To bind an SSL certificate to a Web site, run the following command.

IISCertDeploy.vbs -c WebServerCert.pfx -p CertPassword -q ON

Gacutil.exe in the %systemroot%\Microsoft.Net\<framework_version> folder can be used to register .NET Framework assemblies in the GAC. Gacutil.exe does not have a remote execution capability, so another tool such as Sysinternal's PsExec must be used to execute the command on remote servers. After inclusion in the GAC, any application in any pool can use the assemblies.

To register filename.dll in the GAC, run the following command.

gacutil.exe /if c:\temp\app_name\filename.dll

URLScan is another critical component that the operations team installs on all Microsoft.com Web servers. URLScan is an efficient ISAPI filter that drops anomalous or predefined requests to a Web application. As part of a defense-in-depth strategy, URLScan can be configured uniquely for each Web application, enabling only the specific HTTP verbs and extensions required by a particular application. URLScan can also be configured in alert mode instead of disallow mode to observe what type of traffic would be prohibited in disallow mode.

At Microsoft.com URLScan version 2.5 is a standard part of every Web server, and the operations team has written a script called URLScanSetup.vbs to automate its configuration. URLScanSetup.vbs is included in the sample code download that accompanies the article Inside Microsoft.com at http://www.microsoft.com/technet/technetmag/issues/2005/11/InsideMSCOM/default.aspx

To use URLScanSetup.vbs to configure a specific Web site, run the following command.

URLScanSetup.vbs /SITE:YourSiteName

Deployment of Web Server Configurations

For new servers, deployment to the Microsoft.com production environment requires adding the server to an existing NLB cluster, and then observing the server as it begins to handle live traffic. If any problems are observed, the operations team can easily remove the server from the cluster for offline troubleshooting.

Deploying changes to multiple servers is a more complex process and begins with modifying the XML manifest that defines the reference, or golden server configuration. The update scripts are then applied in a pre-production environment to test the site functionality prior to a full production deployment. After being validated, the updated configurations are deployed to all servers in an NLB cluster using scripts containing FOR loops, one cluster at time. Web content updates are also published in this fashion.

Deployment steps for updating multiple production servers occur in the following sequence:

1. Deployment to staging server
   A. Copy content
   B. Synchronize IIS metabase settings
   C. Test new settings

2. Deployment to production NLB clusters
   A. Remove NLB clusters from global load balancing one at a time
   B. Copy content to all servers in cluster
   C. Synchronize IIS metabase settings to all servers in cluster
   D. Test new settings
   E. Return NLB clusters to global load balancing
   F. Monitor servers in the cluster to confirm operations before proceeding to the next cluster

In IIS 6.0, the metabase can be edited live, but can't be fully replaced live without a service interruption. Also, it is also important to understand metabase settings in detail, being careful not to overwrite server-specific data when pushing changes to multiple servers. Only generic, nonserver-specific settings should be pushed to a group of servers.

The following is a sample of a FOR loop that uses xcopy.exe to copy initial content from a source to an NLB cluster of multiple servers.

c:\scripts\for /f %i in (servers.txt) do xcopy 
   \\sourceserver\share\*.* \\%i\webroot\

Note: The operations team uses xcopy.exe to copy content to servers when they are initially built. Other methods such as the use of Microsoft Content Management Server are used to publish content updates and new content on a continuing basis to the Web servers.

While xcopy.exe is used to copy initial content, iiscnfg.vbs, a script included with IIS, is used to synchronize metabases updates across multiple servers. This script tool can be used to export a known good IIS metabase configuration to an XML format and then deploy or import those settings to a remote server. Some of the common parameters used with this command include /sp, which specifies the metabase path of the Web site; /children, which is used to export all sub keys of that configuration path; and /inherited, which is used to include inherited properties in the export file.

The following code sample shows how to export the metabase to a local XML file.

iiscnfg /export /f c:\scripts\IIS.xml /sp /lm/w3svc/1 /children 
   /inherited /d Y0urP@$$Wo5d

The resulting XML file then needs to be copied to the target servers. Use the following xcopy.exe command.

c:\scripts\for /f %i in (servers.txt) do xcopy 
   \\sourceserver\share\IIS.xml \\%i\webroot\

Alternatively, use IIScnfg.vbs to copy the metabase directly to a single target server using the following command.

Iiscnfg /copy /tu domain\user_name /tp P@$$Wo5d /ts <server_name>

After copying the configuration to a group of target servers, the known good metabase settings must be imported to the server or group of servers.

To import the settings to a group of servers, run the following command.

c:\scripts\for /f %i in (servers.txt) do iiscnfg /import /f 
   \\%i\scripts\IIS.xml /sp /lm/w3svc/1 /children /inherited /d 
   Y0urP@$$Wo5d

Note that this command will replace the target server's metabase settings, which causes a service interruption, so make sure that the target servers are taken offline before making this kind of change.

After the content and metabase settings have been synchronized on the target cluster of servers, be sure to test the servers for functionality and ensure that they have the appropriate level of monitoring in place to send alerts if any problems arise. We also recommend analyzing IIS and event logs for errors after making any changes to the IIS metabase.

Note: IIS 7.0 will make it possible to replace metabase settings by simply copying the new configurations, much like pushing content. This feature will eliminate the need for many of the preceding scripted actions that use iiscnfg.vbs.

Management Commands for Web Servers

After the operations team deploys custom site configurations and the servers are handling live traffic, it is necessary to administer the Web servers remotely from time to time, to perform such common tasks as managing the state of the Web service and recycling application pools.

There will also be times when administrators must start, stop, pause or unpause specific Web sites. Use the adsutil.vbs tool to accomplish these tasks with ease. For example, to start a specific Web site, run the following command.

adsutil.vbs START_SERVER W3SVC/1

In the preceding sample code, the path that follows the START_SERVER command, W3SVC/1, specifies the specific Web site to be started. The W3SVC portion of the path tells the script to access the Web service. The value of 1 tells the script to specifically start the first Web site on that server. To run this command on a remote server or cluster, add the -s switch as in previous adsutil.vbs examples.

To remotely start the first Web site on a single server, run the following command.

adsutil.vbs START_SERVER W3SVC/1 -s:<server_name>

To remotely start the first Web site on a group of servers, run the following command.

for /f %i in (server.txt) do adsutil.vbs START_SERVER W3SVC/1 -s:%i

In addition to START_SERVER, other adsutil.vbs management options include STOP_SERVER, PAUSE_SERVER, and CONTINUE_SERVER.

SQL Server 2005 Database Server Configurations

Unlike Web servers where configurations are standardized across all servers, database servers are not configured to all be identical, although they do share many common settings. The base configuration settings that are configured by using scripts for SQL Server 2005 database servers that support Microsoft.com include:

• Direct attached storage configuration

• Master database location and settings

• TempDB location and settings

• SQL Server memory and CPU settings

Direct Attached Storage

A standard database server for Microsoft.com typically includes 240 GB of direct attached storage. The storage is generally allocated as shown in Table 2.

Table 2. Database Server Storage Configuration

MasterDatabase and MSDB

The operations team makes several changes to the master database configuration. The master database is installed on the D: volume. The primary data file size is set at 100 MB with autogrowth set to off and is configured to contain only system tables. The transaction log size is increased to 40 MB with autogrowth set to off. The msdb data file is also increased in size to 500 MB and 100 MB for the log file with autogrowth set to off for both the data and log files.

TempDB

TempDB is moved to the T: volume, and one data file is created for each physical CPU, plus one log file. The log file is configured to be 25% of the total space available on the T: volume and the data files are configured to be of equal size, consuming the remainder of the space.

The following is a sample of a script for a 4 CPU database server with approximately 130 GB of space dedicated to the TempDB Volume:

ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Data1',
SIZE = 25000MB)
GO
ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Data1',
MAXSIZE = 25000MB)
GO
ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Data1',
FILEGROWTH = 0MB)
GO
ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Log1',
SIZE = 25000MB)
GO
ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Log1',
MAXSIZE = 25000MB)
GO
ALTER DATABASE tempdb
MODIFY FILE (NAME = 'TempDB_Log1',
FILEGROWTH = 0MB)
GO
ALTER DATABASE tempdb
ADD FILE
( NAME = 'TempDB_Data2',
FILENAME = 'T:\MSSQL.1\DATA\TempDB_Data2.NDF',
SIZE = 25000MB,
MAXSIZE = 25000MB,
FILEGROWTH = 0MB) 
GO
ALTER DATABASE tempdb
ADD FILE
( NAME = 'TempDB_Data3',
FILENAME = 'T:\MSSQL.1\DATA\TempDB_Data3.NDF',
SIZE = 25000MB,
MAXSIZE = 25000MB,
FILEGROWTH = 0MB) 
GO
ALTER DATABASE tempdb
ADD FILE
( NAME = 'TempDB_Data4',
FILENAME = 'T:\MSSQL.1\DATA\TempDB_Data4.NDF',
SIZE = 25000MB,
MAXSIZE = 25000MB,
FILEGROWTH = 0MB) 
GO

Memory and CPU

The current standard database server for Micorosoft.com contains four 2.2 GHz AMD Opteron x64-based CPUs and 16 GB of RAM. The configuration setting max server memory is configured to use only 14 GB, reserving 2 GB for dedicated operating system use.

Microsoft.com does have several older 32-bit servers in production. If a 32-bit server is capable of hyper-threading then it is enabled, and the configuration setting max degree of parallelism is configured to be equal to the number of physical CPUs in the server.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:
http://www.microsoft.com
http://www.microsoft.com/technet/itshowcase