Dense Hosting Scenario (Web and Application Server Infrastructure - Performance and Scalability)

Applies To: Windows Server 2003 with SP1

This scenario is the opposite of the Dedicated Application scenario. The objective for the Dense Hosting scenario is to get as many different applications or sites on a physical server as possible, with only one operating system instance to manage, to minimize hardware and management (traditional TCO) costs. This is by far the most efficient way to run. Any form of virtualization and operating system partitioning36 in this scenario is giving away precious processor cycles and RAM resources, which is contrary to the objective of cheap, efficient processing.

This form of dense hosting is prevalent at Internet Service Providers (ISPs), and can also be found in some enterprise environments where a central IT group identifies many non-critical applications executing on separate servers and brings them into a central environment. Note the term non-critical application: hundreds or thousands of applications can be hosted on a single piece of hardware, but should a hardware failure occur, many applications would experience some form of downtime while the servers hardware was being rectified. In an enterprise environment, this could be an extremely cost-effective way of providing service to a number of Tier 3 applications, where a 99.999 up time SLA (Service Level Agreement) is not required.

Windows Server 2003 has been tested with a 20,000+ defined applications or sites per operating system instance. This means that the server can easily support that number of applications defined. It does not mean that if these applications all have high resource needs, it will support them. If one of the 20,000+ application is not gated in its resource usage, it can consume 100 percent of the resources on the server, not allowing the other applications to function efficiently.

Considerations

This section contains relevant considerations for the Dense Hosting Scenario.

Application Pools Shared vs. Independent

One of the central tenets to be observed in this scenario is the level of application isolation. Note that the fundamental unit of isolation in Windows is the process. The process object in the Windows kernel offers operating system and processor instruction protection from one process to another. If one application faults in process A, another application running in process B will not suffer due to application As fault37.

In IIS, the fundamental unit of isolation is the application pool. Administrators need to determine whether all applications/sites need to be put in their own application pool or can some applications share an application pool with other applications.

If you choose to have multiple applications sharing the same application pool, there is a chance that, at some stage, some of them will experience downtime when one application has an issue.

Running Different Application Types Side-by-Side or Within the Same Process Space

Windows Server 2003 offers the ability to mix different kinds of application types together in the same application pool. This means that you can take an existing ASP/COM+ application and run it in the same application pool as an ASP.NET (or managed) application. This also means that you can host native, compiled applications (ISAPI or Win32) in the same application pool as these other application types.

Note that running many different application types in the same process can save system resources (memory and CPU processing), as there is a fixed cost per process for these resources in the Windows operating system. Imagine you had two ASP.NET and one classic ASP/COM+ applications. By pooling these three applications together in the same application pool, you get the following memory footprint:

All Applications Pooled in Single Process	Each Application Per Process
Base Process / IIS fundamentals	Base Process / IIS fundamentals
ASP.NET/CLR Runtime	ASP.NET/CLR Runtime
ASP Runtime	ASP.NET Application 1
COM+ Runtime	Base Process / IIS fundamentals
ASP.NET Application 1	ASP.NET/CLR Runtime
ASP.NET Application 2	ASP.NET Application 1
ASP/COM+ Application	Base Process / IIS Fundamentals ASP.NET/CLR Runtime ASP.NET Application 2 Base Process / IIS Fundamentals ASP Runtime COM+ Runtime ASP/COM+ Application

As you can see, to run the same three applications in separate application pools, you incur a substantially different set of costs in the isolated case vs. the shared case. But, the reliability tradeoff can be a factor in the decision of which application environments you configure in a shared or isolated way.

From an administrative perspective, the decision can be based on a number of different factors:

• End customer. You might decide that each business unit needs to run all of their applications in the same application pool because an application failure would likely be the fault of one of their applications. This would put the emphasis on each business unit to correct their own applications so that they do not incur outage themselves. In this case, the other business units running on the server would not incur outage, because they would be running in different application pools. For an ISP scenario, replace the term business unit above with customer, so that all applications from one customer run together in one application pool.

• Known, problematic applications. You might start off with a policy to keep all applications running together in the same pool to minimize the resources utilized, and when applications start to display issues (or failures), you could partition out the problematic applications. It is quite possible to segregate a specific virtual directory containing a particular application into its own application pool, with the rest of the related applications executing in a separate application pool.

• Everything isolated until the limit. In this scenario, all applications/sites are put in their own isolated application pools. Then, when the server resource utilization exceeds the administrative policy defined for servers in that environment, you consider which applications/sites could be pooled together. An effective strategy at that stage could be to start pooling together ASP.NET (or managed code) applications into the same application pool, or start pooling ASP-only applications into one application pool.

All Application Pools Running with Unique Identity

Another benefit of the IIS 6.0 re-architecture is that the administrator gets to assign the base identity of a worker process. This does not mean that custom code running in the worker process will run with this identity. IIS will still impersonate the caller when calling custom code (ASP, ASP.NET, ISAPI) if the caller is authenticated, or, if the caller is anonymous, IIS will run the custom code under the IUSR_<computername> identity. The application pool identity will be used when the IIS code is running in the worker process.

The default identity is the intrinsic account Network Service, which has rights approximating a member of the Users group of a server, but this identity can be set on a per application pool basis (Figure 9).

Figure 11: Worker process identity tab on application pool dialog

One consideration that needs to be made clear is that there is a limit to the number of concurrently running desktops in Windows Server. In order to scale this to hundreds or thousands of different identities running concurrently, the following registry key can be set to allow the different identities to share the same desktop.

HKEY_LOCAL_MACHINE\System\Currentcontrolset\Services\w3svc\UseSharedWPDesktop to be set to (DWORD)1.

Number of Worker Processes

Depending on the isolation policy for your application or sites, as discussed above, Windows Server 2003 offers some additional tuning parameters to assist in effective resource management (Windows Server 2003 tries to only have applications running if they are processing actual load).

As discussed in the TCP/IP section, physical TCP/IP connections are managed by HTTP.sys, that is, all connections are held by the System process: PID 4. The effect of this is that, if Windows Server 2003 has 20,000+ applications defined on it, it can be listening on the endpoints and can accept TCP/IP connections for any one of those sites, regardless of whether the actual application is physically initialized. In fact, Windows Server 2003 can accept hundreds of connections without even having one physical worker process running.

The other side of this equation is that Windows Server 2003 can idle out the running application if it has been idle, without dropping TCP/IP connections that have sent requests to that application. This is a very useful feature for administrators looking to trim resource usage on a server.

Demand Start and Idle Time-out

Windows Server 2003 always performs a demand start for applications. When IIS 6.0 initializes, no worker processes are running, they only start when a client request for the application pool is received. The default idle time-out for application pools is 20 minutes. This means that if a set of applications running in a specific application pool has not received one request in 20 minutes, IIS 6.0 will spin down the worker process and give those resources back to the operating system.

Administrators can make this setting more aggressive. For applications that have very infrequent use, the idle time-out setting can be set to five or less minutes. Idle time-out can be found on the Performance tab of the Application Pool properties dialog box, or can be set through the adsutil command-line utility:

for a specific application pool:

cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs set W3SVC/AppPools/ApplicationPoolName/IdleTimeout 5

or, for all application pools:

cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs set W3SVC/AppPools/IdleTimeout 5

Process Gating

Some administrators prefer to simplify the administrative model and have all applications or sites totally isolated from each other. This configuration can greatly simplify administration and overall management; however, a downside with this approach can be forecasting the specific traffic pattern and usage of all applications (unless you have a small number of total application pools). This means that there can be a potential issue in having an excessive38 number of concurrently-running processes in the system at a given time.

Therefore, Windows Server 2003 offers a process gating feature, which allows an administrator to set a high water mark for the number of active processes in the system at any given moment. The administrator can simply reset this value to be appropriate for the type of hardware the administrator has (the number of concurrent processes Windows Server 2003 can support is mostly limited by the amount of physical RAM in a system).

How does process gating work?

When a request is sent to an application pool that has not been started (or has idled out), IIS 6.0 makes a check of the number of concurrently-running processes. If the total number of processes, including the new one, would go over the threshold, the worker process is not started and a 503 Service Unavailable error response is sent to the request originator.

The process gating feature is not available through the IIS UI management tool. It is set in the IIS 6.0 metabase only. To override the process gating feature to have a maximum of 100 concurrently active application pools, run the following command at the Windows command prompt:

cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs set W3SVC/DemandStartThreshold 100

to remove the override to get IIS 6.0 back to its standard behavior, run the following command:

cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs delete W3SVC/DemandStartThreshold

IIS Logging Considerations with Thousands of Sites per Server

On IIS 6.0, whenever a site is defined, W3C logging is established for that site. In order to not affect performance, every log file generated gets a 64K buffer assigned to avoid writing log entries to disk too often. This way, many log records are buffered and all written at one time.

When thousands (or tens of thousands) of sites are defined on a server, the memory use due to these log buffers can be substantial (64K per site). Therefore, to conserve memory, there is a way to control the size of the log buffers. Obviously, when the log buffers are decreased in size, the frequency of I/O to the disk is increased, but for sites that get infrequent hits, this may not be a massive impact on performance.

The parameter to manipulate the size of all log buffers on Windows Server 2003 is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Http\Parameters\LogBufferSize (DWORD)

with a range of: 12,288 (12K) 65,536 (64K). The value the parameter is set to must be divisible evenly by 4,096, otherwise it will be discarded and the 64K default will be used.

Centralized Binary Logging

When running thousands of sites on a server experiencing high request rates, the centralized binary logging feature is very useful. Centralized binary logging alleviates a disk I/O bottleneck; it makes the log file size smaller and saves CPU in formatting the log file information.

From a functional perspective, there are some restrictions on the centralized binary log file format (in terms of the fields logged), which cannot be changed. For more details on the binary logging implementation and comparison data, see the New Format in IIS 6.0 Centralized Binary Logging section.

Connection Management Discussion

Running a server with many sites can cause the server to hold onto many connections. The reason for this is that the majority of clients today are browsers. Most browsers send HTTP requests over the HTTP 1.1 protocol with a Connection: KeepAlive header in the request. This directive tells the server that once the browser->server connection is established, it should remain open. The reason for keeping the connection open is to forgo the connection establishment and latency for subsequent requests.

However, from a server perspective, having many connections established (tens of thousands per server possible) means that the server is investing some CPU in managing those connections, and the network infrastructure has an additional load in sustaining them. For these reasons, many very large implementations turn off keep alive connections at the server end, which means that exactly one request is sent per connection. Browsers and other clients need to go through the session establishment process for every request.

Administrators should gain an understanding of how their production servers behave with the KeepAlive header off as well as with it on. The resources to monitor are CPU, memory, network utilization, and latency at the client end. The right decision is based on operational policy for the organization.

For information about how to disable and enable HTTP Keep-Alive connections, see Enabling HTTP Keep-Alives and AllowKeepAlive in the IIS 6.0 Help.

IP Addresses or Host-headers

When placing many sites on Windows Server 2003, it is important to have a strategy for how sites are to be addressed. If the server is to support thousands of sites, the preferred mechanism to address them is via host-headers, rather than TCP/IP addresses or different ports.

For a detailed discussion of the considerations behind the way sites are addressed, see the TCP/IP Configuration and Connections section.

Remote Content Considerations and Tuning

Before you set up an IIS server that will have hundreds or thousands of sites on it, you must first decide where the content is going to be located. IIS 6.0 supports both local and remote content models. Local means that the physical files that make up each site are located on a disk attached to the system. Remote means that the content files are physically located on a remote file system (file server, NAS device, etc.)

The advantage to having remote content in this environment is that it becomes even more straightforward to reorganize which server is handling the requests for a specific site. If there is a need for reorganization, the server configuration for the site and DNS needs updating. The physical content is untouched and unmoved. If the content was local, it would require moving, and backing up that content might also require reconfiguration.

Even though the remote content case looks good, there are some performance and scalability considerations when using content on a remote share such as latency on content and access patterns. The Remote (or Centralized) Content section also introduces a discussion regarding the caches and parameters to control for the caches.

Latency

When content is hosted on a local server, the average access time of a piece of content is usually measured in milliseconds. When a content request is going across a network (even a very fast one), you might get to the content in milliseconds, or it might take seconds. This can have a dramatic impact on the responsiveness perceived by the user. Of course, once IIS 6.0 has seen the content, it will try to optimize its access patterns to make subsequent accesses much more efficient.

Data Volume

Similar to IISs ability to saturate several gigabit network cards, with the data being sent back to many thousands of clients, it is also possible to saturate the network between an IIS server and a NAS (file server or other computer), and hold the raw content. If a network card on either the Web/application server or shared content server is saturated, the overall performance of all the servers accessing that central server can be affected. This must be planned for in your design.

Questions to Answer When Testing a Remote, Centralized Content Scenario

• What is the likely profile of requests over a period of time to the Web/application servers? (LogParser v2.0 can use existing, typical IIS logs to help deduce a reasonable answer to this question.)

• What is the networking between my Web/application server and my central store?

• How many Web/application servers hit my central store at once?

• How will a server cope that is in a cold state,39 just added to production? How does the server cope when all of the content is local? What happens when all of its content is on a remote server?

Errors with UNC Content Under High Load

If the Web server and the remote file server are not configured properly for high stress conditions, administrators might notice some errors. The following errors can show up on browsers, in the system event log on the IIS server/remote file server, and in the IIS logs.

• Not enough server storage is available to process this command. Error code: 0x8007046.

• Windows cannot find the network path. Verify that the network path is correct and the destination computer is not busy or turned off. If Windows still cannot find the network path, contact your network administrator. Error code: 0x80070033.

• The network location cannot be reached. For information about network troubleshooting, see Windows Help. Error code: 0x800704CF.

MaxUserPort Setting

The last two errors can be returned when the IIS server has a shortage of outbound TCP ports while making the UNC connection to the remote file server/NAD device. The Web/application server will then return HTTP 500 errors for all requests, until TCP ports are available again. To fix this problem, you can use the MaxUserPort registry setting (on the Web/application server) to increase the number of available ports.

For instance, if there are a large number of concurrent authenticated users (Basic, Digest, Kerberos/NTLM, Microsoft .NET Passport) a different Server Message Block (SMB) connection is opened for each individual user, from the Web/application server to the remote file server. Every SMB connection uses ports. By default, you are limited to ports from 1,024 to 5,000 for outbound TCP connections (a little less than 4,000 ports available), so you may need to increase this value to a higher number, such as 10,000 or 20,000, to allow for additional ports. Be aware that this may cause problems with other applications creating sockets on these higher ports. The following MSDN document discusses this further: https://www.microsoft.com/windows2000/techinfo/howitworks/communications/networkbasics/tcpip_implement.asp.

Windows-based Remote Content Store

This section only applies when a Windows Server is being used to house the content that several Web/application servers are accessing over Windows UNC file shares.

The first error message above (Not enough server storage is available to process this command. Error code: 0x8007046A.), requires a check of the Windows file server settings. Usually, that IIS error accompanies event log messages like the examples below:

• The server was unable to allocate from the system non-paged pool because the pool was empty.

• The server was unable to allocate a work item <n> times in the last 60 seconds.

Normally, those messages mean that the file server was set to use too much non-paged system memory. This can be quickly verified by opening Windows Task Manager, going to the Performance tab, and looking at the non-paged value under Kernel Memory frame. You should throttle it back and limit the amount of kernel memory the Server service should use.

When troubleshooting this class of errors, you will need to check these performance counters on the Web/application server:

• Server\Files Open

• Server\Server Sessions

• Server\Work Item Shortages

The last class of errors above can typically be fixed by tuning the following registry entries related to SMB40 configuration, namely: MaxCmds, MaxMpxCt, MaxWorkItems.

Between two Windows servers, for each user, one SMB connection is used; however, within that SMB connection, there are a finite number of work items. Work items can be consumed in a variety of ways and for varying amounts of time. For example, doing an operation against a file (for example CreateFile or GetFileAttributes) only takes up an I/O work item for a short amount of time, but asking for a change notification on a directory structure will consume a work item for as long as the connection is established.

Here are some general details about work item configuration:

• Only one SMB connection is created per Web server / file server connection, per user.

• Each change notification instance uses a work item until the connection is closed.

• Each request from the Web/application server to the file server is going to temporarily use a work item.

• The registry settings at this location: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters can be updated on the remote file server, and the registry settings at: HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters can be updated on the Web/application server.

• For these registry values to take effect, you must stop and start the server service on the remote file server and the workstation service on the Web/application server. The servers do not require a reboot.

• Each work item typically takes up about 20K of non-paged pool memory. Although certain KB articles may specify setting the MaxWorkItems value to 16,000 this could consume up to around 320 MB of non-paged pool memory, which could starve the system. Always set this number lower to increase the reliability of your system.

• The rule of thumb for calculating MaxCmds is as follows: (# of distinct physical directories that IIS needs to monitor for change notifications) * (1 (if static files exist) + 1 (if ASP content exists) + 1 (if ASP.NET content exists)) + 50 (for concurrent default/regular File I/O).

The following are the default and maximum registry values for some SMB settings on Windows Server 2003 (same as on Windows 2000):

Registry Setting	Default Value	Maximum Value
MaxCmds	50	65535
MaxMpxCt	50	65535 (Windows 2000 SP2+; max 125 on W2Kpre-SP2)
MaxWorkItems	scaled on memory size, usually between 1K 4K	65535

ASP.NET Web Applications and Web Services in Dense Hosting Environments

The richness of the managed code environment also means that the memory cost of running that environment is bigger than the cost of running applications in an unmanaged environment. When running managed code applications in a hosted environment, the first consideration is the deal the customer signed up for. If part of that deal is to have a very high level of reliability, it would be pragmatic to put that application in its own application pool.

If applications do not have stringent isolation requirements, it is fine to have a strategy of putting managed code applications into the same application pool. In this case, ASP.NET will place these applications into their own application domain within the worker process regardless; however, the risk is that if they are sharing application pool with an unmanaged component that happens to cause a failure, all applications in that application pool are be affected. Additionally, maintenance on any application in the application pool that requires a recycle will also temporarily affect the responsiveness of all other applications in that application pool.

It is very important to understand what the maximum number of processes and memory footprint is on a server. This information can help an administrator size the server adequately so that the server has resource when many of the applications are initialized. This can be done by selectively pinging each possible application pool on a server and seeing what the impact is on memory, CPU utilization, and disk I/O overall responsiveness. When a comfortable threshold is reached, an administrator can consider using the (worker) Process Gating parameter41 to lock down the maximum number of concurrent worker processes on a system. See the section entitled Number of Worker Processes for more information about Process gating.