Tuning the Servers for UNC Content (IIS 6.0)

Applies To: Windows 7, Windows Server 2008, Windows Server 2008 R2, Windows Vista

High stress conditions over a UNC path may require specialized tuning to optimize server performance. Under such circumstances, certain errors can show up on the browser, in the event log on the IIS server and the remote file server, and in the Win32 error messages of the IIS logs. The following discussion presents these errors and discusses their solutions.

Shortage of Outbound TCP Ports

Using UNC paths under load, you may see the following event viewer error messages:

“Windows cannot find the network path. Verify that the network path is correct and the destination computer is not busy or turned off. If Windows still cannot find the network path, contact your network administrator. Error code 0x80070033.”

“The network location cannot be reached. For information about network troubleshooting, see Windows Help. Error code 0x800704CF.”

These errors can be returned when the IIS server has a shortage of outbound TCP ports when making the UNC connection. Under these conditions, IIS returns HTTP 500 errors for all requests, until TCP ports are available again. To fix this problem, use the MaxUserPort registry setting (on the IIS server) to increase the number of available ports. For more information about MaxUserPort, see Registry Keys: https://www.microsoft.com/resources/documentation/isa/2000/enterprise/proddocs/en-us/isadocs/cmt_regkey.mspx.

For instance, if you have a large number of virtual directories, or if you use pass-through authentication and have a lot of authenticated users, you can potentially have many concurrent authenticated users using resources on a UNC path. A different SMB connection is opened for each individual user from the IIS server to the remote file server. Each SMB connection uses a port. By default, you are limited to ports 1,024 to 5,000 for outbound TCP connections (a little less than 4,000 ports available), so you may need to increase this value to a higher number, such as 10,000 or 20,000, to allow for additional ports to be used for each SMB connection. Be aware that this may cause problems with other applications creating sockets on these higher ports. For more information, see the following TechNet document: Windows 2000 Server.

File Server Nonpaged Memory

In some cases, errors reported by IIS related to UNC pathnames are caused by problems on the file server rather than IIS. As stated above, a UNC connection results in a SMB connection. This results in the creation of one or more work items to be used by the SMB connections. Work items are data structure objects used by SMB for I/O operations and can be consumed in a variety of ways and for varying amounts of time. For example, doing an operation against a file such as CreateFile or GetFileAttributes only takes up an I/O work item for a short amount of time, but asking for a change notification on a directory structure will take up a work item for the duration of the connection to the IIS server. So, the kind of work performed by a work item can impact the overall availability of work items to serve SMB connections.

Work items are created in the file server’s nonpaged pooled memory, usually referred to as the nonpaged pool. This memory is referred to as nonpaged because it cannot be dynamically swapped out or paged to the system pagefile. Consequently, it is vital that your server have a sufficient amount of free nonpaged pool to function properly.

A large number of work items on a file server can result in over allocation of nonpaged pool memory. In these circumstances, you may see the following event viewer error messages:

“Not enough server storage space available to process this command.”

”The server was unable to allocate from the system nonpaged pool because the pool was empty.”

”The server was unable to allocate a work item <n> times in the last 60 seconds.”

You can quickly check memory use on the file server by opening Task Manager, selecting the Performance tab and examining the nonpaged value under Kernel Memory frame. There are two ways to manage these kinds of problems: reducing the number of work items or tuning the IIS and file servers to manage more work items.

Before making a decision about how best to proceed, you should run the server under a simulated workload for several hours and examine the performance monitor counters for the following counters on the file server:

• Server\Files Open provides information that can be helpful in estimating appropriate SMB settings.

• Server\Server Sessions provides information that can be helpful in estimating appropriate SMB settings.

• Server\Work Item Shortages if this counter increases, it indicates a need for more Work Items.

• Server\Pooled NonPaged Bytes if this counter is too low, the server has nearly exhausted its available nonpaged pool. It may be necessary to decrease SMB work items.

Monitoring these settings while you tune the server will help you assess the effectiveness of your adjustments.

Reducing the Number of Work Items

Nonpaged pool memory consumption on the file server can be affected by reducing the number of work items allocated on the file server. This can be accomplished using a number of techniques.

• Changing the caching algorithm for IIS; using the default last-modified time method for cache updates results in fewer work items being created.

• Reducing the number of authenticated users connecting to the file server. A connection is created for each unique user. If you designate a user for the connection instead of using pass-through authentication, the number of connections is reduced.

• Reducing the number of virtual directories or Web sites using remote resources.

Tuning the Servers

Adjustments made to the IIS server affect the load on the file server. Consequently, when tuning these parameters, the settings on the file server are determined by the settings on the IIS server. Specifically, you can control the number of work items required on the file server by adjusting the registry setting MaxCmds on the IIS server and then modifying MaxMpxCt and MaxWorkItems on the file server to accommodate the MaxCmds setting. MaxCmds determines the number of simultaneous SMB connections that are allowed from the IIS server to the file server, MaxMpxCt is configured on the file server to limit the number of simultaneous connections to that server, and MaxWorkItems specifies the maximum number of receive buffers that a file server can allocate. It is important to determine the proper value for MaxCmds in to order to know what values should be entered for MaxMpxCt and for MaxWorkItems on the file server. After MaxCmds is determined, the setting for MaxMpxCt on the file server should equal MaxCmds on the IIS server. Additionally, the setting for MaxWorkItems must be at least (MaxCmds * the number of IIS servers).

Table 2 shows the default and maximum registry values for some SMB settings on Windows Server 2003 (the same as on Windows 2000):

Table 2 SMB registry settings

To assist with determining the correct values for these settings, here are some general details about work item configuration:

• When using pass-through authentication to access remote content (as shown in figure 5), each uniquely authenticated user creates an SMB connection. When designating a user for access to remote content (as shown in figure 4), only one SMB connection is used.

• Using file change notification as the cache update algorithm increases the number of work items consumed for a long period of time, because each change notification instance uses a work item until the connection is lost.

• MaxWorkItems is configured on the remote file server at the registry location HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters.

• MaxCmds and MaxMpxCt are configured on the IIS server at the registry location HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters.

• As a general rule, MaxCmds cannot exceed 12,000 due to memory constraints on the x86 platforms with over 512 MB of RAM.

• For the MaxWorkItems, MaxCmds, and, MaxMpxCt registry values to take effect, you must stop and start the server service on the remote file server and the workstation service on the IIS server. You may also need to restart dependent services. You do not need to restart the servers.

The following formulas will help you estimate a starting value for MaxCmds. To make it clear how to apply this information, these formulas are used in a few scenarios in the next section IIS and File Server Configuration Scenarios for SMB Scaling.

Estimating MaxCmds When Using File Change Notification

The formula to estimate MaxCmds when using file change notification for caching is as follows: (number of distinct physical directories that IIS needs to monitor for change notifications) * (1 [if static files exist] + 1 [if ASP content exists] + 1 [if ASP.NET content exists]) + 50 (for concurrent default/regular file I/O).

The value provided by this formula presumes that 100% of the IIS server connections would be active at any given time. Generally, this is not the case, so MaxCmds may be reduced by 25 to 50 percent, depending on the characteristics of your activity.

Estimating MaxCmds When Using Last-Modified Time

The formula to estimate MaxCmds when using last-modified time for caching is as follows: (peak number of requests per second / staleness time) / 2. Staleness time refers to how frequently IIS checks for a file change, which is five seconds by default. The value provided by this formula presumes that a high percentage of IIS resources are in use at any time. If that is not the case, you may be able to reduce MaxCmds by 50 to 75 percent, depending on the characteristics of your activity.

Calculating MaxWorkItems and Nonpaged Pooled Memory Use

After determining the value for MaxCmds, you would set the value for MaxWorkItems to equal the value of (MaxCmds * [number of IIS servers connecting to the file server]). If you used one IIS server, then MaxWorkItems would equal MaxCmds. If you have four IIS servers, then MaxWorkItems = 4 * MaxCmds.

On a Windows 2003 Server with over 512 MB or more of RAM, each work item used takes up 20K of nonpaged pooled memory. If there is less than 512 MB of RAM in the server, each work item uses 8K of nonpaged pooled memory. In order to assure that there is sufficient nonpaged pooled memory available for other server resources, try to minimize the amount of nonpaged pooled memory used by work items.

In some circumstances, you may find that you need a high number of work items, but at a cost of 20K per work item, you would consume too much nonpaged pooled memory. If your server has 512 MB of RAM or more, you can configure the work item’s size to be 8K instead of 20K. On the file server, set the registry key SizReqBuf, REG_DWORD at HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters to 0x1104 (4356d). This allows more work items to be used on the file server without consuming as much nonpaged pooled memory. The costs for this workaround are slower enumerations of large directories and potentially slower network performance.