Production Debugging for .NET Framework Applications

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

 

Introduction to Production Debugging for .NET Framework Applications

November 2002

Summary: This chapter provides an introduction to the guide, defines the scope of the chapters, and provides background material. It also introduces the system requirements for the walkthroughs, and instructions for installing the software required.****

Contents

Production Debugging vs. Development Debugging

Debugging Tools

ASP.NET Process Model and Health Monitoring

System Requirements

Summary

Every developer or support engineer debugs applications at some stage in his or her career, yet debugging is often viewed as an arcane and difficult topic. While it is often difficult to determine why an application is hanging, leaking memory, or crashing, there are techniques you can apply to make the debugging process more productive and efficient. The Production Debugging for .NET Framework Applications guide aims to equip you with the mindset, tools, and techniques that will help you successfully identify and resolve common application debugging.

The techniques presented here are not simplified classroom exercises. They are based on the experience of both Microsoft customers and Microsoft internal development teams. These techniques represent proven, best-practice approaches to debugging a variety of issues.

This guide presents walkthroughs of three scenarios that ASP.NET applications may encounter when running in production environments: memory consumption, contention (also known as "deadlock"), and unexpected server crashes.

These scenarios concentrate on debugging Microsoft® .NET framework applications in a production environment, focusing on low-level details such as thread states and memory allocations. However, the thought processes and techniques discussed will help you understand debugging on a much broader scale. This document helps you learn how to think about debugging in a general sense, and equips you with the mindset to successfully approach unknown and unforeseen debugging situations in the future.

Although the example scenarios target ASP.NET and the Visual C#™ development tool, the thought processes and techniques discussed are common across all .NET technologies and languages, and problems encountered are common themes. The scenarios also introduce the tools that Microsoft provides for the Microsoft Windows® operating system and .NET debugging, including the use of WinDbg, Core Debugger (CDB), and debugger extension DLLs for displaying managed call stacks and object data.

Production Debugging vs. Development Debugging

The primary goal of debugging a system is to isolate and determine the root cause of a performance, configuration, or abnormal error condition that impedes normal system operation. Generally speaking, this goal holds true whether you are debugging in a development environment or in a production environment. However, there are crucial differences between the two environments.

When debugging in a production environment, determining the root cause may be less important than simply getting the system into an operational state.

Time constraints and business cases also differ between development and production environments. Debugging within a production environment is usually done within much tighter time constraints, given that the end user or customer may be losing revenue as a consequence of a non-operational system.

The walkthroughs presented in this guide demonstrate many non-invasive techniques that can be used in a production debugging environment.

Environment

In a production environment, the system experiences live operational loads and timing situations, which may be difficult or impossible to reproduce in a development environment. In addition, scenarios can arise that have not been anticipated during system design. Timing-sensitive or load-sensitive scenarios that lead to abnormal behavior can be difficult to reproduce or debug successfully, even during live operation in a production environment.

Because physical access to a production system is often restricted, offline or remote-access techniques, such as Terminal Services sessions and remote debugging sessions, are required.

Tools

In a development environment, you usually have access to an integrated development environment (IDE), source code, and debug versions of libraries and symbols. In a production environment, the IDE and source code are often not present, and you may only have access to release versions of libraries and symbols. Note that release versions of symbols are an invaluable resource, but are often not built during a normal product development lifecycle. Symbols are not as necessary when debugging .NET code, but they are still beneficial. This is because, while the .NET metadata contains the function names and parameter types, symbols provide source file names, line numbers, and local variable names.

For more information on symbols and how they are used, see article Q121366, "PDB and DBG Files–What They Are and How They Work," in**the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q121366.

The Debugging Tools for Windows toolkit contains a wealth of tools that are helpful to many debugging approaches and processes. The toolkit is small and non-intrusive and can usually be installed in a development or production environment pending end user or customer approval. The rest of this guide describes many of the tools that this toolkit contains.

You can download the latest version of the Debugging Tools for Windows toolkit from the Microsoft Web site at https://www.microsoft.com/ddk/debugging.

Actions

In a development environment, you often have the luxury of using a full range of techniques to debug the system. That's not the case in a production environment, where it is essential to keep the system functioning while performing debugging. A production environment requires the following low-risk actions:

• Using non-intrusive debugging techniques, such as capturing performance information, and saving hang or crash dumps.
• Restricting the use of code changes, except in the case of critical problems. This means that you can't normally experiment using code modification to isolate root causes in production environments.
• Adjusting the testing level to the user's comfort level. This may mean passing a system through a full battery of tests in a development environment, or simply asking the question, "Is the system still running?"

Debugging Tools

Debugging a problem typically involves two phases. The initial discovery phase is used to gather information about the problem. This leads to a debugging phase, when you attempt to determine the cause of the problem. These two phases usually require the use of different tools.

Discovery Phase

Most developers are familiar with the debugging phase, yet the discovery phase is equally important to production-level debugging. The goal is usually to capture the state of the server and then allow the server to continue functioning.

Two particularly useful tools that are included with Windows operating systems are:

• Task Manager, which enables a system administrator to obtain values for system metrics such as CPU or memory usage and virtual memory size.
• System Monitor (known as Performance Monitor in Windows 2000), which enables you to log the values of a number of performance counters in order to gather information for trend analysis.

Other tools, which are not included with Windows operating systems but are available as downloads, are also useful in the discovery phase:

• Autodump+ (ADPlus) is included in Debugging Tools for Windows version 6.0 and later. It is the primary data gathering tool for post-mortem analysis. It takes the form of a Microsoft Visual Basic® Scripting Edition (VBScript) file, which instructs CDB how to gather data, such as user dumps and logs for post mortem analysis. For more information about ADPlus, see article Q286350, "HOWTO: Use Autodump+ to Troubleshoot 'Hangs' and 'Crashes,'" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q286350.
• ADPlus_AspNet.vbs is a modified version of ADPlus, and its use is explained in Chapter 3, "Debugging Contention Problems." ADPlus_AspNet.vbs can create a full dump when a deadlock occurs. It attaches to Microsoft Internet Information Services (IIS) and waits for the health monitoring thread to shut down the ASP.NET process. When it does, ADPlus_AspNet.vbs breaks in and creates a user dump of the ASP.NET worker processes and the IIS processes running them.
• ADPlus_KernelExit.vbs is a modified version of ADPlus, and its use is explained in Chapter 4, "Debugging Unexpected Process Termination." ADPlus_KernelExit.vbs creates a full dump when a process terminates unexpectedly.
• Allocation Profiler provides a way to view the allocation of managed memory in graphical form.

Debugging Phase

The debugging phase may involve live or post-mortem debugging. Live debugging involves stepping through the code as it executes and looking for bugs or defects. Post-mortem debugging involves the analysis of data or program dumps produced by tools such as ADPlus. One advantage of post-mortem debugging is that it uses data gathered from a real problem occurrence, instead of relying on a dedicated programmer to interactively step through the code and try to reproduce the problem.

There are a variety of tools available to help you gather and analyze data during the debugging process:

• WinDbg is a native debugger with a graphical user interface. It attaches to a live local or remote process or opens and helps you to analyze dump files during post-mortem analysis. WinDbg is part of the Debugging Tools for Windows toolkit, along with several other console-based debuggers, notably CDB (the Console Debugger) and KD (the Kernel Debugger).

• CorDbg is a console-based debugger that ships with the .NET Framework SDK and is used to debug managed code. It requires the installation of three DLLs: MSDis130.dll, MSvcP70.dll, and MSvcr70.dll. This tool is useful for production debugging and data gathering because of its small number of dependencies and its size.

• SOS.dll is a debugger extension that is capable of displaying managed call stacks and object data. It currently works only with the debuggers provided as part of the Driver Development Kit (DDK), such as WinDbg, and it requires a .bin file that matches the type and version of the .NET runtime.

  Note   At present, SOS only works with the DDK debuggers. It is intended, however, that the functionality provided by SOS will be supported in the upcoming version of Visual Studio .NET. Also, the .bin file will not be required to match the type and version of the .NET runtime.

• SieExtPub.dll is a debugger extension used to output COM threading information.

• The Visual Studio .NET debugger is both a managed and native debugger that you can use to debug code written in the Visual Basic .NET and Visual C++® .NET development systems, and with Visual C# .NET. It also supports debugging of mixed language solutions and scripts. The debugger supports remote debugging, although this needs several files to be deployed on the target system.

Choosing the right debugger depends on the environment (for example if Visual Studio.NET is installed on the machine) and whether you're dealing with native or managed code. Table 1.1 summarizes the capabilities of the various debuggers. Variations include allowing you to examine managed and/or native data through live debugging or through dumps.

Table 1.1: Debugger capabilities

Situation	Visual Studio .NET	CorDbg	WinDbg	WinDbg and SOS	Upcoming version of Visual Studio .NET and SOS
Native call stacks (live attach)	Yes	No	Yes	Yes	Yes
Native call stacks (post mortem)	Yes	No	Yes	Yes	Yes
Managed call stacks (live attach)	Yes	Yes	No	Yes	Yes
Managed call stacks (post mortem)	No	No	No	Yes	Yes
Native and managed call stacks (live attach)	Yes	No	No	Yes	Yes
Native and managed call stacks (post mortem)	No	No	No	Yes	Yes
Examine .NET memory	No	No	No	Yes	Yes
Trap process exit	No (need breakpoint)	No	Yes	Yes	No (need breakpoint)

Before starting the walkthroughs described in this guide, download the Debugging Tools for Windows toolkit (which includes WinDbg and ADPlus) from the Microsoft Web site at https://www.microsoft.com/ddk/debugging/

ASP.NET Process Model and Health Monitoring

The scenarios described in this guide involve ASP.NET. Therefore, before discussing them in detail, you need to understand something about how ASP.NET works, and in particular, how different versions of IIS monitor the health of an ASP.NET process. This section starts with a discussion of IIS versions 5. x, and concludes by discussing how health monitoring works in IIS version 6.0.

What Is the ASP.NET Process Model?

The ASP.NET process model refers to the path an HTTP request takes through IIS, along with the response that is generated and returned to the client. The process model is configured by editing the machine.config .NET machine-level configuration file.

The machine.config file is located in the \Windows Directory\Microsoft.NET\Framework\Framework Version\Config folder.

This XML configuration file contains a <processModel> element, whose attributes specify the parameters used by the process model. For more information on the <processModel> element and its attributes, see the "ASP.NET Process Model" entry in the .NET Framework SDK documentation, which is available from the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/gngrfprocessmodelsection.asp.

IIS 5.x Process Model

The IIS 5.x process model controls how the ASP.NET request travels through IIS and is finally served by the Aspnet_wp.exe process. Figure 1.1 illustrates how this works.

Figure 1.1. IIS 5.x process model

Both the InetInfo and Aspnet_wp executables load the Aspnet_isapi.dll. The steps below provide a more detailed look at the logic used to process an ASP.NET request.

1. IIS receives the incoming ASP.NET request and passes it to the ASP.NET ISAPI code. The request is added to the request table and is then passed to the Aspnet_wp.exe worker process by named pipes.
2. After the worker process receives the request, it sends an acknowledgement back and updates the request table to "executing." A request in the "executing" state cannot be reassigned to a different worker process if the original worker process does not execute the request to completion.
3. The worker process is now responsible for executing the request, but may call back into IIS to retrieve items, such as server variables.
4. The page response is sent asynchronously through IIS to the client. Then the request table is updated to reflect that the request is complete.

Process Model Configuration in IIS 5. x

When using ASP.NET with IIS 5. x, the aspnet_isapi.dll unmanaged DLL reads the process model settings in the machine.config file. If changes are made to the <processModel> element, IIS must be restarted for the changes to take effect. (When using IIS 6.0, the <processModel> section, by default, is not used. Instead, the Internet Services Manager UI allows you to configure applicable settings for the IIS worker process.)

Runtime Configuration

The ASP.NET HTTP runtime settings can be configured using the <httpRuntime> element, which can be defined in configuration files at the machine, site, application, and subdirectory levels. Several attributes can be used to control the runtime behavior of ASP.NET.

The following table describes the available attributes.

Table 1.2: Runtime attributes for ASP.NET

Attribute	Description
appRequestQueueLimit	The maximum number of requests that ASP.NET will queue for the application
executionTimeout	The maximum number of seconds that a request is allowed for execution
maxRequestLength	Maximum upload file size
minFreeThreads	Minimum number of threads set aside for request execution
minLocalRequestFreeThreads	Minimum number of threads set aside for execution of local requests

Health Monitoring in IIS 5. x

The health monitoring provided by the process model aims to provide maximum application uptime by proactively responding to possible problems before they become critical. It offers the ability to control the way in which applications behave by specifying values for parameters, such as memory usage limits and idle process timeouts.

Health monitoring occurs when the ASP.NET process model is enabled; this is controlled by the "enable" attribute of the <processModel> element in machine.config.

Every two seconds InetInfo checks the memory size, number of requests completed, and the time since the last response was received from the ASP.NET worker process. When the size of any one process reaches a certain percentage of total system memory (by default 60 percent of physical RAM, configurable through a <processModel> attribute), InetInfo suspends all requests to that Aspnet_wp.exe process, routes requests to a new worker process, and then recycles the old one. A clean restart occurs, and entries are written to the application event log (enabled by the logLevel attribute of the <processModel> element).

If process model health monitoring is enabled while debugging, the process may be recycled before you have a chance to capture an adequate picture of the problem. You can use a registry DWORD value, HKLM\Software\Microsoft\ASP.NET\UnderDebugger, to control whether the process recycles when it detects that a debugger is attached. The walkthroughs later in this guide discuss how to use this value.

IIS 6.0 Process Model in Windows Server 2003 Release Candidate 1

The IIS 6.0 process model controls how the ASP.NET request is processed by IIS and then served by the worker process. Figure 1.2 illustrates how this works.

Figure 1.2. IIS 6.0 process model

With "Worker Process Isolation Mode" enabled in IIS 6.0, all configured Web applications are grouped into application pools and each pool runs in a single w3wp.exe process. In this new model, the Aspnet_Isapi.dll is loaded into the worker process serving the ASP.NET content, rather than into InetInfo.exe itself. This provides greater stability, since a failure in Aspnet_isapi.dll only impacts one worker process, rather than bringing down the entire Web server.

Requests from clients for .aspx or .asmx files are received by HTTP.sys, which is the kernel mode listener that handles all HTTP traffic for IIS 6.0. HTTP.sys then forwards requests directly to the worker process for the specific ASP.NET application. After ASP.NET has finished processing the request, the response is sent back through HTTP.sys and on to the client.

IIS 6.0 Application Pools

Proper configuration of the application pool is essential to ensure the health and reliability of an ASP.NET application. All the relevant settings are found on the application pool property sheet.

Similar to IIS 5. x, IIS 6.0 distinguishes between proactive and reactive recycling. IIS performs proactive recycling for known conditions and reactive recycling for unknown or dynamic conditions. For proactive recycling, IIS 6.0 checks the elapsed time, the number of requests completed, the scheduled time, and the amount of memory used. By default, events are only logged in the application event log if the recycling occurs based on the memory limit trigger. If you want IIS to log all proactive recycling events to the application event log, run the following command from the \Inetpub\Adminscripts directory:

cscript adsutil.vbs set w3svc/apppools/<defaultapppool>/LogEventOnRecycle 0xffffffff
  

Note   Make sure that you change <defaultapppool> in the preceding command to the name of the application pool that you want to log proactive recycling events.****

The following table describes the available IIS 6.0 application pool settings by means of the IIS MMC snap-in.

Table 1.3: Application pool settings for IIS 6.0

Setting	Description
Idle Timeout	Controls whether IIS shuts down idle worker processes. Any worker process serving the application pool is shut down after being idle for the specified amount of time.
Request Queue Limit	Controls the size of the request queue. This setting prevents large numbers of requests from queuing up and overloading the Web server.
CPU Monitoring	Specifies what action is to be taken if a CPU usage threshold is reached.
Web Garden	Controls the number of worker processes for the application pool.
Enable Pinging	Specifies how often the Web Administration Service (WAS) pings each worker process in the application pool to detect its status.
Enable Rapid Fail Protection	Configures IIS to remove the application pool from service if a specified number of crashes occurs within the specified time period. If a worker process is removed from service, HTTP.sys responds to any incoming request with a "503 Service Unavailable" message.
Startup Time Limit	Specifies the amount of time a process is allowed for startup, before IIS assumes that it has not started correctly and terminates it. Terminated processes are logged in the system event log.
Shutdown Time Limit	Tells the WAS how to cope with worker processes that hang during shutdown. If a worker process has not shut down during the specified time limit, it is terminated by the WAS.

Orphaning Failed ASP.NET Worker Processes

Web Administration Service (WAS) shuts down ASP.NET worker processes if they fail to meet established health criteria or if they fail to respond to pings. This can complicate debugging, since the failed worker process shuts down before you can attach a debugger to it.

You can reconfigure WAS to prevent the failed process from serving any more requests, while ensuring that the process continues to run; this is known as orphaning the failed process. An orphaned worker process is removed from the application pool, but left in its failed state for later debugging. In this case, the WAS starts another worker process so that the application can continue to serve requests. To configure WAS to orphan worker processes upon failure, run the following command from the \Inetpub\Adminscripts directory:

cscript adsutil.vbs set w3svc/apppools/orphanworkerprocess 1
  

This command applies this setting at the master level for all application pools. To apply this setting to a specific application pool only, run the command as follows:

cscript adsutil.vbs set w3svc/apppools/<nameofapppool>/orphanworkerprocess 1
  

Note   Replace <nameofapppool> in the preceding command with the name of the application pool to which this setting should be applied.****

After WAS is configured in this way, the failing instance of W3wp.exe will not be terminated by WAS, and you will be able to hook up a debugger and investigate the cause of the problem.

System Requirements

The stack traces in this guide were created on a computer running Windows XP Professional. If you are using Windows 2000, you may find some differences in the names of function calls listed in your output.

The walkthroughs in this guide were also tested on a computer with the following configuration:

• Single processor
• 512 megabytes (MB) RAM
• Windows 2000 Advanced Server
• IIS version 5.0

Using a machine with a different configuration may lead to slightly different results, but the process to debug and analyze remains the same.

Additional Software

To perform the walkthroughs in this guide, you need to download and install the following software:

• Sample applications
• Debugging Tools for Windows
• Allocation Profiler

To install the sample applications

1. Download DebuggingWalkthroughs.msi from https://www.microsoft.com/downloads/release.asp?ReleaseID=44273
2. Double-click DebuggingWalkthroughs.msi to start the .NET Debugging Walkthroughs Setup Wizard and then follow the instructions to complete the wizard.

To build the sample applications

1. Start Visual Studio .NET, and then open Debugging.sln from C:\Inetpub\wwwroot\Debugging. This opens both the DebuggingCOM and DebuggingWeb applications.
2. On the Build menu, click Configuration Manager.
3. In the Configuration Manager dialog box, change the Active Solution Configuration to Release, and then click Close.
4. On the Build menu, click Build Solution.

To create folders for the debuggers

• In Windows Explorer, create the following folders:
  • C:\Debuggers
  • C:\Debuggers\ZipFiles
  • C:\Debuggers\SOS
  • C:\Debuggers\AllocProf
  • C:\Symbols\DebuggingLabs

To install the Debugging Tools for Windows toolkit

1. Download the Debugging Tools for Windows toolkit from the Microsoft Web site at https://www.microsoft.com/ddk/debugging/.
2. Double-click Dbg_x86_6.0.17.0.exe.
3. On the Welcome to the Debugging Tools for Windows Setup Wizard page, click Next.
4. On the End-User License Agreement page, if you agree with the terms and conditions, click I agree, and then click Next.
5. On the User Information page, enter your details, and then click Next.
6. On the Select an Installation Type page, click Custom, and then click Next.
7. On the Select an Installation Location page, change the path to C:\Debuggers, and then click Next.
8. On the Custom Installation page, click Next.
9. On the Begin Update page, click Next.
10. On the Completing the Debugging Tools for Windows Setup Wizard page, click Finish.

To install additional tools

1. Download SOS.zip, SieExtPub.zip, and ADPlus_Scripts.zip (stored in dbgnetfx.exe) from https://www.microsoft.com/downloads/release.asp?ReleaseID=44274[](https://www.microsoft.com/downloads/release.asp?releaseid=44274)to C:\Debuggers\ZipFiles.
2. Extract the contents of SOS.zip to C:\Debuggers\SOS.
3. Extract the contents of SieExtPub.zip to C:\Debuggers\SOS.
4. Extract the contents of ADPlus_Scripts.zip to C:\Debuggers.
5. Download the current version of the Allocation Profiler from the Microsoft .NET Framework Community Web site at https://www.gotdotnet.com/userarea/keywordsrch.aspx?keyword=allocation%20profiler
6. Extract the contents of AllocationProfiler.zip to C:\Debuggers\AllocProf.

Documentation Conventions

This guide uses the following style conventions and terminology.

Table 1.3: Style conventions

Element	Meaning
Bold font	Characters that you type exactly as shown, including commands and switches. User interface elements are also bold.
Italic font	Placeholder for variables for which you supply a specific value. For example, Filename.ext could refer to any valid file name for the case in question. New terminology also appears in italic on first use.
Monospace font	Code samples.
Note	Alerts you to supplementary information.

Summary

This chapter introduced the difference between debugging in production and development environments and introduced a number of debugging tools. The chapter also explained and compared ASP.NET process models used by IIS 5. x and 6.0, as well as how ASP.NET uses health monitoring to maximize the reliability and robustness of applications.

The next few chapters focus on specific debugging problems and how to solve them. The chapters present three specific debugging scenarios that illustrate typical debugging problems and the use of the tools just described.

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

© Microsoft Corporation. All rights reserved.