Linux to Windows Migration

Article
02/10/2009

Debugging Windows 2000 Web Applications

By Anand Rajagopalan

Abstract

This white paper describes different tools used to debug Distributed Web Applications and includes procedures for using the tools in debugging tasks. It is geared toward debugging the middle-tier Component Object Model (COM) components in a Web-based development environment using Internet Information Services (IIS) Active Server Pages (ASP), with additional explanation provided for those with a Linux background.

Acknowledgements

I would like to thank Aaron Barth who is the Tech Lead in the Systems Integration Engineering group. I think of Aaron as a debugging guru; he has taught me debugging and was also a tremendous help in writing this paper.

Preface

The Focus of This Article

This article describes the different tools used to debug Distributed Web Applications and includes procedures for using those tools. The debugging scenarios and tasks focus on the middle-tier Component Object Model (COM) components in a Web-based development environment that use Microsoft Internet Information Services (IIS) Active Server Pages (ASP). After reading this article, the user should understand the following:

The IIS architecture
In-process, out-of process, and pooled applications, as well as the advantages and disadvantages of each.

The article should also show the user how to use the following tools:

Exception Monitor, to monitor IIS applications and COM components.
User Dump, to dump a process's memory to disk and examine it.
WinDBG, to debug a live process.
Auto Dump Plus(AD+), to create mini dumps and crash dumps
The Performance Monitor, to troubleshoot a 100 percent CPU problem.
Console-Based Debugger (CDB) and Microsoft Windows NT Symbolic/Systems Debugger (NTSD).

Target Audience

Developers, support professionals, and Web managers involved in development and maintenance of Web sites would better understand the IIS architecture and the debugging process after reading this article.

Introduction

This section will help you become familiar with the debugging environment for COM components in a Web-based development environment that use IIS Active Server Pages (ASP). Detailed information is presented about the following important areas:

IIS architecture
Internet Services
ISAPI filters and extensions
Active Server Pages (ASP)
Web Application Manager (WAM)
Asynchronous Thread Queue

In addition, an introduction to debugging in general is presented, included an overview of the debuggers and when to use which ones. You may be familiar with the Linux debugging tool, GDB (also known as GNU debugger), which has an X-based interface, an EMACS-based interface, and a console-based interface. The GDB tool lets you add the symbol search path, supports source level debugging, and is remotable. Several debugging tools are available to use when you're dealing with COM components in a Web-based development environment that use IIS Active Server Pages (ASP), including WinDBG. WinDBG is similar to the Linux GDB (it's also remotable), but has a couple of advantages:

Symbol serverWinDBG features one database that redirects to the right symbols for what you're debugging, regardless of what server/share has the symbols.
Debugger extension DLLsVendors can ship an extension DLL that understands internal data structures of their applications without making those data structures publicly available. The extension DLL runs in-process inside the debugger and uses its object model to both read and manipulate the item that's being debugged. This let's you write powerful extensions, like strike, because it lets you view the state of the .NET VM, including unassembled managed code, from within NTSD and WinDBG.

Internet Information Server Architecture

To understand the process of debugging a component, you first must have a good grasp of the IIS architecture. The figure below will give you insight into the relationships between IIS, ASP, Internet Server Application Programming Interface (ISAPI), and components.

Figure 1: Internet Information Server architecture

Internet Services

The inetinfo.exe process contains the many Internet services (HTTP, FTP, SMTP, NNTP) that IIS provides. Inetinfo.exe receives requests from the TCP/IP subsystem and directs the requests to the respective services, which in turn interpret the requests, execute them, and return the results to the client. Because all of these services run within the same process (inetinfo.exe), they are able to share cached data such as file handles, account information, and log file data.

ISAPI

There are two types of ISAPI DLLs: ISAPI filters and ISAPI extensions.

An ISAPI filter is a DLL that runs in the inetinfo.exe process to filter data traveling to and from the server. The filter registers for notification of events. When the selected events occur, the control is transferred to the filter and you can monitor and change the data (on its way from the server to the client or vice versa). ISAPI filters can be used to provide enhanced logging of HTTP requests (for example, to track who is logging on to your server), custom encryption, custom compression, or additional authentication methods. If ISAPI filters are implemented, all the requests and responses have to go through the filters, which can have a great impact on site performance.

An ISAPI server extension is a DLL that can be loaded and called by an HTTP server. Internet server extensions, also known as Internet server applications (ISAs), enhance the capabilities of an ISAPI-compliant server. An ISA is invoked from a browser application and provides similar functionality to a Common Gateway Interface (CGI) application, but has considerable performance gains over a CGI.

Active Server Pages

ASP is implemented as an ISAPI extension DLL. IIS and ASP have separate thread pools. IIS uses an internal thread pool called the Asynchronous Thread Queue (ATQ), also called pool threads. IIS uses ATQ for accepting the socket connection and sending static content to clients. Once the request is determined to be an ASP page, IIS sends the page request to the ASP/MTS thread pool for processing.

All of the ISAPI management and ASP code is moved into a component called the Web Application Manager (WAM). All of the information enters and exits the application service layer through the WAM.

Web Application Manager (WAM)

WAM is a COM component layer above Web services that houses the components for ISAPI DLLs and provides services like support for multiple ISAPI DLLs, crash recovery, and process isolation. WAMs are registered as MTS/COM+ packages/applications.

Asynchronous Thread Queue

The ATQ provides a pool of reusable threads to process each client request. The scheduling is performed using the Completion Port Logic of the operating system.

IIS In-Process Applications

If an IIS application is configured to run as in-process (low application protection in IIS 5.0), it runs in the Web server's (inetinfo) process space. Running in-process offers a performance advantage, in that it makes costly cross-process marshalling avoidable. The drawback is that a single malfunctioning application could bring down the entire Web server, seriously decreasing the stability and availability of the Web server. All applications are run as in-process in IIS 4.0 and pooled out-of-process (medium application protection) in IIS 5.0 by default.

IIS Out-of-Process Applications

Running an application out-of-process prevents a single application error from bringing down the inetinfo process. This kind of server protection is invaluable for managing complex production Web sites and hosting multiple applications. If an application malfunctions, its process is automatically terminated without affecting the inetinfo process.

Another advantage of running an application out-of-process is that you can unload the application without having to restart IIS.

An application running out-of-process also gets its own ASP Worker threads, meaning that every time a processor is added to the box, the out-of-process application gets another 10 ASP Worker threads. This is the default case; changing the ProcessorThreadMax and ASPProcessorThreadMax values would affect this number. We will discuss these counters in detail a little later in this article.

The main disadvantage of out-of-process applications is that they each have their own process space, so there is overhead that affects performance.

Also, configuring a number of applications to run out-of-process in a single server box can be quite resource-intensive and affect the performance of the server.

IIS Pooled Out-of-Process Applications (IIS 5.0 Only)

A new option in IIS 5.0 occurs between out-of-process and in-process applications: All IIS applications configured as pooled out-of-process will run under one server package/application created under MTS/COM+.

This option shares the advantage of the out-of-process option, because pooled apps have their own process space, and a failure would not affect the availability of the Web server. In addition, since all the pooled applications share the same process space, this option is not as resource-intensive as the out-of-process option.

The disadvantage is that since all the pooled applications share the same process space, a failure in one application would bring down all the applications that share the pool.

In IIS 5.0, applications run as pooled out-of-process by default.

Figure 2: Process isolation: performance vs. robustness

Figure 2: Process isolation: performance vs. robustness

Why Debugging Is Important

Debugging is a method used by software developers to find and fix problems in their code. Even after bugs are found and fixed during unit and integration testing, some bugs remain in production versions. Even products coming out of a high-quality software life cycle that involves thorough unit and integration testing are not immune.

Debugging can be the most frustrating part of the Web application developer's job. Unlike desktop applications, which are self-contained, Web applications can be spread over several systems and frequently combine different programming languages and technologies. Although some application errors can be avoided with careful planning, others result from unexpected side effects of complex component interactions. Tracking down these problems often requires the use of a variety of debugging and development tools.

The most important thing to keep in mind while debugging is that you are debugging the data, not the code. Asking the right questions to get the right data is a crucial part of a successful debugging process. Data can be either global or local. The most common mistake people make in debugging is to assume that the symptoms they are seeing are the actual cause for a problem.

Before diving into the process of debugging itself, you will want to familiarize yourself with certain terms that will be used frequently in this document pertaining to debugging, as follows:

Attaching means monitoring a live process in order to debug it.

Checked/Debug Build has debugging information included in the process.

Free Build omits all debugging information so that the process is lightweight and more efficient.

Dumping creates a snapshot of the process memory to the hard disk when an exception occurs. A dump is a raw snapshot of the process address space. It is roughly the same size as the amount of memory that the processor is using. A debugger can be used to analyze the dump files to determine what the process was doing at the time it was dumped.

Setting Up a Debugging Environment

Why Install Debug Tools and Symbols?

In a development environment, debugging is made easier by using the Microsoft Visual Studio debugger. You also have the source code handy, which makes the debugging process simpler. If you run into problems in a production environment, on the other hand, you don't have the source code or the powerful Visual Studio debugger to come to your rescue. You have to deal with Assembly language and raw binaries an uncommon skill set. You also need special tools and files called symbols.

Symbol files contain information that allow a descriptive way of referring to resources and user-interface objects, both in your source code and while you are working with them in the resource editors. Symbols simplify the process of debugging by resolving the cryptic assembly codes into meaningful Function\API names.

Consider a situation such as reporting a non-functional Web site to a Microsoft support engineer. To analyze and determine the exact reason for the crash, the support engineer may need additional information from you. To get this information, you may need debugging tools and symbols. If the debugging tools and symbols are not installed, you will first need to install them.

You need to install symbols for Microsoft products, third-party vendor products, and your own components. You should have these symbols already available when deploying the site, as looking for the right symbols may take some time. Delaying this activity will only increase downtime in case of an error.

Note: Microsoft provides symbols for most of its products and object models (such as ADO, MSXML, CDO, and so forth). To obtain symbols for third-party providers, you will need to contact the providers. Please note that the vendors may refuse to provide the tools unless you are already experiencing a problem.

The symbols for your own code are generated when you compile your components. If your component is built with Microsoft Visual Basic, you need to specify the Generate Symbolic Debug Info in the project properties. For more information, see Microsoft Knowledge Base article 281630, "HOWTO: Configure Visual Basic DLL to Run in IIS, MTS, or COM+".

If the debugging tools and symbols are installed, you can provide information immediately and the support engineer can start troubleshooting the reason for the crash. This reduces the time to resolution of the problem.

Debug Symbols (Symbolic Debug Information)

Symbols provide a way to resolve global variables and function names in the loaded executable file. Symbols are nothing but function\API names and global variables mapped to offsets from the base DLL loading address. During the compilation process for a component, the linker produces symbols. Having symbolic information inside the component itself (internal symbols) makes it easy to work with because there is no chance of having mismatched symbols. Typically, symbols are stripped out of components in production, reducing the size of the component considerably, decreasing the memory consumed by the component when loaded, and making it more efficient.

A DBG file is created by splitting symbols from a process image into a separate file using SPLITSYM or REBASE. The DBG file generally does not contain Code View-Style symbols; hence it cannot be used for source-level debugging.

The current versions of Microsoft Visual C++ and VB are capable of creating a symbol format called Program Database (PDB). Debuggers can use PDB files directly.

Apart from installing the symbols for the custom components, having symbols for Microsoft products will come in handy.

Symbol Locations

The following sites point to the symbol locations.

Windows NT 4 Service Pack 6 symbols:

https://www.microsoft.com/ntserver/nts/downloads/recommended/sp6/debug/default.asp

Windows NT 4 Service Pack 5 Symbols:

https://www.microsoft.com/ntserver/nts/downloads/recommended/sp5/debug/default.asp

Windows NT 4 Service Pack 3 Symbols:

Windows NT Option Pack Symbols:

https://www.microsoft.com/ntserver/nts/downloads/recommended/NT4OptPk/default.asp

Windows NT Option QFE Update Symbols:

https://www.microsoft.com/ntserver/nts/downloads/archive/ntopqfe/default.asp

Microsoft Windows 2000 Server Symbols:

https://www.microsoft.com/windows2000/downloads/tools/symbols/default.asp

The following symbol is also on the Windows 2000 Customer Support and Diagnostic Tools CD.

Symbols for Windows 2000 Server SP1, EM 7.0, User Dump:

https://www.microsoft.com/windows2000/downloads/tools/symbols/default.asp

Overview of Commonly Used Debuggers

CDB

CDB is a console-based debugger that runs within the command window in which it was started. It is very efficient because of its very small memory footprint. It can be used to debug Win32 applications on Windows NT.

NTSD

NTSD is similar to CDB, except that it launches a new command window and runs in the new window instead of the one in which it was started.

WinDBG

WinDBG is a popular GUI lightweight debugger. It can be used to debug Win32 applications on Windows NT, Windows 95, and Windows 98. It is currently included on the Windows 2000 Customer Diagnostic and Support Tools CD. It can perform source-level debugging, can read dump files, and has excellent support for symbols.

The latest version of WinDBG can be found at

Exception Monitor 7.x

The newest version of IIS Exception Monitor is a part of the DBGPLUS suite, which contains all the debugging tools discussed here. It uses WinDBG to gather information about the process it is attached to. It is included with the Windows 2000 Customer Diagnostic and Support Tools CD. You can also download the tool from Windows 2000 Customer Support Diagnostics

User Mode Process Dump

The User Mode Process Dump tool attaches to a particular process and creates a snapshot of the process memory to the hard disk when an exception occurs. This is called dumping. A dump is a raw snapshot of the process address space and can be used to determine what the process was doing just before the exception was thrown. It is roughly the same size as the amount of memory that the process was using at the time of the crash.

You can configure the User Mode Process Dump tool to monitor a particular process for exceptions. The user dump thus created can be later viewed in a debugger like WinDBG and analyzed at leisure, instead of debugging a live process.

Note: Windows NT has a tool called Dr. Watson that can create a user dump when an exception occurs in the system. However, the User Mode Process Dump tool is easier to configure and more flexible than Dr. Watson.

You can also download the tool from Windows 2000 Customer Support Diagnostics.

Auto Dump Plus (AD+)

AD+ is the latest debugging tool from Microsoft. A command line debugging tool, it can create both mini dumps like the Exception monitor and full-fledged crash dumps like the User Dump tool. It serves as a replacement to Exception Monitor and User Dump.

The latest version of AD+ can be found at https://www.microsoft.com/ddk/debugging/

Note: It is strongly suggested that you use AD+ to create memory dumps as opposed to using User Dump and Exception Monitor. Whether you use Exception Monitor, User Dump, or AD+, you still need WinDBG to analyze the dumps.

The Debug Process

Verifying Symbols

The first step in the debug process is to verify that the symbols are installed properly. The CHECKSYM tool can be used for this. For example, to verify that symbols installed for all the DLLs running in a process are valid, you can use the following command:

checksym -p processname.exe -r -v -y c:\symbols

Wildcards (*.*) can be used to specify all files in a directory. The -v switch instructs the tool to verify the symbols for this component and the -y switch specifies the path to the symbol files.

For more information about the switches and options that can be used with CHECKSYM, type the following command:

checksym -??

Choosing a Debugger

Debuggers vary in their capabilities, so it's important to have a good grasp of the capabilities of the different tools and which tool to use in different situations.

Exception Monitor works best for tracking down Access Violations only. For complex problems like 100% CPU utilization, Hangs, and Dead-Locks, among others, you should use AD+ to create a full dump, and then analyze the dump thoroughly using WinDBG.

Debugging scenarios can be broadly classified under three headings: live debugging, postmortem debugging, local debugging, remote debugging, and debugging the user dump.

Live Debugging

In live debugging, you attach the debugger to a live process (debugee) and analyze the memory of the process to determine if any problem exists and get some insight into what it might be. You use live debugging if you know how to reproduce the erroneous behavior of an application.

Post-mortem Debugging

In post-mortem debugging, you attach the debugger to a process and it creates a memory dump whenever the process causes a violation. You then use the dump generated in a debugger to analyze the cause of the crash and the tasks that the process was performing at the time of the crash.

Post-mortem debugging is typically required when a process suddenly exhibits an erroneous behavior that cannot be reproduced at will. This is the most frequent debugging scenario.

Local Debugging

When you are able to reproduce a problem locally, it's a good idea to use local debugging; because it provides full control of the environment, you may be able to solve the problem more quickly. The choice of the debugger and its setup are also relatively easy in this scenario. The disadvantage of local debugging is that it is sometimes difficult to accurately duplicate or reproduce ("repro" in support professionals' language) a problem locally.

Remote Debugging

When the debugger and the process being debugged do not reside on the same machine, the scenario is called remote debugging. Remote debugging can be done either using a debugger attached to a remote process through an agent, or by running the debugger on the remote machine and sharing the command line of the debugger on the local machine. In this case the agent runs on the remote system, and the debugger runs on the local system; the agent attaches to the process, reads the process memory, and passes it over to the local debugger. The advantage of remote debugging with an agent is that it is simple and fast to set up. The disadvantage is that it is prone to network problems that sometimes cause the agent on the remote machine to crash. Since the symbols are installed only on the local machine, they may not match those on the remote machine. Consequently, remote debugging can be quite slow.

Remote debugging with a shared command line is accomplished by using the remote.exe tool distributed with the Windows 2000 Customer Diagnostic and Support Tools CD. The remote.exe tool can run both as a server and a client. Named pipes are used to connect the server and the client. WSRemote.exe is a socket version/variation of the remote.exe tool. Remote.exe allows the sharing of the STDIN (standard input) and STDOUT (standard output). On the remote machine you can run the server instance of the remote with the following command:

remote /s applicationpipename

On the local machine you can run the remote.exe as a client by using the following command:

remote /c machinenamepipename

This setup is very reliable and tolerant of network failures.

Debugging the User Dump

User dump creates a snapshot of the process memory to the hard disk when an exception occurs. You can configure the user dump tool to monitor a particular process for exceptions. The user dump thus created can be later viewed in a debugger like WinDbg and analyzed at leisure, instead of debugging a live process.

Using the Debuggers

Using NTSD

The NTSD debuggers require the user to specify a process to connect to when starting the debugger. Using TLIST or PVIEWER, you can get the process ID for an existing process and type NTSD -p pid to debug that process. The NTSD command line uses the following syntax:

NTSD [options] imagefile

Where imagefile is the name of the image to be debugged and options is one of the following:

Table 1 NTSD imagefile options

Option	Description
-2	Opens a new window for debugging character mode applications.
-d	Redirects output to the debugging terminal.
-g	Causes execution past the first breakpoint automatically.
-G	Causes NTSD to exit immediately when the child terminates.
-o	Enables debugging of multiple processes. The default is for one process directly spawned by the debugger.
-p	Specifies debugging of the process identified by process ID.
-v	Produces verbose output.

For example, let's assume that the process ID (PID) for the inetinfo.exe process is 104. To attach the NTSD debugger to the inetinfo process (IIS), type the following:

NTSD p 104

NTSD may also be used to start a new process for debugging. For example, NTSD notepad.exe will start a new notepad.exe process and attach to it.

Once attached to a process, you can use various commands to view the stack, set breakpoints, dump memory, and so forth.

Table 2 Commonly used commands

Command	Meaning
~	Show a list of all the threads
KB	Show the stack trace for the current thread
*~KB**	Show the stack trace for all the threads
R	Show the register output for the current frame
U	Un-assemble the code and show the procedure name and offset
D[type][<range>]	Dump memory
BP[#] <address>	Set break point
BC[<bp>]	Clear break point(s)
BD[<bp>]	Disable break point(s)
BE[<bp>]	Enable break point(s)
BL[<bp>]	List break point(s)

Using WinDBG

To launch WinDBG, click Start, point to Programs, point to Debugging Tools for Windows, and then click WinDBG.
On the File menu, click Attach to a Process, select the process you want to debug from the list, and then click Select.

WinDBG lists the DLLs related to the inetinfo process in the command window and also shows if symbols were loaded for a particular DLL. At this point the inetinfo.exe application (IIS) that is being debugged is stopped. To start it again, type g in the command window.
In the command window, type ~*kb.

This lists all of the stacks for all of the threads in the process.
To change the symbols path, on the File menu, click Symbol File Path and specify the path to the symbols files delimited by a semicolon.

For Example: C:\winnt\symbos;C:\MySymbols
In the command window, type kb.

This lists the stack for the current thread.

Another way to view the stack of the current thread is to view the Call Stack window by clicking Call Stack on the View menu.
In the command window, type lm.

This lists all of the modules (DLLs) in the process with their names and their base loading addresses.
To quit, type q in the command window.

Using Exception Monitor

In the following example, Exception Monitor is used to attach to the inetinfo (IIS) process and debug it.

Click Start, point to Programs, point to Debugging Tools, point to Exception Monitor and then click Exception Monitor.
In the Welcome dialog box, click Next.
In the Select Application State dialog box, select Monitor a Running Service/Application, and then click Next.
In the Process Options dialog box, select Service, click a service in the list, and then click Next.
In the Session Options dialog box, select Automatic, and then click Next.
In the Start Monitoring dialog box, click Run This Monitoring Session, and then click Next.
The Session Status dialog box lists all the processes that were monitored previously or are being monitored currently. Click Finish.

You will see WinDBG being launched. This monitors the inetinfo.exe process. Whenever an exception occurs, the Exception monitor creates the log file (.dbl), restarts the IIS Service, and closes WinDBG.

Using the Log File to Identify the Problem

The log reader program Readlogs.exe is installed when the Exception Monitor is installed. You can use Readlogs to open the newly-generated .dbl log file.

Figure 3: Readlog dialog box

When Readlogs opens the log file, it displays the stack of the thread that caused the exception. If the symbols are installed properly, Readlogs resolves the function names and displays them under the Function column as shown in the preceding figure. The function name is separated from the DLL name by the "!" symbol. For example, in the preceding figure the function OutputDebugStringA resides inside the KERNEL32 DLL.

Use the buttons in the Readlog dialog box for the following functions:

To view the faulting thread stack, click Fault Stack. To scroll through the stack list, use the << and >> buttons. You can also choose to execute any of the listed commands in the Choose a Command drop-down list. If you do not want to see Microsoft files, you have an option to hide them. This helps to quickly identify the involvement of any third-party DLLs involved in the process. Apart from listing the stack, the log file also contains information about the list of loaded DLLs or modules, any locks and their owning thread IDs, and errors that were detected.
To list all the DLLs that were loaded in the process, click DLL.
To list all the errors, including the warnings that occurred when the process was running, click Errors. You can also customize the IIS Exception Monitor to trap certain errors.
To display a list of locks and their owning thread Ids, click the Locks button. Whenever a thread wants to access a resource exclusively, the thread will request that a lock be placed on the resource. When two or more threads lock a particular resource and wait for the other lock to be relinquished, a phenomenon called "deadlock" occurs. You can examine the list of locks and their owning threads to identify potential sources for a deadlock problem.
To list the last 20 lines in the log just before the process broke down, click P.T.C. (Prior to Crash).
To paste a copy of all the information to the clipboard in a report format, click Report. To display some useful information about the system that you can use for debugging, click Sys Info.
To customize the read logs to suit your needs, click Config.

Using the User Mode Process Dump Tool

The User Mode Process Dump tool attaches to a particular process and dumps the process memory to the hard disk whenever an exception occurs.

Before doing the following procedure, you need to set up the User Mode Process Dump service by running setup.exe in C:\Program Files\Debuggers\bin\userdump\. The User Mode Process Dump setup program is installed as a part of the DBGPLUS suite installation.

In the Welcome dialog box, click Next.
In the Confirmation dialog box, click Finish.
Setup installs a control panel icon named Process Dump that can be used to configure the User Dump service.
Double-click the Process Dump icon to bring up the User Mode Process Dump Configuration dialog box.
Click New, and then enter the name of the application that you want the user dump tool to monitor.

To create rules that customize the tool to trap only certain exceptions, click Rules.

If an exception occurs in the process that the tool is monitoring, it will dump the process memory to the disk. Unlike the IIS Exception Monitor, the User Mode Process Dump tool does not restart the IIS Service after dumping the process.

The user dump thus created can be opened in the WinDBG debugger and analyzed to identify the cause for the crash:

Start WinDBG. On the File menu, click Open Crash Dump and select the Crash Dump file.
In the command window, type g (go) to start debugging the process.
To view the faulting stack, type kv.

Using the AD+ Tool

Whenever you want AD+ to monitor a process for Access Violations(AV), open up a command prompt, switch to the debugger installation directory (c:\Program Files\Debugging Tools For Windows), and type the following:

adplus.vbs quiet crash iis o c:\Dumps

If an exception occurs, then AD+ would trip and create a dump in the C:\Dumps directory. There should be a .log file and two memory dumps. The memory dumps should end with *1st_chance_access_violation__mini.dmp or *2nd_chance_access_violation__full.dmp.

For a complete list of commands that you can use with AD+, type the following at the command prompt:

adplus -?

Which Process to Debug?

Identifying the process to which the debugger should be attached can be difficult. In a Web server, there are different scenarios under which COM+ components and IIS applications are used. The following table provides information about which process to attach to, based on component type and your Web site configuration.

Table 3 Component Types and IIS Configurations

Component Type	In-Process IIS Application (Low)	Out-of-Process IIS Application (High)	Pooled Out-of-Process IIS Application (Medium)
Component in a library application or Component not in COM+, called by ASP pages	Attach the debugger to inetinfo process.	Attach the debugger to the DLLhost instance of the application under which the IIS application resides.
Component in a server application	Attach the debugger to the DLLHost instance of the application under which the component resides to debug a problem in the component Or Attach the debugger to the DLLHost instance of the application under which the web site resides to debug a problem in the ASP or ISAPI applications in the web site.

Every server COM+ Server application running on a system has its own DLLHost.exe application. There might be several COM+ applications being used at the same time, and hence there might be many DLLHost.EXE instances running on a server.

To determine the ID of the DLLHost instance that you need to attach the debugger to, go to Component Services in the MMC, and make sure you see the COM+ Applications on the right pane. Select Status view from the MMC toolbar, and you will see a column labeled PID.

There are other mechanisms that help you identify the process that needs the debugger attached. However, they require extra tools:

You can use Exception Monitor to find the matching COM+ application names for each DLLHost.exe.
You can use the Emcmd.exe application, which is installed when you install the DBGPLUS suite of debuggers. The Emcmd.exe application can be found in the C:\program files\debuggers\bin directory. To retrieve the DLLHost.exe process IDs (PIDs) and their corresponding COM+ application name, do the following:

At the command prompt, type emcmd.exe /P, and then press Enter.

Debugging Common Problems

The most common problems that occur in Distributed Web Applications are:

ASP 0115-access violation
ASP server too busy error
100 percent CPU problem

In this section, we will analyze the possible causes for these problems and discuss how to debug and identify the line of code that is causing each problem.

ASP 0115-Access Violation

Access violations are primarily caused by heap and stack corruption.

Access violations occur if you do something like allocate a block of memory and try to read/write beyond that block.

For example:

char* pszTemp = (char*) malloc(strlen("BOMB THIS"));

strcpy(pszTemp,"BOMB THIS");

In this example the user allocates 9 bytes of memory but fails to include the NULL terminator ("\0"). When trying to copy the string to the memory location, an error occurs because the function is trying to access more than the allocated segment of memory.

Another scenario would be trying to access an invalid location in memory.

For example:

FunctionA() 
{ 
m_pAV = NULL; 
FunctionB(m_pAV); 
} 
FunctionB(CAccessViolate* pAV) 
{ 
pAV->BombThis(); 
}

In this example FunctionB uses the pAV pointer passed to it without checking its validity, causing an access violation.

Most access violations occur when an ASP page is trying to access a component, and the component is trying to do something wrong. For simplicity, let's assume that you have a component installed as a library package. When you try to access an ASP page that is using this component, the ASP page may return the following

error 'ASP 0115' 
Unexpected error 
/<Web Server Name>/<ASP file name>.asp 
A trappable error occurred in an external object. The script cannot continue running.

The error message states that there was an error in a particular ASP page and it was caused by an exception that occurred in an external object. This means that you are trying to access some component in your ASP page that is failing. You can try to isolate the problem further by removing all references to any components in the ASP page and then adding them back one at a time until you find the culprit.

If the isolation process is tedious or you do not get a clear error message stating the ASP page name, you can use WinDBG, User Mode Process Dump, or IIS Exception Monitor to attach to inetinfo and start debugging it.

Finding the Line of Code That Caused the Access Violation

Let's assume that you have used the User Mode Process Dump tool to attach to the inetinfo process and have created a dump file. Let's examine the dump file in WinDBG and try to figure out what happened.

Open the crash dump in WinDBG. On the View menu, click Call Stack.

Figure 4: Windows Debugger dialog box showing the stack

Even though the topmost function in the stack in the preceding figure is a function inside the KERNEL32 DLL, it doesn't necessarily mean that the misbehaving code is inside that function. Most of the time this is not the case. The next step is to identify if any components other than the standard Windows system components are listed, and determine what they were doing at the time of crash.

If you read the stack from the bottom up in the preceding figure, you can see that lines 8 and 7 indicate that ASP used the Microsoft VBScript engine to launch the component. Lines 6 and 5 indicate that the VBScript engine was using the IDispatch interface methods to invoke a method inside the component (CRASHATLAPP). Since IDispatch interface was used, it is natural for the OLEAUT32.dll to come into the picture. Lines 4 and 3 indicate this. Line 2 in the stack shows that the BlowOff method was invoked, which in turn invoked the OutputDebugString API inside the KERNEL32 dll. It would be safe to assume that the standard Windows components KERNEL32 dll, the OLEAUT32.dll, and the VBScript engine are innocent (from the perspective of problem isolation). So we are left to deal with the only non-Windows component listed in the stack, the CRASHATLAPP.

The BlowOff method inside the CRASHATLAPP has made a call to the OutputDebugString API. We can examine the values of the parameters that BlowOff method passed to the OutputDebugString by typing kb in the command window.

The BlowOff method has passed NULL as the parameter to the OutputDebugString method inside the KERNEL32 DLL. Now you need to see what is being done inside the BlowOff method that caused it to pass a NULL value. If you have the source code for the CRASHATLAPP component handy, you can double click on line 2 in the stack.

Figure 5: Windows Debugger dialog box showing the stack

This brings up the source code for the BlowOff method.

Note: You can specify the path to the source code in WinDBG by clicking Source File Path on the File menu.

Figure 6: Windows Debugger dialog box showing BlowOff method source code

The source code clearly illustrates that a NULL value was passed as the parameter to OutputDebugString.

Although you won't come across such mistakes in production code, the example gives you an idea of the steps involved in debugging and identifying the cause of an access violation error.

ASP Server Too Busy Error

An ASP Server Too Busy error is generally due to the ASP request queue filling up. Significant amounts of queuing may occur when an IIS Web server experiences a very high load or is waiting on a blocking thread. This normally occurs when a component on which the IIS Web server is waiting is invoked at a rate greater than the component can normally satisfy. All the incoming requests are placed in the ASP Request queue and processed in the order in which they were received. If the blocking lasts for a very short period, the queuing mechanism will eventually get rid of the problem and start servicing the requests in a timely fashion. But if the blocking problem lasts for a longer period of time, the number of requests that are queued increases and eventually peaks at the ASPRequestQueueMax metadata setting, which is by default 3000. If the queue size is under 3000, the user with the 2999th request will see an hourglass as the request is in the queue and waits for the all the other requests to be filled. Even at this point, if the queue clears up quickly the request will be serviced. On the other hand, if the queue size reaches the ASPRequestQueueMax value and a new request comes in, the IIS Server returns the ASP Server Too Busy error.

ASPProcessorThreadMax and ASPRequestQueueMax Settings

The ASPProcessorThreadMax and ASPRequestQueueMax metadata settings greatly affect the performance of the site. The ASPProcessorThreadMax parameter specifies the maximum number of threads that a processor can serve. The ASPRequestQueueMax parameter specifies the maximum size of the request queue. These two settings can be changed by the following commands:

adsutil.vbs set w3svc/AspProcessorThreadMax NewValue

adsutil.vbs set w3svc/AspRequestQueueMax NewValue

For well-written scripts, the ASPProcessorThreadMax value should be low. The primary goal of tuning the ASPProcessorThreadMax is to get processor utilization above 50 percent during peak load.

The physical configuration of each site varies widely, so it is difficult to predict an optimum value for these parameters. You should use Performance Monitor to monitor critical counters to arrive at a decision regarding these values. The statistics using the Performance Monitor should include:

Processor time: %Processor time

ASP: Requests per second, Requests rejected, Requests queued

You can monitor these values in a production site during peak load. Typically, in a high-volume site you can observe that the number of requests queued goes up and down. If the processor utilization is low and the total queue length never goes up, it means that you have a site with a capacity more than you actually need. If the total queue length is going up and down and processors are running below 50 percent, it is an indication that some of the requests are blocking and that you can benefit from increasing the number of threads.

If after increasing the threads you observe that the number of requests queued is worse than before, and that processor utilization is also getting low, then you have serious blocking problems. A typical blocking problem might be caused by an SQL query made to the database that is taking very long to execute. Increasing the number of threads allows more requests to come in for the page that executes the query; more pressure is put on the database, which makes the query response time even worse.

To arrive at a good number for ASPRequestQueueMax, select a maximum acceptable response time, calculate the size of a queue that would clear within that time, and set the value of ASPRequestQueueMax just below that size.

A Debugging Example

Having dealt with the ASPProcessorThreadMax and ASPRequestQueueMax parameters in detail, let's try to simulate the ASP Server Too Busy error and try to debug which component's thread is blocking and why. Assume that you have an ASP page that is making a call to a component that is blocking.

To simulate the problem, let's set the ASPRequestQueueMax parameter to 20 by using the following command:

cscript c:\inetpub\AdminScripts\adsutil.vbs SET w3svc/ASPRequestQueueMax 20

To start the Performance Monitor:

Click Start, point to Programs, point to Administrative Tools, and then click Performance Monitor. Right-click the chart and then click Add Counters.
In the Add Counters dialog box, select Active Server Pages from the Performance Object list box.
From the Select counters from list, select Requests Executing and Requests Queued. Click Add, and then click Close.

Assume that you make a call to a component in your ASP code that is blocking the request.

After adding these counters, watch the values of the counter. You can observe that when the requests queued counter reaches the ASPRequestQueueMax value (20), the server returns the ASP Server Too Busy error.

Figure 7: Performance Monitor showing the Requests Queued counter value

Figure 7: Performance Monitor showing the Requests Queued counter value

To debug the inetinfo process to determine which thread is blocking and why, start the WinDBG and attach to the inetinfo process. The most common cause for a thread blocking is a deadlock. You should try to identify the critical section that has a lock count greater than 0 and its owning thread ID. This thread is the culprit. Once you have the thread ID, you can debug the thread stack to find out which specific method is blocking. To begin, type the following command in the command window:

!locks

This should list the threads that have a lock count greater than zero. The output will be something like this:

CritSec ?g_cs@@3U_RTL_CRITICAL_SECTION@@A+0 at 100355A0 
LockCount 24
RecursionCount 1
OwningThread 6e8
EntryCount 18
ContentionCount 18
*** Locked

Look for the critical section that has lock count greater than zero and note the owning thread ID.

Sometimes the !locks command does not provide the required information, usually because symbols for the ntdll.dll are not lined up correctly. In that case, you can use the following crude but effective method.

In the command window, type ~*kb to display the stacks of all the threads. Look for the thread stack that contains the NTDLL!RtlEnterCriticalSection. Once you have identified the thread that has the entry, note the value of param1 in the line that contains NTDLL!RtlEnterCriticalSection. This value is the critical section object passed to the EnterCriticalSection API**.** Using that value, type the following command in the command window:

!Critsec 100355A0

Using either of the above methods, you should be able to identify the thread ID that has a lock count greater than 0. Convert the thread ID from hexadecimal to decimal.

Let's examine the thread and identify the method that is blocking the thread.

On the Debug menu, click Threads.
In the Threads dialog box, click the thread ID you identified in your previous step, and click Select.
On the View menu, click Call Stack. This displays the stack of the thread. Start debugging the stack as we did in the previous (Access Violation) example.

100 Percent CPU Problem

The 100 percent CPU problem occurs when one thread consumes all the processor time. One possible cause is a program loop that monopolizes CPU resources. Performance Monitor is the perfect tool for troubleshooting 100% CPU utilization problems.

First, we need to create a Performance Monitor log.

Preparing to Log

To start the Performance Monitor:

Click Start, point to Programs, point to Administrative Tools, and then click Performance Monitor.
Right-click Counter Logs, and then click New Log Settings. Give the log a name of your choice and click OK.
On the General tab, click Add to add counters to the log.

Add the following counters:
- ASP: All counters
- Memory: All counters
- Process: All counters and all instances
- Processor: All counters and all instances
- Thread: All counters and all instances
- Web service: All counters and all instances
After adding all the counters, change the Sampling Interval to every 1 second, and then click OK.

Now you are all set to start logging the data. Right-click the log name and then click Start. Make sure that the log captures all the data during the time when processor utilization shoots up to 100 percent. Start WinDBG and attach it to the inetinfo process. Once the processor utilization value comes back to normal, you can stop logging by right-clicking the log name and clicking Stop.

Examining the Log

Now that we have created the log, we can start examining it closely to identify the process that was using up most of the CPU time, the thread inside that process that was eating the CPU time, and why it was doing so.

The first step is to identify which process was using up most of the CPU time. To open a log file, do the following:

Click the System Monitor folder under the console root.
In the detail pane, right-click the chart, and then click Properties.
In System Monitor Properties, click the Source tab. Select Log file and browse to select the log file you just created.
Right-click the chart, and then click Add Counters.
In the Add Counters dialog box, select Process from the Performance object list. Click Select counters from list, and select %Processor Time from the list box. Select All instances. Click Add, and then click Close.

This gives us a list of all the processes and their CPU utilization. Now you need to find out which process caused the 100% CPU problem.

Make sure that the Highlight tool button (with a bulb-like icon) is clicked. Now you can scroll through the counters and delete all except the one that is nearest the 100% mark. Make sure that you select only the counters that are relevant to you; for instance, if you are debugging the 100% CPU utilization, you should probably be looking at inetinfo or other out-of-process COM applications/packages (MTX.EXE/DLLHOST.EXE) or IIS applications.

Once you have isolated the problem at the process level, you can concentrate on the threads that were active inside that process during the 100% CPU problem and try to isolate the thread that caused the problem. For that purpose, let's add more counters to display the CPU utilization of all the threads inside that process.

For this example let's add counters for all the threads under the inetinfo process.

Right-click the chart and click Add Counters. In the Add Counters dialog box, select Thread from the Performance object list**.** Select Select counters from list, and select %Processor Time from the list box. Select Select instances from list, and select all the instances that begin with inetinfo/*. Click Add, and then click Close.

This will display the CPU utilization for all the threads inside the inetinfo process in the graph.

Now you can scroll through the counters and remove all the counters except the one that is nearest to the 100% mark. This will bring us down to the exact thread that was causing the problem.

Figure 8: Performance Monitor showing CPU utilization counters

We are left with a single thread with the instance ID 34 that was utilizing the maximum CPU time. Now we have to find the thread ID of this thread and look it up in WinDBG and debug it.

To find the thread ID, right-click the chart and click Add Counters. In the Add Counters dialog box select Thread from the Performance object list**.** Click Select counters from list, and select ID Thread from the list box. Click Select instances from list, and select the instance inetinfo/34. Click Add, and then click Close.

This gives us the thread ID of the thread that is causing the problem. Select the ID Thread counter. Note the Last, Maximum, Minimum, and Average values of this counter. Typically all four values are equal. Note the value (764 in this example) and switch to the WinDBG application.

Figure 9: Performance dialog box showing ID Thread counter

On the WinDBG View menu, click Threads. This displays a list of threads that were running inside the inetinfo process when the 100% CPU problem was encountered. Look for the thread ID number you had noted from the Performance Monitor. If you find it in the list, click it and then click Select. Click OK.

On the View menu, click Call Stack. This displays the stack of the thread that is causing the problem. The stack isolates the problem method. Once we find the method, we can go back to the source code and see which line of code caused the problem.

Conclusion

You can use the IIS Exception Monitor, WinDBG, the User Mode Process Dump tool, and Performance Monitor to effectively solve the most common problems in Distributed Web Applications. These tools do not necessarily have all the answers and solutions to a problem, but they can point you in the direction that will eventually lead to a solution.

References and Additional Resources

Debugging Resources

Once your applications are deployed in the test lab, you will begin to uncover bugs. Thorough testing and debugging prior to deployment is critical. Windows 2000 with IIS 5.0 provide new features for debugging applications. The Customer Support Diagnostics package contains symbols and debugging tools for diagnosing your system. These can help you resolve bugs on your own, or help you successfully pinpoint and resolve bugs with support services.

In addition to the traditional Windows NT debugging tools (DrWtsn32, Windbg/WindbgRM, kernel debugger (KD), NT software debugger (NTSD)/CDB, and so forth) a new set of debugger-related tools are included with the Windows 2000 Support Tools. This tool set includes debugger extensions that examine kernel data structures, tools for checking the memory pool, and an intelligent kernel memory-dump analysis tool.

These troubleshooting and development tools extend the functionality of existing debugging tools such as Kernel Debugger, provide new features upon which new debugging methodologies can be based, and provide development support.

Recommendation: Download the Windows 2000 Customer Support Diagnostics for obtaining symbols and debugging tools.

Windows 2000 Customer Support Diagnostics:
Debugger Related Resources on the Microsoft Web site:

Debugging COM Applications

As their needs grow, many dot-com businesses add more powerful Component Object Model (COM)based applications. Developers who take up COM to meet certain objectives for their online businesses may be more familiar with scripting models and often have questions about the more complex issues of debugging COM. The following online resources relate to debugging COM in Windows 2000 with IIS 5.0.

Recommendation: Check out the following reading resources for help debugging COM components in IIS.

192754, HowTo: Debug InProc COM Components Inside IIS Using WinDbg
166275, HowTo: Debug a Native Code Visual Basic Component in VC++
177182, Debugging a VB DLL Under Active Server Pages
183480, Debug ISAPI and Filter DLL Under IIS 4.0
Monitoring Events Using Visual Studio Analyzer

Calling Product Support

When you need Microsoft for technical support, you can try self-help by searching the Personal Online Support site's extensive collection of articles from the Microsoft Knowledge Base. When you need assistance from a support professional, the following tips will help you reach a successful resolution to your problem(s):

Describe/Isolate the problem. The better the description of the problem, the easier it will be to isolate. If possible, make sure you can isolate the problem to a specific area.
Reproduce the problem. Describe in detail, the steps the Support Professional can use to reproduce the problem.
Assess the impact of the problem. What is the impact of the problem on your product or business?

Recommendation: Gather information to prepare before contacting Microsoft Product Support. The following articles offer advice for using online self-help.

HowTo: Find Answers Online
246453, HowTo: Search for ASP Knowledge Base Articles -
242971, HowTo: Use the Developer Support Microsoft Knowledge Base
252318, HowTo: Search for COM+ Knowledge Base Articles

Troubleshooting Blue Screens

When a fatal system error occurs in Windows 2000, it enters debug mode for troubleshooting purposes. This appears as a blue screen and the first few lines look similar to the following:

Stop 0x0000001e (c000009a 80123f36 02000000 00000246) 
Unhandled Kernel exception c000009a from 8123f26 
Address 80123f36 has base at 80100000 - ntoskrnl.exe

The following Knowledge Base articles contain information to help you identify the cause of the Blue Screen and gather more information about it before you contact Microsoft Product Support. You can also search the Microsoft Product Support site for more information about fatal system errors, and troubleshooting tips, using the keywords: "kberrmsg" "kbtshoot" and "ntstop."

Recommendation: Read KB articles on Blue Screen information.

129845, Blue Screen Preparation Before Contacting Microsoft
192463, Gathering Blue Screen Information After Memory Dump

Additional Reading

Understanding the IIS Architecture

Debugging Active Server Objects

Additional Knowledge Base Articles

229814, Configuring IIS to Handle Heavy Usage

150934, HOWTO: Create a Performance Monitor Log for NT Troubleshooting

288965, HOWTO: Create a VBScript Class or Jscript Object to debug ASP Pages

183480, HOWTO: Debug ISAPI DLLs Under IIS 4.0 and IIS 5.0

258929, HOWTO: Debug ASP Applications Manually Against Windows 2000 Web Server

253603, HOWTO: Troubleshoot a High-CPU-Utilization "Hang" in Internet Information Services

160360, INFO: Troubleshooting Exceptions in Internet Server Products