Monitoring and Troubleshooting Performance

As an edge server connecting many networks, Microsoft® Internet Security and Acceleration (ISA) Server 2004 handles large amounts of traffic compared to other servers in an organization. For this reason, it is built for high performance. Yet, there are circumstances where high performance is not realized due to misconfiguration or inadequate resources. This article provides information about how to monitor and troubleshoot perceived performance problems.

Diagnosing Performance Problems

Solving Performance Problems

Monitoring Performance

Resource Counters

Diagnosing Performance Problems

In some deployment situations, the performance of ISA Server may be perceived to be slow or unacceptable. The end-to-end user experience that is measured in terms of the total time required to complete a transaction, load a Web page, or download a file is unsatisfactory.

This could be the result of:

• A resource on the ISA Server computer that is inadequate in capacity and therefore becomes a performance bottleneck.
• A deployment problem in the interaction of ISA Server with other components in the system or network.
• A security incident such as a denial of service (DoS) attack consuming a resource to its full capacity.
• In most cases, performance problems can be solved with proper diagnosis based on:
• Accurate definition and measurement of the performance problem (the symptom).
• ISA Server monitoring and system performance counters.
• ISA Server log events.
• Network captures.

Sources of Performance Problems

Performance problems can be based on inadequate capacity, deployment problems, or security incidents. The following sections describe these situations in detail, and how to diagnose each situation using available tools.

Inadequate Capacity

ISA Server capacity depends on CPU, memory, network, and disk hardware resources. Each resource has a capacity limit, and as long as all resources are consumed below their limit, the server as a whole functions properly, fulfilling its performance objectives. Performance drops considerably when one of these limits is reached, causing a bottleneck. Each bottleneck has symptoms that can help detect the resource that has inadequate capacity.

Solving a capacity problem is achieved by tuning the hardware or adding more of the resource that is inadequate. For more information about ISA Server performance tuning, see the document ISA Server 2004 Performance Best Practices.

A resource bottleneck is not always an indication of a capacity problem. When the problem is not a result of inadequate capacity, adding or tuning the hardware may not help solve the problem.

Deployment Problems

As a central network infrastructure component, ISA Server has many interactions with other network components, servers, and applications. The following are some examples:

• To enforce policy, ISA Server must communicate with other types of servers in various situations. If a Domain Name System (DNS) server or an authentication server fails to respond in a timely manner, ISA Server will fail to respond quickly on those requests that require server intervention.
• ISA Server is commonly connected to many physical networks. Improper configuration of these networks may lead to various performance problems, such as Maximum Transfer Unit (MTU) size conversions, packet loss, and retransmissions.
• Serving as a secure traffic pipe, ISA Server interacts with numerous server applications, such as Web, mail, and media. Improper configuration of these servers and the services they execute may lead to performance problems, such as a high Transmission Control Protocol (TCP) connection rate or limited packet rate.

The first step in solving a deployment problem is to identify the component that is causing poor performance.  

Security Incidents

ISA Server implements numerous methods to defend against many network attack vectors. Nevertheless, as an edge server and a central network infrastructure component, it is the first target for attack attempts. Securing at the application level requires ISA Server to manage compound protocol states that have larger resource consumption as compared to lower levels of filtering, such as packet and transport layers.

It may be possible that a performance problem surfaces as a result of an attack. The most common attacks that consume system resources are denial of service (DoS) attacks and distributed DoS attacks (DDoS). The sources of these attacks can be hackers on the Internet accessing through the External network, as well as clients on Internal networks that are infected by viruses and worms that are trying to propagate or attack network resources.  

Step-by-Step Procedure

When you encounter a performance problem, use the following procedure to diagnose the problem:

1. Accurately define the problem in measurable terms. An accurate definition leads to a simple measurement. For example, loading a page from a published Web site takes 30 seconds.
2. Examine resource performance counters and record those that have suspect levels (for example, more than 75 percent total processor utilization). For more information, see Resource Counters in this document. If a resource is at a suspect level, it may be a capacity problem. Go to step 4.
3. If all resources are within normal operation limits, it is probably a deployment problem. Search the ISA Server log and the operating system event log, and record any suspicious events. If logs do not show the problem, create a network capture on all network interfaces. Search for large time gaps between a request and a response. For a list of most common cases, see Solving Deployment Problems in this document.
4. If at least one resource is at full capacity, search the ISA Server log and record any suspicious events that may indicate a DoS or DDoS attack. To identify possible known attacks and ways to mitigate them, see Solving Security Problems in this document. If there is no sign of a security incident, continue with step 5.
5. If at least one resource is at full capacity, to identify possible deployment problems that correlate with high resource consumption, see Resource Intensive Deployment Problems in this document. If no deployment problem is identified, continue with step 6.
6. If some resources are at full capacity, and require adjustment and tuning to identify and solve the problem, see Solving Capacity Problems in this document.

Measurement Problems

There are situations in which ISA Server seems to be performing poorly as compared to other deployment options. This happens when using improper measurement methods and metrics to compare the response time of a system that has an ISA Server computer between a client computer and a target application server to a system that has no ISA Server computer deployed.

Web Publishing Performance

One example of measurement problems occurs when attempting to measure the throughput capacity in bits per second (bps) or requests per second of an ISA Server computer using a fixed number of best-effort simulated Web clients hitting a published Web server. These clients work independently, sending the next HTTP request immediately upon receiving the previous response. This method is common in many load generation tools such as Microsoft Application Center Test (ACT).  

The results of this test show that a Web server published by an ISA Server computer has less throughput than hitting the Web server directly. This is a true outcome but not a true statement. The problem with this method is that it does not measure throughput but rather measures round-trip time (RTT), because the number of bps that the client can generate is dependant on the average RTT for each HTTP transaction:

bits/sec = average_bits/transaction * average_transactions/sec =

average_bits/transaction / average_RTT

The average RTT is expected to increase in this scenario when deploying an ISA Server computer, as compared to hitting the Web server directly. This is because the ISA Server computer acts as another network hop between the clients and the Web server. Every hop has a penalty on RTT, especially if the TCP stack is fully traversed upon entrance and exit of each packet through the application-layer Web Proxy filter.

A true metric that should be measured for ISA Server throughput is bps at 80 percent CPU utilization. Because bps per simulation agent decreases as a result of an increase in average RTT, you need to add more simulation agents to reach the full processor capacity of the ISA Server computer.    

To summarize, ISA Server is not expected to improve the response time in Web publishing scenarios. With properly tuned caching and Web content, it will enable considerable offloading from the back-end Web server, allowing for Web server consolidation.  

Forward Web Proxy Response Time

The same situation as described in the previous section arises when measuring Internet access speed with speed measurement sites. These Web tools measure round-trip time (RTT) of HTTP requests sent to many popular Web sites around the world bypassing all the Web caches on the way. Similar to Microsoft ACT, it does so with best-effort clients. As with Web publishing, ISA Server adds several milliseconds to the RTT, which in most cases is unnoticeable because the majority of RTT is spent over the Internet and Web server. But due to the best-effort behavior of these clients, an expected increase in RTT is mistakenly considered as lower throughput.   

Solving Performance Problems

The following sections describe various performance problems, their symptoms, and how to solve them. These sections are divided into the following categories:

• Deployment problems. Caused by various misconfigurations in ISA Server or the network infrastructure. These problems are characterized by limited throughput, slow response, and low CPU utilization.
• Resource intensive deployment problems. These are more difficult to identify because they occur and amplify intensive CPU utilization conditions.
• Security problems. Caused by DoS or DDoS attacks.
• Capacity problems. Where some resource is at full capacity and requires tuning, or the capacity needs to be increased.

Solving Deployment Problems

Solving deployment problems involves domain name resolution, domain controllers, TCP Nagle algorithms and delayed acknowledgements, and network problems.

Domain Name Resolution

ISA Server requires Domain Name System (DNS) for various name resolutions. For example, when receiving an HTTP request with a host name that is an IP address, ISA Server must perform a reverse DNS lookup to get the domain name of this IP address, because it could be blocked by some URL set.

When DNS does not respond in a timely manner, worker threads will be blocked on pending DNS responses, and the number of backlogged packets will consequently increase. The symptom is characterized by:

• \ISA Server Firewall Packet Engine\Backlogged Packets > 10
• \ISA Server Firewall Service\Worker Threads > 100
• Network captures show gaps of several seconds between DNS queries and their responses.

There are various ways to solve the problem depending on its nature. For more information, see:

• ISA Server Site and Content Rules Are Not Enforced for HTTP Content, at Microsoft Help and Support.
• Only the First Web Site Is Returned Using Web Publishing for Multiple Sites, at Microsoft Help and Support.

Domain Controller

ISA Server interacts with the domain controller in various authentication configurations. When the domain controller does not respond in a timely manner, worker threads will be blocked on pending authentication requests, and the number of backlogged packets will constantly increase. The symptom is characterized by:

• \ISA Server Firewall Packet Engine\Backlogged Packets > 10
• \ISA Server Firewall Service\Worker Threads > 100
• Network captures show gaps of several seconds between authentication requests to and the domain controller responses.

To solve this problem, increase the number of concurrent pending authentication requests as described in HOW TO: Configure your ISA Server for a Very Large Number of Authentication Requests, at Microsoft Help and Support.

TCP Nagle and Delayed ACK

Transmission Control Protocol (TCP) combines various delays to optimize TCP performance by reducing small packet traffic. On the sending side, TCP uses the Nagle algorithm that delays outbound packets for interactive applications. On the receiving side, TCP uses delayed acknowledgements (ACK) to increase probability of piggybacking the ACK on data that is sent back. However, in some circumstances these two mechanisms can cause a fixed delay of 200 milliseconds resulting in five packets per second throughput. ISA Server does not contribute to this effect, but because of its role in the middle, ISA Server can mistakenly be blamed for it. The symptom can be seen in a network trace when sent packets are acknowledged after a 200 millisecond delay.

The solution to the problem is changing the behavior of the application. For details about how to avoid this issue in applications, see INFO: Design Issues - Sending Small Data Segments Over TCP w/Winsock, at Microsoft Help and Support. For overcoming common TCP performance problems, see Common Performance Issues in Network Applications Part 1: Interactive Applications, on the MSDN Web site.

Network Problems

Network problems affecting ISA Server performance include various hardware and configuration problems. Like routers, ISA Server is commonly deployed at network junctions, so the same types of networking problems that are encountered in router configurations are likely to be encountered in ISA Server deployments. In these cases, it is useful to refer to operating system deployment and troubleshooting guides.

Solving Resource Intensive Deployment Problems

Solving resource intensive deployment problems involves packet fragmentation, application and Web filters, and server-side HTTP Keep-Alive.

Packet Fragmentation

Packet fragmentation happens when two networks use a different Maximum Transfer Unit (MTU) size to transmit data. For example, a local area network (LAN) commonly uses an MTU of 1,460 bytes, and a wide area network (WAN) may use a smaller packet size (536 bytes). When packet fragmentation happens, the TCP stack on the ISA Server computer consumes many processor cycles in an effort to manage buffers and transfer data between buffers of various sizes.

The symptom can be seen in a network capture looking at the packet sizes on each network interface. In many cases, it is possible to tune MTU size. For a list of MTU sizes on different network media types, see Default MTU Size for Different Network Topology, at Microsoft Help and Support. For information about tuning MTU and other TCP registry values, see TCP/IP and NBT configuration parameters for Windows XP, at Microsoft Help and Support.

Application and Web Filters

ISA Server application filters and Web filters provide the application logic filtering extensions. Without these filters, ISA Server provides only stateful filtering with various security measures that are common to all applications (such as quota restrictions or access policy). As logic extensions to ISA Server, filters have a direct effect on ISA Server performance.

One way to measure the effect of a filter on the performance of ISA Server is to compare the CPU utilization of ISA Server under realistic load with the filter enabled and disabled. If the difference is not reasonable for the amount of computation that the filter performs, you have identified one possible cause to the performance degradation.

The filter developer should be notified about filter performance problems. In many cases, the diagnosis outlined will provide the required information to the filter developer to identify and fix the problem.

Server-Side HTTP Keep-Alive

Opening and closing TCP connections causes performance overhead for any network server. To lower this overhead, HTTP includes a Connection: Keep-Alive header that instructs both client and server sides to reuse a connection for many requests. ISA Server Web Proxy takes this optimization one step further because requests from many clients are routed through a single connection to the upstream Web server, thus increasing connection reuse. Yet, misconfiguration of upstream Web servers, especially in a Web publishing scenario, may cause every request to end with a connection close. This has a direct effect on performance that is difficult to see without network tracing.

The symptom can be seen in a network trace on the upstream network interface, when every connection is closed after a single request. Solving the problem starts with understanding which side closes the connection, and why it does so. If an upstream Web server closes all connections after sending a single response, check whether it is possible to configure it to reuse its connections.  

If ISA Server closes the connection, it could be as a response to incompatible or ambiguous HTTP syntax.

Solving Security Problems

Security problems affecting ISA Server performance are DoS and DDoS attacks. These attacks are characterized by the full consumption of one or more resources of ISA Server. From a performance view, there is no difference between a capacity problem and a security problem, because in both cases the performance of ISA Server suffers due to a resource bottleneck. Still, there are many indications that can lead to a conclusion that the source of a performance problem is a security incident.

ISA Server uses various mechanisms to automatically detect and block security incidents that lead to DoS conditions:

• TCP SYN attacks. Automatic detection and protection.
• UDP or raw IP flood. Automatic detection and protection by use of per-rule connection quota.
• Virus or worm propagation. Automatic detection and protection by use of per-IP connection quota.

In these cases, alerts are triggered, enabling the ISA Server administrator to examine the nature and source of the attack, and use preventive measures to eliminate it.

Identifying a DoS or DDoS attack requires input from all monitoring sources:

• Performance counters show how much a resource is consumed, as well as other numbers that have suspect levels triggering further examination with other sources.
• ISA Server logs show irregular denial patterns that correlate with a set of ports or IP addresses that are denied access. In most cases, looking at the ISA Server logs provides the necessary information to identify and solve a security incident.
• Network captures can also show irregular traffic patterns but at the lower network level. Use network captures in cases where ISA Server logs do not provide adequate information.

When identifying a DoS security incident that is not automatically detected and blocked by ISA Server, contact Microsoft Help and Support.

Solving Capacity Problems

Capacity problems occur when a resource reaches full capacity. The following sections describe each resource, its capacity limit, and how to solve a capacity problem. Note that at high loads, more than one resource will reach full capacity. In this case, solve them one by one, according to the section order.

For best practices concerning performance tuning and capacity planning, see ISA 2004 Performance Best Practices.

Hard Page Faults

Hard page faults occur when processes compete for physical memory space. Whenever a process is running and allocated processor time, it uses memory. If there is not enough physical memory to hold all the virtual memory it requires while running, hard page faults are triggered by the operation system, causing physical memory pages to be swapped with virtual pages in a page file. Hard page faults cause unwanted input/output (I/O), resulting in enormous response delays. The symptom is easily identified by:

• Average \Memory\Pages/sec > 5 for over several minutes.

To solve a memory problem, do one or more of the following:

• Identify which processes are using this memory by looking at \Process(*)\Private Bytes, \Process(*)\Virtual Bytes and \Process(*)\Working Set, and consider disabling high memory consuming processes.
• If Web caching is enabled, consider reducing the memory Web cache size.
• Add more memory.

Network at Full Utilization

Network at full utilization means that the underlying bandwidth of a network link is fully saturated. The indication that this is the case is:

Average \Network Interface(*)\Bytes Received/sec or

\Network Interface(*)\Bytes Received/sec is more than 80% of \Network Interface(*)\Current Bandwidth

If the majority of this bandwidth is normal application traffic, solving this problem can be achieved as follows:

• For outbound Web proxy traffic, consider using a Web cache if not already deployed. This will lower bandwidth requirements on the upstream Web link, and will improve the average response time on the downstream links.
• Consider adding more bandwidth to the saturated network link. In most cases, this will mean increasing the bandwidth of the WAN Internet link.

Disk Transfers at Maximal Rate

Physical disk transfers are limited to approximately 100 random accesses per second. (This is true for disks spinning at 10,000 revolutions per minute (RPM). At 15,000 RPM, the limit is approximately 130 I/O per second). When attempting more than this limit, all responses that are dependent on I/O will suffer enormous delays.

A disk that has reached this limit will be characterized by:

Average \PhysicalDisk(*)\Disk Transfers/sec > 100 over several minutes for some disk

This can happen in the following cases on an ISA Server computer:

• Disk Web cache is not tuned to serve all the I/O generated by requests for content in the disk.
• Logging creates too much I/O for a single disk to handle. This may happen in extreme cases of high load when using Microsoft SQL Server™ 2000 Desktop Engine (MSDE) logging.

In the first case, the solution is to add another physical disk. For redundant array of independent disks (RAID) storage systems, it is important to select RAID-0 to be able to reach the maximal I/O on each disk.

In the second case, check whether logging can be disabled for some rules, lowering the total log entries to be written. Another option is to switch to text logging, which requires considerably less I/O.

Processor at Full Utilization

A processor at full utilization is characterized by high percentage of processor time. For ISA Server, the recommended maximum is 80 percent. When processor utilization is high for periods of several minutes, this could lead to longer response times, especially if there are peaks reaching 100 percent utilization:

Average \Processor(*)\% Processor Time > 80 over several minutes for some processor

\ISA Server Web Proxy\Average Milliseconds/request > 30,000

When this happens, it is important to record how this utilization is divided between user time (\Processor(*)\% User Time) and kernel time (\Processor(*)\% Privileged Time). In most scenarios, except for intensive content processing scenarios such as Secure Sockets Layer (SSL), user time will be less than kernel time. If this is the case, it is likely that the bottleneck is not caused by inadequate processor capacity. In this case, use \Process(*)\% Processor Time to determine if some other process is extensively using the CPU at the same time.

Before increasing the processing power of the ISA Server computer, consider the following:

1. Check filters and filter configurations for performance intensive options that may be disabled or relaxed. For example, a Web filter performing virus scanning could be configured not to scan some content types, such as images or text files that are not harmful from a security view.
2. Replace MSDE logging with text logging.
3. Review policy and check whether it is possible to use stateful filtering instead of application filtering for traffic that is considered harmless.

Monitoring Performance

To maintain and manage the health of ISA Server, it is necessary to constantly monitor its performance. The following sections list resource counters and ISA Server counters that help troubleshoot ISA Server performance problems. We recommend monitoring these counters on a regular basis at a reasonable sampling rate of several samples per minute.

Resource Counters

The following table describes resource counters.

ISA Server Counters

The following table describes ISA Server counters.