Best Practices for Logging in ISA Server 2004

This guide is designed to provide you with essential information about logging for Microsoft Internet Security and Acceleration (ISA) Server 2004 Standard Edition and ISA Server 2004 Enterprise Edition. The guide reviews the logging formats, describes specific logging maintenance considerations, details capacity guidelines, and outlines special considerations when logging to a Microsoft SQL Server database.

This guide focuses explicitly on best practices to follow when configuring logging as part of your ISA Server deployment. You should use this guide as part of your overall deployment strategy for ISA Server 2004. Specifically, this guide provides detailed answers to the following questions:

• What is the most appropriate logging format for my specific deployment?
• How should I optimally secure the logs?
• How should I maintain the logs?
• What special considerations are there when logging to an SQL database?
• What special considerations are there when logging to a Microsoft SQL Server 2000 Desktop Engine (MSDE 2000) database?

Log Storage Format

General Logging Best Practices

SQL Logging

MSDE Logging

Additional Information

Log Storage Format

You can use the ISA Server 2004 log viewer to monitor and analyze traffic and troubleshoot network activity. The log viewer can display log entries as they occur (live). In this case, each time an event is logged, it is displayed in the log viewer.

ISA Server creates the following logs:

• Firewall log
• Web Proxy log
• SMTP Message Screener log

The fields that can be logged in these files are detailed in online Help.

ISA Server log information can be viewed in a log viewer, directly from ISA Server Management. In addition, the log information can be stored in one of the following formats:

• File
• MSDE database
• SQL database

Selecting Log Format

Each log format supported by ISA Server features different advantages. Use the table that follows to select the optimal log format, based on your specific deployment.

Issues	File	MSDE	SQL
Format	Two modes: Internet Information Services (IIS) and World Wide Web Consortium (W3C) standardized text formats	Format used to store Firewall and Web Proxy log entries	Format used to store Firewall and Web Proxy log entries
Network bandwidth consumption	Because logging is local, no network bandwidth consumption	Because logging is local, no network bandwidth consumption	Because logging is to remote server, sufficient network bandwidth is required, preferably 1 gigabyte (GB) connectivity between ISA Server and computers running SQL Server
Log size	Limited to 2 GB and switched automatically	Limited to 1.5 GB and switched automatically	No limit, and configured by the user, based on retention and maintenance policy
Maintenance	Log maintenance feature enforces log size and cleans out log, as appropriate	Log maintenance feature enforces log size and cleans out log, as appropriate	Database administrator responsible for maintenance
Security	Log failure stops Firewall service	Log failure stops Firewall serviceMSDE runs on the ISA Server computerMSDE instance can only be accessed locally	Log failure stops Firewall serviceAccount used for logging must have permissions on the computer running SQL ServerData is encrypted on the connection to the computer running SQL ServerSQL Server and ISA Server are mutually authenticated
Historical or offline log viewer	Not supported	Supported	Supported (ISA Server Enterprise Edition only)
Online log viewer	Supported	Supported	Supported
Performance	Best	Good	Depends on the following:Number of ISA Server computers loggingSQL Server settingsBandwidth allocation
Centralized logging (ISA Server Enterprise Edition only)	Central log for all array members	Central log for all array members	Central log for all arrays in the enterprise

File

You can save ISA Server logs to a file, in one of the following formats:

• World Wide Web Consortium (W3C) format
• ISA Server format

The SMTP Message Screener log information is saved by default in file format. It cannot be saved to a database.

Log files are limited to 2 GB. When a file exceeds this limit, ISA Server automatically creates a new file. Similarly, a new log file is created at the beginning of every day.

W3C logs contain both data and directives, describing the version, date, and logged fields. Because the fields are described in the file, unselected fields are not logged. The tab character is used as a delimiter. Date and time are in Coordinated Universal Time (UTC).

ISA Server format contains only data with no directives. All fields are always logged. Unselected fields are logged with a dash, to indicate that they are empty. The comma character is used as a delimiter. The date and time fields are in local time.

By default, the log information for log files is stored in the ISALogs folder, under the ISA Server installation folder. You can change the location. If you specify a relative directory, the log is saved in the ISALogs folder, under the ISA Server installation folder. If you specify an absolute path, the actual log folder may be different on every server.

MSDE Database

MSDE 2000 logs are limited to 2 GB. When a log exceeds this limit, ISA Server automatically creates a new database. Similarly, a new log is created at the beginning of every day. The log viewer, however, displays all the data as if it were in a single database.

When you select to save the logs to an MSDE 2000 database, logs are saved in databases named ISALOG_yyyymmdd_xxx_nnn where:

• yyyy represents the year that the log database refers to.
• mm represents the month that the log database refers to.
• dd represents the day that the log database refers to.
• xxx represents the type that the log database refers to. This can be one of the following:
  • FWS. Represents the Firewall log.
  • WEB. Represents the Web Proxy log.
  • EML. Represents the e-mail (SMTP) log.
• nnn is a number that distinguishes between log databases that refer to the same day.

For each log database, two files are created: ISALOG_yyyymmdd_xxx_nnn.mdf and ISALOG_yyyymmdd_xxx_nnn.ldf.

ISA Server prepares, in advance, log databases for the next day. When you save logs to MSDE 2000, a database that refers to the next day always exists.

By default, the log information for MSDE 2000 logs and log files is stored in the ISALogs folder, under the ISA Server installation folder. You can change the location. If you specify a relative directory, the log is saved in the ISALogs folder, under the ISA Server installation folder. If you specify an absolute path, the actual log folder may be different on every server.

SQL Database

You can save log information to an SQL database. Saving the log information to an SQL database is useful for remote logging.

When you configure logging to an SQL database, you specify the database connection parameters, and credential information.

The system policy rule named Allow remote logging using NetBIOS transport to trusted servers must be enabled to log to an SQL database.

General Logging Best Practices

This section details recommended general best practices to follow when using the ISA Server logs. It also describes techniques to implement in case of log failure or connectivity failure.

General Security Best Practices

Follow these guidelines to help secure ISA Server logs:

• Save the logs to a separate NTFS disk partition for maximum security. Only administrators of the ISA Server computer should have access to the logs.
• If you are logging the information to a remote database, configure encryption and data signature for the log information being copied to the remote database.

General Capacity Planning Guidelines

Regardless of log format, we recommend that you allocate 8 GB for logging. Depending on your specific logging capacity, in addition to the 8 GB, we recommend that you further allocate enough space for an additional day and a half of logging. The amount of space required for a day and a half of logging depends on your specific logging requirements.

Log Failure

Because ISA Server is deployed to secure your network, it is critical that logging information is always available and accurate. You should carefully monitor alerts and verify that their activity is always being logged. Check for alerts that indicate failure to log for a variety of reasons, including disk space, SQL Server connectivity issues, and others.

If log information cannot be saved for any reason, the ISA Server computer should be locked down. For this reason, a preconfigured alert for the Log Failure event stops the Microsoft Firewall service.

By default, if ISA Server cannot log activity, the Microsoft Firewall service is stopped. You can change the default behavior by configuring the log failure alert to not stop ISA Server services.

If the ISA Server computer fails, the last log records may be lost.

Maintaining Logs

After you select and configure a specific logging mechanism, follow the best practices listed in this section to maintain the logs.

Reviewing Logs

Review the logs regularly and carefully, checking for suspicious access and usage of network resources.

Log Maintenance

ISA Server has a log maintenance feature, which you can configure so that log files do not exceed specific space requirements. Use the log maintenance feature to ensure that the disk on which log information is stored does not become full.

When you log to an MSDE 2000 database or to a file, you can configure how long log information should be stored on the local disk, and how much disk space should be allocated for logging.

To configure the Log Storage Limits alert definition to stop the ISA Server services, perform the following steps

1. In the console tree of ISA Server Management, click Monitoring:
   • For ISA Server 2004 Enterprise Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Arrays, expand Array_Name, and then click Monitoring.
   • For ISA Server 2004 Standard Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Server_Name, and then click Monitoring.
2. In the details pane, click the Alerts tab.
3. On the Tasks tab, click Configure Alert Definitions.
   
4. In Alert Definitions, click Log storage limits, and then click Edit.
   
5. On the General tab, select Enable.
6. On the Actions tab, click Stop selected services, and then click Select.
   
7. In Services, select Microsoft Firewall and Microsoft ISA Server Job Scheduler.

To configure log storage limits, perform the following steps

1. In the console tree of ISA Server Management, click Monitoring:
   • For ISA Server 2004 Enterprise Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Arrays, expand Array_Name, and then click Monitoring.
   • For ISA Server 2004 Standard Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Server_Name, and then click Monitoring.
2. In the details pane, click the Logging tab.
3. On the Tasks tab, select the appropriate task:
   • Configure Firewall Logging. Used to configure the firewall log limits.
   • Configure Web Proxy Logging. Used to configure the Web Proxy log limits.
     
4. On the Log tab, select File or MSDE Database, and then click the Options button.
   
5. To limit the size of the logs, select Limit total size of log files (GB). Then, type the maximum log size.
6. To maintain a specified amount of free disk space on the disk where the logs are stored, select Maintain free disk space (MB). Then, type how much free disk space to maintain.
   
7. If you selected either Limit total size of log files (GB) or Maintain free disk space (MB), select one of the following:
   • Deleting older log files as necessary. Used to delete the oldest log files when you exceed the limits specified previously.
   • Discarding new log entries. Used to stop ISA Server from adding any new log entries (while keeping all the old log information).
     
8. To delete old log information, select Delete files older than (days). Then, type how long to keep log information.

Configuring Logs During Flood Attacks

A flood attack occurs when an attempt is made to deny services to legitimate users by intentionally overloading a network. Flood attacks might occur, for example, when a worm tries to propagate outside of your corporate network.

The first symptoms that show that ISA Server is experiencing a flood attack are a sudden surge in CPU utilization, increased memory consumption, or very high logging rates on the ISA Server computer.

If you determine that the ISA Server computer is experiencing a flood attack, use the log viewer to determine the source of the offending traffic. Specifically, look for the following:

• Log entries for denied traffic. Pay special attention to traffic that is denied because the quota is exceeded, spoofed packets, and packets with corrupted CHECKSUM. These usually are indicative of a malicious client. In ISA Server 2004 Standard Edition, connections that are terminated due to exceeding the connections limit will have a result code of 0x80074e23. In ISA Server 2004 Enterprise Edition, the result will appear as text, which clearly indicates the connection termination reason.
• Logs that indicate numerous connections that are created and then immediately closed. This often indicates that a client computer is scanning an Internet Protocol (IP) address range for a specific vulnerability.

Another way to detect and list the offending computers is to temporarily reconfigure the Connection Limit alerts to be triggered every one second (instead of using the Manually Reset option). A list of alerts is generated, each one indicating the offending IP address in the alert text. After you identify the list of offending IP addresses, perform the procedure to log requests matching a rule (described later in this document), to improve the performance of ISA Server during the flood.

When an attack occurs, many events will be logged to the computer running SQL Server. To continue logging despite the large number of events, follow these guidelines:

• By default, the log failure alert is configured to stop the Microsoft Firewall service when it is generated. Consider reconfiguring this alert to send an e-mail message to an administrator's e-mail address. Also, use the ISA Server software development kit (SDK) to create a script that does not drop connections for which traffic is not logged. For example, you can use the script located at the Coding Corner. For more information about using these properties, see ISA Server SDK Help, available in the SDK folder on the ISA Server CD.
• Do not log traffic that is denied by the Default rule.
• Disable logging either on the specific rule that matches the flood or altogether until the flood attack is stopped.
• For example, if a large amount of data is being logged from a specific protocol or source, you can create a new rule, which applies to that type of traffic, for which requests are not logged. For example, suppose your policy does not allow Dynamic Host Configuration Protocol (DHCP) requests, and as a result, there are many DHCP requests that are being denied. You can create a new access rule that denies DHCP requests, but does not log the requests.
• Reconfigure the Connections Limit alerts (or any other types of alerts that may be triggered repeatedly as a result of the specific attack) back to manually reset.

To log requests matching a rule, perform the following steps

1. In the console tree of ISA Server Management, click Firewall Policy or Enterprise Policies (for enterprise-level requests in Enterprise Edition):
   • For ISA Server 2004 Enterprise Edition, for a specific enterprise policy, expand Microsoft Internet Security and Acceleration Server 2004, expand Enterprise, expand Enterprise Policies, and then click Enterprise_Policy.
   • For ISA Server 2004 Enterprise Edition, for array-level firewall policy, expand Microsoft Internet Security and Acceleration Server 2004, expand Arrays, expand Array_Name, and then click Firewall Policy.
   • For ISA Server 2004 Standard Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Server_Name, and then click Firewall Policy.
2. In the details pane, click the rule for which logging should be enabled.
3. On the Tasks tab, click Edit Selected Rule.
4. On the Action tab, select the Log requests matching this rule check box.
   

Connectivity Failure

Network issues, such as floods or congestion, may cause connectivity failure between the ISA Server computer and the logging server. Such connectivity issues will cause ISA Server to enter lockdown mode. To avoid such issues, do the following:

• Use a private network between the ISA Server computer and the logging server.
• Protect the logging servers from receiving traffic from untrusted sources, by allowing them to receive traffic only from ISA Server computers and arrays.
• For optimal security, configure Internet Protocol security (IPsec) for the communication between the ISA Server computer and the logging server.

SQL Data and Transaction Log Location

Best practices on databases recommend separating the physical drives to be used for the data file from the transaction log file. Doing so will improve the overall performance of the SQL-based logs.

SQL Logging

This section describes specific guidelines for SQL logging.

Security Best Practices for Logging to SQL Server

Follow these guidelines to help secure ISA Server logs:

• When you save log information to an SQL database, use Windows authentication (and not SQL authentication).
• Save the logs to a separate NTFS disk partition for maximum security. Only administrators of the ISA Server computer should have access to the logs.
• Configure appropriate security when logging to an SQL database.

Specifically, limit the privileges allowed to the SQL account that has access to the ISA Server log tables. Even if the ISA Server firewall is compromised, the malicious attacker cannot delete old log entries. Additionally, allow use of the SELECT and INSERT SQL commands only when the designated SQL account is set for ISA Server logging purposes.

Connectivity Failure

Network issues, such as floods or congestion, may cause connectivity failure between the ISA Server computer and the logging server. Such connectivity issues will cause ISA Server to enter lockdown mode. To avoid such issues, do the following:

• Use a private network between the ISA Server computer and the logging server.
• Configure IPsec for the communication between the ISA Server computer and the logging server.
• Deploy the computer running SQL Server in a separate network and configure a special firewall policy allowing traffic only between the SQL Server network and ISA Server 2004.

The Remote Logging (SQL) system policy configuration group must be enabled to log to an SQL database.

Capacity Planning for SQL Logs

We recommend that if you are using an SQL database, you ensure that there is sufficient bandwidth available from the ISA Server arrays to the computer running SQL Server. We also recommend that the computer running SQL Server is configured to handle simultaneous, large requests.

If the computer running SQL Server might not have the necessary capacity to handle large requests, do one of the following:

• Use local MSDE logging, rather than centralized SQL logging.
• Use centralized SQL logging, but do not generate daily summary reports. Instead, generate reports directly from the SQL database using SQL Reporting Services.

The following minimal network connectivity is required between the ISA Server firewall and the computer running SQL Server:

• 100 megabits for up to three array members
• 1 gigabit for four or more array members

Remote SQL Logging

You can use remote SQL logging to log all records to a centrally managed SQL database. As compared to MSDE and file logging, Remote SQL logging consumes CPU resources somewhere in between those used by MSDE and file logging, and practically uses no disk I/O. However, remote SQL logging introduces other capacity requirements that must be taken into account, because all log records are written to a central remote database:

• Network connections between ISA Server and the remote SQL database must dedicate a gigabit bandwidth to accommodate the capacity of the log traffic.
• Network connections between ISA Server and the remote SQL database must utilize IPsec to secure the log records when sent to the remote SQL database.
• Sufficient redundant array of independent disks (RAID) hardware must be available to support the logging rate of several ISA Server computers.

The following table provides an estimate of the transaction rate and the bandwidth required for logging for the Internet link bandwidths. The table shows megabits per second (Mbps) and kilobits per second (Kbps).

Internet link bandwidth	1 Mbps	5 T1 (7.5 Mbps)	25 Mbps	T3 (45 Mbps)
SQL transactions/sec	25	188	625	1125
SQL transaction bandwidth	92 Kbps	700 Kbps	2.3 Mbps	4.2 Mbps

For larger bandwidths, the numbers in the table can be extrapolated linearly.

Unlike the MSDE logs, an SQL database is a central log for all of the members of various ISA Server arrays in an enterprise. ISA Server array members are preconfigured to generate the daily summary reports at 00:30 (12:30 A.M.). In a scenario with many array members, the simultaneous requests for gigabyte-sized logs from the computer running SQL Server will generate heavy network traffic and a significant load on the computer. The daily summary generation time can be changed, so you could stagger the summary generation time from server to server. We recommend that you configure the time of the summary generation when ISA Server is not busy with other tasks (for example, late at night or early in the morning).

To configure time of daily summary generation, perform the following steps

1. In the console tree of ISA Server Management, click Monitoring:
   • For ISA Server 2004 Enterprise Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Arrays, expand Array_Name, and then click Monitoring.
   • For ISA Server 2004 Standard Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Server_Name, and then click Monitoring.
2. In the details pane, click the Reports tab.
3. On the Tasks tab, do one of the following:
   • For ISA Server 2004 Enterprise Edition, click Configure Log Summary and Report Preferences.
   • For ISA Server 2004 Standard Edition, click Configure Log Summary.
     
4. On the Log Summary tab, select the Enable daily and monthly summaries check box.
5. In Specify the generation time, type the time of day that the report data should be generated.

Special Considerations for SQL Logging

If you select to log to an SQL database, note the following:

• Microsoft SQL Server with Service Pack 3 (SP3) must be installed. SQL Server SP3 is available at the Microsoft Download Center.
• We recommend that you do not configure the autogrow and autoshrink parameters for the SQL database. Follow the guidelines stipulated in INF: Considerations for Autogrow and Autoshrink Configuration at the Microsoft Help and Support Web site.
• If you are logging the information to a remote database, configure encryption and data signature for the log information being copied to the remote database.

Data Encryption for the SQL Log

By default, ISA Server uses a Secure Sockets Layer (SSL)-encrypted connection to the computer running SQL Server, to help secure the sensitive data in the log files. You can configure whether the connection should be SSL-encrypted.

If you are logging the information to a remote database, we recommend that you configure encryption and data signature for the log information being copied to the remote database.

To use data encryption when connecting to an SQL database, perform the following steps

1. In the console tree of ISA Server Management, click Monitoring:
   • For ISA Server 2004 Enterprise Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Arrays, expand Array_Name, and then click Monitoring.
   • For ISA Server 2004 Standard Edition, expand Microsoft Internet Security and Acceleration Server 2004, expand Server_Name, and then click Monitoring.
2. In the details pane, click the Logging tab.
3. On the Tasks tab, select the appropriate task:
   • ConfigureFirewall Logging. Used to configure the firewall log.
   • Configure Web Proxy Logging. Used to configure the Web Proxy log.
4. On the Log tab, select SQL Database.
5. Click Options.
6. Select Force data encryption.
   

We recommend that you use Windows authentication (and not SQL authentication).

MSDE Logging

This section describes specific guidelines for MSDE logging.

Capacity Planning for MSDE Logs

MSDE uses more system resources than file logging. Specifically, you can expect an overall 10 to 20 percent improvement in processor utilization when switching to file logging from MSDE.

MSDE logging also consumes more disk storage resources. MSDE logging performs about two disk accesses on every megabit. File logging will require the same amount of disk accesses for 10 megabits. One way to improve ISA Server performance is to switch from MSDE to file logging. This is recommended only when there is a performance problem caused by saturated processor or disk access.

Compressed Drives

Compressed drives cause severe database performance degradation and disk fragmentation. This can slow MSDE performance by 500 percent or more, potentially causing the ISA Server firewall to lock down when experiencing above normal traffic.

Additional Information

Additional ISA Server 2004 documents are available at the ISA Server 2004 Guidance page.