Overview of the Logging Improvements in Forefront Threat Management Gateway (TMG)

Authors:

Yuri Diogenes - Security Support Engineer (ISA/IAG Team)

Bala Natarajan - Security Support Engineer (TMG Beta Team)

Tech Reviewers:

Jim Harrison - Program Manager, Forefront Edge CS

Mohit Saxena - Security Tech Lead (ISA/IAG Team)

Why Changing?

One of the main aspects of a Firewall is the capability of auditing access and ensuring that logs are in place for each and every request. For many reasons this is one of the most important criteria that a Firewall needs to meet in order to be recognized in an industry.

By default in ISA Server if a logging repository (Text file, SQL or MSDE) fails, the built-in Log Failure alert shuts down the Firewall service. Many companies require the ISA logs to be of evidentiary quality; e.g., they need assurance that if the traffic cannot be logged, it will not flow across ISA. Threat Management Gateway (TMG) enhances the core logging functionality of its predecessor (ISA Server) by offering logging that is much more error-tolerant.

Logging Queue

The new logging feature in TMG is called Large Logging Queue (LLQ) and it is composed of two main components that run in Kernel mode (FWENG) and User mode (Dispatcher). The process in user mode only reads from the disk while the process in kernel mode writes to it. Figure 1 illustrates this relationship:

Figure 1 - Logging in TMG

When TMG is configured to log to a SQL instance and that instance is slow or unresponsive, the firewall engine triggers an alert and writes the log data to the LLQ in Kernel mode. When the SQL connection is restored, the dispatcher reads the Binary LLQ data and logs it to the SQL Database after formatting it.

The Log queue file is a binary file and the file name has following format: ISALOG_xxxxxxxx_yyyyyyyy.LLQ. Where:

• xxxxxxxx - it means the session ID, which is incremented every time the firewall engine driver is restarted.

• yyyyyyyy - it means the file ID, which is incremented every time that the Log queue from memory is written to disk.

The LLQ is stored in memory as well as on the Disk. If the dispatcher sees no delay in sending to the log formatter the LLQ data is formatted directly from memory and written in formatted log database. The amount of time and amount of data that can be kept in memory is controlled by the following registry settings.

HKLM\System\CurrentControlSet\Services\Fweng\Parameters\LogQueueMaxLossCount

HKLM\System\CurrentControlSet\Services\Fweng\Parameters\LogQueueMaxLossTimeInSeconds

Since the above settings controls how long the LLQ data stays in memory before it is written to disk, the raw LLQ files written to the disk will be of varying size. LLQ binary data is kept in memory in batches until either the LogQueueMaxLossCount or LogQueueMaxLossTimeInSeconds registry values are reached. The LogQueueMaxLossCount controls the number of batches kept in memory and the LogQueueMaxLossTimeInSeconds controls the time it can stay in memory. After the configured threshold, the LLQ data is stored in the configured directory on the disk during SQL log failure. If there is a disk failure or the disk is full or the folder doesn’t have correct permissions then it will stop the Firewall service and traffic cannot flow.

When SQL service returns to normal operation the dispatcher reads the binary LLQ file , and the formatter logs it in the SQL database. If the binary LLQ file cannot be read by dispatcher due to a disk corruption, the dispatcher simply deletes the file and moves on to the next binary file. Since the Maximum amount of data that can be present in a single binary file is controlled by the value in the registry key LogQueueMaxLossCount and LogQueueMaxLossTimeInSeconds, a single binary file corruption will not be more than the acceptable LogQueueMaxLossCount or LogQueueMaxLossTimeInSeconds value.

The LLQ files by default are temporally saved in the TMG Server Log folder but you can change it using the Configure Log Queue option as show below:

Figure 2 - Changing Log Queue folder.

The LLQ files will keep growing inside the Log Folder but as soon as the SQL Server Service (MSFW) becomes available again, all the LLQ files will be committed to the database and deleted.

Figure 3 - LLQ Files being created when SQL Service is unavailable

TMG Alerts

When TMG loses connectivity with SQL Server Service, the following log will be recorded in TMG Alerts tab as Log Formatting Failure:

Figure 4 - Warning Log

When connectivity with SQL is reestablished and log formatting is resumed then there it will be another alert but at this time it will only be an informational alert as show below:

Figure 5 - Informational Log

Enhanced Security

The new logging feature on TMG is a major change in the product’s architecture, robustness and security. The reason why security is also listed as a major change is because in ISA 2006 the Firewall Service has write access to the log directory while TMG Firewall Service has no access to the log directory which prevents Firewall Service to write in this directory.