Exchange Server 2007 Process Tracking Log Tool

As a Microsoft Exchange Server 2007 administrator, you can experience situations that may require parsing, monitoring, and analyzing Message Tracking logs. These situations include the following scenarios:

• Growth of a transaction log or of a database

• A server experiencing many unsolicited commercial e-mail messages (also known as spam)

• A looping message

• Transport queue backup

• Poor server performance

In such scenarios, you may have to review and analyze the Exchange Message Tracking logs. The following factors or combination of factors can make it difficult or impossible to analyze the message tracking logs manually:

• Many fields

• A large amount of data in message tracking logs

• Large message tracking log files

Additionally, if you have to review message tracking log files from multiple servers, the task becomes even more complex.

Process Tracking Log tool simplifies reviewing message tracking logs by automating the parsing and analysis of Message tracking logs, and then generating an .xls or .txt output file for reporting. This tool provides useful data about monitoring, administration, and troubleshooting. This tool also provides important data about end-to-end delivery latency distribution for all Messages Delivered, Server Latency Distribution for all Messages Processed, and Server Latency Distribution for all Individual Deliveries.

To download the Process Tracking Log tool (Processtrackinglog.vbs), see the Exchange Server Team Blog article Processtrackinglog.vbs script

To run this tool, use the following script:

cscript ProcessTrackingLog.vbs <LogFilePath> <NumFiles> <hub|edge|all> [ <mm/dd/yyyy> | today | yesterday ]

The following command-line examples provide usage scenarios for the Process Tracking Log tool.

cscript ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking\MSGTRK20070804-1.LOG" 1 all

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 1 all

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 0 all

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 0 all

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 3 all

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 0 all yesterday

cscript \data\scripts\ProcessTrackingLog.vbs "D:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\MessageTracking" 0 all 10/29/2007

The results from the Process Tracking Log tool are stored in the c:\temp\msgtrack\output directory. If you install the Winzip command-line tools in the default path on drive C, the Process Tracking Log tool automatically archives the results in a compressed (.zip) file.

The following sections describe each output file that the Process Tracking Log tool generates, together with interpretations of example data.

The MTDsnFailureResults.csv file includes details of all failure reasons.

The file includes fields such as Timestamp, ServerName, Reason for failure, ReturnPath, and Recipients. The MTDsnFailureResults.csv includes NDR reason codes and recipients together with other information. If you are troubleshooting a particular NDR code or specific recipient issues, you can use the filtering features in Microsoft Office Excel to obtain statistical data about a particular server or organization. To do this in Excel, click Filter on the Data menu. This data may give you information that is relevant to the delivery issue.

For example, you may be troubleshooting a "550 5.2.2 STOREDRV.Deliver: mailbox full" NDR reason code. During your investigation, you may want to know how many users on a server actually have their mailboxes full.

The MTDuplicateDeliveryResults.csv file is useful to help troubleshoot duplicate mail delivery issues that occur on a particular server or across an organization.

The MTNextHopResults.csv file includes a count of messages together with the average message size for each next hop server through SMTP and the StoreDriver component. The file includes statistics that are based on the following fields:

• ClientName,Source

• Server

• MsgCount,AvgMsgBytes

• MsgCountServerSLAMet

• PercentServerSLAMet

• MsgCountDeliverSLAMet

• PercentDeliverSLAMet

This file is one of the three most important files for data analysis purposes. You can use pivot tables that are based on this file to analyze the destination of server traffic. Also, you can group multiple servers into a pivot table that is based on the MTNextHopResults.csv file to obtain an overall traffic analysis. It is best to analyze the data based on the following three fields:

• StoreDriver

• IntraOrg SMTP

• External SMTP delivery

The MTExpandResults.csv file includes the distribution group SMTP address, the number of recipients per distribution group expansion operation, and the number of distribution group expansions performed.

In the example results that are displayed in the following table, the MTExpandResults.csv file shows that the DL100@contoso.com distribution group has a recipient count of 49,426. The distribution group was expanded 21 times, and has an average expansion latency of 3,045 seconds (50.75 minutes). This amount of latency could cause server performance issues. You can use this output file to help troubleshoot server performance issues that you may experience.

To obtain a graphical view of the data, use Microsoft Excel to plot this data in a chart.

The MTLogStatistics.csv file includes many statistics for each log file that is processed. The file contains the following fields:

• LogFilePath

• ServerName

• iLogFileEventCount

• Events/MsgId

• LogDateTimeStart

• LogDateTimeEnd

• LogDuration(Min)

• iAvgStoreDriverReceiveRatePerMin

• iAvgSmtpReceiveRatePerMin

• iAvgStoreDriverDeliverRatePerMin

• iAvgSmtpSendRatePerMin

• iAvgDsnFailRatePerMin

• iAvgFailRatePerMin

• iServerLatencyTotalCount

• iPercentServerLatencySLAMet

• iPercentServerLatencyRecipientsCounted

• iDeliveryLatencyTotalCount

• iPercentDeliveryLatencySLAMet

You can use these values to verify the percentage of messages that are counted in latency SLA measurements.

The "percent SLA met" numbers are the most significant because they indicate the percentage of messages that are below the server default of 30 seconds and the delivery default of 90 seconds. These values are default levels that you can modify to obtain the appropriate percentiles.

The server latency and delivery latency SLA goals are defined as variables in the script. The default values are 30 seconds and 90 seconds respectively. Server latency represents the delay from when the message is submitted to transport to when the message is delivered or relayed. For the server latency calculation, each set of recipient deliveries is counted separately. Delivery latency represents the end-to-end latency based on the difference between the organization's submission time and the final delivery event time stamp.

For example, the combination of iAvgStoreDriverReceiveRatePerMin and iAvgSmtpReceiveRatePerMin in the MTLogStatistics.csv file indicates the average rate at which the transport component receives messages.

The combination of iAvgStoreDriverDeliverRatePerMin and iAvgSmtpSendRatePerMin specifies the average delivery rate for the transport component. This rate assumes that there are no deliveries to the Drop directory.

The comparison of these two combinations may give you an indication of whether the transport component is effectively processing messages or whether the transport or one of its dependent components is a performance bottleneck. To determine this status, you may have to examine other factors that can affect the results. For example, you may have to adjust the calculation to take the iAvgDsnFailRatePerMin value into consideration.

The MTDeliveryLatencyExceptions.csv file includes the MinDeliveryLatency and MaxDeliveryLatency fields. These fields contain latency measurements, in seconds. The fields are used to measure deliveries that have occurred for a particular message. These measurements are valid if only one message delivery is made. You can use this information to help determine whether a single next hop in the message delivery process was the cause of latency for a subset of recipients.

The MTMbxFullRecipResults.csv file contains the results of "mailbox full" events. This file includes the SMTP addresses and NDR counts for full mailboxes.

The following table shows some example output from this file. In this output, the full mailbox for User2 has caused 4,283 NDRs.

You can use this information to notify users to delete messages from their mailboxes. This reduces server overhead.

The MTReceiveResults.csv file includes the number of messages together with the average message size for each host that submits messages by using SMTP or by using the StoreDriver component. The .csv file includes the following fields:

• ServerName

• Client IP

• MsgCount

This file helps display the StoreDriver receive results for the Hub role. The following table contains example output from the MTReceiveResults.csv file. The output indicates that IP address 192.168.0.190 has submitted a very high number of messages (349,230) to the server that is named GCS1E2K7.

This kind of message count requires additional investigation. It could indicate a spam issue. For example, a server may have a receive connector that is configured incorrectly so that the volume of spam increases.

The MTTopRecipientResults.csv file includes the SMTP addresses, mailbox server name, and messages that are received for the top recipients. Top recipients are those who receive more than the average number of messages. The following table contains example output from the MTTopRecipientResults.csv file. In this output, User1245@contoso.com stands out.

You can use the charting feature in a spreadsheet program such as Excel to obtain a graphical view of the top message recipients.

The MTTopSendersbyDeliverResults.csv file includes the SMTP addresses together with the number of sent messages for the top message senders. Top senders are those who send more than the average number of messages. This includes messages from senders who are outside the organization. The following table contains example output from the MTTopSendersbyDeliverResults.csv file. In this output, User1176@contoso.com stands out.

The MTTopSendersbySubmitResults.csv file includes the SMTP addresses, mailbox server name, and messages that are sent for top senders. Top senders are users who send more messages than average. This file only includes messages that are submitted from mailboxes by using the StoreDriver (not SMTP). The following table contains example output from the MTTopSendersbySubmitResults.csv file. In this output, User2389@contoso.com stands out.

The MTEventTimeDistribution.csv file displays Event distributions based on an hourly count. This file helps provide a means to graph events over time. This may help determine which source is submitting messages, and the message destination. The following table contains example output from this file.

The MTMessageSizeDistribution.csv file shows the distribution of messages based on message size. The distribution is broken down according to the number of messages, a percentage of the total messages, and the percentage of messages that are smaller than the current message size. This may help determine the spectrum of message size distribution in an organization. The following table contains example output from this file.

The MTRecipientNotFoundResults.csv file contains statistics about recipients who could not be located in the global address list (GAL) or Active Directory directory service. The following table contains example output from this file. In this output, the high count for User1001 may indicate that this user is no longer associated with the organization.

The MTDomainExpiredResults.csv contains entries that may be helpful to troubleshoot DSN failures. For example, these entries may help you troubleshoot if a DSN failure occurs because of a misspelling, or if a destination server is unavailable for more than two days (the default expiration time-out). The following table contains example output from this file.

The MTMessageSizeExceptions.csv output file is generated only if there is a tracking log event for a message whose size exceeds the value for iMaxMessageSizeThresholdKB. The default threshold is 64MB. If there is no such tracking log event, the file is not generated.

The MTRunTimeLog.log file is the log file for the Process Tracking Log tool. This file contains the run-time log for the Processtrackinglog.vbs script.

The MTMailSubmissionDistribution.csv file contains the distribution of submissions by MessageClass and ClientType. This includes the ability to parse the SUBMIT event SourceContext field to determine the distribution of submissions by ClientType and by MessageClass. This file relies on the Mailbox role tracking logs.

You can use this file to help determine the distribution of client types when you analyze the Mailbox role tracking log.

The MTSummaryResults.txt file includes summary statistics for all the log files that are processed. This file contains many items. Some more significant items are as follows:

• Total Events processed

• Total Message Id's Processed

• Total Messages received

• Total Messages sent

• Total Messages delivered

• Total Messages delivered (duplicate)

• Total Resolve

• Total Transfer

• Total Expand

• Total Fail

• Total Fail with NDR

• Total Fail (valid Return Path)

• Total Fail (Recipient)

• Avg Fail Events/Min

• Max Fail Events/Min

• Max DSN(Fail) Events/Min

• Total DSN Generated

• Total DSN Badmail

• Total DSN

• Total DSN Delivered from CORP

• Total Null Reverse Path Delivered

• Total Null Reverse Path Delivered from Internal

• Total Null Reverse Path Delivered from External

• Total Defer

• Total Poison

• Total SMTP Receive

• Total StoreDriver Receive

• Total SMTP Send

• Total StoreDriver Deliver

• Total Expand

• End-To-End Delivery Latency Distribution for all Messages Delivered

• Server Latency Distribution for all Messages Processed

• Server Latency Distribution for all Individual Deliveries

• Unique Encapsulated Addresses

• Unique DSN Source Context

• Unique Defer Source Context

• Unique StoreDriver Sender Domains

• Transfer Source Context

• FAIL Event Sources

• Unique FAIL Recipient Status codes

The following example is an excerpt from an MTSummaryResults.txt file. This example illustrates some of the information that is collected.