Message Analyzer Tutorial
This section begins with some background concepts about Microsoft Message Analyzer and then goes into several mini-tutorials or Getting Started Primers that will help you get started with using this unique tool. Links are provided throughout so that you can navigate to more information about the described features as needed.
To go directly to procedures that demonstrate how to use Message Analyzer, see the following topics:
Quick Start Procedures
Using the Network Tracing Features
Using the Data Retrieval Features
Using the Data Viewing Features
Using the Data Filtering Features
Using the Asset Management Features
Using the Chart Configuration Features
Message Analyzer enables you to capture, display, and analyze protocol messaging traffic, and to trace and assess system events and other messages from Windows components. It also provides the capability to retrieve, aggregate, and analyze data from saved traces, which includes .etl, .cap, .pcapng, .tsv/.csv, event logs, text logs, and the Message Analyzer native files in the .matp or .matu format. In addition, Protocol Engineering Framework (PEF) extensibility features enable Message Analyzer to retrieve data from any custom text log file as long as you create a log file adapter for it, as described in Parsing Log Files. Message Analyzer also enables you to extend the functionality of the graphic Charts feature so that you can customize the data view in the UI to your own specifications, as described in Extending Message Analyzer Data Viewing Capabilities.
Message Analyzer makes use of two different types of sessions to acquire input data, as described in Starting a Message Analyzer Session. These consist of a Live Trace Session and a Data Retrieval Session, which provide data from the live capture of network traffic, events, and system messages; and saved traces, logs, and text logs, respectively. In a Live Trace Session, Protocol Engineering Framework (PEF) provider-drivers and/or other system ETW Providers listen for and capture protocol messages and events at various stack layers or from other components. The messages and events are passed to the PEF Runtime where they are decoded by Open Protocol Notation (OPN) parsers and then saved. To access and display these messages, Message Analyzer consumes the PEF Runtime data, as described in the PEF Architecture Tutorial. Messages are displayed by default in the Analysis Grid viewer, where you can begin your data analysis process, however, other data viewers are also available to streamline message analysis.
In a Live Trace Session, you have the option to capture data from the local computer and/or multiple remote computers in concurrent subsessions that return all data to the common initiating live session that you configure with a chosen data viewer. Moreover, the local computer is the default host on which a Live Trace Session captures data; however, if you specify valid connection/authentication credentials for other remote computers, you can capture data simultaneously on those computers as well. Message Analyzer also provides you with the flexibility to run multiple concurrent Live Trace Sessions, each with different message provider configurations, to target different computers. You can do this by simply adding one or more Live Trace data sources in the New Session dialog and specifying the hosts from which to capture the data.
In a Data Retrieval Session, Message Analyzer enables you to retrieve and aggregate saved message collections from multiple sources, including traces and logs, in any combination. This means you can mix and merge data from any of these sources and display it in the Analysis Grid or other selected data viewer. If you know that certain events have occurred at a particular time in a collection of data sources, you can configure a Time Filter to view data in a window of time that you specify to eliminate extraneous data and improve performance. You can also set Time Shifts to accommodate for different time zones or skewed machine times across different data sources. You might also select or configure a Session Filter that enables you to return specific data that is based on the filtering criteria that you specify, while at the same time further improving performance.
Although Message Analyzer enables you to capture messages from many system components, the PEF providers used by Message Analyzer enable you to capture data at several different layers, which provide unique inspection points into the protocol stack. For example, by specifying any Trace Scenario that uses the Microsoft-Pef-WFP-MessageProvider, you can focus on network/transport layer messages and above, while minimizing lower-level messages that are filtered out by the Windows Filtering Platform (WFP). Message Analyzer also enables you to set a Viewpoint that filters, reorganizes, and redisplays the data from the perspective of a selected protocol or module type, such as HTTP, TCP, SMB, or ETW, to quickly enable close examination of the traffic specified by the Viewpoint.
You can also select a predefined Parsing Level that controls the stack level to which Message Analyzer parses, while passing certain messages in these scenarios that are useful to your data analysis perspective, as described in Setting the Parsing Level. In addition, you can make use of Aliases, as described in Using and Managing Aliases, to configure user-friendly names for cryptic field values; or you can take advantage of the Unions feature, described in Configuring and Managing Unions, to correlate differently named fields that have identical values in different data sources. You can even capture and analyze loopback traffic for local application communications that use the IPv4 or IPv6 loopback addresses, in the Loopback and Unencrypted IPSEC and Local Loopback Network Trace Scenarios, as described in Default Trace Scenarios. Furthermore, Message Analyzer enables you to decrypt data that is encrypted with the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols, for example Remote Desktop Protocol (RDP) and HTTPS messages. The Decryption feature also provides a Decryption tool window that presents summary and statistical data for the decryption session for analysis purposes, as described in Using the Decryption Feature.
These capabilities solve many inherent capture, data display, and analysis problems, such as the visibility of encrypted data, assessment of loopback traffic that is enabled by the Local Loopback Network scenario, and seeing traffic from the Viewpoint of a protocol. The underlying technologies that support Message Analyzer also machine-validate message structure and values, behavior, and architecture based on protocol specifications; and if errors occur, they are surfaced very quickly as diagnosis messages. Message Analyzer also provides a Diagnostics Tool Window that summarizes all the diagnostic messages in a trace, which interactively drive selection of corresponding messages in the Analysis Grid viewer.
Note Message Analyzer is also an effective tool for testing and verifying protocol implementations. See the Open Specifications documentation library for more information about protocol technical specifications.
Getting Started Primers
The sections that follow provide brief conceptual tutorials that serve as getting started primers for Message Analyzer functionality. These tutorials are provided within the context of the major tasks that you perform from the Message Analyzer UI, where you can:
Capture Message Data
When capturing data live, Message Analyzer makes use of various message providers, which can include the following:
Common Microsoft message provider-drivers — all PEF drivers are instrumented with Event Tracing for Windows (ETW) provider technology, which enables them to take advantage of the ETW event tracing, buffering, logging, and event delivery infrastructure. In addition to numerous system ETW providers and other message capture components, all Message Analyzer installations contain the following PEF and other provider-drivers, the configuration for which is accessible after you select a Trace Scenario from the Select a trace scenario drop-down menu on the Live Trace tab of the New Session dialog for a Live Trace Session.
Note Some of the message providers in this list may be different from what you will find in Message Analyzer installations on other computers, because of an operating system version dependency. For example, on Windows 8 and earlier, the Microsoft-Windows-NDIS-PacketCapture provider does not exist for Local Network Interfaces tracing. Instead, the Microsoft-PEF-NDIS-PacketCapture provider in the Local Network Interfaces scenario is included for that purpose on those computers.
- Microsoft-PEF-NDIS-PacketCapture provider — an ETW-instrumented, Network Data Interface Specification (NDIS) light weight filter (LWF) driver that captures ethernet frames at the Link Layer and delivers them to Message Analyzer as ETW events. Also provides the capability to configure Fast Filters that operate efficiently at the driver-level to isolate specific message types, thereby passing less data and reducing system loads and resource consumption.
To learn more about the PEF-NDIS-PacketCapture provider, see PEF-NDIS-PacketCapture Provider.
- Microsoft-PEF-WFP-MessageProvider — an ETW-instrumented filter driver that captures traffic in the Windows Filtering Platform (WFP) at the Transport layer and delivers them to Message Analyzer as ETW events. This provider also enables you to configure Fast Filters to isolate specific messages of interest and improve trace performance. In addition, you can set the Select Discarded Packet Events option when configuring this provider to log discarded packets.
To learn more about the PEF-WFP provider, see PEF-WFP-MessageProvider.
- Microsoft-PEF-WebProxy — an ETW-instrumented provider that uses the Fiddler API and acts as an HTTP proxy to intercept and capture all HTTP traffic to and from a client web browser in unencrypted format. Also provides the capability to configure driver-level Hostname and Port filters to isolate specific messages and improve performance.
To learn more about the PEF-WebProxy provider, see PEF-WebProxy Provider.
Note For information about usage configurations for PEF providers, see Default Trace Scenarios.
- Microsoft-PEF-NDIS-PacketCapture provider — an ETW-instrumented, Network Data Interface Specification (NDIS) light weight filter (LWF) driver that captures ethernet frames at the Link Layer and delivers them to Message Analyzer as ETW events. Also provides the capability to configure Fast Filters that operate efficiently at the driver-level to isolate specific message types, thereby passing less data and reducing system loads and resource consumption.
Microsoft-Windows-NDIS-PacketCapture provider — an ETW-instrumented provider that has remote capabilities along with special NDIS stack and Hyper-V-Switch extension layer filtering, adapter configurations, packet traversal path directivity, and other filters and specifiers that you can configure.
Note The Windows-NDIS-PacketCapture provider with remote capabilities is used on the Windows 8.1 and Windows Server 2012 R2 operating systems only, as described in Default Trace Scenarios.
To learn more about capturing messages from one or more remote hosts and configuring the Microsoft-Windows-NDIS-PacketCapture provider, see Capturing Data Remotely.
System ETW providers — write events for various components on your system that have been instrumented as ETW event providers. This includes ETW providers that use managed object format (MOF) schemas to define their events, as described in MOF-Based ETW Providers. It can also include Windows WPP-generated events that are generated by software components, as described in Processing WPP-Generated Events.
Specifying Message Providers
You can specify the message providers that you want to use to capture data from the network or other components by configuring a Live Trace Session, as shown in the figure that follows.
Figure 1: Message Analyzer Live Trace Session configuration
These provider configurations are contained in predefined Trace Scenarios that you can select from the Select a trace scenario drop-down menu on the Live Trace tab of the New Session dialog. These Trace Scenarios are templates that contain predefined message provider configurations that are tailored for capturing data from various components and at different stack layers. Optionally, you can enhance the scope of data retrieval by adding other system ETW providers to the scenario. If you have created and saved any custom Trace Scenarios in the Trace Scenario Library by using the Save Trace Scenario feature, these are also available for selection from the Select a trace scenario drop-down menu. You can also modify the capture configuration of PEF and other ETW Providers from the Live Trace tab of a New Session to isolate specific message traffic and realize performance enhancements. For example, by clicking the Configure link for a selected message provider in the ETW Providers list on the Live Trace tab, you can display a configuration dialog and specify Fast Filters that work very efficiently at the kernel level. These low-level filters enable you to quickly retrieve specific messages that meet the filtering criteria that you specify, which reduces the scope of the data to be returned by the trace. In turn, this accelerates data capture and minimizes the parsing time. You also have the option to select or create a Session Filter for a Live Trace Session (or a Data Retrieval Session) to reduce the scope and count of messages that you retrieve, and as a result realize performance improvements. The difference between a Fast Filter and a Session Filter is that Fast Filters work at the driver level and are therefore not subject to the Runtime parsing process, which makes them faster, whereas Session Filters are subject to parsing, which makes them a little slower.
Other ETW Provider settings that you can configure for a Live Trace Session consist of the following:
System Network adapter filters and logically ANDed Fast Filter group settings — the configuration is accessible from the Provider tab of the Advanced Settings – Microsoft-PEF-NDIS-PacketCapture dialog for Local Network Interfaces Trace Scenarios, as described in Using the Advanced Settings - Microsoft-PEF-NDIS-PacketCapture Dialog.
NDIS stack filters, extension layer filter settings for Hyper-V-Switches that service virtual machines (VMs); and Direction (packet traversal), EtherType, IP Protocol Number, MAC Address, and IP Address filter settings — the configuration is accessible from the Provider tab of the Advanced Settings – Microsoft-Windows-NDIS-PacketCapture dialog for the Local Network Interfaces Trace Scenario, as described in Using the Advanced Settings - Microsoft-Windows-NDIS-PacketCapture Dialog.
WFP Layer Set and Fast Filter settings — the configuration is accessible from the Provider tab of the Advanced Settings - Microsoft-Pef-WFP-Message Provider dialog, as described in Filtering Live Trace Session Data.
Hostname and Port Filter settings — the configuration is accessible from the Provider tab of the Advanced Settings – Microsoft-Pef-WebProxy dialog forUnencrypted HTTPS Trace Scenarios, as described in Filtering Live Trace Session Data.
Keyword event and error Level filters — the configuration is accessible from the ETW Core tab in the Advanced Settings of all provider configuration dialogs; however, not all ETW Providers make Keyword and Level filter settings available, as some providers are not instrumented with them. See System ETW Provider Configuration Settings for additional details.
Note You can very quickly start capturing data with Message Analyzer by clicking a Trace Scenario in the Quick Trace submenu of the File menu.
To learn more about configuring a Live Trace Session, see Capturing Message Data.
Message Analyzer provides several global options that enable you to specify certain default values or make default selections that can affect Message Analyzer performance, display configurations, or feature activation. You can set these options at any time; however, you would typically do so prior to starting a Live Trace Session or a Data Retrieval Session. Specific details about these options are described in Setting Message Analyzer Global Options.
Message Analyzer can display message traffic that is captured from specific protocol modules only if the protocol object model (POM) repository within the PEF architecture contains compiled OPN descriptions representing the architecture, behavior, and data for those protocols. Message Analyzer ships with OPN descriptions for a large number of protocols, such as Microsoft Windows and other common public protocols, in addition to Office, Exchange, SharePoint, and SQL protocols. This enables you to capture a wide array of network protocol and application messages. In addition, to support your data analysis process, Microsoft makes the technical specifications for these Protocols available on the Microsoft Developer Network (MSDN) web site, while you can find other standard RFC protocol specifications on the Internet. You can use the technical documents (TDs) provided by Microsoft as references that depict protocol architecture, behavior, and data, as it was designed, to facilitate analysis of the messages you capture with Message Analyzer. For example, you could verify the value of a particular field or confirm the presence of required parameters for a particular method of a specific protocol that is failing to perform properly, although Message Analyzer has a built-in message validation feature that does this automatically.
The Message Analyzer trace model uses ETW to enable integrated capture and display of messages and events from a large number of system components. Whenever you start a Live Trace Session, the underlying message provider/s in the Trace Scenario that you select are enabled to the ETW Session Controller, which determines if there are any specific Keyword event or error Level settings that modify which events are to be returned to the ETW Consumer, which in this case is Message Analyzer. If there are no such settings, then the ETW Session Controller returns all events generated by the component that is instrumented for ETW. Message Analyzer then displays detailed, human-readable information for ETW traces.
For this to be possible, OPN must be generated for any manifest-based system ETW Provider that you employ in a Live Trace Session so that ETW events can be properly parsed by the PEF Runtime. To generate the OPN, manifests for system ETW Providers in use are retrieved so that OPN descriptions can be inferred from them to provide the basis for Message Analyzer to successfully parse event structures. To facilitate this process, the PEF architecture contains an ETW Manifest Import Adapter. This is a POM adapter that converts an ETW manifest for a given ETW Provider into a POM model, and then publishes it to the PEF Runtime so it can parse and dispatch ETW messages generated by that provider. The OPN actors and endpoints that enable parsing and dispatching messages for an ETW Provider that you specify in a Trace Session are dynamically generated at runtime by the ETW Manifest Import Adapter.
Note If you want to extend your system with additional system ETW Providers, you will need to create an OPN configuration file for each new ETW Provider so that the associated provider manifests imported into the system can correctly infer an OPN description that provides proper parsing of ETW events. An ETW Provider manifest defines the event descriptions and format in which events are written by the provider. For more information about how to create an OPN configuration file for a new system ETW Provider, an OPN Configuration Guide for ETW Manifest Imports document will be available in the future. However, you might consider one of the many system ETW Providers that are accessible from the Add Provider drop-down menu on the Live Trace tab of the New Session dialog, to meet your needs.
MOF-Based ETW Providers
Message Analyzer also supports registered event providers on your system that use the MOF schema as the basis of generating their events. These event providers typically appear in the Add Provider drop-down menu on the Live Trace tab of the New Session dialog for a Live Trace Session. With MOF support, events that are captured by Message Analyzer from MOF-instrumented providers can be fully parsed. Without MOF support, messages that are captured from MOF-based providers are displayed as simple ETW messages with no additional parsing.
To provide support for MOF-instrumented providers, including fully parsing events from such providers, Message Analyzer uses an extension to the existing ETW adapter. This adapter normally handles ETW providers that have a manifest that is created at the time the provider is instrumented for ETW. When an ETW event arrives, Message Analyzer checks to see whether an OPN description exists that can parse the event. If an OPN description cannot be found, then Message Analyzer attempts to retrieve the manifest-based event schema, from which it can generate OPN. In a similar manner, Message Analyzer does the following to support MOF when events arrive:
Verifies whether events are generated by an MOF-based provider.
Checks the local system for an existing OPN description that can parse the events.
Uses the extended version of the ETW adapter to generate an OPN description based on the MOF schema of the provider, if an existing OPN description was not found.
Detecting MOF Schema
In Message Analyzer, there are typically three sources from which MOF events can derive, including live traces, saved trace files such as the native Message Analyzer parsed format (.matp), and saved trace files in other supported formats such as .matu, etl, and .cap. As previously indicated, if there is an existing OPN module (see Protocol Modules) that can consume the events, then the events are parsed according to the OPN description and background generation of OPN is not required. However, if there is no existing OPN module to parse the events, Message Analyzer then attempts to locate the MOF schema as follows:
Live trace — when you perform a Live Trace that utilizes MOF-based event providers, the locally installed MOF schemas are retrieved from the appropriate event provider/s that are installed on the local machine, and OPN descriptions for the provider events are automatically generated for parsing the event fields.
Saved .matp files — if one or more MOF schemas were used to parse messages from an MOF provider when a trace is taken with Message Analyzer, the schemas become part of the .matp trace file when it is saved. The schema is thereafter provided to Message Analyzer at the time the .matp trace file is loaded, making it independently available to facilitate event parsing whether or not MOF schemas exist on the local system or were deployed during Message Analyzer installation.
Saved non-.matp files — these files will not contain the embedded schema information, therefore Message Analyzer looks up local files deployed during installation. If a local .mof file is discovered, it is used as the MOF schema from which an OPN description is generated for parsing events. Otherwise, the system MOF schema is retrieved and used in a similar manner.
Note If Message Analyzer requires a MOF schema for a provider that is installed on the local system and cannot find one, then Message Analyzer will display simple ETW messages only, with minimal parsing for that provider’s messages.
Deploying a Custom MOF Provider
If you have a custom MOF-based provider that you want to deploy on your local system, you can use the WMI compiler tool mofcomp.exe to register your provider and its MOF schema. Thereafter, Message Analyzer will be able to locate the MOF schema, should an OPN description need to be created to parse the MOF-based events of the provider. You will find the mofcomp.exe tool in the following directory on your computer:
To learn more about using the mofcomp.exe tool, see mofcomp in the WMI Command Line Tools topic on MSDN.
ETW Session Performance
Message Analyzer also enables you to modify certain aspects of ETW Sessions to focus on capture of specific events and/or to improve performance as follows:
ETW Provider — you can specify the events that you want to receive from a system ETW Provider by configuring Keyword and Level filtering. You can configure Keyword and Level filters from the ETW Core tab in the Advanced Settings dialog for the particular message provider that underlies the Trace Scenario that you selected, as described in Specifying Message Providers, that is, for system ETW providers that permit Keyword and Level filter configuration. Configuring system ETW provider filtering for event tracing enables you to decrease the event volume and capture time by isolating specific types of events to retrieve in the trace.
ETW Session Configuration — you can configure certain aspects of the underlying ETW Session in which an ETW Provider participates to enhance session performance. This mainly involves adjusting settings for the ETW buffer configuration of the ETW Session that is managed by an ETW Session Controller. These adjustments are available from the Message Analyzer ETW Session - Advanced Configuration dialog that is accessible by clicking the ETW Session Configuration button on the Live Trace tab of the New Session dialog, as shown in Figure 1.
To learn more about configuring a Live Trace Session, see Capturing Message Data.
To learn more about how system ETW Providers function in the ETW framework, see the ETW Framework Tutorial.
To learn more about configuring system ETW Providers, see Adding a System ETW Provider and System ETW Provider Configuration Settings.
To learn more about optimizing an ETW Session, see Specifying Advanced Session Configuration Settings.
Retrieve Message Data
When you start a Data Retrieval Session, the configuration of which is shown in the figure that follows, you can load data from saved trace files and logs into Message Analyzer, which includes .matu, .matp, .etl, .cap, .pcap, .log files, and others, as described by the table in Locating Supported Input Data File Types. After clicking the Add Files button on the Files tab in the New Session dialog for a Data Retrieval Session, you can navigate to target files that contain the data you want to load into Message Analyzer. After the files containing the target data display on the Files tab, you can also specify subsets of those files in your Files list to create message collections that target specific data to be loaded into Message Analyzer and parsed. To create a subset, you simply select the check box to the left of the file that contains the data you want to load. Note that a Data Retrieval Session enables you to aggregate and merge message data from multiple data sources that include various types of log files and traces.
Figure 2: Message Analyzer Data Retrieval Session configuration
You can also select specific data to retrieve from a target message collection while blocking all other messages that do not meet the filtering criteria that you define with a Session Filter or Time Filter. A Session Filter narrows the scope of data retrieval to only the message types that meet the criteria of a filter that you manually define, or one that you select from the centralized filter Library. A Time Filter enables you to specify a window of time in which to view data in a correlated target message collection that can consist of one or more sources from which you load data into Message Analyzer.
To learn more about configuring a Data Retrieval Session, see Retrieving Message Data.
Parsing Log Files
If you have a text-based log file that contains log entries in a unique format, PEF architecture enables you to load and view the data from the log file, but you might need to create an OPN configuration file to drive the process. However, the configuration file type that you need might already exist in the Text Log Configuration column drop-down that appears on the Files tab toolbar in the New Session dialog for a Data Retrieval Session. This drop-down only appears on the toolbar after you click the Add Files button to configure the log data to be loaded into Message Analyzer. The predefined configuration file types that are currently available consist of the following:
CheckSymCSV — provides the OPN configuration that parses the CSV output of the CHECKSYM utility, which is commonly used to perform file version and checksum comparisons of binaries and configuration files.
Cluster — provides the OPN configuration that parses event cluster text logs.
Dcdiag — provides the OPN configuration that parses the output of the Domain Controller Diagnostics Tool (Dcdiag).
EventLogCSV — provides the OPN configuration that parses traces that are exported as a CSV file, but with more value than a regular CSV file.
IIS — provides the OPN configuration that parses text logs generated by web servers.
Lync — provides the OPN configuration that parses UCCAPI logs from the Lync client application.
Netlogon — provides the OPN configuration that parses logs for diagnosing logon issues on domain controllers.
Registry — provides the OPN configuration that parses exported Windows Registry files that are in .REG format.
SambaSysLog — provides the OPN configuration that parses text logs generated on Samba Linux machines.
SCCM — provides the OPN configuration that parses System Center logs.
SQLServerError — provides the OPN configuration that parses SQL Server error logs.
SQLServerSetup — provides the OPN configuration that parses SQL Server setup logs
ULS — provides the OPN configuration that parses SharePoint logs.
VMM — provides the OPN configuration that parses System Center Virtual Machine Manager logs.
DefaultSimpleLogFileReaderConfig — a generic configuration file that can parse most text logs, for example those from a domain controller.
Note The EventLogCSV, NetLogon, SCCM, and ULS configuration files are no longer included in the AdditionalTextlogConfigurations Version 1.1 asset package, that was previously accessible on the Downloads tab of the Message Analyzer Start page. These configuration files, along with all others are now included in the Devices and Log File Version 1.2 asset package for Message Analyzer versions 1.2 and later.
If none of the predefined configuration files meet your requirements, then you can create a new one that is specifically designed to parse the data in your text log, as described in Opening Textual Log Files. Whenever you create a new configuration file for a text log, it is added as an item in the Text Log Configuration drop-down menu on the Files tab toolbar in the New Session dialog, and also to the Log File Module drop-down menu in the Default Log File Module pane on the Options dialog that is accessible from the File menu. From the latter drop-down, you have the option to set a specific configuration file as the global default for all text log data that you load into Message Analyzer.
Configuration File Contents
A configuration file contains a description of the log's messages in OPN and RegEx notation, which ensures that text log data that is loaded into the system can be properly parsed and then displayed in Message Analyzer. The text-based data is loaded into the Runtime through a Log File Adapter and the OPN configuration file drives the process. The message definitions contained in the OPN configuration file are compiled by the OPN Compiler to confirm the validity of the configuration file and the integrity of the OPN description that will reside in the POM.
When you create an OPN configuration file, you need to identify each unique log entry and map it to a message structure. You can do this with Regex notation, which is designed for matching strings of text. Regex provides the functionality you will need to match data through the mechanism of capture variables, which you can use to map extracted log file data to message fields.
Note Message Analyzer supports loading regular comma-separated-value (CSV) and tab-separated-value (TSV) data file formats directly, without the need for an OPN configuration file.
To learn more about how to create an OPN configuration file, download the OPN Configuration Guide for Text Log Adapter document.
To learn more about other OPN configuration file requirements, see the Addendum 2: Configuration Requirements for Parsing CustomText Logs topic.
View Message Data
In Message Analyzer, data is reassembled as part of the PEF Runtime parsing process. It is then displayed in a selected Message Analyzer data viewer, such as the default Analysis Grid viewer, which is shown in Figure 4 and described in the Analysis Grid Viewer topic. The Analysis Grid viewer is the primary analysis surface in Message Analyzer. It displays message data as expandable top-level parent nodes that contain all the child node (message stack) messages and message fragments that were involved in a particular transaction or operation.
By default, the Analysis Grid viewer hides message fragments in the message stack (for example, TCP virtual segments), while presenting top-level messages such as request/response message pairs for a particular transaction in a single expandable top-level message node that is known as an operation. By organizing messages this way, you can easily determine such important values as the ResponseTime, which can tell you how long it is taking to receive the first server response to a request message, without having to search through hundreds of messages to find it. This is important to analysis because it can indicate how long a service is taking to respond, which can rule out network issues. Another important value is the ElapsedTime, which can tell you how long an operation is taking to complete, including all the associated message fragments. Also, by performing a Message Analyzer sort on ElapsedTime, you can determine which operations (with fragments) took the longest to complete. In addition, by sorting on ResponseTime, you can determine which request/response pairs had the longest response times.
Grouping Messages in the Analysis Grid Viewer
Another feature that is important to data analysis is the Group feature. By right-clicking selected Analysis Grid viewer columns in succession and selecting the context menu Group item for each one, you can create a data display of nested groups that provides a convenient way to organize and explore targeted trace data. For example, you could add the IPv4 Network and TCP Transport columns to the Analysis Grid viewer from the Field Chooser Tool Window and then Group them, to quickly organize the data into groups of IP conversations that took place across a trace. The TCP messages that supported those conversations will be nested within each IP group.
Grouping Messages in the Grouping Viewer
You can also make use of the Grouping viewer which has a set of predefined View Layouts that render Analysis Grid message data into predefined nested Group configurations to create unique analysis contexts. You can also create and save your own Grouping View Layouts that you customize to your environment based on message fields that you select from the Field Chooser Tool Window. The Grouping viewer is accessible from the Common category of the New Viewer drop-down list in the Session group on the Ribbon of the Message Analyzer Home tab.
With the Grouping viewer, you can organize your traffic into summary hierarchies based on predefined or custom Grouping View Layouts that are configured with message field groups in nested configurations. You can also manually adjust (pivot) your group layout to obtain different message correlation configurations that result in unique analysis contexts. Some of the advantages of viewing data with the Grouping viewer are the following:
Organize data into unique hierarchies to expose targeted information that you can quickly extract from large data sets, which can otherwise be difficult to do.
Arrange nested Group configurations so you can isolate messages of interest at defined levels and drill down into the nested Groups to obtain a concise analytical focus.
Locate the Group(s) with the highest message volumes for performance assessment.
To simplify troubleshooting, Message Analyzer provides the Viewpoints feature that enables you to examine network traffic from the perspective of a protocol. By setting a Viewpoint, you can focus on specific messages at top-level in the Analysis Grid viewer with no layers above them. Moreover, because the Viewpoint temporarily removes all messages above the applied protocol Viewpoint, only the protocol messages associated with the applied Viewpoint appear at top-level in the Analysis Grid viewer. This feature is advantageous when you have higher-layer traffic that obscures the underlying messages that you want to troubleshoot.
Viewing Message Details
You can obtain a full visual representation of message details in the Analysis Grid viewer, including field names and values, by double-clicking any top-level parent message node or nested child message node. This information is presented inline on a Fields tab with other selectable data tabs that can include the Stack, Diagnosis, and Embedded tabs. You can also select any message and view its details data in a separate Tool Window called Details, which includes field, value, and payload data. Other separate Tool Windows are also available to enhance your data analysis perspective, for example, the Message Data, Field Data, Diagnostics, and Decryption windows. You can also view stack information in a separate Tool Window known as the Message Stack. Many of these Tool Windows are interactive, because they either drive or are driven-by message or data selection in other windows.
Viewing Data from Multiple Sessions
Message Analyzer also provides session viewer functionality from the Session Explorer Tool Window, to enable you to easily explore the data in multiple data viewers, which can include Charts that employ top-level summaries in graphic and tabular formats, a sequence match view, and additional timeline-oriented statistics for protocol communications or other messages and events. The figure that follows shows an example of a Chart viewer known as the Protocol Dashboard, which contains top-level data summaries. The Session Explorer is shown to the left of the Analysis Grid viewer.
Note By right-clicking a session node in Session Explorer, you are presented with a context menu that enables you to select other data viewers that display data in separate session viewer tabs. Thereafter, any new data viewer that you specified is listed in the Session Explorer navigation area. If you then select a Session Explorer node, Message Analyzer responds by immediately displaying the data on the session viewer tab that corresponds with the selected node.
Figure 3: Message Analyzer Protocol Dashboard viewer
Message Analyzer data viewers are integrated in that changes you make to data in one viewer, for example with filtering, can be reflected in other viewers that are displayed for the same session. Message Analyzer data viewers are also interactive in that data selection in one viewer drives the display of data in another viewer (or Tool Window). For example, you can double-click a bar chart column or a timeline module node in the Protocol Dashboard viewer that represents the messages of a particular protocol that were captured in a trace, and display only those messages in a new Analysis Grid viewer tab for data assessment purposes. This could involve isolating a group of messages that reflect a high message count for a particular protocol or module to expose any underlying issues.
To learn more about the Message Analyzer data viewer infrastructure, see Data Viewer Concepts.
To learn more about Message Analyzer data viewers that you can work with during data analysis, including the Protocol Dashboard and Chart viewers, see Data Viewers.
To learn more about Viewpoints and operations, see Understanding and Managing Viewpoints.
To learn more about the Group feature, see Using the Analysis Grid Group Feature.
To learn more about the Grouping viewer, see the Grouping Viewer topic.
To learn more about Message Analyzer tool windows, including the Session Explorer, see the Tool Windows topic.
To learn more about the Field Chooser, see Using the Field Chooser.
Filter Message Data
Message Analyzer provides numerous filtering capabilities to enhance data retrieval, capture, and assessment processes. Filtering is critical for focusing on specific messages and enhancing performance. For example, if you were unable to filter message data in a Live Trace Session, you might need to examine potentially tens of thousands of messages to isolate a specific problem. What most Message Analyzer users need to observe is usually related to a specific protocol, error message, conversation, or process. By providing the ability to filter while retrieving, capturing, or viewing data, Message Analyzer provides a convenient way to reduce the scope of the data that you are working with and more effectively pinpoint your issues.
Using a Session Filter
When capturing data or loading data into Message Analyzer through a Live Trace Session or a Data Retrieval Session, as shown in figures 1 and 2, respectively, you can use the common Session Filter to isolate specific data that you want to work with from the live trace or saved message collection. However, you should carefully note that you can never recapture the data that you filter out with a Session Filter in a Live Trace Session, whereas with a Data Retrieval Session, you can always click the Edit button on the Message Analyzer Home tab to return to session configuration, remove or recast your filtering criteria, and then reload the data from the originally specified saved files.
For instance, when configuring a Session Filter, you can specify a Filter Expression that isolates messages to a specific network address, port, or protocol, or that contains a particular field value or other text. Similarly, you can apply a View Filter or Color Rule to streamline the examination of data in the Message Analyzer default Analysis Grid viewer, which is shown in the figure that follows.
Figure 4: Message Analyzer Analysis Grid viewer
Using Special Filters for a Live Trace
You also have the option to use many other types of filters in a Live Trace Session, depending on the Trace Scenario and operating system you are running, as follows:
Fast Filters and WFP Layer Set filters — accessible from the Provider tab of the Advanced Settings – Microsoft-PEF-WFP-MessageProvider configuration dialog. You can display this dialog by clicking the Configure link to the right of the Microsoft-PEF-WFP-MessageProvider listing on the Live Trace tab of the New Session dialog after you select one of several Trace Scenarios that contain this provider from the Select a trace scenario drop-down on the Live Trace tab. For example, the Network Tunnel Traffic and Unencrypted IPSEC, Loopback and Unencrypted IPSEC, and Local Loopback Network Trace Scenarios all use the Microsoft-PEF-WFP-MessageProvider.
Fast Filter Groups and Adapter filters — accessible from the Provider tab of the Advanced Settings – Microsoft-PEF-NDIS-PacketCapture configuration dialog. You can display this dialog by clicking the Configure link to the right of the Microsoft-PEF-NDIS-PacketCapture listing on the Live Trace tab of the New Session dialog after you select the Local Network Interfaces Trace Scenario from the Select a trace scenario drop-down on the Live Trace tab. The Microsoft-PEF-NDIS-PacketCapture provider is available only on Windows 7, Windows 8, and Windows Server 2012 computers.
HostName and Port filters — accessible from the Provider tab of the Advanced Settings – Microsoft-Pef-WebProxy configuration dialog. You can display this dialog by clicking the Configure link to the right of the Microsoft-Pef-WebProxy provider listing on the Live Trace tab of the New Session dialog after you select the Unencrypted HTTPS Trace Scenario from the Select a trace scenario drop-down on the Live Trace tab.
Keyword event and error Level filters — accessible from the ETW Core tab of any Advanced Settings dialog for any Trace Scenario that you select from the Select a trace scenario drop-down menu. You can display this dialog by clicking the Configure link to the right of any provider listing on the Live Trace tab of the New Session dialog.
NDIS stack, Hyper-V-Switch extension layer, and Host adapter filters — accessible from the Provider tab of the Advanced Settings – Microsoft-Windows-NDIS-PacketCapture provider configuration dialog. You can display this dialog by clicking the Configure link to the right of the Microsoft-Windows-NDIS-PacketCapture provider listing on the Live Trace tab of the New Session dialog after you select the Local Network Interfaces Trace Scenario from the Select a trace scenario drop-down on the Live Trace tab. The Microsoft-Windows-NDIS-PacketCapture provider has remote capabilities and is available only on Windows 8.1 and Windows Server 2012 R2 and later computers.
Quick Filter — you can utilize a Quick Filter to configure a window of time in which to view Live Trace Session results data. This is particularly useful if you can approximate a time frame in which some issue occurred that you need to detect. The major advantage of using a Quick Filter is that you can remove it, modify the time window, and then reapply it. You can access the Quick Filtering dialog by clicking the Quick Filter button in the Filter group on the Ribbon of the Message Analyzer Home tab.
To apply a Session Filter, View Filter, or Color Rule, you will need to select a predefined Filter Expression from the centralized filter Library or you can manually create one. This Library is accessible from the New Session dialog when configuring a Live Trace Session or Data Retrieval Session; and from the View Filter Tool Window that displays below the Analysis Grid viewer when you click the View Filter button on the Ribbon of the Message Analyzer Home tab.
Note To simplify the process of configuring and applying a View Filter, Message Analyzer enables you to right-click certain columns in the Analysis Grid viewer and add the column value as a filter expression, which you can then apply to trace results.
To create your own Filter Expressions, you will need to understand the Message Analyzer Filtering Language. Also, before a Filter Expression such as a Session Filter is actually applied to any data, Message Analyzer performs a simple verification check to ensure that it compiles as a valid filter expression; otherwise, you will need to reconfigure the expression or abandon it. To assist you in creating your own Filter Expressions, Message Analyzer provides the Filter IntelliSense service, which is an interactive and intelligent statement completion service that responds to the text you enter in any Filter Expression text box.
To learn more about filter expressions, including how to write Filter Expressions, see Filtering Message Data.
To learn more about the Filter IntelliSense feature, see Filter IntelliSense Service.
To learn more about Fast Filters and WFP Layer Set filters, see the PEF-WFP-MessageProvider topic.
To learn more aboutFast Filter Groups and System Network Adapter filters, see the PEF-NDIS Fast Filters and Using the Advanced Settings - Microsoft-PEF-NDIS-PacketCapture Dialog topics.
To learn more about HostName and Port filters, see the PEF-WebProxy Provider topic.
To learn more about Keyword event and error Level filters, see the System ETW Provider Configuration Settings topic.
To learn more about NDIS stack, Hyper-V-Switch extension layer, host adapter, and other special filters, see the Capturing Data Remotely and Using the Advanced Settings - Microsoft-Windows-NDIS-PacketCapture Dialog topics.
Analyze Message Data
When analyzing data that you have either captured live on the network, loaded into Message Analyzer, or retrieved from a device such as a Bluetooth, you have the option to apply various types of filters to manipulate the way data is presented for analysis purposes. This includes applying View Filters, Quick Filters, Color Rules, and Column Filters to the trace data, in addition to using the Sequence Match capability to detect message patterns in a set of trace results. See More Information at the end of this section, for links to topics that provide further details about using these filtering features.
Other options for manipulating data include the use of features such as the following:
View Layouts — you have the option to apply predefined View Layouts that contain an arrangement of data columns that are designed to assist you in data analysis and troubleshooting processes, for example, the TCP diagnosis and HTTP diagnosis layouts. You can select a View Layout from the View Layout drop-down list in the Analysis Grid group on the Ribbon of the Message Analyzer Home tab.
Viewpoints — you have the option to apply a Viewpoint to enhance your data analysis and troubleshooting perspectives. When a Viewpoint is applied, you can examine network traffic from the perspective of a protocol because all messages above the “viewpoint” protocol are temporarily removed from display. This feature is advantageous when you have higher-layer traffic that obscures the underlying messages that you want to troubleshoot. For example, you might apply a TCP viewpoint to display TCP messages at top-level for diagnostic purposes. You might even add the TCP diagnosis View Layout to expose additional data field values that are particularly important to your analytical proceedings. You can do the same thing with an HTTP viewpoint and the HTTP diagnosis View Layout. To access the Viewpoints feature and apply a Viewpoint, click the Viewpoints drop-down list in the Viewpoints group on the Ribbon of the Message Analyzer Home tab.
Field Chooser — for any given set of data that displays in the Analysis Grid viewer, you should be aware that there are many more columns of data that you can add to the grid beyond the default column layout, to expose the values of message fields that could be critical to troubleshooting processes. You simply find the protocol or module of interest in the Field Chooser Tool Window, expand the protocol node, and navigate to the data field you want to add to the Analysis Grid viewer column layout. After you double-click the field name, it is added to the Analysis Grid as a new column, if the grid is in focus; at which time, data should display in the new field for the particular protocol or module of interest, unless the field is not used or contains no data.
If the Grouping Viewer is in focus, then double-clicking a field name in Field Chooser will add a new nested Group to the current Grouping View Layout configuration. Note that you can also right-click any field in Field Chooser and choose the Add as Column or Add as Grouping context menu command to add a column to the Analysis Grid or a new nested Group to the Grouping viewer, respectively.
Note To access the Field Chooser, click the Add Columns button in the Analysis Grid group on the Ribbon of the Message Analyzer Home tab when the Analysis Grid is in focus. If the Grouping viewer is in focus, you can access the Field Chooser by clicking the Add Grouping button in the Grouping group on the Ribbon of the Home tab.
Note View Layouts and column layouts are different terms that essentially describe the same feature or function in Message Analyzer. Also note that the Grouping viewer has its own set of View Layouts that are independent of the Analysis Grid viewer.
Aliases — if you have any data fields that are difficult to work with, due to their cryptic or complex naming, Message Analyzer enables you to convert their data values to a more user friendly name for ease of recognition. You can configure an Alias by right-clicking a field value that you want to convert and selecting the Create Alias for ‘<columnName>’… context menu item. This action causes the Alias Editor to display, from where you can configure a new Alias.
Unions— if you have multiple data sources that relate to a common environment or service from which you have run traces or generated logs, it is not uncommon for the different data sources to specify different names for fields that have identical values. If this occurs, you can create a Union to correlate the field values into a single, newly-named entity that you specify to reflect those values. This makes it easier to locate and analyze data in an interlaced set of messages from multiple sources.
Other techniques that you can use to analyze data consist of the following:
Using tools — you can display additional Tool Windows to dramatically enhance the scope of analysis capabilities.
Filtering columns — you can apply a Column Filter to any Analysis Grid viewer column, to filter your trace results according to search text that you specify for a column. You can also do the same for Tool Window columns, to filter for specific message field names or other data values in such windows. Be aware that Column Filters search only on data that displays in top-level parent nodes; child nodes will not be included in the search unless you first expand them.
Finding messages — you can use the Find feature to locate individual messages. The Find command is designated by the Find binoculars icon in the Find Messages window that displays when you click the Find Messages button in the Analysis Grid group on the Ribbon of the Message Analyzer Home tab. This command enables you to locate the next message that matches a specified Filter Expression, while still retaining visibility and context of all the messages in the original trace results. You can select a Filter Expression from the Library drop-down in the Find Messages window, or you can manually configure one, such as
contains “bing”or some other string, or
#MessageNumber==messagenumber, to locate the next message that contains the value that you specify.
By using this command, you can dramatically impact your data analysis experience. Although Message Analyzer already provides a View Filter capability that works similarly, the disadvantage of a View Filter is that all messages surrounding the target message are hidden unless they match the filter criteria. However, in many cases the context of the surrounding messages is key to the analysis. In these cases, it might be better to employ a Find filter. A Find filter highlights the next top-level message that matches the filtering criteria, however, note that the match might be to a message that is within the message stack of the highlighted top-level message.
Sorting — you can sort data columns in the Analysis Grid viewer to expose values or trends that can identify potential issues.
Applying time shift values — Message Analyzer provides a Shift Time dialog that enables you to apply a specific incremental time shift value to a message collection from a selected data source, when you know beforehand that a time shift is required. You can also specify a time shift for a particular message when you discover through analysis that a shift is required. Applying a time shift to a selected message then causes a recalculation of time stamps for all messages in a selected data source.
You can use this feature to synchronize multiple traces that you load into Message Analyzer, for example to adjust for machine skew or time zone changes across traces. You might also want to simply match the Timestamp of one message loaded from a particular data source to that of another message loaded from a different data source.
Adding bookmarks and comments — you can add Bookmarks and Comments for annotation purposes to coordinate data analysis with other team members.
To learn more about working with filters and other data manipulation features, see the following topics:
Applying and Managing View Filters
Applying Quick Filters
Using the Find Messages Feature
Filtering Column Data
Using and Managing Color Rules
Understanding and Managing Viewpoints
Sequence Match Viewer
Using the Analysis Grid Group Feature
Applying and Managing Analysis Grid View Layouts
Using the Field Chooser
Using and Managing Aliases
Configuring and Managing Unions
Setting Time Shifts
Save Message Data
After you have performed analysis of your message data, you have the option to save it in the Message Analyzer native file format, as described in Saving Message Data. Thereafter, if you want to work further with the data or share it with others, you can quickly load the data back into Message Analyzer through a Data Retrieval Session, or by using the Quick Open feature on the Files menu.
The figure that follows illustrates the Save/Export Session dialog, in which you can choose the messages you want to save. You have the option to save all messages that you captured in a Live Trace Session or loaded from a Data Retrieval Session, filtered messages only, or you can select specific messages to save.
Figure 5: Message Analyzer Save/Export Session dialog
Note that there are two different ways to save specifically selected messages, as follows:
Use the Windows Save As dialog — highlight one or more messages in the Analysis Grid viewer, right-click the group of messages, and then select the Save Selected Messages… context menu item to display the Windows Save As dialog. After you specify a name for the session messages you are saving, click the Save button to save the session in the native .matp file format only.
Use the Message Analyzer Save As dialog options — highlight one or more messages in the Analysis Grid viewer and then click Save As in the File menu to display the Save As dialog. Note the additional save options when using this dialog, as indicated in Step 1 and Step 2. In the dialog, you will see three radio button options for saving the data; including All Messages (), Filtered Messages for Analysis Grid view (), and Selected Messages forsessionName session (). Select the third option, which parenthetically indicates the number of messages that you highlighted in the Analysis Grid viewer.
Thereafter, click the Save As button in Step 2 of the dialog to open the Windows Save As dialog, from where you can navigate to an appropriate directory location for saving the data in the native Message Analyzer .matp file format. To export the selected messages to a .cap file, click the Export button in Step 2 of the dialog to display the Windows Save As dialog.
If you have a session configuration that consists of an aggregation of data from multiple sources that you have analyzed, Message Analyzer enables you to save your results to a single file in the default .matp format. Prior to saving your data, you can add rich-text comments and you can specify a unique session file name. Note then when you export your data as a .cap file, it will be compatible with the Microsoft Network Monitor tool and other applications.
To learn more about saving Message Analyzer data, see Saving Message Data.