Exchange MTA in Exchange Server 2003 Architecture

 

The Exchange MTA is a core component of Exchange Server 2003 and is responsible for all non-SMTP message transfer. This includes message transfer to external X.400 messaging systems and Exchange servers connected through X.400 connectors. Message transfer to non-Exchange messaging systems, such as Lotus Notes and Domino or Microsoft Exchange Connector for Novell GroupWise, is controlled by the Exchange MTA through MAPI-based connectors, such as Microsoft Exchange Connector for Lotus Notes or Microsoft Exchange Connector for Novell GroupWise. Exchange MTA is also responsible for remote procedure call (RPC)-based communication with Exchange Server 5.5. It is best to implement Exchange MTA on all servers running Exchange Server 2003.

Exchange MTA Communication Partners

The Exchange MTA is implemented in a Microsoft Windows service named Microsoft Exchange MTA Stacks. When you display the properties of the Microsoft Exchange MTA Stacks service in the Services tool and click the Dependencies tab, you find that the Microsoft Exchange MTA Stacks service depends on System Attendant. System Attendant hosts the Directory Service Access (DSAccess) component that the MTA uses to obtain recipient and configuration information from Active Directory. However, System Attendant is not the only dependency, as illustrated in the following figure.

MTA communication partners

The Exchange MTA communicates with the following components:

• Active Directory   Communication with Active Directory is required because the MTA configuration object and X.400 connector objects reside in the configuration directory partition of Active Directory. The Exchange MTA cannot start if Active Directory is inaccessible.

• Registry database   Not all MTA configuration settings are maintained in Active Directory. Important settings for the Microsoft Exchange MTA Stacks service are also stored in the server's registry in the following location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMTA. This section explains numerous settings that you can find under the Parameters key. By configuring registry parameters manually using Registry Editor, you can fine-tune the MTA performance.

  Warning

  Using Registry Editor incorrectly can cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk.

• SMTP service   In Exchange Server 2003, the Exchange MTA does not perform message routing. Message routing is the task of the routing engine, which is part of the SMTP service. The Exchange MTA communicates with the routing engine through Mtaroute.dll to make its routing decisions.

  Note that the communication with the SMTP service is bi-directional. The MTA communicates with the SMTP service for message routing purposes. The SMTP service initiates communication when the routing topology changes, to inform the MTA that all messages must be rerouted. For more information about the interaction between the MTA and the SMTP service, see Connecting Routing Groups Using X.400 Connectors.

• Exchange store   As illustrated in the following figure, the SMTP service passes outbound messages to the Exchange MTA through the MTA's message queue in the Exchange store. The Exchange MTA also passes inbound messages to the SMTP service through the Exchange store. Communication with the Exchange store is required if messages must be transferred over MAPI-based messaging connectors for which the Exchange MTA is responsible. MAPI-based messaging connectors, similar to the SMTP service, maintain message queues in the Exchange store, as explained in Gateway Messaging Connectors Architecture.

  MTA communication through the Exchange store

  

  To communicate with the Exchange store, the MTA uses an internal API named XAPI, which is a wrapper around MAPI. To start the Microsoft Exchange MTA Stacks service successfully, the Exchange store must have at least one mailbox store to host the message queues. You can read more about outbound and inbound message handling later in this section.

  Note

  If your bridgehead server running Exchange Server 2003 hosts many X.400 connectors or connects to non-Exchange messaging systems, such as Lotus Notes and Novell GroupWise, you should consider placing the transaction logs and database files of the Exchange store on separate disks, even if the Exchange store does not contain user mailboxes. By spreading the database files across multiple physical drives, you can improve system performance.

• File system   The Exchange MTA uses two important directories on the file system. These directories are named the MTA database directory and the MTA run directory. The database directory contains queue files and messages during transfer in the form of .dat files. This collection of .dat files is named the MTA database. The run directory contains template files (named run files). Run files determine how the MTA formats data using Abstract Syntax Notation One (ASN.1). By default, both the database directory and run directory point to the same physical directory on the file system. As indicated in Figure 7.2, this is the \Program Files\Exchsrvr\Mtadata directory. It is best to split the database and the run directory, as explained later in this section.

  Note

  The run directory does not contain the actual executable file (Emsmta.exe) and dynamic link libraries (DLLs) of the Microsoft Exchange MTA Stacks service. Emsmta.exe and DLLs, such as Mtaroute.dll, are stored in the \Program Files\Exchsrvr\bin directory.

• Remote X.400 MTA   The Exchange MTA communicates with remote X.400 MTAs through X.400 connectors. An X.400 connector is a configuration object in Active Directory that describes the communication parameters and message formats that the Exchange MTA must use for message transfer. A remote X.400 MTA can be a non-Exchange MTA or an Exchange MTA connected through an X.400 connector. For more information about X.400 communication, see MTA Transport Stacks and X.400 Connectors.

• Exchange 5.5 servers in the same routing group   X.400 connector objects are not required for the Exchange MTA to communicate with servers running Exchange Server 5.5 in the local routing group. These types of MTAs are also named LAN-MTAs to emphasize the fact that an explicit X.400 connector is not involved in their communication. LAN-MTAs use RPCs to communicate with each other. For more information about communication between Exchange Server 2003 and Exchange 5.5, see Exchange MTA in a Mixed-Mode Organization.

Exchange MTA Configuration Settings in Active Directory

The Exchange MTA obtains its configuration settings from the local registry and from an MTA object in Active Directory. The MTA directory object is located under each Exchange server object in the configuration directory partition. For example, the following is the distinguished name of an MTA object on a server called SERVER01 in the tailspintoys.com domain: CN=Microsoft MTA,CN=SERVER01,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=Tailspin Toys,CN=Microsoft.

The location of the MTA configuration object is slightly different in Exchange System Manager. You can find the MTA configuration object in the Protocols container under the server object, as shown in the following figure. The object is named X.400, although you are actually configuring the MTA and not just X.400 settings. You can use the X.400 configuration object to define the name of the MTA and its local password. You can also specify the MTA database directory in which the message queues reside. On the Messaging Defaults tab, you can configure general communication parameters, such as retry values, which the MTA uses for communication with LAN-MTAs. You can override these settings when you configure X.400 connectors.

The MTA configuration object in Exchange System Manager

MTA configuration objects are instances of the MTA object class. For example, you can use this information in a Lightweight Directory Access Protocol (LDAP) filter to export all settings from all MTAs in an Exchange organization to a text file. The following LDAP Data Interchange Format Directory Exchange (LDFIDE) command demonstrates how to perform this step: ldifde -f c:\AllMTAs.ldf -s localhost -d "CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com" -p subtree -r "(objectClass=mTA)". You must have read permissions on all administrative groups in the organization.

The following table lists the important attributes of the MTA directory object and their purposes.

Important attributes of the MTA directory object

Directory Attribute	Purpose
objectClass	Identifies the directory object as an MTA object.
cn	Contains the common name (cn) of the MTA.
transRetryMins	Defines the period of time between transfer retries in minutes. The default is five minutes.
transTimeoutMins	Defines the timeout period for message transfer in minutes. The default is 20 minutes.
mTALocalDesig	Specifies the X.400 name of the local MTA. This name, of up to 32 characters, is used to identify the local Exchange MTA to remote MTAs and LAN-MTAs.
delivContLength	Defines the maximum deliverable message size, in kilobytes (KB), that can be sent over the MTA.
diagnosticRegKey	Specifies the location of the diagnostic registry key. If this key does not exist, the MTA uses the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMTA\Diagnostics.
expandDLsLocally	Corresponds to the setting Expand remote distribution lists locally, in the MTA properties. If expandDLsLocally is True, and a user sends a message to a remote distribution list, the MTA expands the distribution list locally. The Exchange MTA then determines the best route for the message, based on the location of recipients in the list. This method guarantees the most efficient message handling. However, processing large distribution lists can affect server performance.
msExchHomeRoutingGroupDNBL	A back link to the routing group of which this MTA is a member.
associationLifetime	Specifies the amount of time in seconds that the system keeps an association open to a remote X.400 MTA after a message is sent. The default is 300 seconds.
numOfOpenRetries	Specifies the maximum number of times that the Exchange MTA tries to open a connection before it sends a non-delivery report (NDR). The default is 144 times.
numOfTransferRetries	Specifies the maximum number of times that the Exchange MTA tries to transfer a message across an open connection. The default is two times.
openRetryInterval	Specifies the number of seconds that the system waits after an error before attempting to reopen a connection. The default is 600 seconds.
rTSCheckpointSize	Specifies the amount of data in KB that is transferred before a checkpoint is inserted. If an error occurs and the message must be resent, the process restarts from the most recent checkpoint. A value of zero indicates that no checkpoints are inserted.
rTSRecoveryTimeout	Specifies the amount of time after a broken connection that the MTA waits for reconnection before deleting the information (with its checkpoints) and restarting the transfer from the beginning.
rTSWindowSize	Specifies the number of checkpoints that can go unacknowledged before data transfer is suspended. The window size has no meaning if the checkpoint size is zero.
sessionDisconnectTimer	Specifies the amount of time in seconds that the Exchange MTA waits before ending a connection after all associations over this connection are ended.
tempAssocThreshold	Specifies the maximum number of queued messages that the system can send to a remote system. When this is exceeded, the MTA opens another association.
transferRetryInterval	Specifies the number of seconds that the system waits after a message transfer fails before re-sending a message across an open connection. The default is 120 seconds.
transferTimeoutNonUrgent	Specifies the amount of time in seconds per kilobyte that the system waits before sending an a non-delivery report (NDR) for a non-urgent message.
transferTimeoutNormal	Specifies the amount of time in seconds per kilobyte that the system waits before sending a non-delivery report (NDR) for a normal message.
transferTimeoutUrgent	Specifies the amount of time in seconds per kilobyte that the system waits before sending a non-delivery report (NDR) for an urgent message.
msExchMTADatabasePath	Indicates the path to the MTA database directory.
msExchResponsibleMTAServerBL	Contains the distinguished name of the server hosting the MTA.
msExchEncryptedPassword	Specifies the MTA password that remote MTAs must use when connecting to this MTA. The password can be up to 64 characters long and is stored in Active Directory in encrypted form. Note The MTA password is stored in encrypted form in Active Directory, but this does not mean that MTA names and passwords are secure. X.400 MTAs exchange their names and passwords in clear text when establishing connections.

Internal Exchange MTA Architecture

The internal architecture of the Exchange MTA is based on thread pools, message queues, and database queues. Threads perform the actual processing within the Exchange MTA process (Emsmta.exe). Message queues, indicated by arrows in the following figure, handle interactions and notifications between threads. Database queues contain messages waiting for processing by one of the thread pools. For example, the work queue, shown in the following figure, is a database queue that holds messages until the MTA has transferred them successfully or until they expire and are returned to the sender with a non-delivery report (NDR). Running multiple threads allows the Exchange MTA to perform multiple tasks concurrently. This figure shows the internal architecture of the Exchange MTA.

Internal Exchange MTA architecture

The internal Exchange MTA architecture is based on the following elements:

• XFER IN and XFER OUT threads   These threads handle the actual message transfer in and out of the MTA process. This communication takes place through the following mechanisms:

  • X.400   Communication with a remote X.400 MTA or an Exchange server in a remote routing group using an X.400 connector.

  • RPCs   Communication with a LAN-MTA (that is, a server running Exchange Server 5.5 in the local routing group).

  • XAPI   Communication with the Exchange store using MAPI.

  You can control the number of transfer threads using the following registry parameter.

  Location	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ MSExchangeMTA\Parameters
  Value	Transfer threads
  Type	REG_DWORD
  Value Data	0x1
  Description	Determines the number of MTA transfer threads. This value is multiplied by two for the two subtypes of transfer threads (XFER IN and XFER OUT). The default value is 0x1. The recommended value is 0x3, if this MTA communicates with multiple remote MTAs, either within a routing group or between routing groups.

• Work Queue   The Exchange MTA writes all messages to .dat files in the MTA database directory on the file system. It then creates a pointer for each .dat file in the work queue. The work queue maintains a reference to each message for which the MTA is responsible. The MTA database and .dat files are discussed later in this section.

• **Dispatcher   **The dispatcher is at the core of the MTA process and is responsible for message routing and results processing. The dispatcher uses router, fan-out, and result threads for message processing. The dispatcher performs the following key tasks:

  • Message routing, fan-out, and transfer   The dispatcher uses a routing thread to communicate with the routing engine and determine the next hop for message transfer. If a message is addressed to recipients in different destinations that are reached through separate connections (such as two X.400 connectors installed on the same server), the dispatcher uses a fan-out thread. The fan-out thread creates multiple message copies and places them in the appropriate link queues. The dispatcher then triggers XFER OUT threads to process the fan-out message copies.

    Note

    The process of creating multiple message copies in the MTA is named fan-out. This is in contrast to bifurcation, which refers to the process of creating multiple message copies in the categorizer of the SMTP transport subsystem. The categorizer is covered in detail in SMTP Transport Architecture.

  • Results processing   The dispatcher also performs reporting based on the results of routing, fan-out, and transfer actions. For example, if a message is successfully transferred over an X.400 connector, the dispatcher updates the responsibility flags for the message recipients in the work queue to indicate transfer success.

    Each message in the work queue has a bitmap that is made up of one bit per recipient. Initially, the MTA sets the bits for all recipients to indicate that the message is not transferred. Once a fan-out copy of a message is successfully transferred, by an XFER-OUT thread, the dispatcher clears the bits corresponding to the recipients on that fan-out copy. The MTA removes the message from the work queue when the message successfully transfers to recipients, or when the message expires. At this point, the MTA generates an NDR for each recipient that did not receive the message.

  You can control the number of dispatcher threads using the following registry parameter.

  Location	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ MSExchangeMTA\Parameters
  Value	Dispatcher threads
  Type	REG_DWORD
  Value Data	0x1
  Description	Determines the number of MTA dispatcher threads, which are responsible for message processing. This value is multiplied by three for the three subtypes of dispatcher threads (that is, router, fan-out, and result threads). The default value is 0x1. The recommended value is 0x3 if this MTA communicates with more than five LAN-MTAs.

• Submit and deliver threads   These thread pools are a legacy from Exchange Server 5.5. They are not used in Exchange 2000 Server and Exchange Server 2003 under typical circumstances, because in Exchange 2000 Server and Exchange Server 2003, there is no direct submit or delivery. All messages must pass through the SMTP service. The SMTP service delivers messages to local recipients through the Exchange store driver, as explained in SMTP Transport Architecture.

  The Exchange MTA treats the SMTP service similar to a MAPI-based connector with message queues in the Exchange store. XAPI gateways handle the message transfer in and out of the message queues in the Exchange store. XAPI gateways are threads in the Exchange store that interface with the XFER IN and XFER OUT MTA threads.

The Exchange MTA uses the following threads for inter-process communication and to perform system functions, such as updating configuration information when changes occur.

• OSI stack   The Exchange MTA uses a number of thread pools to handle communication tasks between the various layers of the Open Systems Interconnection (OSI) stack. These thread pools include reliable transfer service (RTS) threads, kernel threads, RPC threads, transport threads, and TCP/IP or X.25 threads. For more information about the purpose of these threads, see MTA Transport Stacks and X.400 Connectors.

• Timers   The Exchange MTA runs timer threads in various situations; for example, to wait after a message transfer fails before re-sending a message across an open connection.

• Directory polling   The Exchange MTA uses a separate thread to poll Active Directory every ten minutes for configuration changes.

• Threads not owned by the MTA   The routing engine and the DSAccess module own threads that run in the MTA process, to inform the Exchange MTA when changes occur. For example, the routing engine calls the RoutingReset () function of the MTA if the routing topology changes to trigger message rerouting for all queued messages. DSAccess communicates with the Exchange MTA to provide cached directory information.

Exchange MTA Database

The Exchange MTA maintains all messages that it transfers in the MTA database. The MTA database is a flat-file database that is made up of .dat files, as illustrated in the following figure. There are separate .dat files for database queues and the actual messages. Note that database queues contain references or pointers to the actual messages, but database queues do not contain the messages themselves. The MTA stores each message in a separate message .dat file in the MTA database directory. In an active database, there is a DBREFS file that contains reference counts for objects, which provide tertiary backup information. DBREFS is rebuilt when the MTA is restarted or the MTA check tool is run.

It is difficult to distinguish between the various .dat file types in an active MTA database. The Exchange 2000 Server Resource Kit (available at bookstores) includes a tool named MTAView. You can use MTAView to view the contents of the MTA's queues and the headers of the messages in these queues. The following figure illustrates the MTAView tool with an active MTA database open. The foreground window shows the .dat files in the MTA message queue directory.

Message queues and .dat files of the Exchange MTA

The MTA .dat files are divided into two categories:

• Core .dat files   The core .dat files act as the reference rows and columns of the MTA database. There are 38 core .dat files in the message queue directory (\Program Files\Exchsrvr\Mtadata), and all are required for the Exchange MTA to run. Most core .dat files ship with Exchange Server 2003. You can find these files on the product CD in the \Setup\i386\Exchange\Bootenv directory. The Exchange Server 2003 Setup program copies these files to the \Program Files\Exchsrvr\Mtadata directory during installation. The core .dat files that ship with Exchange Server 2003 are numbered DB000001.dat through DB000026.dat (in hex numbers).

  The Exchange MTA dynamically creates additional .dat files for important database queues. For example, the MTA rebuilds the MTA work queue each time the MTA is restarted. During this rebuild, no messages are delivered, because the MTA must first account for all .dat files. When all files are accounted for, message transfer begins.

  The most important core .dat files on an active MTA database are:

  • DB000001.dat   This is the most important database queue, named the master queue. The master queue contains a list of queue object IDs and references to other work queues, which each have their own individual .dat files.

  • DB000020.dat   This is the XAPI work queue that maintains references to messages bound for the SMTP service or MAPI-based gateway connectors, such as Connector for Lotus Notes and Connector for Novell GroupWise, which have their own message queues in the Exchange store.

  • DB000025.dat   This is the out of office information queue that contains objects for out of office notifications.

  • DB000026.dat   This is the reference data queue. For redundancy, .dat files are referred more than once in the queue .dat files.

  • DB000027.dat   This is the MTA work queue that contains a list of messages waiting for transfer.

• Message and placeholder .dat files   The remaining .dat files in the database directory are message and placeholder files. The MTA creates a message .dat file for each message it must transfer. Because the number portion of the file name is assigned at random (DB######.dat), duplicate file names are possible. To avoid overwriting an existing file, the Exchange MTA creates a placeholder .dat file together with each message .dat file. The placeholder .dat file is one byte in size and is used if a message .dat file cannot be created because of a duplicate file name. The figure above illustrates a message .dat file of two megabytes (DB00002D.dat) and a placeholder .dat file of zero kilobytes (DB00002E.dat). The MTA creates two .dat files for each message.

  After a message is transferred, the Exchange MTA erases the data from the .dat files and resets the files to one byte (instead of deleting the .dat files). The Exchange MTA can then reuse the .dat files for future messages. This mechanism enhances the performance of the MTA, because creating and deleting files is time consuming. The number of .dat files that you find in the message queue directory reflects the maximum number of messages in the queue at any one time. On a busy bridgehead server running Exchange Server 2003, you might find hundreds of .dat files in the MTA database directory.

Relocating the MTA Database Directory

Exchange System Manager enables you to move the MTA database directory to another location in the file system (in the server's Protocols container, right-click the X.400 configuration object, click Properties, and then on the General tab, under Message queue directory, click Modify). When you modify the location of the queue directory in Exchange System Manager, you move the database files to the new location, and you change the msExchMTADatabasePath attribute of the MTA directory object to point to the new location. However, the msExchMTADatabasePath attribute is for informational purposes only. More important is the following registry key, which Exchange System Manager also updates.

Location	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ MSExchangeMTA\Parameters
Value	MTA database path
Type	REG_SZ
Value Data	C:\Program Files\Exchsrvr\Mtadata
Description	Points to the location of the database files (queue files and message files) for the Exchange MTA. If the path is not valid or points to an empty directory, the MTA cannot start.

You should not place the MTA database directory on a file allocation table (FAT) partition. FAT16 partitions support a maximum of 65,535 files. This size limitation can be an issue on a busy bridgehead server. When a queue starts to fill, available entries with which to create new files can become depleted in only a single day. This problem is compounded because each message requires two .dat files. Moreover, FAT does not perform well on large volumes, provides only minimal fault tolerance, and has no recoverability after an unexpected system halt.

On a busy X.400 bridgehead server running Exchange Server 2003, you can optimize performance by placing the MTA database directory on a high-performance disk subsystem, such as a redundant array of independent disks (RAID). A RAID 0+1 with eight to ten disks is a good starting point for high-volume message transfer. A RAID controller with a cache larger than 64 megabytes (MB) also helps performance.

Secured and Unsecured Message Copies

Messages in the MTA work queue are named secured messages, because they are always saved in .dat files. These .dat files persist even if an unexpected termination of the Exchange MTA process occurs. The dispatcher uses secured messages in the work queue as source files and creates unsecured message copies. The dispatcher then attaches the appropriate link queues for the connectors or next hop links to transfer the messages. If a message is intended for multiple destinations, which are reached over separate connections, the dispatcher creates a fan-out copy for each connection. Each fan-out copy references the secured message in the MTA work queue. The MTA removes secured and unsecured message copies from the queues when the message is transferred successfully.

Messages on link queues are represented as reference pointers to message objects and not as the objects themselves. Secured messages are available in .dat files, but this is not necessarily true for unsecured copies. Unsecured copies are kept in memory primarily to increase MTA performance. The dispatcher saves unsecured copies in .dat files only if there is not enough cache space available to hold the objects in memory. The cache is a fixed pool of 4k buffers and per-object structures, based on thread pool sizes. You can adjust the number of data buffers per object in memory using the following registry parameter.

Location	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ MSExchangeMTA\Parameters
Value	DB data buffers per object
Type	REG_DWORD
Value Data	0x3
Description	Determines the number of database server buffers that are configured for each database object. More buffers require more memory but make it less likely for a database object to be rolled out to disk because of a lack of buffer space. The default value is 0x3. The recommended value is 0x6 if this MTA communicates with multiple MTAs, either within a routing group or between routing groups. Additionally, you should tune this setting if a MAPI-based connector is installed on this server.

MTA Database in a Server Cluster

It is not possible to partition the MTA database and .dat files so that multiple MTA instances can use the MTA database concurrently. This is a requirement, for example, if you want to run MTA on multiple cluster nodes in a server cluster, based on the Microsoft Windows Server 2003 Clustering service. Because only one MTA can own the MTA database, the Exchange MTA can run only on the first initialized node in the cluster. Upon failover, the cluster service stops the MTA on the failed node and starts it on another node, which then recovers from the same .dat files and re-establishes connections.

Corrupted Messages in Gateway Queues

Exchange Server 2003 transfers messages, destined for MAPI-based connectors, to the MTA through the Exchange store. Messages travel back from the MTA, through the Exchange store, to the connector mailbox. By default, there are only two threads (one for incoming transfers and one for outgoing transfers) communicating with the MTA. This means that a corrupt message, at the top of a gateway link queue in the MTA, can block all message flow to or from the Exchange store. To unblock the message queue, you can use the Queue Viewer snap-in in Exchange System Manager to delete critical messages.

You can also increase the number of gateway in-and-out threads in the Exchange store process for each mailbox store by using the following registry parameters. If the Exchange store communicates with the Exchange MTA using multiple threads, a corrupt message might block one thread, but another thread might still be able to process further messages. If all gateways are blocked by multiple corrupted messages, this might indicate a serious problem (for example, a hard disk controller malfunction).

Location	HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services \MSExchangeIS\<server_name>\Private-<database_guid>
Value	Gateway In Threads Gateway Out Threads
Type	REG_DWORD
Value Data	0x1
Description	The default value for both of these parameters is 0x1. The recommended setting is 0x3 if this MTA communicates with multiple LAN-MTAs or is responsible for MAPI-based messaging connectors. You must add these parameters to all keys for mailbox stores in the registry. Each gateway-in and gateway-out thread consumes about one MB of virtual memory. This could be an issue on servers with many private databases. For example, if you have ten private databases, and you increase these parameters from 0x1 to 0x3 (a total increase of four threads), you create 4 x 10 = 40 threads, which together consume 40 MB of virtual memory.

Fixing MTA Database Corruption

If deleting a corrupted message from a message queue does not fix all problems relating to MTA queues, use the MTA Check tool (Mtacheck.exe) to analyze and correct problems in the MTA database. Run MTA Check if you suspect corruption in the MTA database or see errors written to the Event Log. The MTA Check tool must be used directly on the server, you must stop the Microsoft Exchange MTA Stacks service. The MTA Check tool checks the pointers in the database queues and ensures that the .dat files indicated in the queue files exist. The tool removes corrupt messages (that is, messages with inconsistent pointers). Corrupt messages are moved to the \Program Files\Exchsrvr\Mtadata\Mtacheck.out directory. You can use command-line parameters to delete public folder replication messages and other system messages, but you must use them with care. For further information, see the MTA Check tool documentation. The MTA Check tool and its documentation are available from the Downloads for Exchange Server 2003 Web site.

Wiping the MTA Database

On a busy bridgehead server with multiple X.400 or MAPI-based connectors, the MTA database directory can fill with more than 10,000 .dat files. This issue is compounded if there are transfer problems due to corrupt messages or network problems. The relevant physical limit to the number of .dat files that can be written to disk is the amount of available disk space. The Microsoft Exchange MTA Stacks service shuts down when the available disk space drops below ten on the hard drive where the MTA database is located.

To restart the Microsoft Exchange MTA Stacks service, you must make sure that more than ten megabytes of disk space are available. This might require that you temporarily move all of the .dat files out of the MTA database directory to another drive or a network share. The process of moving files to create an empty database directory is known as an MTA wipe. An MTA wipe is helpful when troubleshooting MTA database problems, such as numerous corrupted messages stuck in database queues.

For detailed instructions about how to wipe the MTA Metabase, see How to Wipe the MTA Database.

Replaying DAT Files

To deliver messages that were in the MTA database at the time of a MTA database wipe, you must replay the .dat files. There are three different ways in which you can replay DAT files.

• You can perform a complete replay   The easiest way to replay the .dat files is to replay them all at one time on the server where they originally resided. To do this, the MTA on the Exchange server of origin should have nothing in its queues. If there are messages in the MTA queues, the MTA should be allowed to finish sending the messages until all of the queues are empty.

• You can perform a remote replay   The .dat files do not have to be replayed on the Exchange server where they originally resided. If, for example, the original Exchange server is a busy bridgehead server that continuously transfers large quantities of e-mail messages and does not reach an empty MTA work queue, you can choose to replay the messages on another server running Exchange Server.

• You can perform an incremental replay   An incremental replay is performed by dividing the .dat files into several smaller groups, which are replayed one at a time. This approach is more complicated than a complete or remote replay, but can be helpful when dealing with a very large number of .dat files. An incremental replay might also be a good idea when important messages must be delivered, but a corrupt message somewhere in a message queue is causing the MTA to stop unexpectedly. You can perform an incremental replay on the original Exchange server or on another Exchange server in the organization.

For detailed instructions about how to perform each of these procedures, see How to Replay DAT Files After an MTA Database Wipe.

Examining Exchange MTA Processing

The primary tools used to monitor the MTA are System Monitor (in the Performance Monitor tool) and Event Viewer. You can use System Monitor to monitor the behavior of messaging connectors in real time. You can determine the number of messages waiting in inbound and outbound message queues. You can determine the total number of messages that have been transferred by a connector since the MTA was started. It is a good idea to check message queues using System Monitor, because numerous messages queuing up in a messaging connector might indicate a performance bottleneck or a malfunctioning connector component. The Event Viewer tool can help you identify and diagnose the source of system problems or help you predict potential issues. The Exchange MTA writes critical events in the Application Event Log. An event is an occurrence of activity in the Exchange MTA that might require administrator attention.

Checking the Exchange MTA Using System Monitor

Performance monitoring involves looking at discrete components of a system. In Windows, an object represents an individual process, a section of shared memory, or a physical device. An object can have several performance counters associated with it. For example, the MSExchangeMTA object has many performance counters, including a counter named Work Queue Length that indicates the number of messages in the MTA work queue.

To add performance counters to System Monitor, start the Performance monitor tool and then, in the toolbar, click Add, indicated by a plus sign (+). You can then select the MSExchangeMTA object from the Performance object drop-down list in the Add Counters dialog box. According to your selection, appropriate performance counters are listed in the Select counters list.

The following table lists the performance counters available for the MSExchangeMTA performance object.

MSExchangeMTA performance counters

Performance counter	Description
Adjacent MTA Associations	The number of open associations that this MTA has to other MTAs.
Messages/Sec	The rate at which messages are processed.
Message Bytes/Sec	The rate at which message bytes are processed.
Free Elements	The number of free buffer elements in the MTA pool.
Free Headers	The number of free buffer headers in the MTA pool.
Admin Connections	The number of Microsoft Exchange Administrator programs connected to the MTA.
Threads In Use	The number of threads in use by the MTA. This number can be used to determine whether additional processors might be beneficial.
Work Queue Length	The number of outstanding messages in the Work Queue. This indicates the number of messages not yet processed to completion by the MTA.
XAPI Gateways	The number of XAPI gateways connected to the MTA using the XAPI interface. A single gateway can have multiple XAPI gateway sessions.
XAPI Clients	The number of XAPI clients connected to the MTA using the XAPI interface. A single client can have multiple XAPI client sessions.
Disk file deletes	The number of disk file delete operations per second.
Disk file syncs	The number of disk file sync operations per second.
Disk file opens	The number of disk file open operations per second.
Disk file reads	The number of disk file read operations per second.
Disk file writes	The number of disk file write operations per second.
ExDS Read Calls/sec	The rate of read calls to the Directory service.
XAPI Receive Bytes/sec	The rate at which bytes are received over a XAPI connection.
XAPI Transmit Bytes/sec	The rate at which bytes are transmitted over a XAPI connection.
Admin Interface Receive Bytes/sec	The rate at which bytes are received over an admin connection.
Admin Interface Transmit Bytes/sec	The rate at which bytes are transmitted over an admin connection.
LAN Transmit Bytes/sec	The rate at which bytes are transmitted over a LAN to MTAs.
LAN Receive Bytes/sec	The rate at which bytes are received over a LAN from MTAs.
RAS Receive Bytes/sec	The rate at which bytes are received over a RAS connection. RAS-based X.400 connectors are not available in Exchange Server 2003.
RAS Transmit Bytes/sec	The rate at which bytes are transmitted over a RAS connection. RAS-based X.400 connectors are not available in Exchange Server 2003.
TCP/IP Receive Bytes/sec	The rate at which bytes are received over a TCP/IP connection.
TCP/IP Transmit Bytes/sec	The rate at which bytes are transmitted over a TCP/IP connection.
TP4 Receive Bytes/sec	The rate at which bytes are received over a TP4 connection. TP4-based X.400 connectors are not available in Exchange Server 2003.
TP4 Transmit Bytes/sec	The rate at which bytes are transmitted over a TP4 connection. TP4-based X.400 connectors are not available in Exchange Server 2003.
X.25 Receive Bytes/sec	The rate at which bytes are received over an X.25 connection.
X.25 Transmit Bytes/sec	The rate at which bytes are transmitted over an X.25 connection.

The Exchange MTA also provides a performance object for each connection established by the MTA. In the Add Counters dialog box, select the MSExchangeMTA Connections object from the Performance object drop-down list. You can then find the individual connection instances under Select instances from list. Each MSExchangeMTA Connections instance describes a single connection, but you can also choose to monitor all instances together.

Performance counters for individual connections established by the MTA

The following table lists the performance counters that are available for MSExchangeMTA Connections performance objects.

MSExchangeMTA connections performance counters

Performance counter	Description
Associations	The number of associations between the MTA and the connected MTA. MTAs can open multiple associations, if additional transfer throughput is necessary.
Receive Bytes/sec	The rate at which bytes are received from the connected entity.
Send Bytes/sec	The rate at which bytes are sent to the connected entity.
Receive Messages/sec	The rate at which messages are received from the connected entity.
Send Messages/sec	The rate at which messages are sent to the connected entity.

Exchange MTA Event Logging

Troubleshooting MTA issues should include an inspection of the Application Event Log. The Exchange MTA tracks critical events automatically, but you can also set diagnostics logging levels by category for the MTA, to increase the amount of information that the Exchange MTA writes in the event log. In Exchange System Manager, display the properties of the server object of interest and then click the Diagnostics Logging tab. Under Services, select MSExchangeMTA, and then select Categories to list the categories for the MTA. Each MTA category has a diagnostics logging level. When the MTA generates an event with a significance number less than or equal to the logging level, the event is recorded in the Event Log. If the significance number of the event is greater than the logging level, it is not logged. The following table lists the Exchange MTA categories that can be enabled for diagnostics logging.

During normal operation, all categories of the Exchange MTA should have a logging level of None. At this level, only error events and critical messages are written to the event log. When increasing the logging level, select only those categories relevant to the issue. Setting logging levels too high, for too many categories, can quickly fill the event log with irrelevant information. Do not forget to reset the logging level on all categories to None when you are done examining the MTA.

Diagnostics logging categories for the Exchange MTA

Category	Description
X.400 Service	Reports events related to X.400 protocol handling, such as delivery of messages to a remote MTA.
Resource	Reports events related to the use of MTA resources.
Security	Reports events related to MTA security and security violations.
Interface	Reports events related to communication between the MTA and the Exchange store and communication between MTAs using RPCs.
Field Engineering	Reports events related to debugging MTA operation. These events reveal details about internal processing in the Exchange MTA and are useful to a Microsoft Product Support Services support specialist.
MTA Administration	Reports events related to message queues. Uses the Queue Viewer snap-in, in Exchange System Manager, to work with MTA queues to generate these events.
Configuration	Reports events related to problems with MTA configuration settings. Corrupt run files in the \Mtadata directory or invalid registry parameters can cause these events.
Directory Access	Reports events related to Active Directory communication.
Operating System	Reports events related to operating system services that the MTA uses to create and manage files and threads.
Internal Processing	Reports events related to the internal processing in the Exchange MTA, which can be useful to a Microsoft Product Support Services support specialist.
Interoperability	This category does not cause the MTA to write information to the Application Event Log. Instead, it tracks the binary content of protocol messages. Use this category in conjunction with the Interface category to log messages sent over internal interfaces to AP*.log files in the \Program Files\Exchsrvr\Mtadata directory. For example, you can use AP*.log files to create MTA stack traces and XAPI traces. Interoperability logs can be instrumental in tracking down configuration problems on MTAs. Increasing diagnostic logging levels to Medium or higher for both the Interoperability and Interface categories generates AP*.log files. The current log is always named Ap0.log. Prior logs are named AP#.log, with the # increasing with the age of the log. When a log reaches five megabytes, it is renamed and a new AP*.log is created. Note AP*.log files can grow very quickly and have a performance impact on the Exchange MTA.
APDU (Application Protocol Data Unit)	This category does not cause the MTA to write information to the Application Event Log. Instead, it tracks the message envelope content (the P1) for messages the MTA sends and receives, as well as the fully-encoded application protocol data units (APDUs) that are transmitted between the MTAs. Use this category in conjunction with the X.400 Service category to log information to BF*.log files in the \Program Files\Exchsrvr\Mtadata directory. BF*.log files are useful when diagnosing interoperability or conformance problems, for example, when messages from remote MTAs are bad or formatted incorrectly. An ASN.1 Parser tool, such as the Aspirin tool (included in Exchange 2000 Server Resource Kit, available in bookstores), can be used to decode the data in a Bf*.log. Increasing diagnostic logging levels to Medium or higher for both the APDU and X.400 Service categories generates BF*.log files. The current log is always named Bf0.log. Prior logs are named BF#.log, with the # increasing with the age of the log. When a log reaches 5 megabytes in size, it is renamed and a new BF*.log is created. Note BF*.log files can grow very quickly and have a performance impact on the Exchange MTA.

Text Event Logging

To log all MTA event information to EV*.log files in the \Exchsrvr\Mtadata directory, set the following registry parameter. The EV*.log files are a non-localized text copy of the same MSExchangeMTA events that are logged to the Application Event Log. The categories and levels of events written to the log files are controlled, as described in Table 7.4. EV*.log files are easier to read, search, and copy when troubleshooting MTA issues than the corresponding Application Event Log.

Location	HKEY_LOCAL_MACHINE\CurrentControlSet\Services\MSExchangeMTA\Parameters
Value	TextEventLogging
Type	REG_DWORD
Value Data	0x1
Description	Setting this value to 0x1 enables text logging to the EV*.log files.

For more information about the various logs that the Exchange MTA can create, see Microsoft Knowledge Base article 153188, "XCON: Description of MTA Diagnostics Logging Options."

Trace Level Logging

The higher the diagnostics logging level, the more MTA-related events you find in the application event log. This additional information improves your ability to determine the cause of a message flow issue. To obtain the most detailed information about the Exchange MTA, set the diagnostics logging level for the MTA categories to trace level. Trace level logging is not exposed in Exchange System Manager and can only be set using Registry Editor.

To set the diagnostics logging level for MSExchangeMTA categories to trace level, use the following steps:

1. Start Registry Editor and open the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMTA\Diagnostics

2. Double-click each entry for the individual MSExchangeMTA categories and set the values to 0x7. For example, double-click the 1 X.400 Service entry in the right pane, and then change the value to 0x7.

3. Restart the Microsoft Exchange MTA Stacks service. Services typically do not have to be restarted in order for the change to take effect. However, when editing the registry manually, you might have to perform this step.

MTA Check Logging

One underused troubleshooting tool is a current, verbose Mtacheck.log file. This log file shows the results of running Mtacheck.exe. You can run the MTA Check tool manually, but it also runs automatically when the Microsoft Exchange MTA Stacks service determines that the MTA was previously shut down incorrectly. If the MTA Check tool is run automatically, events are logged to the application event log and an Mtacheck.log file is generated in the \Program Files\Exchsrvr\Mtadata\Mtacheck.out directory. You can use the Mtacheck.log file to perform the following tasks:

• Obtain a quick list of all the secured queues and their associated object IDs.

• Quickly identify which queue a message object resides in at MTA startup.

• Associate data and reference objects with each other that are in the work queue at startup.

For more information about the Mtacheck.log file, see Microsoft Knowledge Base article 163323, "XCON: Mtacheck.log."

Object IDs and Message IDs

For every object that the Exchange MTA processes, there is an associated eight-digit object ID. The first two digits of the object ID identify the object class. The last six digits of the ID correspond to a DB<6digit>.dat file, if the object is written to disk. MTA object classes range in hex from 01 to 0E. The two most important classes are 01 (queues) and 06 (messages). For example, the following event refers to a message object with an ID of 0600002D.

Event ID: 272

Source: MSExchangeMTA

Type: Information

Category: X.400 Service

received from entity /DC=COM/DC=CONTOSO/CN=CONFIGURATION/CN=SERVICES/CN=MICROSOFT EXCHANGE/CN=FIRST ORGANIZATION/CN=CONNECTIONS/CN=SMTP (SERVER01-{43D5C017-4A4B-4AFD-85AF-06EAB90057AA}). The entity is a XAPI-Gateway , object is a Normal priority Message, the MTS identifier is C=US;A= ;P=First Organizati;L=SERVER01-040503155933Z-4, and content length is 1719. The number of objects sent so far = 1. External trace information (first 100 bytes) = 3080638061801302555300006280130120000013104669727374204F7267616E697A61746900003180800D3034303530333136303234315A8201008302060000000000. (10)

Each message also has an associated ID. This is referred to as either the message ID or Message Transfer Service (MTS) identifier. Unlike object IDs, which are used only by the local Exchange MTA, the message ID is part of the message itself and can be tracked across MTAs.

A typical message ID generated by an Exchange MTA has the following format: C=<country>;A= ;P=<organization>;L=<server>-<date><greenich mean time>-<message number>. An example is in boldface in event 272, as just shown. There are several variations of the L= value, depending on the message source. Message IDs from outside Exchange might differ, but their purpose is the same. MTS identifiers help to associate message copies with a particular message source. For example, if one message is sent to a distribution group with hundreds of recipients, each generated fan-out copy of the message has the same message ID, even after the messages leave the MTA.