Understanding back pressure

 

Applies to: Exchange Server 2016

Topic Last Modified: 2016-03-28

Learn how back pressure monitors system resources on Exchange 2016 servers to prevent servers from being overwhelmed by the volume of incoming messages.

Back pressure is a system resource monitoring feature of the Microsoft Exchange Transport service that exists on Mailbox servers and Edge Transport servers. Back pressure detects when vital system resources, such as hard drive space and memory, are over-utilized, and takes action to prevent the server from becoming completely overwhelmed and unavailable. For example, when a system resource utilization level on the Exchange server is determined to be too high, the server delays accepting new messages. If the resource utilization gets worse, the server stops accepting new messages to work exclusively on processing all existing messages, and might even stop processing outgoing messages. When the system resource utilization returns to an acceptable level, the Exchange server resumes normal operation by accepting new messages and processing outgoing messages.

Contents

Monitored resources

Actions taken by Exchange Transport when under resource pressure

Back pressure configuration options in the EdgeTransport.exe.config file

Back pressure logging information

The following system resources are monitored by back pressure:

  • DatabaseUsedSpace[%ExchangeInstallPath%TransportRoles\data\Queue]   Hard drive utilization for the drive that holds the message queue database.

  • PrivateBytes   The memory that's used by the EdgeTransport.exe process.

  • QueueLength[SubmissionQueue]   The number of messages in the Submission queue.

  • SystemMemory   The memory that's used by all other processes.

  • UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue]   Hard drive utilization for the drive that holds the message queue database transaction logs.

  • UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data]   Hard drive utilization for the drive that's used for content conversion.

  • UsedVersionBuckets[%ExchangeInstallPath%TransportRoles\data\Queue\mail.que]   The number of uncommitted message queue database transactions that exist in memory.

For each monitored system resource on a Mailbox server or Edge Transport server, the following three levels of resource utilization or pressure are defined:

  • Low or Normal   The resource isn't overused. The server accepts new connections and messages.

  • Medium   The resource is slightly overused. Back pressure is applied to the server in a limited manner. Mail from senders in the organization's authoritative domains can flow. However, depending on the specific resource under pressure, the server uses tarpitting to delay server response or rejects incoming MAIL FROM commands from other sources.

  • High   The resource is severely overused. Full back pressure is applied. All message flow stops, and the server rejects all new incoming MAIL FROM commands.

Transition levels define the low, medium and high resource utilization values depending on whether the resource pressure is increasing or decreasing. Typically, a resource utilization level that's lower than the original level is required as the resource utilization decreases. In other words, there really isn't a static value for low, medium and high resource pressure. You need to know if the utilization is increasing or decreasing before you can determine the next change in resource utilization level.

The following sections explain how Exchange handles the situation when a specific resource is under pressure.

Resource   DatabaseUsedSpace[%ExchangeInstallPath%TransportRoles\data\Queue]

Description   Monitors the percentage of total drive space that's consumed by all files on the drive that holds the message queue database. Note that the message queue database file contains unused space, so an accurate description of the total drive space that's consumed by all files is drive size - free disk space - free space in the database.

To change the default location of the message queue database, see Change the location of the queue database.

Pressure transitions (%):

  • LowToMedium   96

  • MediumToHigh   99

  • HighToMedium   97

  • MediumToLow   94

Comments:

The default high level of hard drive utilization is calculated by using the following formula:

100 * (<hard drive size in MB> - 500 MB) / <hard drive size in MB>

This formula accounts for the fact that there's unused space in the message queue database

1 GB = 1024 MB. The result is rounded down to the nearest integer.

For example, if your message queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-500)/1048576) or 99%.

As you can see from the formula and the rounding down behavior, the hard drive needs to be very small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 25 GB or less.

Return to top

Resource   PrivateBytes

Description   Monitors the percentage of memory that's used by the EdgeTransport.exe process that's part of the Microsoft Exchange Transport service. This doesn't include virtual memory in the paging file, or memory that's used by other processes.

Pressure transitions (%):

  • LowToMedium   72

  • MediumToHigh   75

  • HighToMedium   73

  • MediumToLow   71

Comments:

By default, the high level of memory utilization by the EdgeTransport.exe process is 75 percent of the total physical memory or 1 terabyte, whichever is less. The results are always rounded down to the nearest integer.

Exchange keeps a history of the memory utilization of the EdgeTransport.exe process. If the utilization doesn't go down to low level for a specific number of polling intervals, known as the history depth, Exchange rejects incoming messages until the resource utilization goes back to the low level. By default, the history depth for EdgeTransport.exe memory utilization is 30 polling intervals.

Return to top

Resource   QueueLength[SubmissionQueue]

Description   Monitors the number of messages in the Submission queue. Typically, message enter the Submission queue from Receive connectors. For more information, see Mail flow and the transport pipeline. A large number of messages in the Submission queue indicates the categorizer is having difficulty processing messages.

Pressure transitions:

  • LowToMedium   9999

  • MediumToHigh   15000

  • HighToMedium   10000

  • MediumToLow   2000

Comments:

When the Submission queue is under pressure, the Exchange throttles incoming connections by delaying acknowledgement of incoming messages. Exchange reduces the rate of incoming message flow by tarpitting, which delays the acknowledgment of the SMTP MAIL FROM command to the sending server. If the pressure condition continues, Exchange gradually increases the tarpitting delay. After the Submission queue utilization returns to the low level, Exchange reduces the acknowledgment delay and eases back into normal operation. By default, Exchange delays message acknowledgments for 10 seconds when under Submission queue pressure. If the resource pressure continues, the delay is increased in 5-second increments up to 55 seconds.

Exchange keeps a history of Submission queue utilization. If the Submission queue utilization doesn't go down to the low level for a specific number of polling intervals, known as the history depth, Exchange stops the tarpitting delay and rejects incoming messages until the Submission utilization goes back to the low level. By default, the history depth for the Submission queue is in 300 polling intervals.

Resource   SystemMemory

Description   Monitors the percentage of memory that's used by all processes on the Exchange server. This doesn't include virtual memory in the paging file.

Pressure transitions (%):

  • LowToMedium   88

  • MediumToHigh   94

  • HighToMedium   89

  • MediumToLow   84

Comments:

When the server reaches the high level of memory utilization, message dehydration occurs. Message dehydration removes unnecessary elements of queued messages that are cached in memory. Typically, complete messages are cached in memory for increased performance. Removal of the MIME content from these cached messages reduces the amount of memory that's used at the expense of higher latency, because the messages are now read directly from the message queue database. By default, message dehydration is enabled.

Return to top

Resource   UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue]

Description  Monitors the percentage of total drive space that's consumed by all files on the drive that holds the message queue database transaction logs. To change the default location, see Change the location of the queue database.

Pressure transitions (%):

  • LowToMedium   89

  • MediumToHigh   99

  • HighToMedium   90

  • MediumToLow   80

Comments:

The default high level of hard drive utilization is calculated by using the following formula:

100 * (<hard drive size in MB> - 1152 MB) / <hard drive size in MB>

1 GB = 1024 MB. The result is rounded down to the nearest integer.

For example, if your queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-1152)/1048576) or 99%.

As you can see from the formula and the rounding down behavior, the hard drive needs to be fairly small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 56 GB or less.

The %ExchangeInstallPath%Bin\EdgeTransport.exe.config application configuration file contains the DatabaseCheckPointDepthMax key that has the default value 384MB. This key controls the total allowed size of all uncommitted transaction logs that exist on the hard drive. The value of this key is used in the formula that calculates high utilization. If you customize this value, the formula becomes:

100 * (<hard drive size in MB> - Min(5120 MB, 3*DatabaseCheckPointDepthMax)) / <hard drive size in MB>

noteNote:
The value of the DatabaseCheckPointDepthMax key applies to all transport-related Extensible Storage Engine (ESE) databases that exist on the Exchange server. On Mailbox servers, this includes the message queue database, and the sender reputation database. On Edge Transport servers, this includes the message queue database, the sender reputation database, and the IP filter database that's used by the Connection Filtering agent.

Return to top

Resource   UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data]

Description   Monitors the percentage of total drive space that's consumed by all files on the drive that's used for content conversion. The default location of the folder is %ExchangeInstallPath%TransportRoles\data\Temp and is controlled by the TemporaryStoragePath key in the %ExchangeInstallPath%Bin\EdgeTransport.exe.config application configuration file.

Pressure transitions (%):

  • LowToMedium   89

  • MediumToHigh   99

  • HighToMedium   90

  • MediumToLow   80

Comments:

The default high level of hard drive utilization is calculated by using the following formula:

100 * (<hard drive size in MB> - 500 MB) / <hard drive size in MB>

1 GB = 1024 MB. The result is rounded down to the nearest integer.

For example, if your message queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-500)/1048576) or 99%.

As you can see from the formula and the rounding down behavior, the hard drive needs to be very small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 25 GB or less.

Resource   UsedVersionBuckets[%ExchangeInstallPath%TransportRoles\data\Queue\mail.queue]

Description   Monitors the number of uncommitted transactions for the message queue database that exist in memory.

Pressure transitions:

  • LowToMedium   999

  • MediumToHigh   1500

  • HighToMedium   1000

  • MediumToLow   800

Comments:

A list of changes that are made to the message queue database is kept in memory until those changes can be committed to a transaction log. Then the list is committed to the message queue database itself. These outstanding message queue database transactions that are kept in memory are known as version buckets. The number of version buckets may increase to unacceptably high levels because of an unexpectedly high volume of incoming messages, spam attacks, problems with the message queue database integrity, or hard drive performance.

When version buckets are under pressure, the Exchange server throttles incoming connections by delaying acknowledgment of incoming messages. Exchange reduces the rate of incoming message flow by tarpitting, which delays the acknowledgment of the SMTP MAIL FROM command to the sending server. If the resource pressure condition continues, Exchange gradually increases the tarpitting delay. After the resource utilization returns to normal, Exchange gradually reduces the acknowledgement delay and eases back into normal operation. By default, Exchange delays message acknowledgments for 10 seconds when under resource pressure. If the pressure continues, the delay is increased in 5-second increments up to 55 seconds.

When the version buckets are under high pressure, the Exchange server also stops processing outgoing messages.

Exchange keeps a history of version bucket resource utilization. If the resource utilization doesn't go down to the low level for a specific number of polling intervals, known as the history depth, Exchange stops the tarpitting delay and rejects incoming messages until the resource utilization goes back to the low level. By default, the history depth for version buckets is in 10 polling intervals.

Return to top

The following table summarizes the actions taken by back pressure when a monitored resource is under pressure.

 

Resource under pressure Utilization level Actions taken

DatabaseUsedSpace

Medium

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

  • Message resubmission is paused.

  • Shadow Redundancy rejects messages. For more information about Shadow Redundancy, see Shadow redundancy.

DatabaseUsedSpace

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

PrivateBytes

Medium

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

  • Message resubmission is paused.

  • Shadow Redundancy rejects messages.

  • Processing messages after a server or Transport service restart (also known as boot scanning) is paused.

  • Start message dehydration.

PrivateBytes

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

QueueLength[SubmissionQueue]

Medium

Introduce or increment the tarpitting delay to incoming messages. If normal level isn't reached for the entire Submission queue history depth, take the following actions:

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

  • Message resubmission is paused.

  • Shadow Redundancy rejects messages.

  • Boot scanning is paused.

QueueLength[SubmissionQueue]

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

  • Flush enhanced DNS cache from memory.

  • Start message dehydration.

SystemMemory

Medium

  • Reject message submissions from the Pickup directory and the Replay directory.

  • Start message dehydration.

  • Flush caches.

SystemMemory

High

  • All actions taken at the medium utilization level.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

UsedDiskSpace (message queue database transaction logs)

Medium

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

  • Message resubmission is paused.

  • Shadow Redundancy rejects messages.

UsedDiskSpace (message queue database transaction logs)

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

UsedDiskSpace (content conversion)

Medium

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

UsedDiskSpace (content conversion)

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

UsedVersionBuckets

Medium

Introduce or increment the tarpitting delay to incoming messages. If normal level isn't reached for the entire version bucket history depth, take the following actions:

  • Reject incoming messages from non-Exchange servers.

  • Reject message submissions from the Pickup directory and the Replay directory.

UsedVersionBuckets

High

  • All actions taken at the medium utilization level.

  • Reject incoming messages from other Exchange servers.

  • Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers.

  • Stop processing outgoing messages.

  • Remote delivery is paused.

Return to top

You can use the Get-ExchangeDiagnosticInfo cmdlet in the Exchange Management Shell to view the resources that are being monitored, and the current utilization levels. To learn how to open the Exchange Management Shell in your on-premises Exchange organization, see Open the Exchange Management Shell.

To view the back pressure settings on an Exchange server, run the following command:

[xml]$bp=Get-ExchangeDiagnosticInfo [-Server <ServerIdentity>] -Process EdgeTransport -Component ResourceThrottling; $bp.Diagnostics.Components.ResourceThrottling.ResourceTracker.ResourceMeter

To see the values on the local server, you can omit the Server parameter.

All configuration options for back pressure are done in the %ExchangeInstallPath%Bin\EdgeTransport.exe.config XML application configuration file. However, few of the settings exist in the file by default.

CautionCaution:
These settings are listed only as a reference for the default values. We strongly discourage any modifications to the back pressure settings in the EdgeTransport.exe.config file. Modifications to these settings might result in poor performance or data loss. We recommend that you investigate and correct the root cause of any back pressure events that you may encounter.

General back pressure settings

Key name Default value

ResourceMeteringInterval

00:00:02 (2 seconds)

DehydrateMessagesUnderMemoryPressure

true

DatabaseUsedSpace settings

Key name Default value (%)

DatabaseUsedSpace.LowToMedium

96

DatabaseUsedSpace.MediumToHigh

99

DatabaseUsedSpace.HighToMedium

97

DatabaseUsedSpace.MediumToLow

94

PrivateBytes settings

Key name Default value (%)

PrivateBytes.LowToMedium

72

PrivateBytes.MediumToHigh

75

PrivateBytes.HighToMedium

73

PrivateBytes.MediumToLow

71

PrivateBytesHistoryDepth

30

QueueLength[SubmissionQueue] settings

Key name Default value

QueueLength[SubmissionQueue].LowToMedium

9999

QueueLength[SubmissionQueue].MediumToHigh

15000

QueueLength[SubmissionQueue].HighToMedium

10000

QueueLength[SubmissionQueue].MediumToLow

2000

SubmissionQueueHistoryDepth

300 (after 10 minutes)

SystemMemory settings

Key name Default value (%)

SystemMemory.LowToMedium

88

SystemMemory.MediumToHigh

94

SystemMemory.HighToMedium

89

SystemMemory.MediumToLow

84

UsedDiskSpace settings (message queue database transaction logs)

Key name Default value (%)

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].LowToMedium

89

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].MediumToHigh

99

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].HighToMedium

90

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].MediumToLow

80

noteNote:
Values that contain only UsedDiskSpace (for example, UsedDiskSpace.MediumToHigh) apply to the message queue database transaction logs and to content conversion.

UsedDiskSpace settings (content conversion)

Key name Default value (%)

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].LowToMedium

89

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].MediumToHigh

99

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].HighToMedium

90

UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].MediumToLow

80

TemporaryStoragePath

%ExchangeInstallPath%TransportRoles\data\Temp

UsedVersionBuckets settings

Key name Default value

UsedVersionBuckets.LowToMedium

999

UsedVersionBuckets.MediumToHigh

1500

UsedVersionBuckets.HighToMedium

1000

UsedVersionBuckets.MediumToLow

800

VersionBucketsHistoryDepth

10

Return to top

The following list describes the event log entries that are generated by specific back pressure events in Exchange:

  • Event log entry for an increase in any resource utilization level

    Event Type: Error

    Event Source: MSExchangeTransport

    Event Category: Resource Manager

    Event ID: 15004

    Description: Resource pressure increased from <Previous Utilization Level> to <Current Utilization Level>.

  • Event log entry for a decrease in any resource utilization level

    Event Type: Information

    Event Source: MSExchangeTransport

    Event Category: Resource Manager

    Event ID: 15005

    Description: Resource pressure decreased from <Previous Utilization Level> to <Current Utilization Level>.

  • Event log entry for critically low available disk space

    Event Type: Error

    Event Source: MSExchangeTransport

    Event Category: Resource Manager

    Event ID: 15006

    Description: The Microsoft Exchange Transport service is rejecting messages because available disk space is below the configured threshold. Administrative action may be required to free disk space for the service to continue operations.

  • Event log entry for critically low available memory

    Event Type: Error

    Event Source: MSExchangeTransport

    Event Category: Resource Manager

    Event ID: 15007

    Description: The Microsoft Exchange Transport service is rejecting message submissions because the service continues to consume more memory than the configured threshold. This may require that this service be restarted to continue normal operation.

Return to top

 
Show: