BizTalk Server 2004: A Messaging Engine Overview
M3 Technology Partners Inc.
Applies to: Microsoft BizTalk Server 2004
Summary: This paper describes the architecture and internal workings of BizTalk Server 2004 as a messaging engine. (30 printed pages)
Microsoft BizTalk Server 2004 provides a scalable, manageable platform for building message-based and business process-oriented applications. With this latest version, BizTalk Server provides many improvements in terms of architecture, design, performance, manageability, and scalability. In order to fully take advantage of the platform, many architects and developers need more detailed knowledge about how the platform works. This paper is intended to provide detailed information about the messaging infrastructure of BizTalk Server and about other key areas of the product where customers have typically struggled to gain an understanding.
The goal in providing this information is that it will help customers to design, create, and debug their applications to better take advantage of the BizTalk Server 2004 platform.
Before you review more detailed information about the various messaging components in BizTalk Server, it is important to have an understanding of how the components fit into the overall architecture of the product. BizTalk Server is built on a publish/subscribe architecture in which a message is published into the system, and then received by one or more active subscribers. Different flavors of this architecture exist, but the model implemented in BizTalk Server is often called content-based publish/subscribe.
In a content-based publish/subscribe model, subscribers specify the messages they want to receive using a set of criteria about the message. The message is evaluated at the time it is published, and all of the active subscribers with matching subscriptions (indicated by filter expressions), receive the message. In the case of BizTalk Server, content-based is a bit of a misnomer, however, because the criteria used to build subscriptions do not have to come from the message content and may include contextual information about the message as well. The details of the subscription mechanism are discussed in the topic "Subscriptions in BizTalk Server" in this document.
The following figure provides a high-level overview of the BizTalk Server architecture from a messaging perspective.
Figure 1 BizTalk Server 2004 architecture
In this simplified view, a message is received through a receive location defined in a given receive port. This message is processed by the receive location and then published to the MessageBox database, the main persistence and routing mechanism for BizTalk Server. The MessageBox evaluates active subscriptions and routes the message to those orchestrations and send ports with matching subscriptions. The sections that follow describe each of these components.
Receive Ports and Receive Locations
A receive port is a collection of one or more receive locations that define specific entry points into BizTalk Server. A receive location is a combination of a receive adapter and a receive pipeline. The adapter is responsible for the transport and communications part of receiving a message. Examples include the File adapter and SOAP adapter, each of which receives messages from different types of sources. The receive pipeline is responsible for preparing the message for publishing into the MessageBox. A pipeline is a series of components that are executed in sequence, each providing specific processing to a message such as decryption/encryption, parsing, or validation.
Send Ports and Send Port Groups
A send port group is a collection of send ports and works much like an e-mail distribution list. A message sent to a send port group will be sent to all send ports in that group. A send port is the combination of a send pipeline and a send adapter. The send pipeline is responsible for preparing a message coming from BizTalk Server for transmission to another service. The send adapter is responsible for actually sending the message using a specific protocol such as SOAP, or FTP.
The heart of the publish/subscribe engine in BizTalk Server is the MessageBox database. The MessageBox is made up of two components: one or more Microsoft® SQL Server™ databases and the Messaging Agent. The SQL Server database provides the persistence store for many things including messages, message parts, message properties, subscriptions, orchestration state, tracking data, host queues for routing, and others.
In addition, using stored procedures, the database provides some of the business logic related to routing messages and fulfilling subscriptions. The Message Agent, however, is the component that encapsulates and abstracts the database component and is the interface used by all parts of BizTalk Server to interact with the messaging subsystem. The Message Agent is a Component Object Model (COM) component that provides interfaces for publishing messages, subscribing to messages, retrieving messages, and so on. This interface is the only mechanism used by other BizTalk Server components, including the adapter framework and orchestrations, to interact with the MessageBox.
The Endpoint Manager, not shown in the figure, is the service responsible for managing send and receive ports and acts as the intermediary between the ports and the MessageBox.
Hosts and Host Instances
A host is a logical representation of an application that will host services that provide specific functionality in BizTalk Server. A host can be either an in-process host, which means it is owned and managed by BizTalk Server, or an isolated host, which means that the BizTalk Server code is running in a process that is not controlled by BizTalk Server. A good example of an isolated host is Internet Information Services (IIS), which hosts the receive functionality of the HTTP and SOAP adapters. Hosts are defined for an entire BizTalk Server group; a collection of BizTalk Servers that share configuration, MessageBoxes, ports, and so on.
Each host can have either zero or one host instance running on each server in the BizTalk Server group. For the in-process host, each host instance is a Windows NT Service, BTSNtSvc.exe. This service is installed each time a new host instance is added through the BizTalk Server Administration console and is configured with startup parameters that specify the name of the host which this instance represents. For isolated hosts, each instance represents a running instance of the process that contains BizTalk Server-related processing code, for example an application pool in IIS.
Within each of these host instances, several subservices are running. The exact services can be found in the adm_HostInstance_SubServices table in the BizTalk Server Management database. The services are listed here for reference:
Table 1 Host instance subservices
Used internally by BizTalk Server to cache configuration information for the other services
Responsible for hosting receive and send ports including adapters and pipelines
Responsible for moving data out of the MessageBox database and into the BAM or Tracking databases as appropriate
Host engine for BizTalk Server orchestrations
MSMQT adapter service; serves as a replacement for the MSMQ protocol when interacting with BizTalk Server
The Windows NT Service simply acts as a container to host these other services. When the service is started, each of these subservices is started as well. They handle all of the processing related to the messaging and orchestration engines in BizTalk Server.
The two services that are a bit different are the Tracking and MSMQT services. With the Tracking service, you can specify at the host level whether the host instances for that host participate in the tracking process. This means that some services may not load and start the Tracking service unless the host has been configured to do so. This allows you to move the tracking of data into a particular host, and keep the processing hosts from getting bogged down with tracking data. The MSMQT service, while essentially a transport adapter, is responsible for implementing the MSMQT infrastructure on which the adapter relies. This service is the BizTalk Server version of the classic Microsoft Message Queuing (MSMQ) service.
For isolated hosts, the Endpoint Manager is the only service that is loaded into the process. These types of hosts are only intended to be used for hosting adapters that send and/or receive messages.
BizTalk Server 2004 is, at its core, a message-handling engine. To understand the details of BizTalk Server 2004, it is important to understand messages and how they are represented, stored, and processed by BizTalk Server. The details of how BizTalk Server works with messages become easier to understand after you have gained an understanding of what a message is.
Each message in BizTalk Server is considered a multi-part message and is made up of zero or more parts. Each part consists of a binary chunk of data which can represent an XML document, flat file, a serialized .NET class, or other binary stream of data. Each message with one or more parts has one of these parts identified as the body part. You use the body part of the message to identify the type of the message that can be used for routing.
A very important concept to understand is that all messages are immutable in BizTalk Server 2004. This means that after a message is constructed, it cannot be changed. Any changes to the message require a new message to be created and used from that point forward. This is especially clear in the Orchestration Designer, where compilation rules force you to follow strict guidelines about constructing a message before using it, and not allowing the message to be altered outside of its construction block. If you need to change the message, you must create a new construction block that creates a message of the same type, copy the original message to the new message, and then make any changes to the new message before leaving the construction block.
BizTalk Server follows these same rules in its processing of messages. When a message is processed in a send port, a new message instance is created by pipeline components when the message is to be changed in the processing. This is a practice that should also be followed by any pipeline components that are created by developers working with BizTalk Server.
The reason that messages must be immutable is that for a given message, there may be many subscribers that receive the message. Physically, there is only one message, but many references to that message exist. Allowing the message to be changed by one subscriber would impact all of the other subscribers. For example, a message is received by BizTalk Server and routed to two different orchestrations. If the first orchestration were allowed to change the message, the impact of this change on the second orchestration is unknown and would almost surely be unwanted.
In addition to the parts that make up a message, each message in the system has a set of properties that go along with it in what is known as the message context. These properties can be values extracted from the message itself or values related to the message or the processing of the message. For example, adapters put properties into the context related to the receiving of messages such as the location at which the message was received, and what type of adapter was used to receive the message. Properties can either be written to the context, or promoted into the context. The difference between these two options is that promoted properties can be used as criteria in message routing while written properties cannot.
This concept of writing or promoting values to the context is related to, but not the same as, promoting properties in the BizTalk Editor. In the BizTalk Editor, an element or attribute in a schema can be flagged as a promoted property or a distinguished field. Items that are marked as promoted properties are meant to be promoted into the context, while those marked as distinguished fields are meant to be written into the context.
The design for promoted properties started with the design of message correlation: the ability to relate a message being received to an already running orchestration instance. For correlation, there is a need to define a property or set of properties that provide the link between messages in the orchestration. For example, in a purchasing process, there exists a need to correlate messages based on the PurchaseOrderID. However, in many business cases, the name of the particular field or attribute in the messages may not match. A purchase order schema might have an element named POId, while the companion invoice schema may have an element named OrderID. To correlate messages on named property such as PurchaseOrderID in this situation, the developer must be able to abstract the name of the property to be correlated on from the source of the value. Property schemas allow this abstraction.
A property schema enables you to define promoted properties in a common location and have them referenced by other schemas. Like other schemas, a property schema has a namespace, but unlike other schemas, it can only have defined elements (that is, not records or attributes). Each element that is defined in the property schema has a name and type. Because the property schema may need to be referenced by more than one schema, and because the information in the property schema must be available to components at runtime, the property schema must get deployed to BizTalk Server like all other schemas. In addition to the normal schema deployment steps, information about the promoted properties is extracted and stored in the bts_documentSpec table in the Management database.
After a property schema has been created, elements and attributes with the same type (for example. integer) can be promoted as one of the named properties in the property schema.
To complete the example case from above, a developer would perform the following steps to define the shared property needed for correlation.
Create a property schema and define an element of type xs:int named PurchaseOrderId.
Create a PurchaseOrder schema and add an element or attribute of type xs:int named POId.
Using the show promotions command in the BizTalk editor, the developer adds the POId field to the list of promoted properties and indicates that is should be promoted as the PurchaseOrderId property defined in the property schema by selecting the PurchaseOrderId named property from the list.
Create an Invoice schema and add an element or attribute of type xs:int named OrderId.
Using the show promotions command in the BizTalk Editor, the developer adds the OrderId field to the list of promoted properties and indicates that is should be promoted as the PurchaseOrderId property defined in the property schema by selecting the PurchaseOrderId named property from the list.
Now that this definition exists in the document schemas, pipeline components can properly promote OrderId and POId as the named property PurchaseOrderID so that it can be used for routing and correlation. For more details on this promotion process, see the topic "Message Processing" in this document.
One of the benefits of promoted properties is that the value of the element that is promoted is available in the context of the message. This means that retrieving that value is inexpensive, as it does not require loading the message into memory to execute an XPath statement on the message. Instead, a simple property bag can be used along with a key to get the value. This type of behavior is desirable in situations other than message routing and is the reason for creating distinguished fields. While promoted properties are promoted into the message context, distinguished fields are written into the message context. Unlike promoted properties however, there is no property schema for distinguished fields. This is why distinguished fields cannot be used for routing and are therefore not available as filter criteria in a send port or orchestration receive shape. Distinguished fields can, however, be used in orchestrations to read or write values from the message context instead of having to load the message into memory and extract the value.
In addition to promoting or writing properties into the message context, message predicates can also be added to the context. Message predicates are used as a security measure in BizTalk Server and provide contextual information about the message, which must match values specified for any host that the message is to be routed to. This security measure allows you to configure your BizTalk Server environment in such a way as to allow specific hosts to be the only hosts that can receive and process particular messages. As an example, in the BizTalk Server Administration console, a host can be configured with the thumbprint of a certificate to use for message decoding and decryption. Configuring this property creates an application property for that host in the MessageBox. Secure messages that are received and decrypted using this thumbprint are then only routed to the hosts with the thumbprint configured.
Pipelines are a component of BizTalk Server that provides an implementation of the Pipes and Filters integration pattern. For more information on Enterprise Integration Patterns, see the MSDN article Integration Patterns. During the receiving and sending of messages, there are business reasons to perform transformations on messages to prepare them to enter or leave BizTalk Server. A common example is that you may need to transform a comma-delimited flat file into an XML file in order to take advantage of certain features in BizTalk Server such as maps; the flat file disassembler component does just that. It is common in integration scenarios to have a need to perform several types of transformations to a message before receiving or sending it; that is where pipelines come into play. Pipelines enable the developer to define a series of transformations that will be performed on a message as it is being received or sent.
There are two types of pipelines, send and receive, and these match the ports in which they execute. Send pipelines are executed in send ports and in the response portion of a request/response receive port while receive pipelines are executed in receive locations, and in the response portion of a solicit/response send port. Essentially, receive pipelines are intended to be used to transform messages that are being published to the MessageBox while send pipelines are intended to be used on messages which have been subscribed to and are being sent out of BizTalk Server.
Each pipeline has a set of stages that are executed in order when the pipeline is executed. Each stage can contain zero or more components. The maximum number of components depends on the stage. A receive pipeline, shown in the figure below, has the following stages:
Table 2 Receive pipeline stages
Decrypts or decodes the message data
Disassembles an interchange into smaller messages and parses message contents
Validates the message data, generally against a schema
Identifies the BizTalk Server party associated with some security token in the message or message context
Figure 2 The receive pipeline
A send pipeline, shown below, has three stages for processing components:
Table 3 Send pipeline stages
Performs any message processing necessary before assembling the message
Assembles the message and prepares it to be transmitted by taking steps such as adding envelopes, converting XML to flat files, or other tasks complementary to the disassemble stage in a receive pipeline
Encodes or encrypts the message before delivery
Figure 3 The send pipeline
A stage in a pipeline has an execution mode of either All or First Match, which controls the components that get executed if more than one component is added to a stage. For stages with a mode of All, each component is called to process the message in the order inn which they are configured in the stage. When the mode is First Match, each component is polled to indicate that it is the right component until a match is found, at which point the component that matches is executed, while the remaining components do not get executed.
As an example of execution modes, the Disassemble stage of a receive pipeline is a first match stage, thus each component in the stage is called to see if it recognizes the message and can process it. If the component responds in the affirmative, then no other components in that stage will be queried to see if they can also handle the message. However, the Decode stage of a receive pipeline has an execution mode of All, meaning that each component in this stage is called to process the message in the order in which they were configured. The first decoder might be to decrypt the message, while the second might be to decompress the message from a zipped format.
One common consequence of execution mode in pipeline processing occurs when a developer wants to use multiple disassemblers in a single receive pipeline. Often the disassembling components differ only slightly, for example two flat file disassemblers with similar but different schemas configured. In this case, while the message might actually match the schema defined in the second disassembler configured, the first disassembler might determine through its probing that it can process the message. It is only after processing the message that the error is discovered and the message suspended. In these cases, the choice is to either create a new disassembler which has more specific probing logic in it, or to create two different pipelines and receive the different messages in different receive locations.
All of the components described so far play a part in the processing of messages as they flow through BizTalk Server. This section provides more detail about how these components interact to make the messaging infrastructure work, beginning with receiving a message. The figure below shows the make-up of a receive port and the flow of a message through the receive process.
A receive port consists of one or more receive locations and zero or more maps. Maps are XSLT-style sheets used to transform messages from one structure or format to another and are often used in the receive process to normalize messages into an internal format. A message is received into a receive port by a single receive location. The receive location consists of the receive adapter and a receive pipeline.
Figure 4 A receive port and message flow
The receive adapter initiates the process of receiving messages by reading a stream of data and creating a message. For example, the file adapter sees that a file has been placed in its configured location and reads that file in a stream. The adapter creates a message (an implementation of the Microsoft.BizTalk.Message.Interop.IBaseMessage interface), adds a part to it (an implementation of the Microsoft.BizTalk.Message.Interop.IBasePart interface), and provides the stream of data as the part content.
In addition, the adapter writes and promotes properties into the message context related to the location, adapter type, and other adapter specific properties. After the message and its context have been created, the adapter passes the message to the Transport Proxy, which is also managed by the Endpoint Manager. The message is then processed through the receive pipeline, which has been configured for the receive location. After the message has been processed by the pipeline, a map may be used to transform the message into the format desired before the Endpoint Manager passes the message to the Message Agent for publishing.
While it is the responsibility of the adapter to create the initial message, most of the processing that occurs on a received message happens in the receive pipeline. Pipeline processing deals with message content as well as message context. Message content is generally handled in decoding, disassembling and validating stages, while message context can be handled in all stages. A pipeline, however, does not have to impact either the content or the context; for example, the default pass-through pipeline has no components configured and performs no processing on the message content or context. For simplicity, this document focuses on the disassembling components as they generally have the greatest impact on message routing.
The job of the disassembler is to process an incoming message from an adapter and to disassemble it into many messages, and parse the message data. When an incoming message has many smaller messages, this is known as an interchange. Both the flat file disassembler and the XML disassembler handle interchanges by enabling a developer to configure information about the wrapping content (a header and trailing schema for the flat file disassembler and an envelope schema for the XML disassembler) and the, potentially repeating, body content. In addition, both of these disassemblers parse the original message into XML content. A custom disassembler need not necessarily parse the content into XML if further XML processing in BizTalk Server is not required. An example scenario might include a simple routing situation in which messages entering the system at a particular receive location are sent to a specific send port with no mapping or other XML-based processing.
One of the most common message properties used in routing is the message type. When a developer creates a schema to define the structure of messages, this schema defines the message type for that message. The type is determined by the root node and namespace in the schema definition. For example, an XML document that looks like the following would have a message type of http://tempuri.org/samples/MessageType#Message
<Message xmlns=http://tempuri.org/samples/MessageType> <SomeOtherElement type="sample"/> </Message>
Notice that the message type consists of the namespace, followed by the pound (#) symbol, and the name of the root node. To use message type in routing, it must be promoted into the context. Disassemblers are generally responsible for promoting this value into the message context as well as the pipeline components with the most specific knowledge of message structure. The XML and Flat File disassemblers promote the message type as they are processing messages, and any custom disassembler should also promote this property to ensure proper routing.
It is important to note that a message is not required to have a type. As mentioned previously, the parts of a message can be any binary data and need not have a schema that defines their structure. This type of message part is generally passed through BizTalk Server without much, if any, processing done on it by BizTalk Server itself, though custom pipeline components, adapters, or code called from orchestrations may interact with these parts.
Pipeline components, like adapters, also write and promote properties into the message context. In fact, pipeline components are the most common mechanism most developers use to get properties into the message context. As discussed in the topic "Messages and Message Properties," developers create schemas and can promote properties in the schema or mark them as distinguished fields. This information is stored in the schema as annotations which can then be used by pipeline components. All of the built-in disassembler and assembler components - FlatFile, XML, and BizTalk Framework - use the document schema to retrieve information about the properties that are to be promoted. Using the XPath statement from the annotations, the disassembler knows the location in the document of elements to be promoted. During the process of streaming through the document, the disassembler finds those elements that match one of the XPath statements and promotes or writes the value into the context as appropriate.
Custom pipeline components can also be written to handle getting properties into the context for arbitrary data in a received or sent message. In order to promote a property into the context and have it be useful for routing, which is presumably why the value is being promoted, a property schema with a definition for the property should be created and deployed to BizTalk Server. Before defining a property schema to be used by custom components, you should understand the different types of promoted properties. Promoted properties defined in a property schema can have one of two base types: MessageContextPropertyBase or MessageDataPropertyBase.
A property with a base type of MessageDataPropertyBase indicates that the value for this property comes from the content of the message. This is the default value for properties defined in a property schema and is the most common usage. MessageContextPropertyBase indicates a property that is intended to be part of the message context but does not necessarily come from the message data directly. Properties with MessageContextPropertyBase as their base type are often promoted by adapters and disassemblers and include common properties such as message type and adapter type.
It is important to understand the different types and use them appropriately when defining properties. One of the most significant implications occurs when acessing context properties for a message in an orchestration. If a property is identified as a MessageDataPropertyBase, the orchestration designer examines the schema of the message being received and ensures that it defines a matching promoted property. If no property is found in the schema tied to the promoted property being accessed, then the designer does not allow you to access it. On the other hand, if the property is defined as being a MessageContextPropertyBase, the message type does not matter and the property can be accessed.
In custom pipelines, the mechanism for promoting or writing properties to the context is very similar. For writing properties, you use a call to the IBaseMessageContext.Write method to place the value in the context. For promoted properties, you simply use the IBaseMessageContext.Promote method instead. Each of these methods takes a property name, namespace and value. For the promoted properties, the name and namespace are those of the property defined in the property schema and are most easily accessed by referencing the property schema assembly and using the properties on the class created for the property. Distinguished fields use a common namespace, http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields, and the XPath expression used to retrieve the value is usually used as the name.
The code below shows an example of both promoting and writing properties into the context. Note that in this example, a distinguished field is being written into the context. This is only useful for orchestrations in which the message schema identifies the distinguished field so the orchestration designer knows about the field. It may be useful to write properties into the context for use by other pipeline components on the receiving or sending side.
//create an instance of the property to be promoted SOAP.MethodName methodName = new SOAP.MethodName(); //call the promote method on the context using the property class for name //and namespace pInMsg.Context.Promote(methodName.Name.Name, methodName.Name.Namespace, "theSOAPMethodName"); //write a distinguished field to the context pInMsg.Context.Write("theDistinguishedProperty", "http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields", "theDistinguishedValue");
There are several facts to keep in mind when writing or promoting values into the context. First, writing a value into the context with the same name and namespace that were used previously to promote the property causes that property to no longer be promoted. The write essentially overwrites the promotion. Next, writing a value of null into the context deletes the value, because null-valued properties are not permitted. Finally, promoted properties are limited to 256 characters in length while written properties have no length limitation. Promoted properties are used in message routing and are limited in size for reasons of efficiency in comparison and storage. While written properties have no hard limits on size, using excessively large values in the context will have an impact on performance, because those values must still be processed and passed with the message.
When a message is ready to be sent from BizTalk Server, it undergoes a complementary process in the send port. Maps are applied to messages before the send pipeline is executed, allowing a message to be transformed to a customer or application-specific format before being processed by the pipeline and sent through the adapter. In the send pipeline, instead of promoting properties into the message context, properties are demoted from the context into the message. This only occurs for MessageDataPropertyBase properties because they are the only type of promoted properties that affect message data.
Figure 5 The send port and message flow
In a publish/subscribe design, you have three components: publishers, subscribers, and events. Events in BizTalk Server simply refer to your messages. Publishers include receive ports, orchestrations, and send ports. Publishers include receive ports that publish messages that arrive in their receive locations, orchestrations that publish messages when sending messages or starting another orchestration asynchronously, and solicit-response send ports that publish messages when they receive a response from the target application or transport.
In BizTalk Server, there are two main types of subscriptions: activation and instance. An activation subscription is one specifying that a message that fulfills the subscription should activate, or create, a new instance of the subscriber when it is received. Examples of things that create activation subscriptions include send ports with filters or send ports that are bound to orchestrations, and orchestration receive shapes that have their "Activate" property set to true. An instance subscription indicates that messages that fulfill the subscription should be routed to an already-running instance of the subscriber. Examples of things that create instance subscriptions are orchestrations with correlated receives and request/response-style receive ports waiting for a response from BizTalk Server.
The difference between the two types of subscription at the information level is that an instance subscription includes the unique instance ID, stored in the subscription table in the master MessageBox. When an orchestration instance or receive port completes processing, instance subscriptions are removed from the MessageBox while activation subscriptions remain active as long as the orchestration or send port is enlisted. For more information about how orchestrations interact with the Messaging system, see the topic "Orchestrations and Messaging" in this document.
Subscriptions are created by service classes in BizTalk Server, which are listed in the adm_ServiceClass table in the BizTalk Management database. These services were identified earlier in the document in the section "Hosts and Host Instances" and include the caching service; in-process and isolated messaging, hosted by the Endpoint Manager; orchestrations/XLANG hosted by the XLANG subservice; and MSMQT. Each of these service classes can create subscriptions and receive published messages.
A subscription is a collection of comparison statements, known as predicates, involving message context properties and the values specific to the subscription. For example, Message Type is a context property of messages and many subscriptions specify the message type in their subscription. General information about the subscription is inserted into the subscriptions table by the Message Agent while the specific predicates go into one of the following predicate tables, depending on the type of operation specified for the subscription:
All of this is accomplished by calling the Bts_CreateSubscription_<application name> and Bts_InsertPredicate_<application name> stored procedures in the MessageBox where <application name> is the name of the BizTalk host that is creating the subscription.
When a send port is enlisted, the port creates, at a minimum, a subscription for any message with that send port's transport ID in the context. This allows the send port to always receive messages intended specifically for it. When an orchestration port is bound to a particular send port, the information about that binding is stored in the BizTalk Configuration database. When messages are sent from the orchestration through the port bound to the physical send port, the Transport ID is included in the context so that the message gets routed to that send port. However, it is important to note that this send port is not the only send port that can receive messages sent from the orchestration. When an orchestration sends a message, that message is published to the MessageBox with all pf the relevant promoted properties. The bound send port is guaranteed to receive a copy of the message because the transport ID is in the context, but any other send port, or orchestration, can have a subscription that also matches the message properties. It is very important to understand that any time a message is published directly to the MessageBox, all subscribers with matching subscriptions will receive a copy of the message.
Publishing and Routing
After a subscription is created and enabled, a message must be published before any processing takes place. Messages are published when they are received into BizTalk Server from one of the services mentioned previously. For this discussion of routing, we will focus on messages received into BizTalk Server via an adapter.
When messages go through the receive pipeline processing, properties are promoted into the message context. After a message is ready to be published into the MessageBox after being processed by the receive adapter and pipeline, the first thing that happens is the Message Agent inserts the property values for the promoted properties and predicate values from the message context into the master MessageBox SQL Server database. Having these values in the database enables the Message Agent to make routing decisions.
The next step is for the Message Agent to ask the master MessageBox database to find subscriptions for the current batch of messages being published. Keep in mind that the messages have not yet been written to the database, only the properties from the context. The bts_FindSubscriptions stored procedure in the MessageBox queries the subscription and predicate tables identified above, linking them to the message properties stored for the current batch of messages.
With this information, the Message Agent inserts the message once into each MessageBox database that has a subscription by calling the bts_InsertMessage stored procedure. The bts_InsertMessage stored procedure is first called with a subscription ID. On this first pass, the stored procedure calls the int_EvaluateSubscriptions stored procedure which is responsible for looking up the subscription detail information, validating that the message meets security requirements for the application by checking that message predicates match application properties for the host, and inserting a reference to the message in the application specific queue or application specific suspended queue depending on the state. The message ID, subscription ID, service ID, and other subscription information are inserted into the application specific queue table for each subscription that was found for this application. After the messages are inserted, the message properties and message predicates tables are cleared of the batch related values.
The bts_InsertMessage stored procedure is called subsequently for each part in the message. On the first call, the message context is passed and is then inserted into the SPOOL table along with metadata about the message such as the number of parts, the body part name and ID. In addition the message body part is inserted into the PARTS table using the int_InsertPart stored procedure. The bts_InsertMessage stored procedure is then called for each of the remaining message parts where they are simply passed to the int_InsertPart stored procedure to be persisted in the PARTS table.
When messages are routed, references are added for each service that receives the specific instance of the message and its parts by inserting records into the following tables:
These references keep the messages and parts from being deleted by cleanup jobs that run periodically to keep the MessageBox from getting full with message data for messages that are no longer in the system. Two tables are used to reduce contention and locking issues.
Now that the message has been routed to the right queue, stored in the Spool and Parts tables, and referenced in the application specific queues, the messages must be pulled off the queues by the application instances. Each host instance has a number of dequeuing threads that continuously poll the database on an interval that is configured in the adm_ServiceClass table in the BizTalk Management database. This same table has a column to indicate the number of dequeuing threads to be used. For information on tuning these parameters, see BizTalk Server 2004 Performance Characteristics. Each thread calls into the MessageBox database and calls the bts_DequeueMessages_<application name> stored procedure appropriate for the host application it is running in. This stored procedure uses locking semantics to make sure that only one instance and one dequeuing thread are able to operate on a message in the queue at a given time. The host instance that gets the lock gets the message, and is then responsible for handing the message to the subservice for which it is intended.
If the service receiving the message is the Endpoint Manager, then the send port is invoked (or the response portion of a request/response receive port) and if it is the XLANG/s subservice, an orchestration is either created, or located to service the subscription depending on whether there is an instance ID in the subscription. The service then releases the reference to the message and its part so that if no other services have references, the message data can be deleted.
Orchestrations can subscribe to (receive) and publish (send) messages through the MessageBox. In addition, orchestrations can construct new messages. Receiving messages occurs via the subscription and routing infrastructure already discussed. When subscriptions are filled for orchestrations, a new instance is activated and the message is delivered, or in the case of instance subscriptions, the instance is rehydrated if necessary and the message is then delivered. When messages are sent from an orchestration, they are published to the MessageBox in the same manner as a message arriving on a receive location with the appropriate properties getting inserted into the database for use in routing.
Messages that are constructed in an orchestration must be persisted in the MessageBox and referenced by the orchestration instance, but they should not be published because they have not yet been sent. The XLANG/s subservice makes calls to the Message Agent API to insert messages directly. This allows the engine to insert the message body into the MessageBox and have it directly associated with the running orchestration instance. The persistence of the constructed message in the MessageBox is coordinated with persistence points in the orchestration as an additional optimization of database operations.
The concept in orchestrations that makes the publish and subscribe seem to act differently is binding. Orchestration ports are logical ports that describe an interaction. These logical ports need to be bound to some physical port in order for messages to actually get delivered, but this binding process is nothing more than configuring subscriptions for message routing. There are four basic options for binding these ports:
Specify Now (specifying them directly in the orchestration)
Specify Later (specifying them at deployment time)
Using a dynamic send port where the address is set in the orchestration code
Creating a direct binding from the orchestration to either the MessageBox or another orchestration
When a binding is specified at design time, a physical port that matches the parameters configured in the orchestration gets created when the orchestration is deployed. When the binding is configured at deployment time, any port that matches the requirements of the logical port can be bound to the orchestration port. For dynamic binding, a physical port is created just as with the Specify Now option, but the port is a dynamic send port that has no address information configured.
A confusing concept for many developers is that while a send port in an orchestration is bound to a physical send port, this does not preclude that message from getting delivered to other subscribers. That is, if another send port happens to have a subscription, via its filters, for the message being sent to the bound port, both send ports will receive the message. Binding simply creates the subscription such that the message sent from the orchestration always matches the criteria for the bound send port. Likewise, the orchestration port bound to a receive port creates the appropriate subscription based on message type and receive port ID. The subscriptions guarantee that the messages going in and out of the orchestration get delivered to the bound ports, but the messages still go through the same publish and subscribe mechanism described earlier.
Probably the most misunderstood, and misused or underused binding option is the Direct binding option. Direct binding allows an orchestration to publish messages to the MessageBox with varying routing properties much like messages are published by receive locations. In simple direct messaging, the message is published to the MessageBox with its promoted properties to be routed like any other published message received into BizTalk. This allows any subscriber(s) to receive this message, but requires that at least one subscriber exist or the orchestration will get an exception thrown back to it because a routing failure has occurred. For more information about routing failures, see the topic "Routing Failures" in this document.
Another option for direct binding is to use self-correlating ports. Self-correlating ports are ports that create a unique correlation token and use that token alone in correlating messages between instances. The most common use of a self-correlating port is to call or start an orchestration passing in a port parameter. In the called orchestration, the port can be used to send a message, while in the calling orchestration the same port can be used to receive a message. Because the port has a unique correlation token, the message is routed back to the calling orchestration. Self-correlation ports act as private communication channels between orchestration instances.
The final option is to use a partner orchestration in which, in both the calling orchestration and the called orchestration, the port is configured using the same shared port type and in the port configuration, the same port is selected. For example, in both Orch1 and Orch2, Orch2.MyDirectPort is selected. This type of binding sets up a subscription for the receiving orchestration based on the sending orchestration type, the port name, and the operation name. This again ensures that the messages get routed to the correct instance.
One thing to keep in mind with these direct messaging options, as well as using the Start orchestration shape to asynchronously start an orchestration, is that they all use the underlying publish and subscribe model. The difference between these options is in the properties that are used for creating subscriptions and routing, and in the use cases they help solve.
One common problem encountered when using direct bound ports in orchestrations is that an orchestration may publish a message that it is also subscribed to. For example, an orchestration is configured to be activated by a PurchaseOrder message. This orchestration uses a direct port to publish the PurchaseOrder message to the MessageBox. However, in addition to receiving the message as expected, another instance of an orchestration gets started because it too had a subscription for PurchaseOrder messages. The processing gets into an endless loop and it may take some time for a developer to figure out what has happened.
Correlation in orchestrations is the mechanism for being able to receive related messages into the same running orchestration instance. In Orchestration Designer, in order to use correlation, a developer follows these general steps:
Defines a correlation type that includes the promoted properties that are used to relate messages.
Defines a correlation set that is an instance of the correlation type just defined.
Specifies on send and receive ports whether they initiate or follow a given correlation set.
Instance subscriptions come into play when a correlation set is initiated, as this is when subscriptions are created for all of those ports that follow this correlation set to receive messages. Because the correlation type defines the properties to be used for correlation, the orchestration engine can extract these properties from the message being sent or received by the initiating action. These values are then used to define subscriptions for all of the remaining actions which follow this correlation set.
In the following figure, the send that occurs at point 1 initializes a correlation set. At this point, because the receive shape at point 2 is configured to follow that correlation set, an instance subscription is created for the receive port which includes the message type and the correlation properties.
Figure 6 A send initializing a correlation set
Due to how correlation works, it is important that messages received into BizTalk Server and intended for use in a correlation have their promoted properties correctly defined and promoted to the message context. In order for most properties to get promoted when a message is initially received, a disassembler component is used to extract the values. For this reason, it is not possible to use the PassThrough receive pipeline to receive messages that must be correlated to a running instance of an orchestration. This is a common issue that comes up when using the SOAP receive adapter to receive correlated messages because the default value of the receive pipeline when using the Web Services Publishing Wizard is the PassThrough pipeline.
In a request/response messaging pattern, one party sends a request message and the receiving party returns a response message. Two typical examples of request/response processing are the interaction that a browser has with a web server using the HTTP protocol, and web service processing using the SOAP protocol. In BizTalk Server 2004, both the request and the response messages are handled in a typical publish/subscribe fashion. This is an important consideration to understand for performance tuning a BizTalk application as a system requiring high throughput may be configured differently than one requiring low latency for individual messages.
Figure 7 Request/response messaging
When a message is received by a request/response style receive adapter, BizTalk first publishes the request message to the MessageBox. Next this message is received by the appropriate subscriber, which is likely an orchestration bound to a receive port. This subscriber formulates a response message and publishes it to the MessageBox along with properties that cause it to be sent back to the receive port from which the request came. Finally, the response message is picked up by the publisher of the request, the receive adapter which submitted the request, and is returned to the calling application. The diagram below provides a detailed graphical representation of these steps.
Figure 8 Message processing, step by step
Flow of request/response message received by SOAP adapter
The SOAP adapter submits messages to the Endpoint Manager.
The Endpoint Manager uses the Message Agent API to insert the message into the MessageBox.
The orchestration, which is bound to the receive port and therefore has a subscription for the message, receives the message and processes it.
The orchestration sends a response message that is published to the MessageBox.
The Endpoint Manager receives the response message.
The Endpoint Manager returns the response to the SOAP adapter
The implications of this type of behavior on performance can be overlooked if the internal implementation is not understood. BizTalk Server is initially tuned for high throughput scenarios but may be configured for an environment with lower throughput and a need for lower latency, especially in request/response scenarios. Several components need to be considered for tuning in this scenario. First, subscribers find out about published messages through a polling mechanism. If the polling interval is set too high, this could cause request/response style interactions to have a higher latency than desired.
Note that in this scenario, there are two subscriptions to be filled: the subscription for the initial message, as well as the one for the response message, and this increases the impact of this polling interval. Second, receive adapters are configured to insert messages into the MessageBox in batches of varying sizes. Most adapters enable you to configure the batch size through the typical adapter configuration UI or through parameters in BizTalk Server or the registry. If the batch size is set too high, the latency for individual messages may be increased. For more information about performance characteristics of BizTalk Server 2004 and how to tune for low latency scenarios, see BizTalk Server 2004 Performance Characteristics.
When there are no matching subscriptions for a message that is published to the BizTalk MessageBox, a routing failure occurs. The reason for this behavior is that without a subscriber, the message would get written to the MessageBox and potentially never get picked up. A routing failure is an indication that a message was published to the MessageBox and based on the properties of the message, no subscription was found. The context of the routing failure message is the same context of the message that was published to the MessageBox. This allows a developer or administrator to examine the properties and compare them against the subscriptions that should be active in the system to determine why the message was not properly routed.
In general, routing failures occur for a number of well known reasons. The most obvious reason is that the properties of the message simply do not match an active subscription. There are two situations that most often cause the properties not to match: either the send port or orchestration with the appropriate filter is not enlisted, or the message does not contain the correct data at the time message properties are promoted. In the first case, enlisting the send port or orchestration fixes the problem. In the second case, the process that creates or alters the message needs to be checked to make sure it properly inserts all data needed for promoted properties to be used for routing. One common reason for properties not getting promoted is the use of the passthrough pipeline. This pipeline has no disassembler in it to promote the properties. The XML receive pipeline or a custom pipeline that promotes message properties should be used instead.
Another case where routing failures appear is in a request/response scenario, such as a Web service receive location, when the response message cannot be routed back to the requester. A common cause for this type of failure is when a Web service client connection times out waiting for an orchestration to complete and send a response. In this case, after the Web service client times out, a response message from the orchestration is published, but no subscription will exist for it.
To debug routing failures, a simple comparison of the message properties available at runtime and the subscriptions active in the system usually identify the problem. To examine the message properties, the Health and Activity Tracking tool can be used. The routing failure appears under the Operations/Messages search. Right clicking on this message and choosing message details brings up a property pane that includes the message context. The message context can be opened and all of the properties on the message will be viewable. This exposes the actual properties present, but you can use another tool to examine the subscriptions. The BizTalk Server 2004 SDK includes the Subscription Viewer tool. This executable shows the active subscriptions in the system and enables you to view the properties that the subscriptions include. By comparing these two data points, it is usually apparent why no subscriptions matched the inbound message.
Using the Subscription Viewer
The subscription viewer is a useful tool that ships in the BizTalk Server 2004 SDK. The tool can be found in the <Install Directory>\SDK\Utilities folder. The purpose of the tool is to provide developers and IT professionals with a view into the active subscriptions in a given BizTalk Server installation. This section is intended to provide information on how to read the data presented in the tool and to use it in troubleshooting routing and subscription problems.
Figure 9 The Subscription Viewer tool
The subscription viewer loads all active subscriptions into the window. In the first column, the name of the subscription provides some insight into the type of subscription. Orchestration Activation subscriptions start with "Activate;" instance subscriptions start with "XLANGs." Send port subscriptions are named using both the port name and the transport ID of the send port. Two other types of subscriptions can appear in the viewer: the first is a GetObject subscription that is used for starting or calling orchestrations, and the last is for internal subscriptions and have no name in the viewer.
The viewer shows whether the subscription is enabled or disabled. This indicator shows whether an orchestration or send port is enlisted. If a subscription is not enabled, then messages are not routed to the service; if no other services have subscriptions for the message, a routing failure occurs. The viewer also shows other subservices that created the subscription: EPM for Endpoint Manager, XLANG for orchestrations.
To get details about a subscription, highlight it in the grid and the subscription details appear in the bottom pane. Each group of properties that are in an "AND" grouping of a filter show up their own grouping in the bottom pane. The break between groups represent an "OR" condition in the filter groupings. To debug subscription issues, these properties in the bottom pane can be compared to the message context properties found on inbound messages.
One of the scalability limitations of previous versions of BizTalk Server was related to the suspended and work queue processing. There was a point as volume increased in messaging scenarios where the amount of read and write activity in the database caused performance to stagnate and eventually deteriorate due to lock contention on SQL tables. With BizTalk Server 2004, the MessageBox databases provide scalability not only through using larger and more powerful hardware, but also by allowing for multiple MessageBox databases to be combined into one virtual MessageBox. Customers needing to scale the message processing of their system beyond the capabilities of their MessageBox database can use multiple MessageBox databases to scale out their infrastructure.
As mentioned previously, the Message Agent is responsible for managing all interactions with the MessageBox databases and it is, therefore, the only component that needs to know about and be able to work with multiple MessageBox databases. Because of this, it is not important for a developer to know about how these databases are managed or how to interact with them. Instead, it is important to understand how having multiple MessageBox databases impacts the scalability of the system and how application designs can hinder this scalability.
When a second MessageBox database is added to a BizTalk Server installation, the first MessageBox remains as the master MessageBox. One of the responsibilities of the master MessageBox is to handle all subscription matching and routing of messages to the appropriate MessageBox database(s). Actually, the master MessageBox determines where the message should be routed, and the message agent actually takes care of sending the message to the correct MessageBox database(s). As mentioned previously in the topic "Subscriptions in BizTalk Server," there are two types of subscriptions: activation and instance. Messages being routed to activation subscriptions can be routed to any MessageBox database as they are not tied to any existing instance, whereas messages for instance subscriptions must be routed to the MessageBox to which the activation subscription for that instance was sent.
It is important to keep in mind that the multiple MessageBox database design is intended to provide scalability, but not any additional reliability. Reliability for MessageBox databases is provided through clustering each MessageBox SQL database. If one of the MessageBox databases becomes unavailable, then the entire MessageBox is unavailable. Each MessageBox database has a work and suspended queue for all hosts in the system and each host interacts with all the MessageBox databases, though they do so transparently through the message agent layer. Currently, the Message Agent uses a round robin algorithm to determine which MessageBox database receives a message, though this is open to change and should not be counted on. After an orchestration instance is started, it is locked to the particular MessageBox where it is being processed. If this MessageBox instance were to fail, there is currently no mechanism to recover from this situation, and incoming messages that should be routed to this MessageBox would fail.
Many customer questions about working with BizTalk Server are actually questions about Microsoft® Visual Studio®.NET. Because BizTalk Server development is tightly integrated with the .NET runtime, there are certain deployment, coding, and management issues to be aware of. This section covers the key areas and issues related to BizTalk Server and .NET.
While .NET is the platform of choice for developing in BizTalk Server, it should be noted that some areas of BizTalk Server also allow for development in a COM-compliant language. For example, the pipeline and adapter related APIs have COM-defined interfaces which can be used when building components and adapters. In fact, the messaging engine in BizTalk Server is almost entirely written in native code. Because this part of the application was mostly built when .NET was released, it was not feasible to update it to .NET. The orchestration engine, however, was being redesigned and rewritten and therefore was a prime candidate for being written in .NET.
The most important aspect of BizTalk and .NET is that all BizTalk Server artifacts; maps, schemas, orchestrations, and pipelines, get compiled into .NET assemblies. The two most important implications of this design are that these assemblies must be strong named, and because of that, they also follow .NET versioning rules. The main implication of this is that a BizTalk project, once built against a particular version of another .NET project/assembly (including BizTalk projects), continues to use that version until it has been rebuilt against a newer version.
A common problem that occurs during development related to .NET versioning is when the version numbers on a BizTalk project are not changed and the assembly is redeployed without stopping and starting the BizTalk host instance that the types are loaded into. When the process is run again, the changes do not take effect. This is due to the way in which .NET assemblies are loaded into memory. Because the host already has an in-memory copy of the assembly, it does not reload the assembly when a new copy is put into the Global Assembly Cache. For example, if version 184.108.40.206 of an assembly with an orchestration is deployed and running, and changes are made to the orchestration but the version number is not changed, then the changes do not take effect. After the host instance is stopped, the in-memory copy of the assembly is released and when the host instance starts again it reloads the new copy of the assembly and gets the changes. If a new version was deployed, say version 220.127.116.11, and it was loaded, then the changes would have taken effect.
Deploying assemblies to BizTalk Server is a two part process. The first part is traditional .NET assembly deployment in which the strong named assembly is deployed to the Global Assembly Cache (GAC) on each server where the assembly will be used.The second step is to deploy metadata about the assemblies and their types to the BizTalk Server Management database. When BizTalk assemblies are loaded by BizTalk Server, they are most often loaded using their strong name, found in the Management database.
The BizTalk Server artifacts that a developer creates get compiled into classes which derive from built in BizTalk Server types. For example, an orchestration becomes a class which derives from the Microsoft.BizTalkXLANGs.BTXEngine.BTXService class. It is because these base classes are deployed in assemblies to the Global Assembly Cache, and these assemblies have dependencies on other assemblies in the GAC, that a developer's assemblies must also get deployed to the GAC.
Another important implication of BizTalk Server artifacts being deployed to the Global Assembly Cache and therefore being strong named, is that strong named assemblies cannot call other assemblies that are not also strong named. This means that any assemblies a developer creates that are used by these BizTalk assemblies must also be strong named. Likewise, assemblies deployed to the GAC that load other assemblies without using a specific path, must load those assemblies from the GAC.
Unlike many of the other components that developers create, Pipeline components do not get deployed to the GAC. Pipeline components work entirely with BizTalk Interfaces and therefore do not have base classes deployed in the GAC. Instead, Pipeline components are found and loaded using another common method for deploying .NET assemblies, which is to deploy them to a known location. Pipeline components are deployed to a folder named "pipeline components" in the BizTalk Server installation directory.
Pipeline components are added to a developer's toolbox in Visual Studio .NET to make them available to be dragged onto the pipeline designer. When a BizTalk pipeline is compiled into a .NET assembly, the information about all of the components in the various stages of the pipeline get compiled into the assembly. When this pipeline is deployed to BizTalk Server, the information about the components, including their file name, is inserted into the BizTalk Management database. When the pipeline is executed, these components are loaded from their known location, and the interfaces they implement called as appropriate.
It is important to note two things about pipeline components here. First, any assemblies that the pipeline component depends upon must be deployed to the GAC in order to be found at runtime. This is because only the pipeline component itself has metadata in BizTalk allowing it to be loaded from a known location. Second, due to new features in BizTalk Server 2006, this will be changing, and pipeline components will be deployed into the GAC.
The following set of queries is intended to provide useful information about the health of a BizTalk Server system. These queries make some assumptions about the locations and names of the SQL Server databases. Minor alterations may need to be made to these queries in order to make them work in your environment. If you are not a qualified SQL Server administrator or developer, or do not understand what these queries are doing, you should not modify them or use them in your environment.
Important These queries use locking hints to ensure that no locks are taken on the database tables which are critical to ensuring that these queries do not impact a running system. Additionally, with the exception of inserting into temporary tables, these queries do not update, insert, delete or otherwise modify data or structure in the databases. It is essential that the queries not be modified in any way to alter data or structure in the databases. Finally, because the database structure in BizTalk Server is an internal component, it is open to change and these queries may not work. Future releases of BizTalk Server may incorporate these queries, or similar queries, into the management and monitoring tools.
Count of all orchestration instances grouped by state
Purpose: For each orchestration in the system, provides the number of instances in each state. This information is available for a single orchestration through the properties page in the Administration console, but this query provides summary information for all orchestrations.
How to read the results: If you have a large number of suspended instances, something is failing. If you have a large number of dehydrated instances it could indicate a problem with your backend systems not responding promptly or it could just mean that the host in which messaging services are running is not started. If you have a large number of Ready To Run services then either your BizTalk server is not up or there is a high load on the system and the system is attempting to catch up. The latter situation may not be a problem if the system catches up later. If not, then the load on the system is too high and measures should be taken to scale out your infrastructure.
What to do: If orchestrations are suspended, you need to look at the suspended info in HAT and determine why they are suspended and take corrective action. If orchestrations are dehydrated and there is reason for concern, the message flow should be checked to see what the orchestrations are waiting for and then determine the cause based on the point of stalling. If orchestrations are ready to run, make sure the BizTalk Server services are running and check CPU utilization on these computers and other resource issues to identify potential load issues.
SET NOCOUNT ON SET TRANSACTION ISOLATION LEVEL READ COMMITTED SET DEADLOCK_PRIORITY LOW SELECT o.nvcName AS Orchestration, COUNT(*) as Count, CASE i.nState WHEN 1 THEN 'Ready To Run' WHEN 2 THEN 'Active' WHEN 4 THEN 'Suspended Resumable' WHEN 8 THEN 'Dehydrated' WHEN 16 THEN 'Completed With Discarded Messages' WHEN 32 THEN 'Suspended Non-Resumable' END as State FROM [BizTalkMsgboxDb]..[Instances] AS i WITH (NOLOCK) JOIN [BizTalkMgmtDb]..[bts_Orchestration] AS o WITH (NOLOCK) ON i.uidServiceID = o.uidGUID --WHERE dtCreated > '2004-08-24 00:00:00' AND dtCreated < '2004-08-24 13:30:00' GROUP BY o.nvcName, i.nState
Counts for all host queues
Purpose: This returns the table size for the three queues: work, suspended, and state, for all applications.
How to read the results: If the suspended queue grows, processing is failing. If the work queue grows, the system is being over-driven, the BizTalk services are off, or there may be failures occurring, but retries are pending. If the state queue grows, you either have a lot of dehydrated orchestrations, or you might be having problems with MSMQT.
What to do: If the Suspended Queue is increasing in size, use the error information in HAT and the event log to figure out what is failing. It is important to eventually resume or terminate the suspended instances in order to clear them out of the system. If it is the work queue, services should be checked to make sure they are running and the CPU and other resources should be checked to verify that the system is not overloaded. Also check the event log for warnings indicating transport failures which could cause retries. If it is the State Queue, check for a lot of dehydrated or suspended orchestrations and determine why they are suspended. One of the later queries can also be used to try to determine if the issue is related to an MSMQT instance holding all the references. If it does appear to be MSMQT BizTalk Server may not be sending data to the outgoing queue anymore because it is waiting for an acknowledgement for an earlier message.
SET NOCOUNT ON SET TRANSACTION ISOLATION LEVEL READ COMMITTED SET DEADLOCK_PRIORITY LOW CREATE TABLE #Temp (AppQueue nvarchar(256), QueueSize int) declare @nvcAppName sysname declare MyCursor CURSOR FAST_FORWARD FOR SELECT nvcApplicationName FROM [BizTalkMsgboxDb]..[Applications] WITH (NOLOCK) open MyCursor FETCH NEXT FROM MyCursor INTO @nvcAppName WHILE (@@FETCH_STATUS = 0) BEGIN INSERT INTO #Temp exec ('SELECT ''' + @nvcAppName + 'Q'', COUNT(*) FROM ' + @nvcAppName +'Q WITH (NOLOCK)') INSERT INTO #Temp exec ('SELECT ''' + @nvcAppName + 'Q_Suspended'', COUNT(*) FROM ' + @nvcAppName +'Q_Suspended WITH (NOLOCK)') INSERT INTO #Temp exec ('SELECT ''InstanceStateMessageReferences_' + @nvcAppName + ''', COUNT(*) FROM InstanceStateMessageReferences_' + @nvcAppName + ' WITH (NOLOCK)') FETCH NEXT FROM MyCursor INTO @nvcAppName END SELECT * FROM #Temp ORDER BY QueueSize DESC close MyCursor deallocate MyCursor DROP TABLE #Temp
Count for number of state messages grouped by instances
Purpose: This will give you a count of the number of state messages associated with each instance
How to read the results: This specific query is probably only useful for systems which use MSMQT as a quick warning when things are starting to go wrong. If you have a lot of messages building up for a specific instance and that instance is an MSMQT instances, you could have problems.
What to do: If you have one of these problematic instances and the service class is MSMQT, you can look at the UserState column to determine the name of the queue that you are talking to, and then you need to see why the communication with that computer queue is not working. If this is an orchestration, it is possible this is simply expected. It depends on your business logic as to why so many messages are going to the same instance.
SET NOCOUNT ON SET TRANSACTION ISOLATION LEVEL READ COMMITTED SET DEADLOCK_PRIORITY LOW CREATE TABLE #Temp (AppName nvarchar(256), StateMessages int, InstanceID uniqueidentifier, ServiceClassID uniqueidentifier, ServiceState nvarchar(256)) declare @nvcAppName sysname declare MyCursor CURSOR FAST_FORWARD FOR SELECT nvcApplicationName FROM [BizTalkMsgboxDb]..[Applications] WITH (NOLOCK) open MyCursor FETCH NEXT FROM MyCursor INTO @nvcAppName WHILE (@@FETCH_STATUS = 0) BEGIN INSERT INTO #Temp exec ('SELECT ''' + @nvcAppName + ''', COUNT(*), i.uidInstanceID, i.uidClassID, i.nvcUserState FROM InstanceStateMessageReferences_' + @nvcAppName + ' AS s WITH (NOLOCK) JOIN Instances AS i WITH (NOLOCK) ON s.uidInstanceID = i.uidInstanceID GROUP BY i.uidInstanceID, i.uidClassID, i.nvcUserState') FETCH NEXT FROM MyCursor INTO @nvcAppName END SELECT * FROM #Temp ORDER BY StateMessages DESC close MyCursor deallocate MyCursor DROP TABLE #Temp
Count of all active messages for a specific send port
Purpose: Indicates how many messages are in the work queue for a given send port, and provides information about whether the messages are being sent by the primary or secondary transport and the number of retries attempted for the message.
How to read the results: Having a high number of instances for a certain transport does not necessarily indicate a problem, and may just indicate that the system is backlogged on processing the messages for this transport. However, if the retry count is high it indicates that sending failures are occurring. Having a high number of messages using the secondary transport indicates that the primary transport is failing even with retries and processing has moved to the secondary transport.
What to do: Check the event log for information on why the failures are happening and possibly verify communication with the system or application this send port is communicating with the to identify the problem.
SET NOCOUNT ON SET TRANSACTION ISOLATION LEVEL READ COMMITTED SET DEADLOCK_PRIORITY LOW CREATE TABLE #Temp (MaxRetries int, Active int, SendPort nvarchar(256), IsPrimaryTransport int) declare @nvcAppName sysname declare MyCursor CURSOR FAST_FORWARD FOR SELECT nvcApplicationName FROM [BizTalkMsgboxDb]..[Applications] WITH (NOLOCK) open MyCursor FETCH NEXT FROM MyCursor INTO @nvcAppName WHILE (@@FETCH_STATUS = 0) BEGIN INSERT INTO #Temp exec ('SELECT MAX(q.nRetryCount) AS MaxRetries ,COUNT(*) AS Active ,sp.nvcName AS SendHandlersHostServiceName ,spt.bIsPrimary AS IsPrimaryTransport FROM ' + @nvcAppName + 'Q as q WITH (NOLOCK) INNER LOOP JOIN Subscription as s WITH (NOLOCK) ON q.uidServiceID = s.uidServiceID AND s.uidPortID = q.uidPortID INNER LOOP JOIN [BizTalkMgmtDb]..[bts_sendport] as sp WITH (NOLOCK) ON q.uidServiceID = sp.uidGUID INNER LOOP JOIN [BizTalkMgmtDb]..[bts_sendport_transport] as spt WITH (NOLOCK) ON sp.nID = spt.nSendPortID AND spt.uidGUID = q.uidPortID GROUP BY sp.nvcName, s.uidPortID, spt.bIsPrimary ORDER BY SendHandlersHostServiceName ASC') FETCH NEXT FROM MyCursor INTO @nvcAppName END SELECT * FROM #Temp ORDER BY Active DESC close MyCursor deallocate MyCursor DROP TABLE #Temp
Last duration for SQL Agent jobs
Purpose: This query provides information about the last time each SQL Server job ran and the duration. This information is important to monitor to make sure these jobs are running correctly and that you have them running as they are essential to good system performance.
How to read the results: If the jobs start taking a long time, there could be database resource issues. These issues could be related to CPU performance or high disk activity. If the job failed, it could be because of a number of deadlock issues which were fixed in BizTalk Server Service Pack 1.
What to do: If it is a resource issue such as CPU or disk, the SQL Server may need to be scaled up using to add processors or faster disks. If the failures only happen rarely, it may not be much of an issue because the jobs will run again simply having more work to do. If the servers do not already have Service Pack 1 applied, then it should be applied as it has many valuable fixes in it.
SET NOCOUNT ON SET TRANSACTION ISOLATION LEVEL READ COMMITTED SET DEADLOCK_PRIORITY LOW CREATE TABLE #Temp (JobID uniqueidentifier, JobName sysname, Status int, DateStarted int, TimeStarted int, Duration int, OpEmailed nvarchar(20), OpNetSent nvarchar(20), OpPaged nvarchar(20), Retries int, Server nvarchar(30) ) CREATE TABLE #Final (MessageBox sysname, JobName sysname, DateStarted int, TimeStarted int, Duration int, Status int ) declare @dbName sysname declare MyCursor cursor FAST_FORWARD FOR select DBName FROM [BizTalkMgmtDb]..[adm_MessageBox] WITH (NOLOCK) open MyCursor FETCH NEXT FROM MyCursor INTO @dbName WHILE (@@FETCH_STATUS = 0) BEGIN INSERT INTO #Temp exec ('[msdb]..[sp_help_jobhistory] @job_name = ''MessageBox_DeadProcesses_Cleanup_' + @dbName + '''') INSERT INTO #Final SELECT TOP 1 @dbName, JobName, DateStarted, TimeStarted, Duration, Status FROM #Temp TRUNCATE TABLE #Temp INSERT INTO #Temp exec ('[msdb]..[sp_help_jobhistory] @job_name = ''MessageBox_Message_Cleanup_' + @dbName + '''') INSERT INTO #Final SELECT TOP 1 @dbName, JobName, DateStarted, TimeStarted, Duration, Status FROM #Temp TRUNCATE TABLE #Temp INSERT INTO #Temp exec ('[msdb]..[sp_help_jobhistory] @job_name = ''MessageBox_Parts_Cleanup_' + @dbName + '''') INSERT INTO #Final SELECT TOP 1 @dbName, JobName, DateStarted, TimeStarted, Duration, Status FROM #Temp TRUNCATE TABLE #Temp INSERT INTO #Temp exec ('[msdb]..[sp_help_jobhistory] @job_name = ''PurgeSubscriptionsJob_' + @dbName + '''') INSERT INTO #Final SELECT TOP 1 @dbName, JobName, DateStarted, TimeStarted, Duration, Status FROM #Temp TRUNCATE TABLE #Temp INSERT INTO #Temp exec ('[msdb]..[sp_help_jobhistory] @job_name = ''TrackedMessages_Copy_' + @dbName + '''') INSERT INTO #Final SELECT TOP 1 @dbName, JobName, DateStarted, TimeStarted, Duration, Status FROM #Temp TRUNCATE TABLE #Temp FETCH NEXT FROM MyCursor INTO @dbName END SELECT * FROM #Final ORDER BY MessageBox close MyCursor deallocate MyCursor drop table #Temp drop table #Final
Lee Graber, Suri Narayanan