This paper illustrates how BizTalk Server 2004 deals with convoy message processing. Realize that convoys are not always the best solution to the problem at hand. You should use convoys when the business needs require convoy message processing.
All convoys implement singleton processing with regard to the convoy set they represent. Sometimes, this can be a significant bottleneck in the system. Think back to real world example of the group of vehicles headed to the party. If one vehicle has a flat tire then all the vehicles must stop.
Convoys also have a side effect called zombies. A zombie is a message or orchestration that resides in the "Completed with discarded messages" status. This occurs any time a message routes to a business process that is in the process of ending, or any time unexpected messages arrive into a business process. The sample walk through in the following section covers zombies and covers how to handle them. The best approach available now is the WMI event that commences when a zombie occurs. This shows in the ZombieWMI component.
The three business scenarios discussed earlier are all excellent examples of business processes that require convoy message processing in order to solve the required tasks.
There is a detailed discussion of each sample to help gain a deeper understanding of convoys.
Sample setup: Run the included MSI. All samples install into C:\ConvoyDeepDive directory. If you have not already downloaded the MSI, you can obtain it here.
Scenario 1: Hospital Admits a New Patient
The Hospital Admits a New Patient Scenario is an example of a business scenario that requires parallel convoy message processing. The business requirements dictate the receipt of three different types of message before admitting the patient into the hospital. These three messages are the Insurance, History, and Patient messages. Any one of these messages can be the first message to arrive for the patient and this creates a race condition. To solve this, three Receive shapes are put into a Parallel Action shape and each receive is marked as Activate = True. This enables any one of the three messages to start the orchestration. Shown here is this message receipt pattern.
Figure 8 Admit a new patient
Use a correlation type called corrPatientInfo to identify the properties used to tie the individual messages together. A correlation type is a list of properties that eventually populates with values for use in routing messages. HospitalData.xsd, a custom property schema defines these properties and references each of the three individual message schemas. The nodes that contain the correlation values do not have to have the same name across the various messages. They just need promotion to the same node using the property schema. The properties used for correlation are the patient's date of birth, Property Schema Node Name: DOB, and social security number, Property Schema Node Name: SSN, as these uniquely identify a new patient. What follows is the definition of the correlation type, and patient properties used in this business process.
Figure 9 Definition of the correlation type
Figure 10 Patient properties
Use a correlation set called corrPatientMessage on the Receive shapes to create a new instance of the correlation type based on the properties received by the first message in the set. Remember, this occurs independently of an actual running orchestration instance. Each of the three Receive shapes must initialize the same correlation set.
In order to run this sample, make sure to enlist and start the Scenario 1 Orchestration. At this point, three separate activate subscriptions can be seen in the Subscription Viewer. The Subscription Viewer is located in the Software Development Kit (SDK) at SDK\Utilities\BTSSubscriptionViewer.exe.
Shown here are these three base subscriptions. Additional subscriptions may be present in the table other then what is in the following list.
Figure 11 Base subscriptions
A deeper look into one of the subscriptions shows the properties used for the base subscription. An earlier diagram shows the properties for the InsuranceInfo message. It shows that DOB and SSN must exist but they do not have any specific values.
The sample files used for running this scenario are located at C:\ConvoyDeepDive\Scenario1\SampleFiles. This contains one of each message type for two separate patients, A and B. To start the orchestration, drop any message into the Scenario1\In folder. In this case, the A_MedicalHistory.xml, B_MedicalHistory.xml, and B_PatientInfo.xml are dropped into the In folder.
This action starts two separate orchestrations. You can see these by opening Health and Activity Tracking (HAT), selecting Operations and selecting Service Instances.
You can see the more interesting part of the process by looking in HAT under Operations but this time selecting Messages. This shows two MedicalHistory messages and one PatientInfo message that are consumed and probably dehydrated. This means that the message routed to an orchestration and it is waiting for the next action to happen. On the far right, the Service Instance ID tells with what orchestration instance the messages are associated. Shown here is the compressed message view of HAT.
Figure 12 Compressed message HAT view
The first mesoccurredeived was the A_MedicalHistory.xml message. This contained both a DOB and SSN as described by the MedicalHistory schema so the base subscription filled. After that, a check occurred for the existence of a convoy set with these properties, with none found. Therefore, a new one created. This new virtual subscription had the following properties of DOB=2-23-1950 and SSN=123456789 and created a new orchestration.
The second message, B_MedicalHistory.xml, did exactly the same thing and created another new virtual subscription with properties of DOB=2-23-1950 and SSN-987654321 and another new orchestration.
The third message, B_PatientInfo.xml, handled somewhat differently. It also matches the base subscription with the existence of DOB and SSN. This time however, the test for the existence of a convoy set returns as true. In this case, this message routes to the same orchestration as the previous message with the same convoy set. This is evident by viewing HAT and seeing the two messages routed to the same service instance.
To finish the existing orchestrations, drop the remaining messages into the In folder. This includes the A_InsuranceInfo.xml, B_InsuranceInfo.xml, and A_PatientInfo.xml messages. The orchestrations complete and output messages appear in the Out folder.
The SampleFiles folder also contains a message called C_PatientInfo_NoSSN.xml. This message fails routing on the base subscription because the SSN does not exist. Test this by dropping this message into the In folder. View the results in HAT and in the event log.
Remember, message correlation occurs independently of any running business processes. To test this, open BizTalk Explorer and stop the Scenario 1 orchestration. Make sure the orchestration remains enlisted. Drop all three messages for Patient A into the In folder. HAT shows the three messages in the Suspended (resumable) status and all three already routed to the same Service Instance. View the service instances in HAT. This shows a Scenario 1 orchestration that is pending and ready to run as soon as you start it.
Now, start the orchestration using the BizTalk Explorer and make sure you select the Resume Existing Orchestration Instance check box. This executes the waiting orchestration and output appears in the Out folder. A quick view in HAT shows the messages no longer show up as suspended and the service instance has completed.
If there is receipt of a message two times before the business process ends, it becomes a zombie. Remember that zombies are messages in the "Completed with discarded messages" service status after the orchestration is completed. You can see this by running the process again but this time submitting one of the messages two times before ending the orchestration. You must investigate zombies in order to guarantee no data is lost. Figure 13 shows how this kind of message would look under Messages in HAT.
Figure 13 Message types
The ZombieWMI console application can help solve zombie related problems. This application detects zombie orchestrations and writes the zombie messages to the C:\ConvoyDeepDive\ZombieMessages\. Both the messages and the message contexts write to this folder with the message ID as the message name. Shown here is the output from the console application.
Figure 14 ZombieWMI.exe
Scenario 2: Retail Company Batch Orders
The Retail Company Batch Order Scenario is an example of a business process that requires sequential convoy processing of incoming messages. The business requirements state that all single orders received before 2:00 PM for a specific day should batch together. This requires the first message received to start a new orchestration instance and initialize a correlation set. At that point, all other orders received of that message type route to the already running orchestration instance. To accomplish this, you position a Listen shape (containing a Receive shape and a Delay shape) inside a Loop shape. After reaching the delay, the loop ends. This allows for the receipt of an unknown amount of orders in the same business process. This kind of process could potentially result in routing messages to this orchestration while it is in the process of ending. Therefore, zombie monitoring for this kind of process is very important.
The repeating receive messaging pattern is shown here. Note that this does not show the receipt of the initial message.
Figure 15 Repeating receive messaging pattern
The above illustration uses XML document manipulation to append additional orders to a base XML document. This approach requires an Atomic Scope shape since XmlNode is not serializable.This enables a straightforward approach to batching in this scenario. It also eliminates message retention and build up inside the MessageBox database. In this case, message receipt happens inside the loop and puts the message into a "Delivered, not consumed status". The message remains in this status until the orchestration tells the MessageBox database otherwise at a persistence point. Using the Atomic Scope shape creates a persistence point and releases the messages as they are consumed. Other types of long running processes inside a loop without a persistence point may cause a large amount of messages to remain inside the MessageBox database in the "Delivered, not consumed state" until the orchestration completes or dehydrates.
Use a correlation type named corrMessageType to identify the properties used for correlation. In this case, use a single BizTalk Server 2004 internal property for correlation. That property is BTS.MessageType. This allows for the same orchestration instance to route and process all orders. This creates a singleton behavior with regard to the MessageType of a message and could potentially create a performance bottleneck. Remember, convoys are business process driven and should only be used when required. A correlation set named corrOrderMessage initializes by the first Receive shape named FirstOrder followed by the AllOtherOrders Receive shape inside the loop.
A look into the Subscription Viewer shows two subscriptions for this message. They are the same, except one is marked as not being able to activate a convoy. This does not show in the Subscription Viewer. As part of the subscription matching process, the presence of an existing orchestration is checked and creation of new orchestration instances occur as needed based on the convoy values in the correlation set. Therefore, both subscriptions always evaluate but only one ever fills.
In order to run this sample, make sure to enlist and start the Scenario 2 Orchestration. The files used for this scenario are located in C:\ConvoyDeepDive\Scenario2\SampleFiles. This contains five different orders. Select any one of the orders and drop it into the In folder. In this case, you select and position Order_No101.xml in the In folder. A single orchestration starts with a correlation set based on this message type. This creates a virtual subscription based on the received message type of the message. In this case, it is a single order. Note that for this sample the 2:00 PM time to close a batch changes to a simple one-minute timer. The batch ends when there is no receipt of new messages for one minute. See the Helper Class about how to change this time-out.
Drop the remaining four orders into the In folder. The same orchestration instance accepts each order and adds it to the batch. A look into HAT shows a single message associated with this orchestration. After reaching the time-out, the result is a single batch file in the Out directory.
If messages arrive to the orchestration after the loop has ended, these messages will cause the orchestration to be "Completed, with discarded messages". To simulate this, add a Delay shape after the loop and send messages into the process. You can see this in the RetailCompanyBatchOrdersZombie.odx provided in the Scenario2 directory. This orchestration does not deploy with the initial set up. This orchestration writes a message to the event log as soon as the loop is finished so make sure your BizTalk Server 2004 host has event log write permissions. The delay holds for one minute causing any submitted messages to route to this running instance and become a zombie.
Scenario 3: Standardize Test Taking and Grading
The Standardized Test Taking and Grading Scenario is an example of a business scenario that requires sequential convoy message processing. This differs from Scenario 2 because, in this case, there is the receipt of four different message types. This differs from Scenario 1 because, in this case, the messages must arrive in a defined order.
In this scenario, the receipt of the Registration, Check In, Check Out, and Test Received messages are in order. Shown here is this kind of pattern.
Figure 16 Messages received in order
Use a Correlation Type named corrStudentType to identify the properties for this convoy. In this case, use a single custom property named StudentID for correlation. This definition is in the StudentTrackingID.xsd schema. The Registration Receive shape initializes a correlation set named corrStudentID and is followed by the other three Receive shapes.
Examining the subscription viewer shows four subscriptions, one for each of the different message types. These are all marked as Activate but the ability to start a new orchestration instance receives a check as part of the subscription matching process.
Figure 17 Subscription viewer
To run this sample, make sure to enlist and start the Sample 3 Orchestration. Locate the sample files at C:\ConvoyDeepDive\Scenario3\SampleFiles. Drop the Registration_0004.xml into the In folder. This creates a new instance of the business process for test taking and grading. This also creates a virtual subscription for the student ID that was in the received message. In this case, the virtual subscription is 0004. Any additional messages that match one of the four subscriptions with an ID of 0004, routes to this already running instance.
To complete this process, drop the SiteCheckIn_0004.xml, SiteCheckOut_0004.xml, and TestReceived_0004.xml into the In folder. Although the business process has the files in a defined order, this is not enforced. If these files drop out of order, the message routes to the correct orchestration instance and it waits until all the remaining messages arrive.
As the orchestration receives messages, they then appear in HAT all with the same service ID. The Messages view of HAT looks similar to this.
Figure 18 Wait for remaining messages
After the receipt of the last message, a message appears in the Out folder.
You can create zombies in this process if there is receipt of same message two times for a specific student ID. This causes the orchestration instance and the additional messages to be suspended. Using the ZombieWMI console application a zombie would look similar to the following figure.
Figure 19 Zombie view