Export (0) Print
Expand All

Introduction to BizTalk Adapters for Host Systems

Published: August 2007

Author: Jon Fancey, Pluralsight

Applies to: BizTalk Server 2006

With Microsoft® BizTalk® Server 2006, Microsoft has further extended its reach into enterprise integration by providing a suite of new technology adapters. These include a new set of adapters for IBM Host Integration scenarios covering DB2, WebSphere MQ, Host Applications, and Host Files. Together, these adapters enable powerful integration scenarios to be achieved by lowering the barrier of complexity and standardizing on approach. In this white paper we shall look at each of these adapters in detail and explain their capabilities and use. We will also consider how to build BizTalk solutions that leverage them and some of the tradeoffs in choosing the appropriate integration and configuration choices. Combined, these improvements enable BizTalk solution architects and developers to push the boundaries of the product even further into the enterprise to create truly seamless solutions.

BizTalk Server 2006 is Microsoft's premiere business process and integration server. As the fourth major version of the product, it introduces new capabilities and engine improvements, enabling developers to create more flexible solutions for integrated business processes as well as empowering administrators and business users to more effectively monitor ongoing business processes. In addition to these benefits, a fully integrated adapter story is provided giving out-of-the-box connectivity to a multitude of third-party applications and technologies which had previously required often costly additional software purchases to facilitate. As part of this strategy, Microsoft now provides a set of new adapters aimed specifically at customers with IBM midrange and mainframe investments to enable easier connectivity in a variety of situations. These adapters are known as the Microsoft BizTalk Adapters for Host Systems and include four adapters:

  • Microsoft BizTalk Adapter for WebSphere MQ (MQSC adapter) (Client-Based)
  • Microsoft BizTalk Adapter for DB2
  • Microsoft BizTalk Adapter for Host Files
  • Microsoft BizTalk Adapter for Host Applications

The purpose of this paper is to discuss each of these new adapters and how they may be utilized in your BizTalk Server 2006 solutions to enable rich Host integration. In addition, we will also consider alternative options when these adapters cannot provide the required functionality and how to leverage host technologies using Host Integration Server (HIS) and BizTalk Server effectively.

BizTalk Server Adapters

The runtime architecture of BizTalk Server 2006 is modular and essentially breaks down into a number of logical pieces shown in Figure 1. In the scenario shown there are two adapters, SOAP and DB2, on the receive and send sides, respectively. The job of an adapter is to act as the edge-connector for BizTalk Server to communicate with other systems. An adapter wraps up the necessary logic and details to receive or send messages over a particular transport (such as HTTP) enabling the application to exchange messages with other systems in a consistent way. This is illustrated by the BizTalk Adapter for DB2, which although having a number of dependencies required for its operation, does not encroach on the design of the BizTalk application itself. Individual adapters are at liberty to provide additional features on top of this base requirement such as transactional support, ordered delivery or batching, for example. However, the one thing all adapters have in common is their ability to connect and communicate over a particular transport or technology. This flexibility has enabled the creation of hundreds of adapters from both Microsoft and third-parties to facilitate integration with many diverse applications, platforms, and systems. For more details on the available adapters I have included a reference at the end of this paper.

Figure 1. Architectural View of Adapters

Architectural View

Styles of Integration

Before we get into the capabilities of each of these adapters, let’s take a short detour to consider systems integration and ways to achieve it with an integration server such as BizTalk Server. Integration, and therefore adapters, essentially breaks down into four main categories: network, data, application and messaging. Each of these can be seen as providing a different level of integration as we shall see in a moment. Figure 2 shows each type of integration and how they are achieved in the context of BizTalk adapters.

Figure 2. Categories of Integration

Architectural View

Network integration is concerned with connectivity between one system and another over a particular transport such as TCP/IP or a higher-order protocol such as SOAP/HTTP over that transport. Network integration typically requires the connecting applications to do more work to integrate as it simply provides the path between the two, and issues such as message formatting, encoding and reliable delivery must be implemented on top of this at the endpoints.

Data integration provides access to an application’s data store directly. This facilitates the ability to both read and update data, for example. Reading is relatively straightforward, and typically reporting engines such as Microsoft SQL Server™ Reporting Services use this approach. Update (which includes classic CUD) is more problematic, however, as it must be ensured that the data is kept in a valid, consistent state with regard to the application that sits on top of it. For example, validation, rules and business logic which traditionally have been implemented in application code may require duplication to achieve this level of integration.

Application integration is the ability to connect to an application and actually drive it programmatically from another system. This can take many forms, the most common being through an application-provided API or screen-scraping. Application APIs exist in several forms from modern Web-service-based to COM components. Where no API exists, the application must be driven through its original interaction mechanism which is usually its user interface. Again, depending on technology this may be achieved in different ways from constructing messages that “appear” to have come from a user by emulating terminal input, for example, or by interfacing with the terminal itself and entering data character by character as a human user would. Similar issues exist to data integration where it must be ensured that data cannot be provided that would invalidate the system (and cause the real application to misbehave), but application integration tends to provide a level of isolation compared to data integration but often at the cost of complexity.

Finally, Messaging Integration provides a disconnected style of inter-system communication by sending messages from one system or application to another. This style of integration facilitates scenarios where systems do not have permanent connectivity and is often achieved through a store-and-forward pattern where messages from one system are stored up and delivered when possible to the recipient. This provides for senders and receivers to neither be online nor connected to one another at any point. Messaging middleware such as Microsoft’s MSMQ enable such communication patterns to be implemented.

Now that we have established a delineation in style of integration, it’s time to look at each of the new Host adapters and how they achieve integration between the BizTalk Server and IBM platforms. A common customer requirement is the ability to enable straightforward integration with IBM platforms such as the one Host Integration Server provides, through BizTalk Server. The Microsoft BizTalk Adapters for Host Systems have been specifically designed to address this need. Table 1 lists what this package contains, describing the new adapters.

Table 1. Adapters

Adapter Purpose

IBM WebSphere MQ adapter (MQSC adapter)

Sending and receiving Messages via MQ

DB2 adapter

Full Tx CRUD access to DB2 database

Host Files adapter

Accessing Host file systems

Host Applications adapter

Invoking RPG/COBOL programs

Connecting to the Host - Adapters

As listed in Table 1, there are four new Host adapters available to all BizTalk Server 2006 customers as an additional download (see the references at the end of this paper). We will now look in detail at each of these adapters and how they can be employed to solve the integration demands of customers with diverse estates spanning Microsoft Windows Server® 2003 and traditional IBM platforms.

Messaging Integration - MQ Series

The Client-Based BizTalk Adapter for WebSphere MQ (MQSC) provides an alternative to the Server-Based BizTalk Adapter for WebSphere (MQSeries) that has existed since soon after the release of BizTalk Server 2000. Although it provides similar functionality to the Server adapter (MQS), its architecture is very different. Figure 3 shows this in detail. The principal difference lies in the fact that a WebSphere MQ server on Windows® is no longer required to connect to remote Queue Managers on non-Windows systems, unlike the Server (MQS) adapter. With the Server adapter a component is installed on the MQ computer that interacts with the MQ APIs which essentially boils down to a COM+ application (called MQSAgent). This adapter then uses DCOM to communicate with the MQSAgent to send and receive messages between BizTalk Server and MQ. Negating the need for an additional server simplifies this architecture and instead the IBM MQ Client software (see references for the required versions of IBM software required) is installed on the BizTalk server(s). There are two different client installations, base and extended. Although the base client is non-transactional, the MQSC adapter still implements guaranteed delivery (at least once) as we shall see in a moment while the extended client is fully transactional (once-and-only once).

Figure 3. MQ Client Adapter Architecture

Client Adapter Architecture

In Figure 3, two environments are depicted, the BizTalk Server on Windows Server and an MQ Server which is running on AS/400. The MQSC adapter enables send and receive functionality using the chosen client. In addition, Table 2 lists the features of the adapter. The MQ architecture for the Client works by receiving and sending messages to and from queues using a channel. A channel is a uni or bi-directional securable mechanism to perform reliable communication and is defined on a given Queue Manager on the server. Between the Client and the Queue Manager on both ends is the Message Channel Agent. It is the agent’s responsibility to control the send and receipt of messages.

Table 2. MQ Client Adapter Features

Feature

SSL Encryption

Ordered Delivery

Transactions

MQ Headers

Dynamic ports

The MQ Client adapter supports SSL connections between the client and MQ Server and can achieve this in both static and dynamic binding configurations. In the static case, the required information can be added to the port configuration dialog box. For dynamic sends this is not the case and the SSL details must be supplied some other way as the transport information is not configured on the port. In this case, a Client Channel Definition File (CDF) is required. The resultant table (TAB) file must reside in the install location of the WebSphere MQ Client software on the BizTalk Server(s). When the MQ Client libraries are loaded by the adapter, they locate this file and configure the connection accordingly. You should refer to the IBM WebSphere MQ product documentation for more information on defining CDFs.

It should be noted that the guaranteed delivery with the Base Client comes at a small cost. Due to the additional interaction between the MQ Client and MQ Server to ensure delivery the Base Client configuration will be less performant than the Extended Client. With the Extended Client, MQ participates in a traditional MSDTC 2PC transaction with the SQL Server MessageBox database to ensure consistency and atomicity.

When to Use

We’ve already covered why you might want to use the Client adapter over the original Server (MQS) one. However, there are many additional advantages of using queuing over other integration choices including disconnected operation, decoupling, reliability, and performance. Let’s take a quick look at each of these with respect to the Client adapter.

The sender and receiver do not have to be online at the same time using a direct connection. This is advantageous where permanent connection is not possible and a batch-style message transfer can be achieved easily. With unreliable connectivity the sender’s messages are stored durably while the client attempts to retrieve them. Using queue-based integration exhibits a “pull” rather than “push” model whereby the receiver is in control of the amount of messages they retrieve and process simultaneously. This is advantageous where the number of messages varies as it helps to smooth out peaks and troughs in load.

It is not necessary for a sender or receiver to have any prior knowledge of one another. Indeed, there may be multiple senders or receivers to/from the same queue with the queue as the only shared resource acting as the rendezvous point. The decoupling also provides asynchronous operation while synchronous (request/response) may still be layered on top of this if required.

The MQ Client adapter provides guaranteed delivery whereby messages can never be lost. This is critical for many enterprise-level applications, and the adapter further provides the choice of exactly once or at least once delivery depending on the IBM Client used.

Use of MQ Series in conjunction with BizTalk Server can provide very high throughput in many scenarios - especially for messaging only. A common message pattern with MQ is to use BizTalk Server as a content-based router. This is analogous to BizTalk Server acting as a postman, receiving letters and delivering them to the interested recipients. In addition to this, BizTalk Server is able to actually deliver the same message to multiple parties and apply rules to who receives them. All of this can be achieved without orchestration, enabling very high throughput to be achieved especially when taking messages off one queue and placing them on another, for example.

Now that we’ve looked at when to use the adapter, let’s have a look at how. Essentially there are three main components to using the Client adapter: MQ Client configuration, BizTalk Server port configuration, and the solution design itself. Let’s cover each of these in turn:

Client Configuration

MQ Client Configuration starts with a choice between the Base Client and Extended Client (see references for details). The principal difference between the two is that the Extended Client is fully transactional while the Base Client is not. However, the BizTalk Server adapter is still able to guarantee at least once delivery with the Base Client due to the way it uses the client. Essentially, the adapter first retrieves a batch of messages non-destructively and only once they are safely in the MessageBox database are they removed from the queue. This is achieved using a cursor in MQ to enable locking and browsing of a queue without removing the messages. This is the reason performance differs between the Base and Extended clients as there are additional round-trips to the MQ Server for the Client. The same process occurs in reverse when sending messages, where the adapter checks the messages are safely on the queue before responding to the BizTalk Message Agent (also known as the End Point Manager (EPM) ), which will then remove them from the MessageBox database. Because of this, it is possible in failure conditions to cause duplicate messages to be processed and your solution must take this into account if necessary. The Extended client ensures “once and once only” semantics.

BizTalk Server Configuration

The client adapter has a simplified infrastructure for SysAdmins, especially where MQ is not employed on computers running Windows as is typical in IBM midrange/mainframe environments. Depending on requirements, configuration of the MQ Client is required as well. This is typical for security, where an SSL connection is necessary between the MQ Client and Server. If dynamic ports are used, then the SSL configuration is required as mentioned earlier – see the IBM documentation for full details on this. After the MQ Client is installed and configured, the BizTalk Server Port configuration can be performed. There are a multitude of options here and we will consider the use of the most important ones here. The first thing to mention is that configuration is handled differently on Send and Receive. On the Receive side, the transport configuration is achieved through creating a Receive Location (see Figure 4) while on the Send side a Send Port is created and configured.

Figure 4. Port Configuration

Port Configurations

First, the connection between the BizTalk Server and the MQ Server must be defined. This is covered by four properties: channel name, connection name, queue and queue manager name.

As I’ve already mentioned, client connection to an MQ Server is achieved over a channel. The channel is the mechanism over which messages are sent and received, defined on the MQ server and for a specific MQ queue manager. For TCP/IP the connection name is the IP address and port over which to communicate with the MQ Server. The queue manager is the MQ server queue manager on which the required queue is configured. Finally, the queue name itself is the queue to poll for messages.

The MQ adapter supports full ordered delivery on both the receive and send sides. When both are configured, First In, First Out (FIFO) processing all the way through BizTalk Server is possible (without orchestration – and (using a sequential convoy) with orchestration). This is achieved by BizTalk Server adding certain information in the MessageBox with the Message itself as it passes through, containing ordering details. The impact of ordered delivery is that the adapter switches to a single-thread per queue approach to ensure that messages are pulled and pushed in the exact same order. This yields reduced performance over the multi-threaded operation whereby the adapter is capable of processing queue messages asynchronously across multiple threads.

The size of the batches used by BizTalk Server to receive messages can have a significant overall effect on throughput. Large batches yield higher performance but there is a higher likelihood of errors in the batch which can adversely affect this. Consequently batch size is a trade off of performance and throughput and should be fully tested prior to go-live to ensure that any performance requirements are met. To see why this is so, we need to consider how the adapter processes messages, batches and how errors are dealt with.

It often comes as a surprise to people new to BizTalk Server or MQ processing with it, how batches are processed. Essentially, BizTalk Server polls the configured queue periodically on a polling interval (which is configurable – 3 seconds by default) for messages and when at least one message is present it is retrieved and a new batch created. The adapter performs the poll with a wait (timeout) interval - also 3 seconds by default and configurable. What this means is that if a message is written to the queue during this period it will be retrieved immediately, otherwise it will be picked up on the next poll (after the specified polling interval has elapsed). If the batch size is greater than 1, the adapter continues asynchronously (for unordered receives) filling up the batch until the batch size limit (absolute number or size in KB) is hit. The batch is then submitted to BizTalk Server for processing. Meanwhile, if there are still more messages ready, a second batch is created on another thread and submitted. This continues until there are no more messages – i.e. a non-full batch is submitted. The adapter then waits for the specified polling interval before polling the queue again. What this means for processing is that if a queue is pre-loaded with many messages (in the order of thousands), batches will be submitted in quick-fire succession. In BizTalk Server 2004 this caused a number of problems which have been addressed in 2006 with advances in throttling and control of processing. This is good news as it ensures that with tens or even hundreds of 1000s messages, BizTalk Server is able to process them quickly and efficiently.

Transactions Supported will cause MQ to participate in an MSDTC transaction to ensure once-only delivery. On the receive side, the adapter will create the transaction and pass it to the BizTalk Message Agent which is then used to enlist SQL Server updates together with the MQ Queue. In the event of failure, SQL changes can be rolled back and the message put back on the queue. This causes a problem if it is a bad message as it will cause a loop, where the message is continually retrieved and rolled back. The adapter handles this with the Stop On Error and Suspend as Non Resumable properties. These allow the Receive Location to be disabled if an exception occurs or the message suspended in the MessageBox respectively. In the case, the ‘bad’ message will be published to the MessageBox with error information added to the context to enable failed message routing. In the former case, this is typically required when ordered delivery is essential, as if a bad message cannot be processed then subsequent messages, if processed, would cause a re-ordering.

Request/response with MQ – Correlation

As I mentioned earlier, one of the advantages of using queue-based integration is that it is decoupled. This works well for asynchronous messaging but makes traditional request/response exchanges more difficult as two unrelated messages (from the transport’s perspective) must be correlated with one another. The MQ adapter supports this in two ways depending on whether a solicit-response port is used, or separate Send and Receive shapes. The principal difference lies in who generates the correlation token. If a solicit-response port is used and the MQMD_MsgID context property is left blank, MQ Series will return a message on receipt with a generated MQMD_CorrelID context property value. This ID can then be used in subsequent receives from the same or different queue to correlate the original message sent with the response(s).

As often a different queue is required from the send queue, the second option of manual correlation may be more suitable as it enables more flexibility at the expense of a little more work. It operates as before by using MQ headers to track the correlation id between queues and messages but isn’t limited to a solicit-response-response pattern and therefore can reduce the number of message box hops required. Figure 5 shows the sequence of events.

Figure 5. Manual Correlation

Manual Correlation

Figure 5 shows an orchestration using the MQ client adapter and manual correlation to implement a solicit-response pattern. First, the orchestration must set the MQ header MQMD_CorrelID to a unique ID and a correlation set initialised containing this context property. Next, the message is sent to an MQ queue. After a period of time, a second message is received on a potentially (but not necessarily) different queue. The downstream application that received the original message must have copied the MQMD header’s CorrelationID to the new (response) message’s CorrelationID. The adapter automatically copies this property and the receive port has a following correlation set – the same as that initialised in the send port. This ensures the second message is correlated to the first and the response message delivered to the correct orchestration. Using this approach, two different queues can be used to implement a more flexible approach to the solicit-response pattern. For further details on these techniques see “Correlating Messages Using Request-Reply” at http://go.microsoft.com/fwlink/?LinkId=96585.

Data Integration – DB2 and Host Files

If you recall from earlier, data integration is concerned with the direct access of an application’s data store – circumventing at least part the application itself. There are two data adapters included as part of the BizTalk Adapters for Host Systems download: DB2 and Host File. Let’s look at each of these in turn.

BizTalk Host Adapter for DB2

The BizTalk Host Adapter for DB2 has been designed to access IBM’s DB2 database on a number of platforms (see references) including mainframe, AIX, AS/400, and Windows. Where the database is located and on what platform is completely abstracted from the BizTalk application itself and forms part of the configuration of the transport itself in the normal way. Often, with older systems the easiest way to integrate them into new business processes is to access the data contained within directly. This is because the application itself may have no API to call, may not be modular in structure and it may only be possible to drive the application through its own UI. All of these lead us down the path of direct data integration. Figure 6 presents the architecture of the DB2 adapter.

Figure 6. DB2 adapter

DB2 Adapter

The DB2 adapter leverages the new Managed Provider for DB2 that is provided with HIS 2006. This provider can be used stand-alone as well as course, if desired. The provider is capable of communicating over both TCP/IP and LU6.2 but in the latter case, an HIS Server installation is required as shown above, to handle the communication. The adapter provides a number of features, which are listed in Table 3.

Table 3. DB2 Adapter Features

Feature Description

Transactional

Full XA 2PC Transaction support

DRDA

Industry standard for distributed data access

Connection wizard

Easily create DB2 connection configuration

Schema-based

For XML to relational retrieval and update

SQL, stored proc, command

IBM DB2 SQL support for database operations

Updategrams

Intuitive XML-based mechanism to update data without SQL

LU6.2 / TCP/IP

Multiple Host transport Support

Send-Side Adapter Configuration

Data adapters require a schema to define the messages that will be passed between BizTalk Server and the data store. The DB2 adapter provides a wizard to enable the creation of this schema - shown in Figure 7. There are three types of interaction allowed with DB2: Updategram, Stored Procedure and SQL. The adapter has two components, receive and send and each is dealt with differently.

When sending messages to the adapter, either a one-way (send only) or solicit-response port can be used. The choice depends on whether the results from a database operation are required for subsequent processing or not. The messages sent and received must conform to the configured schema although the developer is at liberty to change the auto-generated schema if required. More than this, because XML messages are being exchanged, the messages themselves can be dynamically generated to perform different database operations, say to call different stored procedures, which affords a great deal of flexibility.

The send port configuration is shown in Figure 7.

Figure 7. DB2 Adapter Send Port Configuration

Adapter Send Port Configuration

The most important piece of information required here is the connection string – see Figure 8. This can be specified from a previously existing string (from the HIS Data Access Tool for example), or the Data Source Wizard can be used to create the connection string from scratch. Remember that these details are stored in the port configuration in Enterprise Single Sign-on (SSO), so user IDs and passwords are kept secure. The remaining pieces of information are the namespace and root node name for the XML documents sent and received from the adapter respectively. The namespace is used to construct the outgoing messages as the required schema must be looked up by this namespace and the root node of the actual root element of the message. The response root node is needed to correlate the outgoing message to the request correctly. It’s important to note that the connection details cannot be set at runtime – using dynamic ports is not currently supported by the DB2 Adapter. The URL appears to allude to the fact that the endpoint address and database can be specified dynamically (by BTS_OutboundTransportLocation) but in fact all that is required is a unique URI. This value is simply used to retrieve the port’s configuration at runtime and cannot be used to specify an arbitrary database server or database.

Figure 8. Sample Connection String

User ID=Jon;Password=***************;Initial Catalog=SAMPLE;Network Transport Library=TCP;Host CCSID=1252;PC Code Page=1252;Network Address=localhost;Network Port=50000;Package Collection=SYSIBM;Process Binary as Character=False;Units of Work=RUW;DBMS Platform=DB2/NT;Defer Prepare=False;Rowset Cache Size=0;Persist Security Info=True;Connection Pooling=False;Derive Parameters=False;

The SQL, Stored procedure, and Updategram options are dealt with differently, each having a different message structure created by the schema wizard. The type of structure actually determines the operation to perform. Some examples should help to clarify this.

Updategrams

When a SQL schema is generated, Updategrams are used to specify the required database operations. An example updategram is shown in Figure 9.

Figure 9. Updategram

<ns0:SalesUpdate xmlns:ns0="http://db2.samples.microsoft.com">
  <sync>
    <before>
      <SALES SALES_DATE="2005-12-31" SALES_PERSON="LUCCHESSI"   
             REGION="Quebec" SALES="1" />
    </before>
    <after>
      <SALES SALES_DATE="2005-12-31" SALES_PERSON="LUCCHESSI" 
             REGION="Quebec" SALES="3" />
    </after>
  </sync>
</ns0:SalesUpdate>

Updategrams work using a declarative model where the database operation, and hence the SQL to be generated are implied. This works by the presence or absence of the <before> and <after> elements. There are three possible combinations, listed in Table 4. In the example, both nodes are present indicating an UPDATE. The name of the immediate child node specifies the table to be affected and in the UPDATE case must be the same. The attributes on this table element specify the before and after values of the columns in the table referred to. At the very least these fields must include the primary key column(s) to identify the rows to operate on.

Table 4. UpdateGram Permutations

<before> <after> Operation

X

X

UPDATE

X

DELETE

X

INSERT

This turns out to be simple, yet very powerful. In addition, by creating the schema manually, even greater flexibility may be achieved. Multiple before and after nodes can be specified in a single message to perform composite operations (all within a single unit of work) while each node can specify a different table to affect. This opens the door for messages to be constructed dynamically to perform arbitrary operations on the database. For example, multiple child nodes specifying different tables to be updated may be specified allowing more complex operations to be carried out.

Using Stored Procedures

Stored procedures are invoked by the adapter using the format shown in Figure 11. The schema wizard handles the generation of this schema, allowing a Stored Procedure to be chosen and static argument values supplied (See Figure 10). Obviously, it is often required that the parameter values are determined at runtime and in this case the message must simply be created from the schema and the default values replaced. All of this can be easily achieved using a map in an orchestration or send port. The name of the stored procedure is the sub node of the <StoredProcedure> element and again, if the schema is manually modified, can be used to specify any stored procedure dynamically.

Figure 10. Stored Procedure Schema Generation

Stored Procedure Schema Generation

Figure 11. Stored Procedure Message

<ns0:therequest xmlns:ns0="http://samples.microsoft.com/db2/cedarbank/sproc">
<sync>
<StoredProcedure>
<BONUS_INCREASE P_BONUSFACTOR="10.4" P_BONUSMAXSUMFORDEPT="10.4" P_DEPTSWITHOUTNEWBONUSES="P_DEPTSWITHOUTNEWBONUSES_2" P_COUNTDEPTSVIEWED="10" P_COUNTDEPTSBONUSCHANGED="10" P_ERRORMSG="P_ERRORMSG_5" /> 
  </StoredProcedure>
  </sync>
  </ns0:therequest>

The final option is to execute an arbitrary SQL SELECT using the adapter. This option simply requires the population of the <Select> node in the message with the SQL required. Figure 12 shows an example message. Again, the message can be created using a map or can be specified in a message assignment shape for example. This option is most commonly used when data is being retrieved from DB2 as part of a business process implementation.

Figure 12. Sample SQL SELECT Message

<ns0:sqlreq xmlns:ns0="http://samples.microsoft.com/db2/sql">
<sync>
  <Select>SELECT * FROM AUTHORS</Select> 
  </sync>
  </ns0:sqlreq>

Hopefully by now you’ve realized that the schema is not what’s driving the SQL generation in the adapter – its’ the messages sent to the adapter themselves. Of course, these messages must conform to the schema, but there’s no reason why untyped messages, using XmlDocument, shouldn’t be sent to the adapter.

Configuring the Receive-Side

Using the receive-side of the DB2 Adapter is achieved in the same way as the Send. The principal difference lies in the fact that the receive-side works by polling the database configured on the Receive Location’s connection details. As such, receive-side is by definition a one-way operation although data may be updated as a consequence of its retrieval through the adapter as we shall see.

Receive-side, two options are provided, calling a stored procedure and executing a SQL select. Both of these are configured as before, and used in your solution by creating messages against the schemas generated. Figure 13 shows the receive location properties.

Figure 13. Receive Location Configuration

Receive Location Configurations

Figure 13 shows that a few more options exist on the receive side. Connection string, Root element name and namespace are configured as before, but this time a SQL command and Update command can be specified. The SQL command property opens up a dialog allowing the entering of a valid DB2 SQL statement. The SQL command could be either a SQL SELECT or stored procedure (with static parameters). Either way, the expected dataset should conform to the receive schema generated above if the intention is to process the messages received against a schema. A message is created by the adapter for each row in the returned dataset. In addition, the Update Command property enables an action to be taken for each selected row. There are three options available: do nothing, delete row, update row. Do nothing is self-explanatory, while delete row will delete the row retrieved. Note that this only occurs once the resultant message has been published to the Message Box to prevent data loss. The final option is Update Command, enabling any of the columns in the SELECTed table to be updated with static values. This is useful for example, when retrieving newly added rows of data, to prevent reselection on each subsequent poll.

The polling period is also configurable on the Receive Location, with the interval and unit of measure specified, the default being 30 seconds.

Using Transactions

You may have noticed that unlike other adapter configuration, there is no mention of transactions here. This is because transactional support is indicated in the connection string specified on the Receive Location/Send Port. The Units of Work property is used to specify this. There are two possible values – RUW (remote unit of work) and DUW (distributed unit of work). Using RUW ensures that any database operations are performed atomically whilst DUW enlists DB2 in a full MSDTC transaction with the Message Box. Although this ensures zero data loss it comes at the expense of performance. RUW still ensure no data loss but can cause duplicates in the event of failures and subsequent retries, but exhibits better performance.

Data Integration as a strategy is not without weakness. The principal issue is that any business logic, validation etc in the application itself is effectively bypassed using this approach. To avoid inconsistencies in the database (and consequent destabilisation of the owning application) this logic must be duplicated causing consequent maintenance issues. However, for retrieving data and working with it from other systems, this approach is very attractive.

BizTalk Adapter for Host Files

The second data integration adapter provided is the Microsoft BizTalk Adapter for Host Files. This adapter, as its name suggests, enables access to files in a host environment. A number of different file types are supported on both AS/400 and mainframe. Table 5 lists the most commonly required file types.

Table 5. Supported File Types

File Type Platform Description

VSAM

Mainframe

ESDS, KSDS, RRDS, VRRDS

SAM

Mainframe

QSAM, BSAM

PDS

Mainframe

PDS/PDSE Partitioned Data Sets

Physical

AS/400

File system access

Logical

AS/400

Logical grouping of multiple physical files

The Host File adapter enables full CRUD (Create, Read, Update and Delete) access using a SQL-like syntax, implemented in a new managed provider. When used stand-alone the provider enables SQL to be used to provide a familiar ADO.NET programming model for data access. The adapter supports usage on both send and receive Ports, with receive port processing occurring in a polling fashion similar to the DB2 adapter.

Defining File Structures

There are two constituents to creating a BizTalk solution using the Host File adapter. The first is specific to the host file adapter, a file definition, and the second is the familiar schema generation discussed above in the context of the DB2 adapter. A schema is required to define the file data sent back and forth using XML whilst the file definition is needed to map the file’s contents to XML. Figure 14 shows this relationship in detail. Taking a look at the file definition first, Host Integration Server 2006 introduces a new Microsoft Visual Studio® project type, Host File, enabling the creation of host file definitions. A definition can be created in several ways; manually, through program import or from an existing Host Column Definition (HCD) file. The last option is provided for backward compatibility with HIS 2004 and the Data Access Tool (DAT).

Figure 14. Moving Parts in Host File Adapter

Moving Parts in Host File Adapter

The easiest way to create the definition though is through the import option which allows an RPG/400 or COBOL program to be specified. The Import Wizard then parses the program and creates the field definitions by extracting the data section for RPG and COMMAREA or Linkage Section for COBOL. The wizard is smart enough to handle redefines and multiple occurrences creating arrays and unions as necessary. Figure 15 shows an imported file definition. Even though the wizard does most of the heavy lifting, the definition may be edited afterwards, if required. When the project is saved, a .NET assembly is created containing no code, just the file definition. For use by the adapter the assembly must be GAC’d and registered using REGASM. This ‘metadata’ assembly can then be referenced during the schema creation, to which we turn next.

Figure 15. Host File Definition in Visual Studio

Bb726925.6f4c38ef-dc11-4a4c-bd6e-121d1c73dd83(en-US,BTS.10).gif

By selected “Add Generated Items” and the File Adapter we can now create the schema to use for transferring data between BizTalk Server and the Host. There are three options, updategrams, SQL SELECT or OS/400 Command. Updategrams work in exactly the same way as the DB2 adapter – except for what the nodes represent. The SQL SELECT option allows the host file system to be queried using a SQL-like syntax. An example is shown in Figure 16.

Figure 16. SQL SELECT in File Adapter

SELECT CONTACT,TITLE FROM CUSTOMER WHERE KEY(CUSTID) = ('1')

Sidebar : SQL Syntax for Files

The adapter supports a subset of the SQL syntax. There are four supported verbs:

SELECT, INSERT, UPDATE, DELETE

It is possible to specify a wildcard (*) or individual column names.

A WHERE clause is allowed and supported as follows by specifying each predicate as a key or absolute file position by prefixing each with KEY or POSITION

e.g.

WHERE KEY (KEY_COL) = '1290'
WHERE POSITION BETWEEN 101 AND 201

The final option is OS/400 command enabling commands to be executed on the OS/400. While useful, this option does not currently enable the results of executing a command to be returned to the BizTalk solution from the Host.

Configuring Ports

Once both the schema and metadata file assembly have been created a port can be configured. As already mentioned, the adapter supports both Send and Receive ports allowing File Polling Receive-side and one-way or solicit-response Send-side. Figure 17 shows the Receive Location property page for the File Adapter. The connection string can be provided manually or through the Data Source Wizard as with DB2. There are some additional options for host files, the principal one being the Metadata property, which contains the name of the metadata assembly containing the file definition. This will contain the full path of the assembly which will obviously cause problems if this connection string is copied from environment to environment. Instead, a better strategy is to remove the path, causing the adapter to load the assembly from the GAC.

Figure 17. Receive Location Configuration

Receive Location Configurations

Unlike the DB2 adapter, the Host File adapter does not support batching and returns only a single message for all data returned from a file in one operation. If single record/message processing is required, an envelope schema can be used to split the message up in the receive pipeline processing. This technique may be used whether a Receive Location or solicit-response Send port is being used to return data. The splitting of messages enables parallelism to be achieved allowing BizTalk Server to process multiple records simultaneously rather than having to de-batch them in an orchestration.

The Root element name/namespace are required to create the response messages from the adapter. The purpose of this is to ensure the incoming message (generated by the adapter) will match a particular schema. The SQL Command property enables a SQL query to be specified (see “Sidebar – SQL Syntax For Files,” earlier in this paper). Update Command allows the retrieved records to be updated or deleted as required. The principal use for this is to ensure that the same data is not picked up twice due to the polling nature of the adapter. If update is specified, the values for each field to update on each retrieved record must be specified. The URI property value specified must be unique. Although it is permissible to have two Receive Locations polling the same file, each much be differentiated with unique URIs. This is because the adapter retrieves configuration for the Receive Location by the URI. Lastly, the polling details allow configuration of the periodic polling of the host file.

Send side, Figure 18 shows the options. As you can see the configuration is much simpler, with only the connection string and document namespace/root name required with the unique URI. This is because all the information required for accessing the host file is provided in the message schema. It is to this configuration that we turn next.

Figure 18. Host File Send Port

Host File Send Ports

Although the BizTalk Adapter for Host Files provides a great deal of functionality to process file data through BizTalk Server, there are a couple of limitations that you should be aware of. Firstly, the adapter is not transactional with respect to the Host File system. Although the Host will ensure failure cannot corrupt the file being updated, updates may be lost. Following on from this, the updates within a message sent to the adapter are not performed as a unit of work (UOW) and as such some may complete and some fail.

The second limitation is that the adapter does not support dynamic sends. A dynamic send is where the URI containing transport and endpoint address (e.g. http://www.microsoft.com) is specified at runtime. The adapter only enables the URI details to be specified on a static Send Port and BizTalk Server will not allow the adapter to be selected on Send Ports defined as dynamic in an orchestration. This limitation is partially mitigated by the fact that the message itself can be used to specify what file and operation is performed using dynamic SQL as discussed above.

Application Integration - BizTalk Adapter for Host Applications

The third type of integration - Application Integration, is addressed by the new BizTalk Adapter for Host Applications. This adapter provides the ability to invoke programs running on host environments remotely through BizTalk Server. Specifically, the adapter is capable of calling COBOL programs in an IMS or CICS environment on the mainframe, or RPG/400 program on midrange. It does this through Host Integration Server 2006, which is now available to BizTalk Server 2006 customers as part of their license. Figure 19 shows a common scenario where a Host program is exposed through BizTalk Server as a Web service. The diagram shows both mainframe and AS/400. Although very different Host architectures the adapter hides this complexity, enabling BizTalk solutions to be created without knowing the details. In fact, the only principal difference this diagram highlights is that connectivity is possible over LU6.2 as well as TCP/IP when communicating with the mainframe. Again, in practice, the BizTalk Server developer does not have to be concerned with this.

There are a number of components shown in Figure 19, spanning BizTalk Server, HIS and the Host. Let’s look at each of these in detail to understand how to create such a solution.

Figure 19. Host Application Adapter Usage

Host Adapter Application Usage

We can see that BizTalk Server exposes a Web service through the SOAP adapter, enabling client applications to make calls. BizTalk Server through orchestration or pipeline processing can then turn the incoming message into a request to the Host Application adapter. We shall look at how to do this in a moment. Next, the adapter loads the HIS-specific assemblies to make the call to the Host – this is the ‘TI Runtime’ box in the diagram. The adapter can then invoke a Host program in the data center which is termed the remote environment. The Host program is passed the XML message from BizTalk Server which HIS has turned into parameters to the program. In the same way, output can be returned from the Host which the adapter will create an XML message from to feed back in to BizTalk Server.

Transaction Integrator

Transaction Integrator (TI) is a component of HIS that allows Host programs to be called from .NET applications, including BizTalk Server. There are three pieces to TI.

  • Design time
  • Deploy time
  • Run time

We’ve just encountered the TI Runtime in Figure 19. This is the TI component that dispatches calls from a Windows application to a Host program – termed Windows Initiated Processing (WIP). This contrasts with the alternative invocation model we shall look at later – Host Initiated Processing, or HIP, allowing communication to be initiated in the opposite direction, with the Host calling down to a Windows application or component. The TI Runtime is also responsible for character encoding and protocol conversion from TCP/IP to LU6.2 if required. All of this is invisible to the BizTalk Server developer.

Figure 20. Transaction Integrator Project

Transaction Integrator Projects

The Design-Time component of TI is a new Visual Studio 2005 project type – Transaction Integrator Project. Figure 20 shows an example of this. It provides a TI Designer which enables the creation of Host program definitions and schema. The definitions are stored in a metadata assembly, similar to the Host File project type. The schema is created from the definition and is saved as an XSD file. The schema defines the XML messages that are passed between BizTalk Server and the adapter while the metadata defines how this message is converted into a Host call.

The Deploy-Time component of TI is the configuration of the Remote Environment definition (the Host environment) and the metadata assembly deployment to that environment. Deployment is managed through an MMC snap-in – TI Manager, found under Start -> Programs -> Host Integration Server 2006. TI Manager essentially provides the bridge between the Windows and IBM Host worlds tying the TI Definitions to the Host programs themselves at runtime.

The TI Designer

Using the TI Designer, Host program definitions can be created manually or automatically through import. The import process takes the provided COBOL or RPG program and attempts to extract the callable definition and create a TI Interface from it. This is illustrated in Figure 21. Depending on the Host environment and the call type (see Table 6), the import wizard will look for the necessary program statements. If the wizard is unable to do this, or if the definition generated is incomplete it can be edited through the Designer. Having said that, you should find the Import Wizard saves a lot of typing as it automatically handles such things as unions (REDEFINES in COBOL, OVERLAY in RPG) and arrays (OCCURS) as well as DataTables and DataSets. When the project is saved, a schema file (XSD) is created and saved alongside the metadata assembly.

Figure 21. Host Program Mapping

Host Program Mapping

Metadata

A metadata assembly consists of at least one Library definition. A library definition contains a single interface on which methods can be defined. The methods are used to represent the Host program to invoke and the parameters to pass. In addition to the interface, DataTables, Structures and Unions may be defined. These may then be used in the method parameter definitions to specify their type. A library may be either client or server and managed (.NET) or unmanaged (COM). Unmanaged support is provided principally for backward compatibility. The difference between a client and server library is in how they are used. A client library is intended for use by a Windows client calling the Host environment. In our case, this is the BizTalk Server adapter. A server library is used for HIP calls from the Host to Windows. An additional file with a TIM extension is created for this type. We will look at how this is used when we cover HIP later. The client library definition also specifies the Host environment (CICS, IMS or OS400) and the programming model. Table 6 lists the different programming models available. Essentially, the programming model dictates the mechanism used to call the program and pass parameter values to it. If it is intended to use the assembly from the Host Application adapter, the library should be marked with the “BizTalk Adapter for Host Applications” type restriction as shown in Figure 22. This prevents the Designer creating schemas with types incompatible with the BizTalk Server schema designer.

Figure 22. Library Definition

Library Definitions

Once the TI metadata assembly has been saved it can be used in a BizTalk solution. There are two steps to this:

1.Deploy the TI metadata assembly (can be done after, or during, BizTalk solution deployment)

2.Use the associated schema in a BizTalk solution with the Host Application Adapter

Deployment is achieved in one of two ways, using rapid deployment or the TI Manager. Rapid Deployment is a new TI feature enabling deployment of TI assemblies directly from Visual Studio. A library can be deployed (or undeployed) by right-clicking it in the TI Designer and selecting Deploy. This improves developer productivity by enabling them to work purely in Visual Studio. For server deployments, TI Manager must be used. TI Manager requires two things, a Host Remote Environment and an Object definition. The Remote Environment specifies the type and connection details of the Host environment itself (IP address, CICS, etc) while the Object specifies the Metadata assembly to add to that environment. In addition, the hosting model for the assembly must be specified as part of the Object Wizard definition. The BizTalk Adapter for Host Applications uses the TI self-hosting model where the TI Runtime is hosted by the adapter. The alternative is to host the assembly via IIS where it can be accessed through .NET Remoting or a Web service – this is not required when used only by the adapter.

Once the assembly has been configured in TI Manager the methods defined in it can be called just like regular .NET method calls. The adapter achieves this by converting the XML messages from BizTalk Server into these method calls. It is these XML messages to which we now turn our attention.

Table 6. Programming Models

Environment Model

CICS

Direct Program Link (DPL)

CICS

User model (TRM)

IMS

Connect

IMS

Implicit

IMS

Explicit

IMS

User Data

OS400

Distributed Program Call (DPC)

TI and BizTalk Server

The XSD generated from the TI Definition can be used directly in a BizTalk solution to create messages. The schema contains two types, one each for the request and response. The names of these types conform to the following scheme:

  • •TILibrary__Interface__Method__Type

This consists of four parts, Library, Interface, Method and Type - Request or Response; delimited by a double underscore. The Library is the name of the assembly, while the rest is self-explanatory. This information is used by the adapter to fabricate the call to the Host. The request and response contain two types, the first containing the parameters, the second the client context. We’ll get to what client context is shortly. The parameters simply reflect what was specified in the TI Designer. There are two main ways to create the message in BizTalk Server, via a Map or a pre-configured document. That document could come from a file, embedded resource, configuration or even code for example. A Map allows a new message to be created with dynamic parameters relatively easily.

Figure 23. BizTalk Adapter for Host Applications Send Port Configuration

Send Port Configuration

The final part of the BizTalk solution is to configure the ports used. Figure 23 shows the Host Application adapter Send Port. Interestingly, there is no Receive Location configuration possible as the adapter does not support receiving calls from the Host – we’ll look at how to make this possible later. There are several properties to configure on the Send Port. The first is Allow Advanced Overrides, enabling you to change values in the TI Client Context. Mentioned earlier, Client Context is similar in concept to the BizTalk Message Context as it is additional information passed with the call to the Host through the adapter.

Client Context

The schema provides the node, TIClientContext, allowing context values to be passed. There are several Client Context properties supported by the adapter, listed in Table 7. However, only two are exposed via the messages passed to the adapter as attributes on the root node - the user’s Host credentials to use for the call and the connection timeout value. The Allow Security Overrides property works in conjunction with Client Context and allows the User and Password to be taken from the TIClientContext node in the message and used to make the Host program call. In order to access the others, a property schema must be created to add them to the BizTalk Server message context. All of the properties are in the http://microsoft.com/HostApplications/TI/WIP namespace. Once created, any of these properties can be set via the BizTalk Message Context.

Table 7. TI Client Context Properties

Property Description

RecvTimeOut

Time (in secs) TI Runtime will wait for Host response

SendTimeOut

Time (in secs) TI Runtime will wait for send operation

TPNameOverride

CICS Transaction Name override

ProgNameOverride

CICS Program Name override

IMS_LTERM

IMS LTERM override

LibNameOverride

AS/400 Executable Library Name override

OverrideSourceTP

IMS Transaction Program override

PortOverride

Override TCP/IP Port Number for connection

REOverride

Remote Environment Override

The Host Type property enables the specification of the type of host environment to use – essentially the programming model discussed earlier. The Host Type is also used to create the URI which if you recall, must be a unique value. Persistent Connections provides an optimization to allow multiple Host calls over the same connection instead of re-establishing a new connection each time. When persistent connections are used, the RE Override has no effect as the initial RE will always be used on each subsequent call. The SSO Affiliate Application property is covered later when we discuss security. Tracing enables the adapter to write to a BizTalk Server trace log for troubleshooting purposes.

Figure 24. Host Application Message

<ns0:CedarBank__BankingFunctions__GetBalance__Request 
TIAssemblyVersion="1.0" 
xmlns:ns0="http://microsoft.com/HostApplications/TI/WIP">
  <ns0:GetBalanceInDocument>
    <ns0:name>Jane Doe</ns0:name>
    <ns0:ACCNUM>1001</ns0:ACCNUM>
    <ns0:ACCBAL>0.00</ns0:ACCBAL>
  </ns0:GetBalanceInDocument>
  <ns0:TIClientContext TIContextKeyword="ConnectionTimeout"
TIContextValue="60" />
</ns0:CedarBank__BankingFunctions__GetBalance__Request>

Because the message specifies the TI Assembly and method it is possible to dynamically specify which Host program to invoke – provided there is a TI Metadata Assembly that defines it already created and deployed. Using the TI Context it is also possible to dynamically control a number of other Host settings as Table 7 shows. This provides additional flexibility for BizTalk solutions that require messages to be routed based on their content or business rules.

After the BizTalk solution has been created it must be deployed. This is straightforward on a developer computer as it can be performed through Visual Studio, but when deployed to a server another approach must be used. The simplest way to achieve this is using the new BizTalk Server 2006 Application Export capability that generates an MSI file containing all the required artefacts which includes any referenced assemblies. This has the advantage that when TI Metadata assemblies are included they are automatically configured in TI when deployed on the target computer.

The “T” in TI

It is often necessary when dealing with updating distributed systems to perform updates between them atomically. This is to prevent an inconsistent state between two systems in the event of one or other of them failing during an update. The BizTalk Adapter for Host Applications provides this capability, allowing the BizTalk MessageBox and the Host to participate in a distributed two-phase commit (2PC) transaction. Figure 25 shows the details. Here, we can see that the Microsoft Distributed Transaction Coordinator (MS DTC) acts as the arbiter, enlisting both the MessageBox database and Host Mirror Transaction Program. The purpose of this program is to determine if the Host program was invoked successfully or not. The Mirror TP acts as a wrapper, calling the actual TP required – the one specified in the TI Metadata assembly. Should the Host program fail, the HIS Resync Windows service ensures that the transaction is aborted via MS DTC. You should be aware that any resources changed by the Host program prior to failure, such as files or databases are not part of this transaction and as such, changes made will not be rolled back. The transaction simply provides the capability to reliably ensure that the outcome of the Host program is known to enable changes to the other resources enlisted through MS DTC to be committed or rolled back. Finally, it should be noted that 2PC transactions are only supported over LU6.2 on the mainframe.

Figure 25. Host Application Adapter Architecture

Adapter Architecture

An issue with developing applications using the Host Application adapter is testing. Often a Host connection or even environment may not be available. HIS 2006 provides an application called SImHost.exe (located in Program Files\Host Integration Server 2006\System) to alleviate this problem. SimHost allows the simulation of Host calls from BizTalk Server and can even return results back to the adapter. Samples are provided in the SDK and you can also use the record and playback features of TI for recording your own Host calls to enable offline testing to be performed as well.

Security

One issue that I have avoided until now with all of the adapters is security. This is because it’s easier to deal with it as a single topic as it affects all the adapters. Specifically, the issue we need to deal with is how to access remote resources, be they data or programs, in a secure manner. All of the adapters we have looked at integrate with SSO, differing only in how they are configured to achieve this. As you may know port configuration is stored in SSO (backed by a SQL Server database) and is considered a secure credentials store because it encrypts all credentials and requires an access key using symmetric encryption to decrypt them. The process is that an IT Pro would deploy a BizTalk application and configure the connection details as required for the target environment prior to commissioning. As neither the DB2 nor File adapters support dynamic sends there is no mechanism other than this to specify the security credentials for the connection.

Enterprise Single Sign-On

One of the features of SSO is to simplify the problem of credential management in distributed integration scenarios. In such solutions, the caller may be required to ‘logon’ to the callee, presenting the required information. In the case of Host systems, typically the caller will be expected to have a user id/password pair (and maybe more) to gain access to applications and data. As well as this, the caller may need additional credentials to access the actual application itself if it has its own independent security model. SSO is based on the idea of tickets. A client can request a ticket from SSO which can later be redeemed to retrieve the stored credentials. The client provides their Windows account information to obtain a ticket and SSO returns the requested details on redemption. Tickets are a useful time and space mechanism. In time, because during long running processes credentials may change and thus a scheme where credentials are presented on redemption that were submitted when the ticket was issued may have become stale. In space, because as encryption is performed using a secret key, a ticket may pass around multiple BizTalk Server hosts without all of them needing to validate the Windows account details each time.

To redeem a ticket, the caller must inform SSO of the system it requires the credentials for. This is known as the affiliate application. There are three affiliate types, individual, group and configuration. The individual type represents each client user having a backend account i.e. a one-to-one relationship between their Windows account and the backend system. The group type allows multiple Windows accounts to map to a single set of backend credentials. Finally, an affiliate may represent not a system, but a configuration store of information required in the course of accessing another system. This provides a useful and secure alternative to holding this information in config files or the Windows registry. During creation, a mapping can be defined between Windows user accounts and Host system account information. The affiliate application name is required by the ticket-redeemer to indicate the system credentials it requires. For a configuration-type affiliate, a unique id is used instead to request access to the stored details.

The disconnection between a ticket being issued and its redemption matches how the adapter architecture is designed in BizTalk Server. It allows a receive-side adapter to request a ticket, providing Windows account details using integrated security. This can be obtained by authenticating the caller, for example via IIS Basic Authentication or other means. On the send-side, the ticket can be redeemed to obtain the system’s credentials. Each issued ticket can be redeemed only once. SSO will actually provide the redeemer with the user and password information (plus anything else stored) in clear text. It is therefore important that this is treated with care by the recipient to ensure that it is not persisted or present in memory any longer than necessary to avoid creating a security vulnerability.

Figure 26. Port Configuration for File and DB2 Adapters

Port Configuration for File and DB2 Adapters

Configuring Adapters for SSO

As I mentioned earlier, the adapters we’ve covered in this paper differ in how they are configured to use SSO. The MQ Client and Host Application adapters allow the affiliate name to be specified in the Send Port configuration whilst this information is provided in the connection string for the Host File and DB2 adapters. The Data Source Wizard which launches when creating a new connection during port configuration takes care of this as can be seen in Figure 26. In this way the Host credentials can be retrieved and used when establishing a connection to access a resource. For this to work, the receive-side must create the ticket to be redeemed either by another adapter that supports this or in code using the SSO API. Note that for all Host adapters SSO support is provided only on the send-side. This makes sense, as tickets can only be created against Windows-authenticated resources such as IIS because a Windows account is required for the mapping in SSO.

For the Host Application adapter, configuration is possible in two places, TI Manager and the BizTalk Server Send Port. This is because TI is not dependent on BizTalk Server and as such it is possible to use SSO with TI Assemblies directly rather than hosted by the adapter. To cater for this there are several options. The first is to configure the SSO Affiliate application to use in the RE settings as shown in the left-hand dialog in Figure 27. In this case, when the port is configured in BizTalk Server, the SSO Affiliate application should be set to use the RE setting as shown in the right-hand dialog. The alternative is to provide the Affiliate name in the Send Port which will override the one configured for the RE if specified there also.

Figure 27. Configuring Host Application SSO Via TI and BizTalk Server

SSO via T1 and BTS

Going It Alone

Despite the comprehensive range of options now available for Host connectivity, certain requirements necessitate the need for custom development. Introduced in Host Integration Server 2004 and improved in 2006, Host Initiated Processing (HIP) enables a Host program to call a Windows server as a peer and execute managed code. The HIP architecture is shown in Figure 28. Unfortunately, no HIP support is provided by the Host Adapters for BizTalk Server (remember the Host Application Adapter is Solicit-Response only). Here we’ll look at a simple receive adapter that enables this scenario, allowing a Host program to invoke a BizTalk Server orchestration. Writing a BizTalk Adapter is fairly straightforward, but there are a certain number of things to bear in mind as we shall see. I have provided full sample code with this article that you can use to experiment.

Figure 28. HIP Architecture

HIP Architecture

Creating a Custom Adapter

For our adapter, we will make use of the sample Cedar Bank application which ships as part of the HIS 2006 SDK. Cedar Bank is a Host-based banking application with functionality to retrieve a customer’s balance or a list of accounts. A sample Host program is also provided to perform Host to Windows communication over TCP/IP. The approach we’ll take is to create a receive adapter that can be used in a BizTalk solution such that requests from the Host can be received and submitted to the MessageBox database. We will implement a BizTalk solution that uses the adapter to receive a balance request, retrieve the customer details (from the Host) and return them. Rather than building the adapter up from scratch, the Base Adapter in the SDK (c:\\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Adapters\Common\BaseAdapter) is provided as a set of classes and interfaces to help construct new adapters. Most of the adapters we’ve been discussing so far are actually built on top of the Base Adapter. The Base Adapter is built on the BizTalk Adapter Framework – itself an API for implementing adapters. The classes are arranged hierarchically, providing standard reusable implementations of the BizTalk Adapter Framework interfaces in a similar way to the .NET BCL.

The Base Adapter

The first thing to do is compile the SDK’s Base Adapter as we will be referencing this from our solution. In order to configure a Receive Location to use our adapter we must implement IAdapterConfig which allows us to return a schema detailing any custom properties we need. The Configuration schema must always contain a uri property (it is case-sensitive). Recall from earlier that each adapter configuration on a Receive Location or Send Port must have a unique URI – it is the key used by the adapter to retrieve its configuration at runtime.

Receive adapters can be in-process or isolated. An in-process receive adapter is created when the first receive port is started that uses one. In a configured system this will be when the BizTalk Server service itself is started. This is because a receive adapter needs to be created by BizTalk Server in order for it to pass the adapter enough configuration for it to bootstrap itself. This includes a transport proxy, allowing it to submit messages. The choice between an isolated or in-process receive adapter is dependent on how requests are expected to arrive. Because in-process adapters are created when the host instance is started they suit polling scenarios such as watching file locations, queues or other resources for example. Isolated adapters are hosted in a separate listener process responsible for processing requests and submitting messages to BizTalk Server. As we require a synchronous request/response model where Host call is received via TI and a dispatched to a specified method we need to implement an isolated adapter.

In order to submit messages via the Transport Proxy, the adapter must register itself passing the URI for each endpoint it is servicing. An endpoint in this case represents a Receive Location. The URI for our adapter is obtained from the name of the TI Metadata

Hosting Model

As Figure 28 shows, HIP runs as a Windows service listening for Host calls. HIP works by calling the specified interface method on a configured class in TI Manager when a Host request is received. This method is in our adapter which will turn the parameters passed into an XML message. The message is added to a new batch and submitted to the Transport Proxy. A delegate is passed with the batch to receive a callback with the batch status. If there is no problem with the message the adapter will create a WaitHandle and when the batch has been processed (and the balance retrieved) the EPM will call back to the adapter’s TransmitMessage method. This method will take the XML message received, extract the balance and return it to the Host. It’s worth noting that although we are using batching we only ever add a single message (for the received Host call) per batch.

In order to make use of an adapter from a port definition is must be registered. Registration involves adding certain keys and values to the Windows registry to provide the required configuration information. The easiest way to do this is to use the Adapter Registry Wizard in the SDK – (C:\\Program Files\Microsoft BizTalk Server 2006\SDK\Utilities\AdapterRegistryWizard). This creates a registry file that can be imported into the target computer’s registry as part of solution deployment. The adapter must then be added to a BizTalk group using the BizTalk Server Administration MMC snap-in, under the Host/Platform Settings/Adapters node. The name of the adapter and an optional description are provided here. Now we’re ready to use it in a BizTalk solution!

The BizTalk solution will obtain the customer’s bank balance and return it to the adapter via an orchestration. This uses the Host Application adapter to retrieve the balance from the Host (or SImHost) and return it to our caller. To call a .NET method from the Host we must create a TI Metadata assembly Server library using the TI Designer in Visual Studio discussed earlier. This library specifies the host type (mainframe or OS/400) and the interface we must implement in the adapter (IBankingFunctions). Methods can then be added to the interface which we can implement. For our purposes, there is only one method GetBalance, which will return the customer balance to the Host. Once this server library is created it must be registered with TI through the TI Manager. This is accomplished by expanding the Objects node under Host-Initiated Processing. A wizard requires the name of the accompanying TIM file and the .NET assembly that implements our defined interface. In this case, it is the adapter itself. A method to call on this interface, GetBalance, is also specified. In this way, when a Host call is received, TI can determine where to send it. An interesting question arises – what is hosting the adapter? The answer of course, is the process that loaded our assembly – the HIP Service process.

Putting It All Together

To run the whole solution we need to configure the BizTalk Server ports and TI Manager for HIP. We looked at the Send Port configuration for the Host Application Adapter so I won’t cover it again here. For our HIP adapter we must create a Receive Port with a Receive Location configured with a URI matching the name of the interface registered with TI Manager – see Figure 29. At runtime, the adapter passes this type name to the Transport Proxy to register itself. This enables us to reuse the adapter with different interfaces (i.e. different host method calls) by simply providing a different calling class that replaces the CedarBankHIP class. For testing the solution without a Host connection I have provided a simple console application that simulates the HIP call. It creates an instance of the CedarBankHIP class in the adapter and calls the cedrbank method passing the required parameters. This is the same method that TI is configured with using a real host.

Figure 29. Receive Location Configuration

Receive Location Configuration

Conclusion

The BizTalk Adapters for Host Applications offer a set of tools to provide a new level of ease in creating rich and deep Host integration solutions in many common scenarios. Importantly, this is possible without the BizTalk Server developer requiring detailed Host knowledge or experience. More than this, by providing Host Integration Server 2006 to licensed BizTalk Server customers the full features of HIS can be used to augment the adapters in creating Host solutions in .NET - including adapters as we have seen. I encourage you to experiment with this technology to see how you can apply it in your own organization to unlock the potential these adapters have.

About the author

Jon Fancey is a co-founder of Affinus, a UK-based software consultancy specializing in Microsoft and other technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. As a technical staff member at Pluralsight, Jon also teaches courses on BizTalk Server 2006 R2 and BizTalk Server 2006. You can contact him via his blog at http://go.microsoft.com/fwlink/?LinkId=96586.

References

  • BizTalk Adapters for Host Systems download at http://go.microsoft.com/fwlink/?LinkId=96762
  • “Microsoft BizTalk Adapters for Host Systems” white paper at http://go.microsoft.com/fwlink/?LinkId=98340
    With Microsoft BizTalk Server 2006 adapters for host systems, you can integrate BizTalk solutions with existing line-of-business programs and data on IBM mainframe zSeries and midrange iSeries computers, extending investments in legacy systems to modern SOA and BPM applications.
  • “Leveraging the Value of Your Host Systems with Real World SOA” at http://go.microsoft.com/fwlink/?LinkId=98342
    This document walks you through an example of implementing a real-world SOA by using Microsoft BizTalk Server 2006 and the BizTalk Adapters for Host Systems.
  • “Legacy Modernization with BizTalk Server 2006 and Host Integration Server 2006: A Scenario-Based Guide” at http://go.microsoft.com/fwlink/?LinkId=98343
    Read details about the tasks associated with exposing functionality on IBM mainframe (z/OS) and AS/400 (iSeries) platforms. DB2 and VSAM host file access, initiating IBM transaction programs, and terminal-based applications will be exposed to the Windows platform by using Host Integration Server and the BizTalk Adapters for Host Systems.
  • Supported versions of IBM WebSphere MQ Client software – available from IBM
    • V5.3 with Fix Pack 12 or
    • V6.0 with Fix Pack 6.0.1.1
  • Supported versions of IBM DB2
    • For SNA communication use: V7 or V8 for z/OS and V5R2 for OS/400 and V5R3 or V5R4 for i5/OS
    • For TCP/IP communication use: V8.2 or V9. Both supported for connection to DB2 UDB on Windows, AIX, Solaris, Linux

For more information about IBM platform interoperability, see the Host Integration Server site at http://go.microsoft.com/fwlink/?LinkId=96763. Here you can also learn more about the technologies on which the BizTalk Server 2006 adapters for host systems are based. Explore even more host interoperability options that come with the standalone product.

Copyright

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2007 Microsoft Corporation. All rights reserved.

Microsoft, BizTalk, Visual Studio, Windows, and Windows Server are trademarks of the Microsoft group of companies.

All other trademarks are property of their respective owners.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft