| uilding business-to-business (B2B) e-commerce systems presents many challenges to the system architect. Often, each company involved stores their data and documents in formats that are different from the formats used by the other participating companies. These companies need a way to integrate what tend to be vastly different systems.|
MicrosoftÂ® BizTalkâ¢ Server 2000 can help organizations quickly establish and manage Internet relationships with other organizations. It makes it possible for them to automate document interchange with any other partner organization, regardless of the conversion requirements and data formats used. This provides a cost-effective approach for integrating business processes across enterprises. Figure 1 describes some of the scenarios in which BizTalk Server 2000 can be used effectively. In this article weï¿½ll describe the important architectural concepts and features of BizTalk Server 2000 that youï¿½ll need to be familiar with before you can implement your own BizTalk Server 2000-based solution.
BizTalk Server 2000 is a data and business process integration server designed to facilitate collaborative e-commerce business processes. The server is built on industry-standard XML technology. It includes a document interchange engine, a business process execution engine, and a set of business document and server management tools. A business document editor and mapper are provided, in addition to useful tools for managing trading partner relationships, administering server clusters, and tracking transactions.
At runtime, BizTalk Server 2000 is a scalable engine for validating business documents, translating data formats, transforming schema, transporting documents, and tracking transactions. Supported data formats include UN/EDIFACT and X12 Electronic Data Interchange (EDI), text delimited, positionally delimited flat files, and XML. Transport protocols supported include HTTP, Secure Sockets Layer (SSL), Microsoft Message Queue (MSMQ), FTP, DCOM, and SMTP.
During the design and development stage, BizTalk Server 2000 provides an enterprise application integration framework for integrating third-party and custom applications. This is achieved by defining COM interfaces for application integration components and by providing out-of-the-box COM components for transport and security services.
The ultimate goal of BizTalk Server 2000 is to integrate trading partners as part of collaborative business processes. Here, an organization represents an endpoint in a document exchange. An organization can have properties associated with it including names and unique corporate identifiers, such as a corporationï¿½s Dun & Bradstreet number. Distribution lists allow the same document to be routed to multiple organizations as part of a single agreement. An agreement is a set of rules that will govern that distribution. Organization profiles may also be exported to facilitate the exchange of trading partner information.
BizTalk Server Basics An agreement defines the rules for exchanging data between two or more organizations. They define the source and destination organizations, at least one document specification, document envelopes, security settings, and transport settings. Basically, the flow of data into and out of BizTalk Server 2000 must be governed by an agreement. Figure 2 shows a typical agreement for a purchase order.
For example, say ESitesRUs, a fictitious online retailer, accepts orders from Web customers and sends purchase orders to Worldwide Importers Inc. for processing. Upon receiving the PO and fulfilling the order, Worldwide Importers Inc. sends ESitesRUs a shipment notification. ESitesRUs integrates the shipment status information into their online customer service site. To configure this traditional fulfillment transaction on BizTalk Server 2000, you create two agreements: an outbound agreement, which defines the rules for sending purchase orders, and an inbound agreement, which defines the rules for accepting shipment notifications.
An outbound agreement defines ESitesRUs as the source organization and Worldwide Importers as the destination organization. This agreement also includes the documentation specifications for the purchase order as well as security and transport settings for the data exchange. The inbound agreement defines Worldwide Importers Inc. as the source organization and ESitesRUs as the destination organization. The shipment notification document specification and security and transport settings are also defined in this agreement.
A third type of agreementâ"an open agreementâ"is a special type that defines a single organization. The missing trading partner information is provided when the document is submitted to BizTalk Server 2000.
Business documents are the fundamental unit of data exchange between trading partners in a BizTalk Server 2000-based solution. A business documentï¿½s physical format may be XML, EDI, or comma-separated values, among others. Logically, they are simply collections of records and fields. Business analysts must review schema initiatives in their industries, perform gap analysis between published specifications and internal business requirements, and negotiate details with trading partners. Industry efforts are underway to ease this process by publishing schema libraries and providing tools for collaboratively defining and refining these schemas. Examples include BizTalk.org (http://www.biztalk.org), OASIS, and RosettaNet.
|Figure 3 Document Specification in BizTalk Editor |
BizTalk Server 2000 treats all document types as XML regardless of the original format. Document specifications are required to define the translation from the original document format to the BizTalk Server 2000 intermediate XML format. Figure 3 shows an example of a document specification in the BizTalk Editor. Figure 4 shows the XML source code for the document specification as generated by the editor. After the document exists in the internal XML format, it will have to be transformed. To perform schema transformations, the XML transformations can be defined through document maps.
|Figure 4 Document Specification Source XDR |
Envelopes are EDI, text, or XML data structures that provide a way to add header information, such as addressing and document identifiers, to business documents. The server also uses envelopes to support processing of inbound and outbound EDI interchanges. Incidentally, the server supports batch processing of inbound EDI, but handles outbound EDI one transaction at a time.
Document maps are typically used to alter the schema or data format of a business document from a source organizationï¿½s native representation to a representation requested by a destination organization. For example, a companyï¿½s business application may be able to produce XML documents, but their trading partnerï¿½s application may require a SAP IDOC or an EDI document. Or perhaps each partner has simply used different XML document structures to define a purchase order.
Pipelines tie together the built-in or custom processing steps during a data interchange. They allow a developer to customize many aspects of an agreement. Pipelines are used to link the document definitions for an outbound agreement or distribution list to a document definition of an inbound agreement. Agreements help the server identify the appropriate pipeline to run, and then provide important document processing rules to the pipeline.
A concept known as pipeline filtering is configured within BizTalk Server 2000 pipelines. This feature allows documents to be processed based on values in user-specified fields of a document. Tracking or storing subsets of documents or entire source or destination documents is also a feature supported within pipelines. Alternatively, document tracking settings may be specified in the document specification (as weï¿½ll discuss later). Specifying these settings in the document specification instead of in a pipeline allows the same settings to be used in multiple pipelines.
Digital certificate identification and processing rules are also defined within a pipeline. Finally, pipelines allow the analyst to select the map for creating a destination document from the source document submitted to BizTalk Server 2000.
Custom components called application integration components (AIC), or simply pipeline components, may also be inserted into a pipeline. AICs allow the line-of-business application developer to easily integrate existing third-party or custom applications with BizTalk Server 2000-brokered transactions by developing pipeline components that implement a set of COM interfaces. BizTalk Server 2000 invokes methods on the IPipelineComponent and related interfaces to access extended functionality during the execution of interchange agreements. Pipeline components frequently implement custom transport services.
There are numerous ways to exchange data between servers on the Internet, and BizTalk Server 2000 supports many standard protocols by encapsulating HTTP, SMTP, MSMQ, FTP, file, and DCOM functionality provided by WindowsÂ® 2000. The server features that support these protocols are called Transport Services and Receive Functions. Transport Services are used for sending business data from BizTalk Server 2000, while Receive Functions allow applications to submit business documents to BizTalk Server 2000 without any programming.
BizTalk Server 2000 supports the following transport services: HTTP, HTTPS, SMTP, MSMQ, FTP, file, and fax. Additionally, documents may be sent programmatically from BizTalk Server 2000 with a custom COM AIC.
FTP, file, and message queuing are supported receive functions. The FTP receive function polls a given location and uses FTP to send files to BizTalk Server 2000. The file receive function is invoked by a file system event when activity occurs in the defined directory. The source file is copied and submitted to BizTalk Server 2000. The message queuing receive function provides event-based integration with MSMQ to read messages from a queue and submit the message body to BizTalk Server 2000. Documents may also be programmatically submitted, either synchronously or asynchronously, through COM interfaces. An ASP receive function may also be easily implemented by creating an ASP page to which partners will use HTTP to post business data. In the ASP, just submit the received HTML form data to BizTalk Server 2000 by calling the serverï¿½s COM interface. Thatï¿½s all there is to it.
MIME is supported as a content encoding type, while S/MIME and PKCS are supported for data encryption and digital signatures. Custom encoding, security, and transport logic may also be plugged directly into BizTalk Server 2000. This encoding architecture conveniently allows data integration to be independent of the network protocol used to communicate with trading partners.
An interchange protocol is a set of rules each trading partner must adhere to in a business document exchange. This protocol may define characteristics such as acknowledgment requirements, retry and timeout thresholds, and performance requirements. Although BizTalk Server 2000 does not directly implement the concept of an interchange protocol, it is an important concept to negotiate among trading partners and to implement using the built-in components and extensibility model provided by the server.
Architecture and Tools The BizTalk Server 2000 architecture uses Windows 2000 Server and Microsoft SQL Serverâ¢ 2000 to implement a horizontally scalable, reliable, and extensible document interchange engine. Figure 5 provides a system-level view of a BizTalk Server 2000 deployment.
|Figure 5 BizTalk Server 2000 Deployment |
A standard deployment may include one or more BizTalk Server 2000 servers configured as a group. Each server will process documents independently of the others to provide horizontal scaling and fault tolerance without a separate clustering solution. Server groups share document specifications and maps through the Windows 2000 Web Distributed Authoring and Versioning (WebDAV) service. They share working interchange data by accessing a set of shared SQL Server queues.
Four SQL Server queues are used during the processing of a document: scheduled queue, work queue, retry queue, and suspended queue. Documents are retrieved from the scheduled queue by the first available BizTalk server and placed on the work queue for processing. The scheduled queue ensures high server throughput by providing server resource management. If a necessary service is unavailable to process a document, the document will be placed onto the scheduled queue so the servers can proceed to another interchange. Documents that fail due to processing errors such as network or validation errors are placed on the suspended queue or retry queue for later attempts. Microsoft Cluster Server may be used in combination with SQL Server to provide scalable and fault-tolerant deployment for the BizTalk Server 2000 shared queues.
BizTalk Server 2000 relies on many Windows 2000 Server platform services. Internet Information Services (IIS) 5.0 and the Network Load Balancing Service provide the components necessary for scalable HTTP and ASP processing. Cluster Service provides a highly available deployment architecture to the shared SQL Server databases. COM+ component services and MSMQ provide DCOM-based integration and MSMQ receive functions and transports. The built-in WebDAV service provides distributed access to document schemas and maps.
BizTalk Server 2000 also uses the Microsoft XML toolset, which includes a validating XML parser that supports XML Data Reduced (XDR) Schemas and Document Type Definitions (DTD) as well as Extensible Stylesheet Language (XSL) and XSL Transformation Language (XSLT). Of course, the server also gains the increased scalability, reliability, and availability that is inherent in Windows 2000 Server.
Document Processing Submitting a business document to BizTalk Server 2000 triggers a series of logical processing steps by the interchange engine: agreement route, parse, validate, transform, serialize, and transport.
BizTalk Server 2000 receives business data from business applications in one of two ways. A document may be submitted directly by a COM-aware application through the IInterchange interface, or indirectly by a file, FTP, MSMQ, or custom receive function. Hereï¿½s an example of direct COM application submission:
The Submit method takes arguments that describe the agreement, the document being submitted, source and destination organizations, custom pipelines, or enveloping information. It then returns a handle to the submitted document. This handle allows the developer to identify this document within queues or the document tracking and activity database.
strDocument = "<CommonPO/>"
Set objInterchange = _
strDocument, "PurchaseOrder", _
"Source Organization Name", _
"Destination Organization Name", _
"DestOrg", [pipeline name], _
[file path], [envelope name], _
Business data may be self-routing or the routing information may be placed explicitly in the COM method call. Routing information may also be described to the server in the configuration of a custom receive function. The server uses this routing information to identify the correct interchange agreement used to process the document.
Business data is submitted to BizTalk Server 2000 as text. As described earlier, this text may be any file format as long as a BizTalk document specification can been created to describe its structure and format. The server ships with parsers for well-formed XML, UN/EDIFACT EDI, X12 EDI, text, or positionally delimited flat files. It also allows for third-party development of new parsers. The server selects the appropriate parser based on the envelope specified by the agreement.
Regardless of the original data format, the parser looks up the document specification defined in the agreement, loads it using the WebDAV protocol, and uses it to create intermediate XML representations of the submitted business data. The server performs internal document processing using this intermediate XML representation prior to serializing the data into the final destination document format. If the parsing step fails, the document will be placed in the server groupï¿½s suspended queue.
Once the server has created the XML representation of the business data, it will validate the structure and grammar of the document instance using rules provided by the document specification. Since all BizTalk Server 2000 agreements require document specifications, the validation will occur for both XML and non-XML business data. Invisible to the developer or analyst, the validation is implemented within BizTalk Server 2000 using standard XML technology. The intermediate XML representation of the business document created during the serverï¿½s parsing step is simply an instance of the XDR document specification. This means that the server can use the validating Microsoft XML parser. In cases where XDR validation is not sufficientâ"such as with X12 and UN/EDIFACT documentsâ"the server validation engine processes a set of extended attributes that correspond to special validation rules.
Validation errors cause the document to be flagged as invalid in the document tracking database and work queues. Since server queues are implemented in a SQL Server database, they can be processed by the developer through published object models or by direct access using Transact-SQL stored procedures.
Transformation is the process of executing the maps created with the BizTalk Mapper (see Figure 6). The server loads the map defined in the agreementï¿½s pipeline configuration and applies the transformation to the document. The maps move the data in the fields and records of a source document instance into the fields and records of a new destination document instance. The business data is in the serverï¿½s private intermediate XML representation before and after this logical processing step.
The serialization process is the opposite of the parsing process. The internal XML representation of a document is serialized to the format defined by the document specification. This format may be well-formed XML, or it may be UN/EDIFACT EDI, X12 EDI, text, or positionally delimited flat files. This serialized format is the document routed to the destination organization specified by the agreement.
After the business data has been transformed into the appropriate format for the destination organization, the transport service defined by the agreement (HTTP, HTTP/S, SMTP, MSMQ, FTP, file, or fax) is selected and the data is sent to the destination location specified in the agreement. If the transport fails, the document will be placed in the retry queue if its count is nonzero; otherwise it will be placed in the suspended queue.
Weï¿½ve briefly mentioned the serverï¿½s method of handling document-processing errors. Much of this is based on moving the erroneous document to the appropriate queue and on providing an object model (and T-SQL scripts) to query the SQL Server queues. In addition, the developer can make use of server-generated Windows Management Instrumentation (WMI) events to handle exception processing. Another technique is to employ stored procedures to analyze document queues.
Business-to-business e-commerce transactions must be secure whether the server is processing purchase transactions for a Fortune 500 enterprise or fulfilling orders for a startup internet company. BizTalk Server 2000 addresses the issue of security by giving the developer a number of out-of-the-box authentication and encryption components that take advantage of security services in Windows 2000. SSL support is provided through the built-in HTTPS transport service. This adds server-to-server authentication and transport layer encryption to the document interchange. Documents may also be encoded using built-in S/MIME encoding components, ensuring document integrity, authentication of the sending party, and payload encryption. The Public Key Cryptography System (PKCS) for encrypting and decrypting document payloads is also supported. Finally, digital signatures may be applied to outbound messages and verified on inbound messages using the BizTalk Server 2000 native support for digital signatures.
Most of these techniques require organizations to get X.509 digital certificates from a trusted certificate authority. If none of these techniques meets the demands of an enterpriseï¿½s security policies, the server can be extended through custom security components that make use of the Microsoft Cryptography API as well as new security features in Windows 2000 such as support for smart cards and the Kerberos protocol.
BizTalk Document Editor and Mapper BizTalk Server 2000 represents business document schemas as document specifications. Document specifications define the structure of a business document in a way that is independent of the underlying data format (well-formed XML, UN/EDIFACT EDI, X12 EDI, text, or positionally delimited flat files) using a familiar metaphor based on records and fields. Document specifications define a way to translate between the documentï¿½s original data format and the serverï¿½s internal XML format.
The BizTalk Editor is a graphical tool that allows analysts and developers to create document specifications in a number of ways. A document specification can be created by manually defining records and fields in the editor tool. The server also provides ready-to-use specifications for many UN/EDIFACT and X12 EDI documents, SAP IDOCs, and sample XDR schemas. Finally, the editor provides instance import functionality that allows the user to import well-formed XML document instances, XML DTDs, and XDR schemas. It allows the user to edit and save the resulting BizTalk document specification.
Although it is transparent to the user, BizTalk Server 2000 represents document specifications internally with standard XDR schema technology plus extended attributes to enable server processing of the documents during an interchange. The document specification author can define attributes on data items such as minimum and maximum number of occurrences, whether it is optional, data types, fixed data values, enumerated lists, and more. BizTalk Server 2000 builds on standard XML to provide rich document specification capabilities.
As described earlier, Figure 3 shows a purchase order document specification in the BizTalk Editor, and Figure 4 is a partial listing of the XDR schema produced by the tool, illustrating the definition of the PurchaseOrder ElementType. Once it is finalized by the W3C, the standard XML Schema Definition Language (XSDL) will replace the current XDR syntax.
In addition to the tools provided for working with business document specifications, BizTalk Server 2000 includes the BizTalk Mapper for transforming a document from the internal XML representation of an inbound document to the internal XML representation of an outbound document. This mapping allows BizTalk Server 2000 to alter the schema (transformation) and data format (translation) of business documents. The final XML document is eventually serialized to the format defined by the outbound document specification, which may not always be XML.
BizTalk Mapper provides the design environment and BizTalk Server 2000 provides the runtime engine to create and execute document maps that translate data formats and transform data schemas. The BizTalk Mapperï¿½s use of standard XSLT technology to internally represent mapping rules is transparent to the analyst or developer. The Microsoft XSLT implementation provides COM and scripting language integration. BizTalk Mapper and BizTalk Server 2000 take advantage of this integration to provide built-in reusable components called functoids that may be inserted onto the BizTalk Mapper design surface and called at runtime.
Functoids that ship with BizTalk Server 2000 are grouped into seven categories: String, Mathematical, Logical, Date, Conversion, Scientific, and Advanced. Examples of common String functoids are Substring and upper or lower-case conversions. If a map needs to extract a manufacturerï¿½s part number from a vendorï¿½s 256-character catalog ID, these String functoids may be valuable. Obtaining a timestamp with the Current Date functoid also has clear value in business-to-business document processing. The Advanced functoids category includes the versatile Custom Visual BasicÂ® Script functoid. As the name suggests, this allows the developer to define custom script logic that will be applied to source data values during the execution of a map.
|Figure 7 BizTalk Mapper Compiled XSLT |
Figure 6 shows a map between two different purchase order schemas in the BizTalk Mapper. Figure 7 is a partial listing of the XML and XSLT produced by the tool. It illustrates the primary components of a BizTalk map. <srctree> and <sinktree> contain the document schema for the source and destination documents. <links> describes the graphical mapping. <functions> includes the functoids (or pre-built mapping components), and <CompiledXSL> contains the XSL transformation language required to execute the transformation at runtime.
Management and Analysis Tools There are two types of management tools included in BizTalk Server 2000. The first is the BizTalk Management Desk, which analysts and developers can use to define the important aspects of a trading partner relationship through a graphical console. The second, which system administrators use, is the Microsoft Management Console (MMC) snap-in environment used to configure deployment characteristics of server groups.
The BizTalk Management Desk allows the recreation and configuration of trading partner agreements and all associated properties (organizations, distribution lists, document specifications, envelopes, transport protocols, security settings, and pipelines). Document tracking and activityâ"the level of auditing the server groups will performâ"is also configured from the Management Desk. Figure 8 shows the Agreement Editor within the BizTalk Management Desk.
|Figure 9 BizTalk Server MMC |
Figure 9 shows the left-hand pane of the BizTalk Server 2000 MMC snap-in. Using this console, groups, queue activity, receive functions and individual servers may be managed remotely or locally. This includes adding or removing servers from groups, checking the status of documents on the shared queues, and creating new receive functions. It is often convenient to customize the MMC to include BizTalk Server 2000, IIS, SQL Server, Event Viewer, COM+ Component Services, and MSMQ management snap-ins so you can manage all of a solutionï¿½s components from a single console.
Business Process Integration Most of our discussion has centered on server capabilities to facilitate the exchange of business documents between trading partners. BizTalk Server 2000 will also provide tools and a runtime to facilitate the modeling, development, and execution of business processesâ"where data exchange is only one key aspect. COM+, of course, will provide the component architecture and services for business process implementation. Message sequencing, receipt correlation, content-based routing, and retry logic may easily be designed and implemented using the BizTalk Server 2000 core document interchange coupled with these advanced business process integration features.