Using EDI with Microsoft BizTalk Server 2002
Summary: Learn how Microsoft BizTalk Server can help both large businesses that are currently using EDI and smaller businesses that do not use EDI but that want to trade with larger businesses. (19 printed pages)
Enhancing an EDI Environment by Using BizTalk Server
Enterprise Application Integration
New Relationships with Smaller Trading Partners
Using BizTalk Server in Smaller Companies
Translating and Transforming Documents
Comparing BizTalk Server with EDI Technology
Many companies today use electronic data interchange (EDI) to exchange business documents. This article discusses how Microsoft® BizTalk® Server can help both large businesses that are currently using EDI and smaller businesses that do not use EDI but that want to trade with these larger businesses.
BizTalk Server can help a company that uses EDI in the following ways:
- Enabling enterprise application integration (EAI).
BizTalk Server automates the exchange of internal business data as well as accepting EDI data and automatically integrating it with internal systems.
- Creating new relationships with smaller trading partners.
BizTalk Server provides a cost-effective way to exchange electronic documents with companies that choose not to use EDI.
- Facilitating future growth.
BizTalk Server provides a cost-effective way to handle the expansion of a company's messaging and document interchange requirements.
Smaller companies can benefit from BizTalk Server by using it to establish electronic document exchange relationships with larger companies that use both EDI and XML. A smaller company can also streamline its internal business processes by employing the EAI capabilities of BizTalk Server.
The ability of BizTalk Server to translate and transform documents is central to its EAI and document exchange capabilities. This article introduces BizTalk Editor and BizTalk Mapper, tools that help to direct the translation and transformation of EDI documents and other electronic documents. BizTalk Editor enables you to create and edit specifications (BizTalk Server-specific schemas). BizTalk Mapper uses specifications to map the structure of one document instance to the structure of another document instance.
This article also discusses the relative strengths and weaknesses of BizTalk Server when compared with EDI technology.
Electronic data interchange (EDI) is a set of standards for controlling the exchange of business documents (such as purchase orders and invoices) between computers. Businesses can use EDI to ensure that the documents they exchange are interpreted correctly, regardless of the platforms or internal applications they use. Because EDI enables electronic documents to move from one computer to another without the need for human intervention, it is faster, cheaper, and more accurate than the exchange of paper documents.
Standardization efforts for EDI formats began in the 1960s, and were led by the transportation industry. The need for a uniform standard that encompassed all industries prompted the creation of the Accredited Standards Committee (ASC) X12, sanctioned by the American National Standards Institute (ANSI), in 1979. The Accredited Standards Committee X12 created the EDI standard commonly referred to as X12, which was used primarily for American domestic trade. Meanwhile, the European community developed its own EDI standard called Guidelines on Trade Data Interchange (GTDI). A new standard that borrowed from both X12 and GTDI, called Electronic Data Interchange for Administration, Commerce, and Transport (EDIFACT), was developed at the United Nations. The International Organization for Standardization (ISO) adopted EDIFACT in 1987. Although ASC X12 members approved the adoption of EDIFACT as the universal EDI standard in 1992, X12 continues to be a widely used EDI standard in North America.
Although EDI has been around for nearly forty years, it has not triggered an explosion in business-to-business electronic commerce. In fact, the number of businesses trading electronically today compared to those using phone or fax is limited. The reasons for this are numerous and include the following:
- EDI server systems are typically expensive.
- The EDI document format is somewhat cryptic.
- EDI document transport was historically a value-added network (VAN) that incurred both an expensive setup fee and ongoing operational costs.
Although many companies have long-term strategies that involve replacing their legacy infrastructure, a company that is heavily invested in EDI might not want to immediately discard its investment and replace it entirely with an XML-based BizTalk Server system. However, BizTalk Server can add a great deal of value to a company that chooses to continue using EDI in the short term. Areas where BizTalk Server can enhance the operations of a company that uses EDI include:
- Enabling EAI.
- Creating new relationships with smaller trading partners.
- Facilitating future growth.
Integrating with business partners is only one of the challenges that face businesses today. Equally important is the integration of internal business applications, such as accounting, inventory, and customer relationship management (CRM) systems.
EDI systems do not typically offer EAI infrastructure, and they support only a limited subset of possible electronic document formats other than EDI. One of the strengths of BizTalk Server is its ability to automate and streamline the flow of a company's business data both internally and externally.
The following illustration shows a simplified representation of how a company that uses EDI might use its EDI server to exchange business documents.
Figure 1. Using EDI server to exchange business documents
Organization A's EDI server sends and receives standard EDI documents to and from Organization B over a VAN. Each transaction over the VAN incurs an expense for Organization A. Organization A's EDI server communicates with its accounting, inventory, and CRM applications either by manual data entry or by custom-built software.
The following illustration shows how a BizTalk Server hub can be added to this system to facilitate the integration of Organization A's internal applications.
Figure 2. Adding a BizTalk Server hub
In this scenario, BizTalk Server becomes the hub of Organization A's internal data exchange. The BizTalk Server hub provides an accurate and cost-effective way to automatically update the organization's line-of-business applications when a transaction with Organization B occurs.
The key to the success of this scenario is the ability of BizTalk Server to be the universal message gateway. BizTalk Server can automatically send data to Organization A's internal applications in XML or flat-file format, or even in custom formats with the introduction of custom parsers and serializers. BizTalk Server uses the TCP/IP communication layer built into Microsoft Windows® 2000, which is commonly used for communication between applications in an organization. With BizTalk Server these EAI processes can be put in place at costs far lower than would be possible by paying developers to create custom communications applications. BizTalk Server enables internal data exchange that is far more accurate and efficient than can ever be achieved by manual processes.
BizTalk Server makes it easy for a company currently using EDI with trading partners to also exchange documents with smaller trading partners who cannot afford or do not want to use EDI. The following illustration shows how Organization A can add a new trading partner to its existing communications network.
Figure 3. Adding a new trading partner
Organization A's BizTalk Server hub can electronically exchange business documents with Organization C using TCP/IP over the Internet. These documents can be delivered in a format that is easy for Organization C to use, such as XML. In this scenario, both Organization A and Organization C enjoy the accuracy and efficiency of the automated exchange of electronic business documents. Neither company needs to incur the high costs of setting up a new EDI relationship or the ongoing expense of a VAN.
Organization A might find that it needs to set up new automated messaging with Organization B beyond what it has implemented with its EDI server. Organization A will find that the least expensive and most direct solution is to use its BizTalk Server hub for exchanging these new messages. In this way it bypasses its EDI server altogether. In time, Organization A might need to make more significant changes in its data-exchange relationship with Organization B. This would be an ideal time for Organization A to forgo its EDI server entirely and replace it with the BizTalk Server hub.
Many smaller companies could benefit from the ability to exchange electronic business documents with larger companies that use EDI, but they cannot justify the setup and operational costs associated with traditional EDI servers. BizTalk Server provides a cost-effective solution to this problem (and can also be leveraged with trading partners that use XML and other non-EDI formats). BizTalk Server enables a small company to automatically transform its business documents into an electronic format that an EDI server of a larger company can use. The following illustration shows such a relationship.
Figure 4. EDI documents sent and received by BizTalk Server
Organization D might be so small that it runs its entire business on Microsoft Office, or it might use another business tools suite that can read and write XML documents or provide adapters for BizTalk Server. BizTalk Server can process Organization D's documents and transport them to and from the EDI server of Organization B. Organization D might have BizTalk Server and other business applications installed on a single computer, or it might be a larger company with business applications distributed across several computers. In either case, Organization D's BizTalk Server hub can serve the dual purpose of exchanging EDI documents with Organization B and automatically integrating the flow of internal business data within the company.
A key strength of BizTalk Server is its ability to accept input in a wide variety of document formats, map that input into almost any document structure, and then output the new document structure into a wide variety of document formats. XML is central to the translation and transformation capabilities of BizTalk Server, which is in large part what makes BizTalk Server such a powerful tool for EAI and business-to-business electronic commerce.
The following illustration and accompanying list show how BizTalk Server internally processes a document.
Figure 5. Processing a document
- The incoming document instance is sent to BizTalk Server.
- The parser uses the source specification associated with the incoming document instance to translate the incoming document instance to XML (if it is not already in this format). The source specification is created in BizTalk Editor.
- The XML file is transformed by an Extensible Stylesheet Language Transformation (XSLT) map into another XML file of the specified structure. (Nodes in the incoming XML file are mapped to nodes in the outgoing XML file.) The XSLT map is created in BizTalk Mapper.
- The serializer uses the destination specification associated with the outgoing document instance to translate the outgoing XML file to the outgoing document instance (if it is not already in this format). The destination specification is created in BizTalk Editor.
- BizTalk Server outputs the outgoing document instance and transports it to a destination.
The parsers and serializers included with BizTalk Server can translate XML, EDI (X12 and EDIFACT), and flat files (delimited and positional). Parsers and serializers for other formats might be available in the future. For more information, go to the Microsoft BizTalk Server Web site.
If you create your own parsers and serializers, BizTalk Server can translate files of any format. Regardless of the format of an incoming document instance, BizTalk Server translates it to an XML file so that the XSLT map can transform the incoming document structure into the structure necessary for the outgoing document. Even if a BizTalk Server hub inputs and outputs EDI documents, internally these documents are translated to XML. This enables BizTalk Server to take advantage of the power and flexibility of XML when transforming documents from one structure to another.
BizTalk Editor enables you to create the specifications used by BizTalk Server to translate document formats to and from XML, and to create the maps that transform the translated XML files from one structure to another. The following illustration shows an EDI purchase-order specification based on an X12_4010_850 schema displayed in BizTalk Editor. The hierarchical structure of a document is displayed in the left pane of BizTalk Editor, regardless of whether the document format is XML, EDI, or flat-file. The right pane contains tabs that display property settings for the nodes in the document hierarchy.
Figure 6. BizTalk Editor window
A specification that is based on an industry standard schema, such as the X12_4010_850 schema, is a subset of that standard schema. For example, the specification in the illustration is a subset of the X12_4010_850 schema because the nodes that ordinarily exist between the CUR and ITD nodes have been removed. With BizTalk Editor you can create a new document specification based on a standard X12 or EDIFACT template and remove the nodes that you don't need.
With no coding, you can use BizTalk Mapper to create XSLT maps that BizTalk Server uses to transform the structure of an incoming document instance to the structure of an outgoing document instance. The source specification in a map is associated with the incoming document, and the destination specification is associated with the outgoing document.
The following illustration shows an X12-based purchase-order specification that is mapped to a purchase-order specification with a different structure. This map represents the document transformation from the incoming EDI document to the XML format acceptable for the accounting application in Organization A. The display is format independent—in this case the document displayed in the left pane is an EDI file, while the document displayed in the right pane is an XML file.
Figure 7. BizTalk Mapper window
The illustration shows links from five nodes in the source specification to five corresponding nodes in the destination specification. If you viewed the map in this illustration in BizTalk Mapper, you could see the remainder of the specifications by scrolling and by expanding nodes in the specifications. BizTalk Mapper uses built-in, reusable functions called functoids to enable more complex transformations than the simple links shown here.
BizTalk Mapper has a grid preview function that is useful for navigating to a particular subsection of a complex map, such as might be required when mapping EDI documents. You can also create multiple grid pages that enable you to associate the layers with logical or physical parts of the map. For example, if you were going to map an X12 810 schema representing an invoice, you might want to create grid pages for the Header, Detail, and Summary sections.
As explained earlier in this article, BizTalk Server offers functionality and advantages that EDI technology cannot provide. For anyone currently using EDI who is considering deploying BizTalk Server in their business, it is important to understand the strengths and limitations of both EDI and BizTalk Server.
- Currently deployed in many businesses.
EDI is a long-established standard, and many large businesses currently use it successfully.
- Uses agreed-upon standards.
EDI standards are recognized by everyone who uses EDI.
- Standards are fairly rigid.
Rigid standards require conformity.
- High cost.
EDI systems are costly to set up and maintain. Hiring and retaining EDI experts is expensive.
- Value-added networks (VANs).
Many companies that use EDI use VANs to exchange documents. VANs are expensive to set up and incur costs each time they are used.
- Document format is not easily human-readable.
It is difficult for a person to read an EDI document.
- Not well suited for EAI.
An EDI server handles connections outside the business. BizTalk Server handles connections both outside the business and within the business.
- Many industry-specific subvariations of standard documents.
In some industries, such as the automotive and aerospace industries, EDI document standards have been extended for industry-specific purposes. This can cause document translation difficulties between variants of standard EDI documents given the expectation for rigid standards interpretation.
BizTalk Server Strengths
- Uses XML as a foundation.
BizTalk Server uses XML to translate and transform documents regardless of the document format required for input and output. This creates an extremely flexible environment for document exchange both now and in the future. XML as a document format has the following advantages:
- XML is self-describing and creates documents that are relatively easy for people to read. This makes it easier for a person unfamiliar with a particular BizTalk Server installation to become familiar with it.
- XML experts are plentiful, and they are less expensive to employ than EDI experts.
- XML is very flexible and extensible.
- Easy setup and maintenance.
BizTalk Server systems are easier to set up and maintain than EDI systems.
- Many schemas available.
There is a large and growing library of schemas available to users of BizTalk Server. For more information about this schema library, go to the Microsoft BizTalk Server Web site.
- EAI capabilities.
BizTalk Server handles EAI. For more information about case studies on BizTalk Server for EAI, go to the Microsoft BizTalk Server Web site.
- Orchestration capabilities.
In addition to the universal messaging capabilities described in this article, BizTalk Server has powerful orchestration capabilities. BizTalk Orchestration enables users to design and execute long-running, loosely coupled business transactions.
BizTalk Server Limitations
- Relatively new product.
BizTalk Server 2002 is the second release of the BizTalk platform. Many EDI servers have been available for many years. While BizTalk Server 2000 and BizTalk Server 2002 have proven to be extremely robust and stable in production environments, some of the less-commonly used EDI functionality is not currently available in the product.
Specifics of BizTalk Server Support for EDI Functionality
BizTalk Server supports a wide variety of features common to most EDI software. As of the recent release of BizTalk Server 2002, the following tables describe the most important features of an EDI system and how those features are supported in BizTalk Server. In the "BizTalk Server support" column, the score is effectively a rating or grade on how well the functionality, out-of-the-box, meets what is typically expected of each feature.
Table 1. Category: Trading partner management
|Feature||BizTalk Server support||Comments|
|Multiple organization IDs||100%||BizTalk Server can easily handle many IDs per trading partner/organization, with the only restriction being that no two organizations are allowed to have the same ID (no ambiguity).|
|Variable use of standards within and between trading partners||100%||BizTalk Server allows multiple transaction types, of the same or different versions, between the same two endpoints. In other words, no point-to-point EDI data flow adversely affects any other (see next point, on routing).|
|Schema reuse||90%||Schemas are completely reusable in the sense that one schema for, perhaps, a purchase order, can be used repeatedly for any point-to-point data flow. However, because some routing data is tied to properties called document definitions, which encapsulate the schema, effectively, there is some proliferation of document definitions, all containing the same schema.|
|Run-time, automatic routing||100%||BizTalk Server is completely capable of routing EDI messages based solely upon envelope content, transaction content (for example, purchase order number), or a combination of both.|
|Unknown source and directed routing||100%||Though BizTalk Server is typically configured to automatically route EDI, it is possible to specify and force a particular routing. It is also possible to process all EDI transactions that conform to a particular schema through a single processing channel, even from unknown sources, provided that the previously mentioned specific routing is known at design time.|
|Uptime maintenance||100%||BizTalk Server allows configuration changes and additions while the server is in operation.|
|Role shifting||100%||BizTalk Server allows processing of EDI data either as an involved party to the transaction, or as an intermediary between the two parties in the transaction. By default, and unless otherwise specified, the server assumes the former.|
Table 2. Category: Data/syntax support
|Feature||BizTalk Server support||Comments|
|ANSI/X12 and EDIFACT transaction types||99%||BizTalk Server ships with schemas for only a small subset of these libraries. However, comprehensive libraries are available from Edifecs, the industry leader in EDI standards data (for more information, go to the Edifecs Web site at http://www.edifecs.com). BizTalk Server can handle all transaction types for all versions provided that a schema can be obtained, and all schemas are available through Edifecs.|
|ANSI/X12 and EDIFACT envelope segments||80%||BizTalk Server handles the addition and removal of envelopes to and from EDI data completely. However, there are limitations to the flexibility of the content in those envelopes. Specifically, X12 group envelopes must use an 8-digit date, this being the only X12 limitation.
EDIFACT, on the other hand, has much greater standards flexibility with respect to the content of the envelopes, and BizTalk Server only automatically handles this variability to a certain point, beyond which the customer might have to write a component for pre- or post-processing of the data to modify the envelope.
|Hierarchy and sectional boundary segments||100%||BizTalk Server can completely handle things like Hierarchical Level (HL) segments in X12, which are present to interpret data where recursion of structure otherwise makes determination of scope (location in the transaction) impossible.
BizTalk Server takes this one step further, and allows qualified segments and loops such that data parsed does not have to exactly match the schema in terms of the sequence expected. In other words, if the schema indicates a ship-to address followed by a bill-to address, using content-qualified tags called trigger fields, data that has the order reversed can still be parsed and validated successfully.
|X12 and EDIFACT data types||95%||BizTalk Server supports the alphanumeric types in both standards. Furthermore, X12-specific items, such as implied decimal, are handled at run-time during the transformation to and from XML.
The one limitation here is in the support for binary segments. Specifically, there is a size restriction (roughly equivalent to 32-bit MAXINT) for how big a binary field's contents are allowed to be.
|Floating segments||0%||BizTalk Server does not enable a mismatch between the sequence of data specified by a schema and the sequence of data present in a document instance, except as noted previously in the case of qualified segments or loops. However, there are unqualified segments in EDI whose position can float according to the standard (for example, the NTE segment in X12) and floating of these is not supported. Obviously, the segments can be used if their location is specified.|
Table 3. Category: Document mapping and translation
|Feature||BizTalk Server support||Comments|
|Validation according to user controlled schema||100%||The EDI schemas can and should be standards compliant, but certainly do not have to be. BizTalk Server validates data at runtime according to the schema, over which the user has complete control (provided schemas are validated in BizTalk Editor).|
|Drag-and-drop, GUI-based mappings||100%||BizTalk Mapper is a visual approach to converting data from one structure to another. These mappings can include value copy, name copy, rich modification of value copy (using functoids), logically gated mappings, and looping control, all accessible through drag-and-drop operations.|
|Uses published standards||100%||BizTalk Server maps are XSLT that conforms to W3C standards for this style sheet language for transformations. As such, the maps can be shared more easily.|
|N-way mappings||70%||BizTalk Mapper addresses the 1-to-1 mapping problem. By simultaneously executing multiple maps per inbound message, BizTalk Mapper also handles 1-to-many mappings. However, BizTalk Mapper cannot currently draw upon multiple sources of input to a map, either for single-message or multiple-message output.|
|Mapping to/from envelopes||20%||BizTalk Server essentially does not handle mapping to and from envelopes in the parser. Pre- and post-processing of the data makes it relatively easy to achieve the desired effect outside of the map itself.|
Table 4. Category: Messaging operation protocols
|Features||BizTalk Server support||Comments|
|Batching||100%||Inbound to BizTalk Server, data can be batched in large, multi-group interchanges, and can also consist of file-based submissions containing multiple interchanges. Outbound from the server, batching of data within interchanges is not built into the core product so that, on first installation and use, outbound interchanges will consist of a single group containing a single document.|
|Control numbers||80%||BizTalk Server can generate control numbers automatically for outbound data based on a seed value. However, there are features in some EDI systems that BizTalk Server does not support. For example, control numbers in a series are always incremented by 1.
Also, relationships between control numbers at various levels (document, group, and interchange) are not supported. For example, the document control number cannot be based on the group control number value.
Lastly, BizTalk Server does not, out-of-the-box, enable duplicate control number enforcement. The user must build a component or BizTalk Orchestration process to provide such enforcement.
|Functional acknowledgment content (X12/997, EDIFACT/CONTROL)||80%||BizTalk Server has full support for these transaction types, and is capable of automatically generating them, as needed. However, there are two limitations in this area:
|Functional acknowledgment status and reconciliation||95%||BizTalk Server can indicate when receipts are overdue (see the following tracking section for more details). Also, BizTalk Server can reconcile inbound receipts to the substantive business transactions to which they correspond. The one limitation in this area currently is that receipts that come in must match the document they are acknowledging by version of the standard (for example, 004010 of X12).|
Table 5. Category: Transports
|Feature||BizTalk Server support||Comments|
|VAN integration||80%||BizTalk Server does not include VAN transport components, but they are available through Covast (a third party partner) and cover the top VANs in use. To the extent that the transport step is also fully customizable, any connectivity that a given VAN offers is possible.|
|TCP/IP transport protocols (HTTP/x, SMTP, FTP, UNC/file)||95%||BizTalk Server has full support for these protocol types. FTP is available through a partner add-on.|
|SNA connectivity||100%||Provided through Microsoft Host Integration Server.|
|Custom||100%||As an integration platform, BizTalk Server enables you to implement custom transport technology in front of or behind the BizTalk Server domain in the point-to-point flow. This is central to the ability of BizTalk Server to integrate with legacy systems. Hundreds of adapters are available to support connectivity to legacy systems.|
Table 6. Category: Tracking and auditing
|Feature||BizTalk Server support||Comments|
|Full tracking of pertinent EDI metadata||100%||BizTalk Server, by default, extracts and tracks information such as where EDI messages came from, where they are destined, what size they were, which version of the standard was used, and what the control numbers were at each envelope level.|
|User interface for tracking query||100%||BizTalk Server includes a Web-based tracking interface that enables users with search capabilities across the previously mentioned tracking data. Tracking queries can be as specific (partner X, on Wednesday, between 1 and 2 P.M., transaction type Y, containing invoice number 123456) or general (all partners, all transactions, and so on) as a user requires. If a user wants, tracking can also be mostly or completely disabled.|
|Legal applicability||95%||Most of what BizTalk Server does in the area of tracking is sufficient for legal requirements in the area of non-repudiation, especially on the outbound side.
However, there is one limitation in this area. Specifically, when data arrives encrypted, BizTalk Server tracks only the post-decryption data, meaning that the initial over-the-wire form of the message is not tracked. If this is a requirement, users must separately track the data prior to its submission to BizTalk Server.
Table 7. Category: Security
|Feature||BizTalk Server support||Comments|
|Other/custom||100%||BizTalk Server allows the use of custom compression or encryption in the processing sequence.|
BizTalk Server can add value to any company that needs to automate its internal data flow or automate the exchange of business documents with other companies. This includes companies that use EDI, as well as companies that don't use EDI but need a way to build business relationships with EDI-based companies. BizTalk Server provides much more functionality than legacy EDI servers. It enables you to use EDI and other formats for business-to-business integration and EAI. BizTalk Server provides a powerful and flexible framework you can use to move your enterprise forward.
BizTalk Server has successfully replaced legacy software for some large EDI operational environments, but is also run alongside such software in other cases. Typically, the decision to switch to BizTalk Server is made after examining both the business and technical goals for the implementation. Some users want to use the flexibility of BizTalk Server to augment their existing EDI servers, while others are looking to provide a single messaging bus for the enterprise replacing their current servers.