Developing a Streaming Pipeline Component

Orchestrations are often the first port of call for most BizTalk developers and are the feature most identified with the product. However, many scenarios can be implemented very well using messaging alone, harnessing the power of port configuration and pipelines.

Any serious messaging implementation is bound to involve one or more custom pipeline components, and these are not difficult to develop. However, too often developers ignore the streaming fashion of the pipeline and write components that read the entire message into memory. Whilst this approach does work, and admittedly is somewhat easier to grasp, it has two significant downsides:

  • Higher memory consumption – the entire message has to be loaded into memory, whatever the message size is. In addition, if the .NET XML DOM is used (via the XmlDocument class, for example) the memory footprint of the component can be considerably bigger than the message size.

  • Latency – any further processing of the message has to wait until the current component has finished processing the entire message.

Both of these downsides can be addressed by taking a ‘streaming’ approach to pipeline component development, which is what this article attempts to demonstrate.

To download a copy of this guide in .docx form, go to

At this same location, you can also download a sample that is designed to demonstrate both the non-streaming and the streaming ways of achieving the solution.