Enterprise Integration Patterns (Middleware)---a Technical Reference Guide for Designing Mission-Critical Middleware Solutions

Patterns are abstraction mechanisms that can be applied repeatedly for providing a practice-proven solution to recurring technical challenges within a given context. Patterns do not map neatly into specific organizational models, or solution implementations, and are usually combined with other patterns. Patterns also inform decisions selecting implementation technologies like BizTalk, AppFabric, MSMQ, WCF, WF, SQL Server and SSIS.

• Enterprise domain patterns: Broker/Hub, Message Bus/Enterprise Service Bus (ESB), and Gateway.

• Process integration patterns: Filter, Pub-Sub, Reliable, Wire Tap, Smart Proxy, Manager, VETO.

• Data integration patterns: Replication, Synchronization, Consolidation, Federation.

• Message Routing Patterns: Message Router, Content-Based Router, Routing Slip, Scatter-Gather, Recipient List, Splitter.

• Full list of integration patterns.

Best Practices

Development

• Use declarative filters in WCF to minimize the quantity of messages transmitted or processed.

• Use the claim check pattern to avoid sending large payloads through BizTalk.

• Ensure that a pattern’s behavior is not inadvertently altered. Pipes and Filters and Wire Tap patterns do not include persistence points such as databases or file locations, which would alter the pattern’s behavior.

• Try to avoid patterns such as Resequencer. If ordered delivery cannot be guaranteed, or duplicates cannot be prevented, then use SQL Server to restore referential integrity before applying messaging patterns to alleviate symptoms.

• Designing an effective exception-management strategy is important for the security and reliability of your integration service or application. Prevent services from exposing information about their internal implementation when an exception occurs such as through unnecessary detailed exception message or enclosed stack trace.

• Implement a service metadata repository for maintaining service components, including schemas, endpoints, policies, and implementation details.

• Take advantage of the service virtualization which hides service endpoint implementation and underlying transport protocol from service consumers.

• Implement a standard interaction model for services that includes enterprise-wide canonical data model, consistent use of standardized message headers across all services, and adoption of appropriate semantics.

• Provide guidelines and standards for the implementation of service logic. Promote the use of reusable templates, such as software factory templates, for service implementation to significantly strengthen this area.

Deployment

• Plan to use Deployment Patterns: Layered Application, Tiered Distribution.

• For cloud guidance, read Enterprise Integration and Distributed Computing: A Ubiquitous Phenomenon.

• Remove hard-coded dependencies on the physical locations of services using a directory service (or service repository pattern) to resolve logical service names for physical URLs.

• Simplify deployment by using dynamic configuration of services and service clients at runtime using one of the available SOA governance products or centralized configuration management tools.

Management and Administration

• SOA Governance Integration outlines the use of run-time governance policies for SLAs, versioning, compliance with regulations and business standards, metrics reporting, validation, and performance thresholds. These aspects of governance are implemented using a model similar to federated, end-to-end security.

• The Microsoft Composite Service Application Guidance provides design and implementation patterns for service discovery, composition, and integration metrics.

Performance

• Perform critical path and performance testing. Enterprise Ready BizTalk - Best practices from the field.

• Performance has improved in WWF 4.0: Windows Workflow Foundation 4 Performance.

• Avoid unnecessary blocking actions such as transactions, file locks, correlation and synchronous connections.

• Always analyze the performance of patterns for exception and fault handling. Exception Handling Patterns in Process-Aware Information Systems. When sizing hardware, factor in additional resource usage when "all fails."

• Use PathPing to analyze network traffic to reconcile logical pattern behavior with physical network behavior.

• Use TCPView and Avoid TCP/IP Port Exhaustion: https://msdn2.microsoft.com/en-gb/library/cc296845.aspx.

• Avoid chatty calls to the service interface, which can lead to very poor performance. Service operations should be coarse-grained and focused on application operations. Consider using the Facade pattern to package smaller fine-grained operations into single coarse-grained operations.

Case Studies and References

Examples of successful architectures are described in the following case studies and white papers:

• This service oriented solution shows how integration patterns are implemented in BizTalk Server 2010.

• Practical Messaging Scenarios with WCF 4: http://www.code-magazine.com/Article.aspx?quickid=1009051

• The Scatter-Gather sample architecture in the ESB Toolkit demonstrates a message bus implementation of this pattern versus the broker implementation in Scatter and Gather using Self Correlating Port.

• Application Architecture Knowledge Base from the Patterns & Practices team.

• SOA Design Patterns publication.

• Grab your own copy of the integration bible.

Questions and Considerations

This section provides questions and issues to consider when working with your customers.

Development

• Be prepared to understand the fundamental concepts of Hub-and-Spoke and Service Bus, and be able to actively engage into a CxO-level dialog possessing Hub-and-Spoke vs ESB arguments.

• Identify and consider patterns for messaging, data integration, and process integration independently of each other while maintaining a coherent end-to-end view as to how the identified patterns are to be interoperable with each other.

• How does one system communicate with the others? Have you determined the communication, trust boundary, transaction, and performance requirements required for each messaging pattern?

• Do chosen communication transports support the required levels of security, reliability, scalability, throughput, and latency?

• Interoperability with heterogeneous services requires adoption of standards-based, interoperable communication protocols. Consider protocol standardization in alignment with selected patterns.

• Evaluate the pros and cons of asynchronous messaging architecture, which in turn may open up the opportunity for introducing additional patterns that overcome limitations in distributed communication, such as latency and unreliability.

• When designing the communication strategy for integration solution, the protocol you choose should be based on the deployment scenario for your solution. For instance, the HTTP/HTTPS-based protocols (Web Services, REST, OData) are the optimal choice for cloud services whereas TCP and UDP-based transports are best when utilized on-premises.

• Measure your level of understanding in the area of business processes that require integration—a good measurement metric is your ability to clearly articulate the key premises of the integrated process to an average developer in the project team.

• As the most common denominator, use the Envelope Wrapper pattern for the effective cross-platform management and access to metadata and state, especially for process integration.

• Access to legacy systems can be accomplished by exposing a service gateway that translates operations between the service interface and legacy system interface.

• Review the key integration styles (File Transfer, Shared DB, Synchronous RPC, Asynchronous Messaging) and choose the combination best suited for addressing particular integration requirements.

• Employ the sufficient level of abstraction when implementing patterns in hybrid or cloud-based applications—this would enable greater code reuse and resilience to platform evolution.

Deployment

• Which versioning strategy to deploy? Side by Side Deployment, Application Versioning, Service Contract Versioning. Do new versions change the original patterns? How will updates be communicated and applied?

• Does your disaster recovery strategy fit a Broker/Hub model, or a Messaging Bus model?

• Identify deployment targets (on-premises, cloud, or hybrid) and ensure selected patterns can be equally applied in the given environment.

• Does the business employ well-defined processes for change management/change control policy in the production/QA environments?

• Centralization of deployment policies reduces the costs associated with reconfiguring integrated applications and services as they migrate between dev, test, staging, QA, and production environments.

Management and Administration

• WCF provides excellent diagnostics for tracing at the service, channel, and message layers.

• Use performance counters to detect patterns that characterize normal and abnormal behavior.

• Use patterns for governance of integration solutions. Examples include: federated searches for data aggregation of diagnostic traces, Microsoft Sync Framework for data synchronization patterns, and Wire Taps for monitoring.

Performance

• Choose hardware-based load balancing over software-based load balancing, because it is more robust and has the capability to work with any operating system or platform.

• In isolation, one pattern may perform better than another, but not when run in parallel. In a Scatter-Gather pattern, load bursts during the Scatter phase not only increases in magnitude but also must be reconciled with latency during the Gather phase, which can alter throughput rates.

• Interactions with remote services may introduce latency. Caching reference data may increase performance of applications by reducing latency. Along these lines, consider providing support to cache service responses on the requester side so that subsequent operations can use the cached response.

• Synchronous request-reply messaging pattern may cause bottlenecks for services that are handling a high number of requests. As a result, this pattern requires careful consideration when designing highly transactional services. Consider using asynchronous communications to improve performance where a service might be handling a high load of requests. The best strategy would to be use a two-way message communication where both the service and the client send messages to each other independently irrespective of the use of one-way or request-reply messaging.

• Sometimes you want order but you also want as much concurrency as possible in an ordered use case.

Appendix

< Full URLS for Hyperlinked Text>