Complex Event Processing (Middleware)---a Technical Reference Guide for Designing Mission-Critical Middleware Solutions

Complex Event Processing (CEP) is characterized by temporal analysis of massive streams of business events resulting in the identification of meaningful patterns in real-time. That analysis is performed in an effort to identify correlated events that happen over a span of time whereby the culmination of those events is significant, such that a complex event is derived. To quote Mark Simms, Microsoft Senior Program Manager, the goal of CEP is "to find the needle in the stack of needles."

For example, among millions of incoming events, a CEP system may receive the following three, from one or multiple sources, over the course of an hour:

• A sales person receives an email with the word congratulations in the body.

• An opportunity owned by that sales person is closed in a CRM system.

• The accounting department receives a purchase order from the company for which the opportunity was closed.

From these events the CEP system may infer the complex event—we have a new customer.

Best Practices

By leveraging an event-driven architecture, organizations are enabled to be more proactive in identifying and responding to business events. StreamInsight is a unique Microsoft product capable of legitimately processing hundreds of thousands of data points per second, and executing complex filtering or incremental aggregations. However, even if your business does not have event producers that send out massive amounts of events, you can still benefit from having an event processing engine like StreamInsight that maintains standing queries, and drives your organization to look for new ways to improve efficiency and make smarter decisions faster.

Development

• Although StreamInsight does not currently ship with adapters out-of-the-box, there are a number of good examples of input, output, typed and un-typed adapters availble on CodePlex and MSDN as well as an excellent how-to article for Building your First End-to-End StreamInsight Application that covers building adapters.

• StreamInsight adapters support three types of event models: interval, point, and edge, which are described in this Exploring StreamInsight’s Adapter Model blog posting.

• If your application uses event sources or sinks that implement the IObservable or IObserver interfaces, the application must target .NET Framework 4 (not the .NET Framework 4 Client Profile). For more information, see Using Observable and Enumerable Event Sources and Event Sinks.

• Event filtering is performed with LINQ and is fairly straight-forward, whether it needs to be done on One Condition or Multiple Conditions.

• Although StreamInsight, and consequently the available pool of relevant resources, is fairly nascent, there is a steadily growing arsenal of code samples, articles, and documentation from the community and Microsoft alike, including the StreamInsight Twitter Feed, Mark Simms’ Blog, the StreamInsight team blog, the AppFabric Customer Advisory Team StreamInsight Blogs, and other sources.

Deployment

• Solutions that require throughput in excess of 5000 events per second and/or latency tolerance that is less than 5 seconds should select Premium Edition whereas solutions with less robust requirements should select Standard Edition. The Choosing StreamInsight Edition article explains which edition of SQL Server ships with which edition of StreamInsight.

• StreamInsight supports two deployment models, including hosted and stand-alone. This Deployment Model article discusses the scenarios in which each model would be a good fit.

• StreamInsight has a somewhat limited hosting model for load balancing and failover, but there are a few options for achieving failover that are described in this discussion thread.

Management and Administration

• Although there is no user interface for managing or monitoring StreamInsight, it is possible to create and access diagnostic views for Monitoring the StreamInsight Server and Queries.

• The only graphical tool that StreamInsight provides is the StreamInsight Event Flow Debugger. The debugger provides the abillity to start and stop queries, view existing applications and show queries included in applications.

• What is not available graphically is the ability to create applications, delete applications and delete queries, but Richard Seroter has created an application that let’s you do this called the StreamInsight Server Manager.

Case Studies and References

Case Study

 EdgeNet StreamInsight Case Study

References

• Monitoring the StreamInsight Server and Queries

• Microsoft CEP Overview

• StreamInsight Server Architecture

• Advanced Microsoft SQL Server 2008 R2 StreamInsight

• 6 things to know about StreamInsight

• StreamInsight First Thoughts

Questions and Considerations

This section provides questions and issues to consider when working with your customers.

• Recognize that events may come from various source systems and may be destined for diverse target systems. Understand what those sources and sinks are and whether or not adapters already exist for those systems.

• Determine if events are typed (known and defined at design time) or untyped (dynamically generated at runtime).

• Understand if events will happen at a point in time, over a predetermined interval of time, or over an interval of time where the start time is known but the duration is not.

• Understand if those source or sink systems implement the IObservable or IObserver interfaces.

• Determine if you need to filter events on one or more conditions.

• Determine if events will need to be aggregated to arrive at averages, sums, counts, or the like.

• Determine if patterns or situations will need to be detected from source events.

• Determine what windows of time are monitored, and at what intervals results will be computed.

• Recognize that StreamInsight is a nascent and emerging Microsoft technology and that documentation and resources may seem somewhat limited relative to other, more mature products.

Deployment

• Determine throughput or event rate—the number of events that must be processed per second.

• Determine the latency requirements—the amount of time in which events must be consumed in to produce the required results.

• Identify the memory footprint constraints of your solution.

• Understand availability and reliability requirements.

Management and Administration

There is currently no user interface for managing or monitoring StreamInsight.

Appendix

< Full URLS for Hyperlinked Text>