What's New (StreamInsight)

  • Article

 

Microsoft StreamInsight introduces new features and enhancements that increase the power and productivity of complex event processing developers and administrators.

For information on backward compatibility, see Backward Compatibility (StreamInsight).

New features in StreamInsight 2.3

StreamInsight 2.3 is strictly a licensing update to version 2.1. StreamInsight has been updated in version 2.3 to allow you to use it with a SQL Server 2014 license. Any code written against StreamInsight 2.1 remains supported in version 2.3.

New features in StreamInsight 2.1

StreamInsight 2.1 is a significant update to version 2.0. While any code written against StreamInsight 2.0 and earlier remains supported, the 2.1 update addresses customer feedback by providing new capabilities.

  • A new object model is introduced that is much more clear and consistent.

  • You can provision, name, and manage data sources and sinks in the server.

  • You can express and run computations that include StreamInsight, Reactive Framework (Rx), or any other LINQ expressions, and manage them as a single unit.

  • Complex input and output adapters are no longer necessary. Instead, sources and sinks can now be implemented using the observable/observer and enumerable/enumerator models. Adapters remain fully supported, but the new model provides a simpler alternative for handling most data streaming scenarios.

  • You can now checkpoint temporal data streams and checkpoint over multiple processes with shared computation.

New features in StreamInsight 2.0

For complete information on StreamInsight 2.0, see Microsoft StreamInsight 2.0 documentation in the MSDN Library.

StreamInsight 2.0 requires SQL Server 2012 license and .NET Framework 4.0. It includes the following performance improvements along with few bug fixes.

Performance Improvements

The following issues that hinder performance of a StreamInsight application are fixed in this version.

  • Queries can deadlock when a join operator has input streams that originate from a common source.

  • The memory consumed by a StreamInsight application may grow infinitely when lifetime of the left-side reference stream overlaps with a right-side continuous real-time stream of a Left Anti Semi Join operator.

New features in StreamInsight 1.2 (June 2011)

For complete information on StreamInsight 1.2, see Microsoft StreamInsight 1.2 documentation in the MSDN Library.

New resiliency features

StreamInsight 1.2 introduces support for built-in resiliency to system failures for data stream processing. This support is available only in the Premium edition of StreamInsight.

The new resiliency features let you run your critical applications with minimal operational downtime, with transparency to system failures, on the Windows platform of your choice, at an industry-leading total cost of ownership.

The new resiliency features also introduce new query states:

  • Initializing. This state typically exists after one of the following events:

    • The query was stopped and the user has called its Start method.

    • A resilient query was running and the server is recovering from failure. To change the state of the query, the user must call its Start method.

  • Checkpointing. The query is running and a checkpoint is in progress.

New developer features

Enhancements to the StreamInsight API include the following features:

User-defined stream operators

You can now provide custom processing of input streams with user-defined stream operators. In contrast with user-defined operators and user-defined aggregates, which let you interact with windows, user-defined stream operators let you interact directly with the contents of a stream, compute and generate results incrementally, and manage state.

LINQ enhancements

  • You can use nested types in event payloads, in adapters and in LINQ queries.

  • You can use the LINQ "let" clause. For more information, see let clause (C# Reference).

  • You can join multiple streams in a single query. You are no longer limited to joining only 2 streams.

  • You can call extension methods inline in "from" clauses other than just the first "from" clause and in joins.

  • You can join streams on composite join keys. For example: where {x.a, x.b} equals {y.a, y.b}.

  • You can project a single field without creating a new anonymous type. For example: from x in xs select x.A instead of from x in xs select new { x.A }.

  • You can use multiple aggregates in an expression. For example, select new {ratio = (double)win.Sum(e => e.good) / (double)win.Count()}.

  • You can group a new anonymous type in a "group by" clause. For example: from x in xs group new { x.A, x.B } by new { x.C, x.D } into g, where group new { x.A, x.B } demonstrates the new capability. Previously, the selector in a "group by" clause was the incoming type; for example, from e in input group e by ….

  • You can now include projections in a Take clause. For example, … select new { myAvg = r.Value / 10, r.SourceId }).Take(2, …). Previously, the selector in a Take clause was the incoming type; for example, from win in input.window(…) from e in … select e).Take(…).

API enhancements

  • There is a new overload for the ShiftEventTime method that behaves as the name of the method implies. That is, the new overload shifts the time of the event by the amount of the provided timespan. The original version of this method required you to specify both the current event time and the timespan to shift the event time. Instead of writing ShiftEventTime(e => e.StartTime + TimeSpan.FromMinutes(15)), you can now write ShiftEventTime(e => TimeSpan.FromMinutes(15)). However you can still use the original version of the method to set the event time to a specific time by providing a datetime value. The original version has been retained for backward compatibility.

  • There is a new output policy, PointAlignToWindowEnd, available for hopping windows. The existing policy, ClipToWindowEnd, yields a window size that corresponds to the lifetime of the set-based operation. The new policy yields a point event whose start time is the end time of the window. This new output policy is useful when you combine the result with another stream, since there is only a single valid result at each point in time, which expresses the most recent aggregation result at that point.

  • All windows now have default output policies, which are ClipToWindowEnd for snapshot windows and PointAlignToWindowEnd for hopping or tumbling windows and for count windows.

New manageability and tools features

You can now use your existing diagnostic infrastructure to monitor and troubleshoot StreamInsight processes and queries with Performance Monitor and Event Viewer. You can also access an expanded list of diagnostic properties.

Performance counters

You can now use Performance Monitor to view counters for StreamInsight server processes, queries, and input streams.

Administrative logging to the Windows Event Log

You can now use Windows Event Viewer to see administrative events related to StreamInsight server processes, adapters, queries, and the dumper in the Application event log.

Additional diagnostic properties

Several new diagnostic properties are available to help with monitoring and troubleshooting. The new properties include the following:

  • AdapterNumberOfRunningWorkers, AdapterNumberOfRunningWorkers, and AdapterNumberOfSuspendedWorkers

  • AdapterFirstCtiTimestamp

  • OperatorNumberOfWorkers

  • OperatorGroupIdField

  • QueryInstanceGroupId

  • StreamNumberOfWorkers

  • StreamMinInputEventCountAcrossWorkers and StreamMaxInputEventCountAcrossWorkers

  • StreamMinMemoryIncludingEventsAcrossWorkers and StreamMaxMemoryIncludingEventsAcrossWorkers

Also, some of the previously available views have been removed. These include all views related to the scheduler.

New features in StreamInsight 1.1 (October 2010)

The following changes and additions were implemented since StreamInsight 1.0 Refresh 1. They were released in StreamInsight 1.1 in October 2010.

  • Support for event sources and sinks that implement the IObservable or IEnumerable interfaces. New classes and members have been added to the StreamInsight API to support this new programming model for input and output, as an alternative to regular input and output adapters.

  • A Stop method on the Adapter class. This method notifies an adapter to stop as a result of stopping or aborting a query.

  • New versions of StreamInsight are installed side-by-side with older versions.

New features in StreamInsight 1.0 Refresh 1 (June 2010)

The following changes and additions were implemented since StreamInsight 1.0 RTM. They were released in StreamInsight 1.0 Refresh 1 in June 2010 concurrently with SQL Server 2008 R2 Cumulative Update 2.

  • The ClipEventDuration API. This new method on the CepStream class lets you to use a secondary stream to determine where to clip the duration of events in the primary stream.

  • Support for the Microsoft .NET Framework 4 in StreamInsight application development

For more information about these changes, or to download StreamInsight 1.0 Refresh 1, see the Knowledge Base article, StreamInsight 1.0 updates that are released together with Cumulative Update package 2 for SQL Server 2008 R2.

New features in StreamInsight 1.0 RTM (April 2010)

The following changes and additions were released in StreamInsight 1.0 in April 2010 concurrently with SQL Server 2008 R2.

Design and Development

You can now use StreamInsight queries as building blocks for your applications. Queries can be built on top of existing and currently running queries.

Support for synchronized input streams has been added.

The maximum size of a single event in StreamInsight has been increased to 16K. Accounting for system fields and event metadata, a safer, pragmatic limit is 14K.

Operators

Support for count windows has been added.

Support for left-anti-semi-join operations has been added.

API

The AdapterFactory class has a new interface definition, IDeclareAdvanceTimePolicy, that returns an AdvanceTimeSettings instance. This is the mechanism by which you can specify advance time settings from the adapter factory.

Management

Diagnostic Views

New diagnostic views are available to monitor the system at the following levels of specificity: server, operator, scheduler, and adapter.

Samples

Samples are available for download at StreamInsight Samples.