What's New (StreamInsight)

Microsoft StreamInsight introduces new features and enhancements that increase the power and productivity of complex event processing developers and administrators.

For information about known issues, see Microsoft StreamInsight 1.2 Release Notes. For information backward compatibility, see Backward Compatibility (StreamInsight).

New features in StreamInsight 1.2 (June 2011)

New resiliency features

StreamInsight 1.2 introduces support for built-in resiliency to system failures for data stream processing. This support is available only in the Premium edition of StreamInsight. For more information about editions, see Choosing a StreamInsight Edition.

The new resiliency features let you run your critical applications with minimal operational downtime, with transparency to system failures, on the Windows platform of your choice, at an industry-leading total cost of ownership. For more information, see the following topics.

• StreamInsight Resiliency

• Building Resilient StreamInsight Applications

• Monitoring Resilient StreamInsight Applications

• Troubleshooting Resilient StreamInsight Applications

The new resiliency features also introduce new query states:

• Initializing. This state typically exists after one of the following events:

  • The query was stopped and the user has called its Start method.
  • A resilient query was running and the server is recovering from failure. To change the state of the query, the user must call its Start method.

• Checkpointing. The query is running and a checkpoint is in progress.

New developer features

Enhancements to the StreamInsight API include the following features:

User-defined stream operators

You can now provide custom processing of input streams with user-defined stream operators. In contrast with user-defined operators and user-defined aggregates, which let you interact with windows, user-defined stream operators let you interact directly with the contents of a stream, compute and generate results incrementally, and manage state. For more information, see User-Defined Stream Operators.

LINQ enhancements

• You can use nested types in event payloads, in adapters and in LINQ queries. For more information about event payloads, see Creating Event Types.

• You can use the LINQ "let" clause. For more information, see let clause (C# Reference).

• You can join multiple streams in a single query. You are no longer limited to joining only 2 streams. For more information about joins, see Joins.

• You can call extension methods inline in "from" clauses other than just the first "from" clause and in joins.

• You can join streams on composite join keys. For example: where {x.a, x.b} equals {y.a, y.b}. For more information about joins, see Joins.

• You can project a single field without creating a new anonymous type. For example: from x in xs select x.A instead of from x in xs select new { x.A }. For more information about projection, see Projection.

• You can use multiple aggregates in an expression. For example, select new {ratio = (double)win.Sum(e => e.good) / (double)win.Count()}.

• You can group a new anonymous type in a "group by" clause. For example: from x in xs group new { x.A, x.B } by new { x.C, x.D } into g, where group new { x.A, x.B } demonstrates the new capability. Previously, the selector in a "group by" clause was the incoming type; for example, from e in input group e by ….

• You can now include projections in a Take clause. For example, … select new { myAvg = r.Value / 10, r.SourceId }).Take(2, …). Previously, the selector in a Take clause was the incoming type; for example, from win in input.window(…) from e in … select e).Take(…).

API enhancements

• There is a new overload for the ShiftEventTime method that behaves as the name of the method implies. That is, the new overload shifts the time of the event by the amount of the provided timespan. The original version of this method required you to specify both the current event time and the timespan to shift the event time. Instead of writing ShiftEventTime(e => e.StartTime + TimeSpan.FromMinutes(15)), you can now write ShiftEventTime(e => TimeSpan.FromMinutes(15)). However you can still use the original version of the method to set the event time to a specific time by providing a datetime value. The original version has been retained for backward compatibility. For more information, see Time Stamp Modifications.

• There is a new output policy, PointAlignToWindowEnd, available for hopping windows. The existing policy, ClipToWindowEnd, yields a window size that corresponds to the lifetime of the set-based operation. The new policy yields a point event whose start time is the end time of the window. This new output policy is useful when you combine the result with another stream, since there is only a single valid result at each point in time, which expresses the most recent aggregation result at that point.

• All windows now have default output policies, which are ClipToWindowEnd for snapshot windows and PointAlignToWindowEnd for hopping or tumbling windows and for count windows.

New manageability and tools features

You can now use your existing diagnostic infrastructure to monitor and troubleshoot StreamInsight processes and queries with Performance Monitor and Event Viewer. You can also access an expanded list of diagnostic properties.

Performance counters

You can now use Performance Monitor to view counters for StreamInsight server processes, queries, and input streams. For more information, see Monitoring StreamInsight Performance Counters and Events.

Administrative logging to the Windows Event Log

You can now use Windows Event Viewer to see administrative events related to StreamInsight server processes, adapters, queries, and the dumper in the Application event log. For more information, see Monitoring StreamInsight Performance Counters and Events.

Additional diagnostic properties

Several new diagnostic properties are available to help with monitoring and troubleshooting. The new properties include the following:

• AdapterNumberOfRunningWorkers, AdapterNumberOfRunningWorkers, and AdapterNumberOfSuspendedWorkers

• AdapterFirstCtiTimestamp

• OperatorNumberOfWorkers

• OperatorGroupIdField

• QueryInstanceGroupId

• StreamNumberOfWorkers

• StreamMinInputEventCountAcrossWorkers and StreamMaxInputEventCountAcrossWorkers

• StreamMinMemoryIncludingEventsAcrossWorkers and StreamMaxMemoryIncludingEventsAcrossWorkers

Also, some of the previously available views have been removed. These include all views related to the scheduler.

For more information, see Monitoring the StreamInsight Server and Queries.

New features in StreamInsight 1.1 (October 2010)

The following changes and additions were implemented since StreamInsight 1.0 Refresh 1. They were released in StreamInsight 1.1 in October 2010.

• Support for event sources and sinks that implement the IObservable or IEnumerable interfaces. New classes and members have been added to the StreamInsight API to support this new programming model for input and output, as an alternative to regular input and output adapters. These features are described in the following topics:

  1. Using Observable and Enumerable Event Sources and Event Sinks (StreamInsight)

  2. End-to-End Example of an Enumerable Event Source and Event Sink (StreamInsight)

  3. API Support for IObservable and IEnumerable in StreamInsight

  4. More Information about IObservable and IEnumerable Interfaces in StreamInsight

• A Stop method on the Adapter class. This method notifies an adapter to stop as a result of stopping or aborting a query.

• New versions of StreamInsight are installed side-by-side with older versions. For more information about installing StreamInsight versions, see:

  • “Installing a New Version of StreamInsight” in Installation (StreamInsight).

  • “StreamInsight Versions” in Programming Considerations.

New features in StreamInsight 1.0 Refresh 1 (June 2010)

The following changes and additions were implemented since StreamInsight 1.0 RTM. They were released in StreamInsight 1.0 Refresh 1 in June 2010 concurrently with SQL Server 2008 R2 Cumulative Update 2.

• The ClipEventDuration API. This new method on the CepStream class lets you to use a secondary stream to determine where to clip the duration of events in the primary stream. For more information, see:

  • “Clipping the Duration of an Event” in Time Stamp Modifications.

  • ClipEventDuration<TSource, TClip>

• Support for the Microsoft .NET Framework 4 in StreamInsight application development

For more information about these changes, or to download StreamInsight 1.0 Refresh 1, see the Knowledge Base article, StreamInsight 1.0 updates that are released together with Cumulative Update package 2 for SQL Server 2008 R2.

New features in StreamInsight 1.0 RTM (April 2010)

The following changes and additions were implemented since the November 2009 Community Technology Preview (CTP) of StreamInsight 1.0. They were released in StreamInsight 1.0 in April 2010 concurrently with SQL Server 2008 R2.

Design and Development

You can now use StreamInsight queries as building blocks for your applications. Queries can be built on top of existing and currently running queries. For more information, see Composing Queries at Runtime.

Support for synchronized input streams has been added. For more information, see Advancing Application Time.

The maximum size of a single event in StreamInsight has been increased to 16K. Accounting for system fields and event metadata, a safer, pragmatic limit is 14K.

Operators

Support for count windows has been added. For more information, see Count Windows.

Support for left-anti-semi-join operations has been added. For more information, see Joins.

API

The AdapterFactory class has a new interface definition, IDeclareAdvanceTimePolicy, that returns an AdvanceTimeSettings instance. This is the mechanism by which you can specify advance time settings from the adapter factory.

Management

Diagnostic Views

New diagnostic views are available to monitor the system at the following levels of specificity: server, operator, scheduler, and adapter. For more information, see Monitoring the StreamInsight Server and Queries.

Samples

Samples are available for download at StreamInsight Samples.

See Also

Other Resources

New and Updated Topics in StreamInsight