Readme_Change Data Capture since Last Request Package Sample

[This topic is pre-release documentation and is subject to change in future releases. Blank topics are included as placeholders.]

This sample shows how to incorporate change data capture technology into an extraction, transformation, and loading (ETL) environment that uses Integration Services. This sample is a complete end-to-end sample that includes both workload generation and the initial load of target tables as part of the sample setup. After the initial load finishes, the master package runs periodically to gather the changes that have occured since the last request. The master package then updates the target replicas to reflect those changes.

The source database is the AdventureWorks2008 database. This sample uses existing tables in the AdventureWorks2008 database to populate comparable destination tables in the CDCSample schema in the AdventureWorksDW2008 database.

Requirements

To run this sample package, the following requirements must be met:

• The sample package and data files that the sample uses must be installed on the local hard disk drive. You can download the sample from Microsoft SQL Server Product Samples: Integration Services.
• You must have installed and have administrative permissions on the AdventureWorks2008 OLTP and AdventureWorksDW2008 databases.
• You must install SQL Server Database Engine to install support for the change data capture feature. You must also install Replication when you install SQL Server Database Engine.
• If you intend only to run the sample from the command line, you must install Integration Services.
• If you want to open and run the sample in SSIS Designer, you must install Business Intelligence Development Studio.

For more information about how to install samples, see "Installing Sample Integration Services Packages" in SQL Server Books Online.

Location of the Sample Files

This sample consists of the following files:

• The solution file and Readme file.
• The project files and packages.
• The Transact-SQL script files.

Location of the Solution File and Readme File

The following folder is the default location of the solution file and Readme file for this sample:

<drive>:\Program Files\Microsoft SQL Server\100\Samples\Integration Services\Package Samples\Change Data Capture LSN Based Sample\

The following table provides the complete name and a short description of the files in this folder.

File	Description
Change Data Capture LSN Based Sample.sln	The solution file for this sample.
Readme_ChangeDataCaptureLSNBasedSample.htm	The Readme file for this sample.

Location of the Project Files and Packages

The following folder is the default location of the project files and packages for this sample:

<drive>:\Program Files\Microsoft SQL Server\100\Samples\Integration Services\Package Samples\Change Data Capture LSN Based Sample\Change Data Capture LSN Based Sample\

The following table provides the complete name and a short description of the files in this folder.

File	Description
Change Data Capture LSN Based Sample.dtproj	The project file for this sample.
SetupCDCSample.dtsx	The setup package and sample test harness. Note If you did not install this sample to the default location, you will have to modify the package variables, SQLServerInstallPath and BasePath, to correspond to the appropriate installation path. The default value of the SQLServerInstallPath package variable is <drive>:\Program Files\Microsoft SQL Server. The value of the BasePath package variable derives from the value of the SQLServerInstallPath package variable.
MasterCDC.dtsx	The master package that performs an incremental load cycle.
CDCCustomerExtract.dtsx	The child package that gathers the changes to the Customer table for the extraction interval, and then applies these changes to the destination.
CDCCreditCardExtract.dtsx	The child package that gathers the changes to the CreditCard table for the extraction interval, and then applies these changes to the destination.
CDCWorkOrderExtract.dtsx	The child package that gathers the changes to the WorkOrder table for the extraction interval, and apply these changes to the destination.

Location of the Transact-SQL Script Files

The following folder is the default location of the Transact-SQL script files for this sample:

<drive>:\Program Files\Microsoft SQL Server\100\Samples\Integration Services\Package Samples\Change Data Capture LSN Based Sample\Change Data Capture LSN Based Sample\Scripts\

The following table provides the complete name and a short description of the files in this folder.

File	Description
CDCSetupTables.sql	The sample setup file.
CDCCustomerInsert.sql	The script that applies Inserts to CDCSample.Customer table.
CDCCustomerModify.sql	The script that applies Updates to CDCSample.Customer table.
CDCCreditCardInsert.sql	The script that applies Inserts to CDCSample.CreditCard table.
CDCCreditCardModify.sql	The script that applies Updates to CDCSample.CreditCard table.
CDCWorkOrderInsert.sql	The script that applies Inserts to CDCSample.WorkOrder table.
CDCWOrkOrderModify.sql	The script that applies Updates to CDCSample.WorkOrder table.
CDCCleanup.sql	The script that removes all objects that the sample creates. To apply this script, run sqlcmd –E –iCDCCleanup.sql.

Running the Sample

To run the package, you can use either the dtexec command prompt utility (dtexec.exe) or BI Development Studio.

To run the package by using the dtexec utility

1. Open a Command Prompt window.

2. Use the Change Directory command, cd, to change the directory that contains the dtexec utility.

   The default location of the dtexec utility is <drive>:\Program Files\Microsoft SQL Server\100\DTS\Binn.

3. At the command prompt, type the following command:

   dtexec /f "C:\Program Files\Microsoft SQL Server\100\Samples\Integration Services\Package Samples\Change Data Capture LSN Based Sample\Change Data Capture LSN Based Sample\SetupCDCSample.dtsx "
   

4. Press ENTER.

   For more information about how to use the dtexec utility run a package, see the topic, "dtexec Utility," in SQL Server Books Online.

To run the package in Business Intelligence Development Studio

1. In BI Development Studio, on the File menu, point to Open, and then click Project/Solution.

2. For Files of type, select Integration Services Project Files (.dtproj). Then, locate the Change Data Capture LSN Based Sample* folder, and double-click the file, Change Change Data Capture LSN Based Sample.dtproj.

3. In Solution Explorer, in the SSIS Packages folder, right-click SetupCDCSample.dtsx, and then click Execute Package.

Sample Packages and Their Components

The sample consists of five packages:

• The setup package—SetupCDCSample
  The setup package acts as a test harness for the sample. As a test harness, this package does the following processes:

  • Performs all the required initialization. This includes creating a database snapshot.
  • Generates a data manipulation language (DML) workload against the source tables while cycling the master package at 10 second intervals. The master package performs an incremental load of the target tables for all changes processed by the capture process since the last request.
  • Monitors the progress of the run, and stops the sample after all the DML workload has been processed and applied to the destination tables.
  • Validates, at the end of the run, the content of the destination tables. The setup package compares the destination tables with the content of the source tables, and then records the results in the event log.

• The master package—MasterCDC
  The master package performs an incremental load of the target tables for the interval provided as input.

  To determine whether to continue with the incremental load, the master package checks the endpoints of the interval:

  • If both endpoints for the extraction interval are in range, the master package starts the three child packages, which perform the incremental loads for the individual tables.
  • If either the low or high endpoint is outside the interval where change data exists for a source table, the master package logs an error and stops the sample.

  After all the child packages have finished running, the master package logs a message to the event log that indicates the sample has finished for the interval. The master package then stops the sample.

• Three child packages—Extract Customer Data, Extract CreditCard Data and Extract WorkOrder Data
  The structure of each of these child packages is identical:

  • Each child package first creates the query that extracts the changed data.
  • Then, the child package runs a data flow task that performs the incremental load of the changed data. The data flow task uses information in the result set that the query returns to split the returned rows. The data flow splits the rows into three distinct flows—insert, update, or delete—that correspond to the operation required to apply the row to the destination. The data flow task then runs the tasks that handle the various flows.

The following sections describe the components found in each package.

SetupCDCSample Package Components

The SetupCDCSample package has several annotated functional groups:

• Initialize Environment and Enable Change Data Capture
• Generate Workload
• Verify Capture Process is Running
• Perform Initial Load for Target Tables
• Prepare for Initial Extraction
• Extract and Process Change Data
• Validate Incremental Load and Report Run Status

These groups help organize the content as described in the following sections.

Initialize Environment and Enable Change Data Capture

This functional group contains a single Execute SQL task. The purpose of this task is to do the following:

• Create the sample source tables from existing AdventureWorks2008 tables, and then enable change data capture for these tables.
• Create the target tables in the AdventureWorks2008DW database.

The following table lists the task that this functional group uses.

Name	Element	Purpose
Create Tables and Enable Change Data Capture	Execute SQL task	Creates the sample source tables from existing AdventureWorks2008 tables: Checks for any objects that were created in previous runs of the sample, and deletes these objects, if present. Enables the AdventureWorks2008 database for change data capture. Creates the CDCSample schema. Creates the tables, CDCSample.Customer, CDCSample.CreditCard, and CDCSample.WorkOrder, in the CDCSample schema. (These tables mirror the comparable tables in the dbo schema.) Creates the target tables in the AdventureWorks2008DW database : Checks for any objects that were created in previous runs of the sample, and then deletes these objects, if present. Creates the CDCSample schema. Creates the target tables, Customer, CreditCard, and WorkOrder, in the CDCSample schema. Initializes the tables, CDCSample.Customer, CDCSample.CreditCard, and CDCSample.WorkOrder, with a subset of the table data from their parent AdventureWorks2008 tables. Enables change data capture on the parent AdventureWorks2008 tables.

Generate Workload

This functional group contains six Execute SQL tasks and a Script task:

• Each of the three source tables has two Execute SQL tasks: one that generates insert activity against the table and one that generates update activity. Three of the six Execute SQL tasks run after change data capture has been enabled on the source tables. The remaining three run as each of the first three workload tasks finishes.
• When all the Execute SQL tasks have run, the Script task runs. The Script task sets the package variables that indicate that the workload has finished.

The following table lists the tasks that this functional group uses.

Name	Element	Purpose
Insert Customer Table	Execute SQL task	Applies inserts to the AdventureWorks2008.CDCSample.Customer table.
Modify Customer Table	Execute SQL task	Applies updates to the AdventureWorks2008.CDCSample.Customer table.
Insert CreditCard Table	Execute SQL task	Applies inserts to the AdventureWorks2008.CDCSample.CreditCard table.
Modify CreditCard Table	Execute SQL task	Applies updates to the AdventureWorks2008.CDCSample.CreditCard table.
Insert WorkOrder Table	Execute SQL task	Applies inserts to the AdventureWorks2008.CDCSample.WorkOrder table.
Modify WorkOrder Table	Execute SQL task	Applies updates to the AdventureWorks2008.CDCSample.WorkOrder table.
Mark Workload Completion	Script task	Sets the following package variables: The WorkloadCompleted package variable that indicates whether the workload has finished. The WorkloadEndTime package variable that indicates the time at which the workload finished. This task uses Visual Basic.

Verify Capture Process is Running

This functional group contains one Execute SQL tasks and one Script task.

• The Execute SQL task runs after the environment has been initialized and the source tables have been enabled for change sata capture. This task verifies that the change data capture process has started automatically.
• If the change data capture process has not automatically started, the Script task runs to log an error in the event log. This error message indicates that SQL Server Agent must be running for the capture process to automatically start. Then, the sample stops running. If the capture process is running, the Script task allows the database snapshot to continue in preparation for the initial load of the target tables.

Perform Initial Load for Target Tables

This functional group contains one Script task and three Data Flow tasks:

• The Script task, Create Database Snapshot, creates a database snapshot of the AdventureWorks2008 database. This snapshot provides the source tables for the initial load of the target tables.
• The three data flow tasks then extract data from the snapshot tables and apply this data to the destination tables in the AdventureWorksDW2008 database.

The following table summarizes these tasks.

Name	Element	Purpose
Create Database Snapshot	Execute SQL task	Creates a database snapshot for AdventureWorks2008.
Load Target Customer Table from Snapshot	Data Flow task	Uses an OLE DB source to extract data from the snapshot table, CDCSample.Customer, and then uses an OLE DB destination to apply the rows to the table, AdventureWorksDW2008.CDCSample.Customer.
Load CreditCard Table from Snapshot	Data Flow task	Uses an OLE DB source to extract data from the snapshot table, CDCSample.CreditCard, and then uses an OLE DB destination to apply the rows to the table, AdventureWorksDW2008.CDCSample.CreditCard.
Load WorkOrder Table From Snapshot	Data Flow task	Uses an OLE DB source to extract data from the snapshot table, CDCSample.WorkOrder, and then uses an OLE DB destination to apply the rows to the table, AdventureWorksDW2008.CDCSample.WorkOrder.

Prepare for Initial Extraction

This functional group contains one Execute SQL task, which determines an appropriate LSN range to synchronize the first incremental load to the initial load.

Name	Element	Purpose
Get End-points for Initial Query Interval	Execute SQL task	Extracts the last LSN that is included in the database snapshot from snapshot metadata. Then, increments this value to serve as the low LSN value for the first incremental load that is applied after initialization. The task then delays until the capture process has processed through this LSN value in the transaction log. After this condition is true, the task identifies the current maximum LSN as the high LSN value for the initial query interval. The task uses the function, sys.fn_varbintohexstr, to convert the binary LSN values for the initial query interval to strings that are saved as the package variables, User::ExtractStartLSN and User::ExtractEndLSN. The task uses the cdc.lsn_time_mapping table to associate an approximate datetime value with the low endpoint, which the task saves as the package variable, User::ExtractStartTime.

Extract and Process Change Data

This functional group contains a For Loop container that includes two tasks:

• An Execute Package task that gathers and applies the change data that is associated with the current query interval to the target tables.
• An Execute SQL task that first delays for 10 seconds, and then computes the LSN range for the next query interval.

The For Loop container maintains the package variable, IntervalID. The For Loop container initializes the IntervalID package variable to 0, and then and increments this variable by 1 on each iteration of the loop. The loop continues to run until the following conditions are true:

• The WorkloadCompleted package variable is set to 1.
• The computed value for the ExtractionStartTime package variable, which will be used in the next iteration, exceeds the recorded value for the WorkloadEndTime package variable.

Name	Element	Purpose
Cycle Master at 10 second intervals	For Loop container	First runs the package to gather changes for the interval, and then applies these changes to the target tables. The container then runs the Execute SQL task that first delays for 10 seconds and then computes the new extraction interval. The container stops processing when all the changes in the generated workload have been applied to the destination tables.
Run Master to Extract Data	Execute Package task	Runs the MasterCDC package that applies the incremental load for a given extraction interval to the destination tables.
Delay 10 Seconds and Set New Query Interval	Execute SQL task	Uses the function, sys.fn_cdc_hexstrtobin, to convert the string representation of the high LSN value for the previous query to a binary(10) value. Then, uses the function, sys.fn_cdc_increment_lsn, to increment this value to serve the low LSN value for the next query interval. After a delay of 10 seconds, uses the function, sys.fn_cdc_get_max_lsn, to obtain the current maximum LSN that has been processed by the capture process. This value serves as the high endpoint of the next query itnerval. If the low endpoint is greater than the maximum LSN, the task continues to delay for 10 seconds, and then obtains the current maximum LSN until the low endpoint does not exceed the maximum. The task uses the function, sys.fn_varbintohexstr, to convert the binary LSN values that act as the lower and upper bounds for the next query interval to strings. The task then saves these strings as the package variables, User::ExtractStartLSN and User::ExtractEndLSN. The task uses the cdc.lsn_time_mapping table i to associate an approximate datetime value with the low endpoint, which is saved as the package variable, User::ExtractStartTime.

Validate Incremental Load and Report Run Status

This functional group contains an Execute SQL task and a Script task:

• The Execute SQL task, Check for Mismatch in Replicas, uses SQL CHECKSUM to compare the source tables with the replicas and sets package variables that indicate the run status.
• At the end of the run, the Script task, Output Run Completion Status, uses the package variables to report the results of the table comparisons in the event log.

The following table summarizes these tasks.

Name	Element	Purpose
Check for Mismatch in Replicas	Execute SQL task	Uses CHECKSUM to compare the replicas to the source tables. Based on this comparison, the task then sets the package variables, CustomerMismatch, CreditCardMismatch, and WorkOrderMismatch, accordingly.
Output Run Completion Status	Script task	Outputs a status message to the event log at the end of the run. (This message reports on the parity of the package variables, CustomerMismatch, CreditCardMismatch, and WorkOrderMismatch.) Then, sets the run status accordingly.

MasterCDC Package Components

When the SetupCDCSample package runs, five package configurations of type, "Parent package variable", enable the SetupCDCSample package to pass package variables to the MasterCDC package. The following table lists these package variables.

Parent	Child
User::ExtractEndLSN	ExtractEndLSN
User::ExtractStartLSN	ExtractStartLSN
User::IntervalID	IntervalID
User::BasePath	BasePath

The following tables list the tasks, containers, data sources and destinations, and transformations that are used in the master package.

Name	Element	Purpose
Wait for Capture Process to Process Extraction Interval	For Loop container	Contains the following components: An Execute SQL task that sets a package variable that indicates the status of the proposed extraction interval relative to the capture process. A Script task that generates an entry in the error log if the query interval is not valid, and causes the package to stop running. The query interval is not valid if either the low endpoint or the high endpoint do not lie in the change data capture validity interval for the database. If the interval is valid, this container also starts the Execute Packages tasks that run the child packages that apply the change data to the destination tables. After the child packages complete, this container also runs a Script task that records that the extraction interval has finished in the event log.
Check for Valid Interval	Execute SQL task	Sets the DataReady package variable to one of the following values: 0 = Query interval is not valid 1 = Query interval is valid (This task runs in the For Loop container.)
Log Extract Error	Script task	Logs an error to the event log when the DataReady package variable is set to 0. (This task runs in the For Loop Container.)
Extract Customer Data	Execute Package task	Runs the CDCCustomerExtract package after the For Loop container finishes, and the DataReady package variable is set to 1
Extract CreditCard Data	Execute Package task	Runs the CDCCreditCardExtract package after the For Loop container finishes, and the DataReady package variable is set to 1.
Extract WorkOrder Data	Execute Package task	Runs the CDCWorkOrderExtract package after the For Loop container finishes, and the DataReady package variable is set to 1.
Log Extraction Complete	Script task	Enters a message in the event log that indicates that the extraction interval has finished. This task runs after all child packages have finished.

CDCCustomerExtract Package Components

When the MasterCDC package runs, two package configurations of type, "Parent package variable", enable the MasterCDC package to pass package variables to the CDCCustomerExtract package. The following table lists these package variables.

Parent	Child
User::ExtractEndLSN	EndLSN
User::ExtractStartLSN	StartLSN

The following table lists the tasks, containers, data sources and destinations, and transformations that the CDCCustomerExtract package uses.

Name	Element	Purpose
Generate SQL Data Query	Script task	Uses the package variables, StartLSN and EndLSN, to determine the LSN values to provide to the generated function that queries for change data. The query is deposited in the package variable, SQLDataQuery.
Process Change Data	Data Flow task	Uses an OLE DB source to query for change data. A data flow component then separates the returned rows into flows—insert, delete, and update—that correspond to the operation required to apply the row to the destination. An OLE DB destination handles the insert flow. Data Flow components that run SQL commands for each row handle the delete and update flows.
Query for Change Data	OLE DB source	Uses the query that the SQLDataQuery package variable supplies to extract change data. The query uses the connection manager, AdventureWorks2008.
Filter Based on Operation	Conditional Split transformation	Divides the results set that the query returns into three different flows. The transformation uses the value of the __$operation column in each returned row to divide the results set: If the value is 2, directs the row to the insert flow. If the value is 4, directs the row to the update flow. If the value is 1, directs the row to the delete flow.
Customer Inserts	OLE DB destination	Processes the insert flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then inserts the specified rows into the target Customer table.
Customer Deletes	OLE DB Command transformation	Processes the delete flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then deletes the specified rows from the target Customer table.
Customer Updates	OLE DB Command transformation	Processes the update flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then updates the specified rows in the target Customer table.

CDCCreditCardExtract Package Components

When the MasterCDC package runs, two package configurations of type, "Parent package variable", enable the MasterCDC package to pass package variables to the CDCCreditCardExtract package. The following table lists these package variables.

Parent	Child
User::ExtractEndLSN	EndLSN
User::ExtractStartLSN	StartLSN

The following table lists the tasks, containers, data sources, data destinations, and transformations that the CDCCreditCardExtract package uses.

Name	Element	Purpose
Generate SQL Data Query	Script task	Uses the package variables StartLSN and EndLSN, to determine the LSN values to provide to the generated function call that queries for change data. The query is deposited in the package variable, SQLDataQuery.
Process Change Data	Data Flow task	Uses an OLE DB source to query for change data. A data flow component then separates the returned rows into flows—insert, delete, and update—that correspond to the operation required to apply the row to the destination. An OLE DB destination handles the insert flow. Data Flow components that run SQL commands for each row handle the delete and update flows.
Query for Change Data	OLE DB source	Uses the query that the SQLDataQuery package variable supplies to extract change data. The query uses the connection manager, AdventureWorks2008.
Filter Based on Operation	Conditional Split transformation	Divides the results set that the query returns into three different flows. The transformation uses the value of the __$operation column in each returned row to divide the results set: If the value is 2, directs the row to the insert flow. If the value is 4, directs the row to the update flow. If the value is 1, directs the row to the delete flow.
CreditCard Inserts	OLE DB destination	Processes the insert flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then inserts the specified rows into the target CreditCard table.
CreditCard Deletes	OLE DB Command transformation	Processes the delete flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then deletes the specified rows from the target CreditCard table.
CreditCard Updates	OLE DB Command transformation	Processes the update flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then updates the specified rows in the target CreditCard table.

CDCWorkOrderExtract Package Components

When the MasterCDC package runs, two package configurations of type, "Parent package variable", enable the MasterCDC package to pass package variables to the CDCWorkOrderExtract Extract package. The following table lists these package variables.

Parent	Child
User::ExtractEndLSN	EndLSN
User::ExtractStartLSN	StartLSN

The following table lists the tasks, containers, data sources, data destinations, and transformations that the CDCWorkOrderExtract package uses.

Name	Element	Purpose
Generate SQL Data Query	Script task	Uses the package variables, StartLSN and EndLSN, to determine the LSN values to provide to the generated function call that queries for change data. The query is deposited in the package variable, SQLDataQuery.
Process Change Data	Data Flow task	Uses an OLE DB source to query for change data. A data flow component then separates the returned rows into flows—insert, delete, and update—that correspond to the operation required to apply the row to the destination. An OLE DB destination handles the insert flow. Data Flow components that run SQL commands for each row handle the delete and update flows.
Query for Change Data	OLE DB source	Uses the query that the SQLDataQuery package variable supplies to extract change data. The query uses the connection manager, AdventureWorks2008.
Filter Based on Operation	Conditional Split transformation	Divides the results set that the query returns into three different flows. The transformation uses the value of the __$operation column in each returned row to divide the results set: If the value is 2, directs the row to the insert flow. If the value is 4, directs the row to the update flow. If the value is 1, directs the row to the delete flow.
WorkOrder Inserts	OLE DB destination	Processes the insert flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then inserts the specified rows into the target WorkOrder table.
WorkOrder Deletes	OLE DB Command transformation	Processes the delete flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then deletes the specified rows from the target WorkOrder table.
WorkOrder Updates	OLE DB Command transformation	Processes the update flow. Uses the connection manager, AdventureWorksDW2008, to connect to the destination and then updates the specified rows in the target WorkOrder table.