Log Files

You use the time overlap property to eliminate redundancy that can occur when log files contain requests that overlap in time. You use the log file rotation property to eliminate redundancy or the loss of data that can occur when Log files are rotated among servers to keep the log files from getting too large.

Without the time overlap and log file rotation settings, it is impossible to import log files into the Data Warehouse without losing activity data because log files are continually created as your site operates. Your files may contain data for overlapping time periods if you run log files on separate servers, interrupt and resume logging on a single server, accidentally re-import a log file, or concatenate distinct log files.

Time Overlaps

Log File Rotation

Time Overlaps

Time overlaps may occur if you have multiple servers associated with a site, interrupt and resume logging on a single server, accidentally re-import a log file, or concatenate distinct log files. (Concatenation makes tracking time overlaps extremely difficult and therefore should be avoided.) The default time overlap period for Commerce Server is 30 minutes.

When a time overlap is detected during import, by default the overlapping requests are discarded and the remaining requests are imported. You can change both the time overlap period and the action you want to occur when an overlap is detected.

If you import a log file accidentally, you can delete the log file using the Data deletion task. For information about deleting log files, see Deleting Data from the Data Warehouse.

The following figure shows how the four overlap actions, Discard records and proceed, Import all records, Stop the import, and Stop all imports, affect the portions of the second and third log files that are retained when an overlap is detected.

Portion of log file not imported

Portion of log file imported

Log file

Log File 3

Log File 2

Log File 1

Discard records and proceed

Import all
records

Stop the
import

Stop all
imports

To configure Web log file time overlaps

1. Expand Microsoft SQL Server, expand SQL Server Group, and then expand the server on which your Data Warehouse is installed.

2. Right-click Data Transformation Services, and then click New Package.

   Alternatively, if you are changing an existing package, right-click Data Transformation Services, click All Tasks, and then select Open Package. Select the package you want to change, and then click Open.

3. On the Task menu, click Web server log import (Commerce Server).

4. In the Import Web Server Logs dialog box, click Advanced.

5. In the Advanced Web Log Import Properties dialog box, on the Log Files tab, do the following:

   Use this	To do this
   Number of minutes that records must overlap	Type the number of minutes that records must overlap.
   If overlap is detected	Select an action to take if an overlap is detected: Discard records and proceed. (Default setting) Discards all overlap requests and imports the remaining requests. Import all records. Imports all requests, including overlap requests. Stop the import. Does not import the log file containing overlap requests. All other log files are imported. Stop all imports. Stops all log file imports. You must delete the imported log file if you do not want to keep the data imported before the overlap request was detected.

Note

  - Select **Stop the import** or **Stop all Imports** to troubleshoot problems you may experience with importing.

6. Click Apply, and then click OK.

The time overlap property is now configured for the Web log file import process. You can continue to configure the Web log file import properties, or you can import data into the Data Warehouse. You must synchronize your site configuration with the Data Warehouse before you import data into the Data Warehouse.

Log File Rotation

Log file rotations cause an arbitrary cutoff of data in log files because data for some visits are divided between the end of one log file and the beginning of the next. You can choose to store, commit, or disregard the open visits at the end of your import process.

To set log file rotation

1. Expand Microsoft SQL Server, expand SQL Server Group, and then expand the server on which your Data Warehouse is installed.

2. Right-click Data Transformation Services, and then click New Package.

   Alternatively, if you are changing an existing package, right-click Data Transformation Services, click All Tasks, and then select Open Package. Select the package you want to change, and then click Open.

3. On the Task menu, click Web server log import (Commerce Server).

4. In the Import Web Server Logs dialog box, click Advanced.

5. In the Advanced Web Log Import Properties dialog box, on the Log Files tab, do the following:

   Use this

   To do this

   When import is completed

   Select an action to take with the open visits at the end of your import process: Store open visits for the next import. (Default setting) Reconstructs the actual visit as if there were one seamless log file. This option takes slightly more time because the open visits must be called up from the cache at each new import. You must import log files in chronological order for this option to work correctly. Commit all open visits to database. The visit count is higher at the opening and closing of the log file period because visits that span those two periods are counted twice. Discard open visits. The report count is lower at the closing of the log file period, because visits that have not been closed are dropped.

6. Click Apply, and then click OK.

The log file rotation property is now configured for the Web log file import process. You can continue to configure the import properties, or you can import data into the Data Warehouse. You must synchronize your site configuration with the Data Warehouse before you import data into the Data Warehouse.

See Also

Data Warehouse Components

Configuring Web Log File Import Properties

Importing Data into the Data Warehouse

All rights reserved.