Site Server Express - Automating Import Usage and the Report Writer Operations

  • Article

Introduction

Microsoft® Usage Import and Report Writer are usually used to produce reports of site activity on a regular basis. The site's operations manager may want to see a report of the previous day's total site activity every morning at 8 a.m., while the managing editor may require a report of visitors by content area every Friday at noon.

Maintenance tasks, such as compacting databases and updating metadata, will also need to be done on a regular schedule.

To make it easy to have tasks performed automatically when they are needed, the Scheduler tool is provided so that any task can be scheduled to occur automatically.

Jobs, Tasks, and Batch Files

In using the Scheduler, you will be scheduling jobs and setting up tasks within those jobs.

A job is a collection of tasks that is scheduled to begin at a certain day and time. A task is a specific activity, such as importing a log or creating a report. A typical job could contain the following elements:

After you enter the required information, the Scheduler creates a batch file, which is a collection of operating-system commands that executes the required tasks. Here is what part of a batch file might look like:

REM MARKETFOCUS START DO NOT MODIFY 
REM MARKETFOCUS TASK "Import stuff" 
%ANALYSIS% -silence messagelog="C:\Program Files\Usage Analyst\Message\Big job.1.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" report="C:\Program Files\Usage Analyst\Reports\trial.html" analysis="C:\Program Files\Usage Analyst\Catalog\Comprehensive\Comprehensive report.mfa" 
REM MARKETFOCUS END DO NOT MODIFY

Although it is not necessary for automating operations, you can edit the batch file directly to add other operations to the job, such as copying a log file from one computer to another or emailing a finished report to a distribution list. For more information about editing the batch file, see "Editing the Batch File" later in this chapter.

The Scheduler Window

You can open the Scheduler window, shown in the following illustration, from either Usage Import or the Report Writer by choosing Scheduler… on the Tools menu or by clicking the Scheduler button on the Toolbar.

 

There is no significance to the order in which jobs are displayed in the tree structure; when a new job is added, it appears directly below the All Jobs icon.

The sequence in which the tasks within a job are shown is significant: Tasks are performed in the order shown.

Icons

Each type of task has a distinctive icon, as shown in the following table.

Icon

Task Type
 

Analysis
 

Compact database
 

Delete
 

Import
 

IP resolution
 

Title lookups
 

Whois organizations

The Scheduler has menus that appear when you click on an icon with the right mouse button. Different menus appear for the All Jobs icon, a job icon, and a task icon.

Setting Up a Job

The process of setting up a job includes two main steps:

  1. Adding a new job to the Scheduler window

  2. Setting the schedule and message-log properties 

Adding a New Job

To add a new job 

  1. With the right mouse button, click on the All Jobs icon or on any job icon.

  2. On the menu that appears, choose New Job….

The Job Properties dialog box, shown in the following figure, appears.

Setting Job Properties

The properties of the job include its schedule and the name of the log file to which it posts messages.

Setting the Schedule

To set the schedule 

  1. In the Occurs section of the Job Properties dialog box, select the option button for the schedule frequency.

    Note If you select Today or Tomorrow, the job will be run only once.

  2. If you selected Every or Next, select the appropriate term in the Days list.

    The list includes the days of the week as well as the numbers 1 through 31. 

  3. In the Time box, type the time of day when you want the job to be run.

Time Format

Times are entered in 24-hour format. For example, for 8 A.M., type 8:00:00; for 2:30 P.M., type 14:30:00.

Note You must always type digits for minutes and seconds, separated by colons. You do not need to type A.M. or P.M.

Schedule Conflicts

When you choose OK, the scheduler will notify you if there is any other job scheduled at the same time as the new job, which could lead to interference between the two. If there is, you have the option of rescheduling the new job or editing the schedule of the other job, or you can leave both schedules as they are.

Active Checkbox

Check this box when you want the job to become active. If the box is not checked, the scheduling facility in the Windows NT 4.0 operating system will not start the job. This allows you to turn jobs on and off without having to delete them or change their schedules.

Setting the Message Log

When the job is run, messages about the results of each task are saved in a log file. In the Message Log box, type the path and name for that file. In the drop-down list box, you can select variables to add to the file name. The variables will be replaced by the current value when the file is created.

Note Drop-down list boxes similar to the one described above appear on several dialog boxes in the Scheduler. They all work the same way: When you drop the list and click on an item to select it, the list closes and the item appears in the box. Without clicking the mouse button, you can move the pointer to a field on the dialog box. The pointer is surrounded by a wide rectangle. With the rectangle positioned over a field, click the left mouse button. The item you selected from the drop-down list box is pasted into the field.

Editing Job Properties

You can edit any of the properties of a job at any time.

To edit properties of a job 

  1. Click on the job with the right mouse button.

  2. On the menu that appears, choose Edit.

  3. Make your changes and choose OK.

Copying Jobs

You can make copies of jobs. The copies contain all of the properties and tasks of the original job. This is an efficient way to perform the same set of tasks on different schedules. After you make a copy, change the schedule for the new job so it doesn't conflict with the original one. You can also change any other properties of the job and the tasks it contains.

To copy a job 

  1. Click on the name of the job with the left mouse button.

  2. On the Edit menu, choose Copy, or press the CTRL and C keys.

  3. On the Edit menu, choose Paste, or press the CTRL and V keys.

    The copy of the job is added to the tree below the original job. 

Changing the Name of the Job

When a job is added to the Scheduler, it is named "New job" (or "New job n," if necessary to avoid a name conflict). You can change the name to something more meaningful.

To change the name of the job 

  1. Click on the name of the job.

  2. When the name appears surrounded by a solid line, type the new name.

Setting Up Tasks

You can think of a job as a container for tasks. The job by itself does nothing; it's the tasks you add to the job that do the work.

Adding a Task

Usage Import and the Report Writer can perform seven kinds of tasks:

  1. Analysis

  2. Compact database

  3. Delete

  4. Import

  5. IP resolution

  6. Title lookups

  7. Whois organizations

You can add any of these tasks to a job in any combination. A task can be added more than once. For example, a job might contain three analysis tasks to produce three reports from the same data.

To add a task to a job 

  1. With the right mouse button, click on the job name or on the name of an existing task in the job.

    On the menu that appears, choose New Task. 

  2. The Task Properties dialog box, shown in the following figure, appears.

  3. In the Task Type drop-down list, choose the kind of task you want to add to the job.

    The Task Properties dialog box changes to show the fields required for the kind of task you chose. 

  4. Type the information required for the task. Choose OK when you are finished.

    If you click on a job name to start this procedure, the new task appears below the job name. If you clicked on an existing task, the new task appears below that task. 

Setting Task Properties

For each type of task (except for IP resolution and Whois), there are one or more properties you need to supply. The following table lists the properties for each type of task along with brief explanations of them.

Task

Property

Description

Analysis

Analysis

The path and name of the .MFA file that defines the report to be prepared
 

Report

The path and file name for the finished report
 

Date Replacement Macros

Variables that will be replaced by the current value when the report is created

Compact Database
(Access only)

Source Database

The path and name of the database to be compacted
 

Destination Database

The path and name for the compacted database

Delete

Boolean Filter

A Boolean expression to be used as the criterion for deleting requests from the database
 

Reference Terms

Boolean terms that can be used to build the Boolean expression

Import

Log Data Source

The name of the log data source for which data will be imported
 

Log Location

The path and name of the log file that contains the data to be imported
 

Date Replacement Macros

Variables that will be replaced by the current value when the report is created

IP Resolution

None

  
 

Date Replacement Macros

Variables that will be replaced by the current value when the report is created
 

Site ID

The identification of the site

Title Lookups

Boolean Filter

A Boolean expression to be used as the criterion for looking up the title of the file
 

Reference Terms

Boolean terms that can be used to build the Boolean expression

Whois Organizations

None

  

Editing Task Properties

You can edit any of the properties of a task at any time.

To edit properties of a task 

  1. Click on the task with the right mouse button.

  2. On the menu that appears, choose Edit.

  3. Make your changes and choose OK.

Note When you edit a task, you can change it to a different kind of task altogether.

Moving and Copying Tasks

Tasks are performed in the order in which they appear in the job tree. You can rearrange the tasks as necessary.

To move a task 

  1. Click on the task with the left mouse button and hold the button down.

  2. Drag the task to its new location.

    As you move the mouse, the pointer changes to an icon that represents the task. The names of other tasks are highlighted as the pointer moves over them. When you release the mouse button, the task you're moving is inserted below the task that is highlighted. 

You can make copies of tasks. The copies contain all of the properties of the original task. This is an efficient way to perform the same task with different properties. After you make a copy, change the properties of the new task as necessary.

To copy a task 

  1. Click on the name of the task with the left mouse button.

  2. On the Edit menu, choose Copy, or press the CTRL and C keys.

  3. On the Edit menu, choose Paste, or press the CTRL and V keys.

    The copy of the task is added to the tree below the original task. 

Changing the Name of the Task

When a task is added to the Scheduler, it is named "New task" (or "New task n," if necessary to avoid a name conflict). You can change the name to something more meaningful.

To change the name of the task 

  1. Click on the name of the task.

  2. When the name appears surrounded by a solid line, type the new name.

Log File Rotation

Your Internet server records all hits in a log file. Log file rotation is the process of stopping logging, renaming the log file to a unique name based on the time or date, and resuming logging to a new log file. For example, suppose you wanted to analyze a day's worth of hits at a time. To do that, you would want to rotate your server log file once a day. If your server records hits in a file called LOGFILE, you could rotate that file at 12:01 A.M. every day, renaming the file to LOGFILE.<date> and resuming logging to an empty LOGFILE.

Log file rotation is not controlled by Site Server Express Analysis, but the facility for log rotation is built into most web servers. Consult your web server documentation for instructions on how to set up log rotation.

Editing the Batch File

For each job, the Scheduler creates a batch file, which is stored in the …\Usage Analyst\Schedule directory. Here's what a typical batch file might look like:

REM MARKETFOCUS START DO NOT MODIFY 
REM MARKETFOCUS SCHEDULER BATCH FILE-- DO NOT MODIFY IN 
REM MARKETFOCUS BETWEEN COMMENTS
REM MARKETFOCUS CREATED 4/11/97 4:38:08 PM
REM MARKETFOCUS ACTIVE False
REM MARKETFOCUS OCCURS 2
REM MARKETFOCUS DAYS 7
REM MARKETFOCUS TIME 12:00:00 AM
SET IMPORT="C:\Program Files\Usage Analyst\IMPORT.EXE"
SET ANALYSIS="C:\Program Files\Usage Analyst\ANALYSIS.EXE"
SET METADATA="C:\Program Files\Usage Analyst\METADATA.EXE"
SET MAIL="C:\Program Files\Usage Analyst\UTILITY\MAIL.EXE"
REM MARKETFOCUS END DO NOT MODIFY

 

This section imports a log file into the Access database. The log file is being accessed from the local file system.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Import"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" log="G:\LogFiles\in$i.log" logsource="LogSource1"
REM MARKETFOCUS END DO NOT MODIFY

 

This section performs title lookups.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Title lookup"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" titles='Title=""'
REM MARKETFOCUS END DO NOT MODIFY

 

This section performs the IP address resolution.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "IP resolution"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" action="IP resolution"
REM MARKETFOCUS END DO NOT MODIFY

 

This section performs the Whois lookup.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Whois lookup"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" action="Who is"
REM MARKETFOCUS END DO NOT MODIFY

 

This section prepares the executive summary report.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Analysis"
%ANALYSIS% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" report="C:\Program Files\Usage Analyst\Reports\executive_summary.htm" analysis="C:\Program Files\Usage Analyst\Catalog\Summary\Executive summary.mfa"
REM MARKETFOCUS END DO NOT MODIFY

 

This section prepares the request detail report.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Analysis"
%ANALYSIS% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" report="C:\Program Files\Usage Analyst\Reports\request detail.htm" analysis="C:\Program Files\Usage Analyst\Catalog\Summary\Request detail.mfa"
REM MARKETFOCUS END DO NOT MODIFY

 

This section deletes old data.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Delete data over 6 months old"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" delete="Date>ThisMonth-6"
REM MARKETFOCUS END DO NOT MODIFY

 

This section compacts the Access database.

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Compact database"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" sourcedb="C:\Program Files\Usage Analyst\MSUsage.mdb" destdb="C:\Program Files\Usage Analyst\temp.mdb"
REM MARKETFOCUS END DO NOT MODIFY

 

In addition to the commands that the Scheduler adds to the batch file, you can add commands to perform other operations. For example, you can add commands to copy a log file from a web server to the computer where Usage Import and the Report Writer are running.

For more information about batch-file commands that are not part of Usage Import and the Report Writer, refer to your Windows NT 4.0 operating system documentation.

To edit the batch file

  1. With the right mouse button, click on the job whose batch file you want to edit.

  2. On the menu that appears, choose Edit Batch File.

    The batch file appears in a WordPad window. 

Note Do not edit any lines in groups that begin with "REM MARKETFOCUS START DO NOT MODIFY" and end with "REM MARKETFOCUS END DO NOT MODIFY." Changes in those groups will be lost the next time you open Scheduler.

Automating Report Distribution

Reports produced by the Report Writer are stored on the local computer. If you want to distribute the reports automatically, you can edit the job batch file to publish them on the web.

The easiest way to distribute reports through the web is to run a web server on the Windows NT system where the Report Writer is running. If you need to post your HTML reports to external web server, then you'll need to follow the steps described below:

To post HTML reports to a web server 

  1. Create all your HTML report files in the same (and preferably empty) directory.

  2. Set up a command-line FTP script with the correct user name, password, and path information, such as the following example:

    user username password
    lcd C:\Progra~1\Usagea~1\Reports
    cd /web/hthome/reports
    bin
    prompt
    mput *.htm *.jpg
    bye 

  3. Modify your job batch file to run a command-line FTP script.

Sample Batch File Modifications

The following example shows the batch file produced by the Scheduler (shown above) with several modifications:

REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS SCHEDULER BATCH FILE-- DO NOT MODIFY IN
REM MARKETFOCUS BETWEEN COMMENTS
REM MARKETFOCUS CREATED 4/11/97 4:38:08 PM
REM MARKETFOCUS ACTIVE False
REM MARKETFOCUS OCCURS 2
REM MARKETFOCUS DAYS 7
REM MARKETFOCUS TIME 12:00:00 AM
SET IMPORT="C:\Program Files\Usage Analyst\IMPORT.EXE"
SET ANALYSIS="C:\Program Files\Usage Analyst\ANALYSIS.EXE"
SET METADATA="C:\Program Files\Usage Analyst\METADATA.EXE"
SET MAIL="C:\Program Files\Usage Analyst\UTILITY\MAIL.EXE"
REM MARKETFOCUS END DO NOT MODIFY

 

Commands added to use FTP to copy a log file from a remote web site.

REM ADD-IN COMMAND
REM DOWNLOAD COMPRESSED LOG FILE FROM FTP SITE USING COMMAND LINE FTP
REM See ftp help screen for more details
REM ftpdown.txt is simply a list of ftp commands
ftp -v -i -n -s:ftpdown.txt www.webserver.com

Commands added to decompress the log file.

REM ADD-IN COMMAND
REM UNCOMPRESS LOG FILE USING COMMAND LINE UNZIP UTILITY
REM You must supply your own unzip utility
unzip C:\LogFiles\logfile.zip
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Import"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" log="G:\LogFiles\in$i.log" logsource="LogSource1"
REM MARKETFOCUS END DO NOT MODIFY

 

Commands added to remove the old log file from the local file system after it has been imported.

REM ADD-IN COMMAND
REM DELETE LOG FILE TO MAKE ROOM FOR NEXT ONE
echo y|erase C:\LogFiles\logfile.log
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Title lookup"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" titles='Title=""'
REM MARKETFOCUS END DO NOT MODIFY
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "IP resolution"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" action="IP resolution"
REM MARKETFOCUS END DO NOT MODIFY
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Whois lookup"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" action="Who is"
REM MARKETFOCUS END DO NOT MODIFY
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Analysis"
%ANALYSIS% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" report="C:\Program Files\Usage Analyst\Reports\executive_summary.htm" analysis="C:\Program Files\Usage Analyst\Catalog\Summary\Executive summary.mfa"
REM MARKETFOCUS END DO NOT MODIFY
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Analysis"
%ANALYSIS% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" report="C:\Program Files\Usage Analyst\Reports\request detail.htm" analysis="C:\Program Files\Usage Analyst\Catalog\Summary\Request detail.mfa"
REM MARKETFOCUS END DO NOT MODIFY

 

Commands added to use FTP to copy reports to a web site for viewing by others.

REM ADD-IN COMMAND
REM UPLOAD REPORTS TO REMOTE WEB SERVER USING COMMAND LINE FTP
REM See ftp help screen for more details
REM ftpup.txt is simply a list of ftp commands
ftp -v -i -n -s:ftpup.txt www.webserver.com

 

Commands added to remove reports from the local file system.

REM ADD-IN COMMAND
REM CLEAN OUT REPORT DIRECTORY
echo y|erase C:\Progra~1\Usagea~1\Reports\*.htm
echo y|erase C:\Progra~1\Usagea~1\Reports\*.jpg
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Delete data over 6 months old"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" db="C:\Program Files\Usage Analyst\MSUsage.mdb" delete="Date>ThisMonth-6"
REM MARKETFOCUS END DO NOT MODIFY
REM MARKETFOCUS START DO NOT MODIFY
REM MARKETFOCUS TASK "Compact database"
%IMPORT% -silence messagelog="C:\Program Files\Usage Analyst\Message\MUA$M$Y.log" sourcedb="C:\Program Files\Usage Analyst\MSUsage.mdb" destdb="C:\Program Files\Usage Analyst\temp.mdb"
REM MARKETFOCUS END DO NOT MODIFY