Analyzing Web Site Usage

If you want to know what kind of impact your Web site has, you need to track how many users visit your site, the type and number of hits your site receives, and other site-usage information. Microsoft's SharePoint Team Services and Microsoft FrontPage 2002 Server Extensions include features that analyze usage of your site. Summary and detailed usage reports supply information such as:

• Total hits on a site

• Total unique hits on a site

• Top page by hits

• Most popular browsers

• Top referring site

• Hits per page

The usage reports rely on a plain-text log file generated by the Web server that the Web site is hosted on (either Internet Information Services or FrontPage-patched Apache Web server), which keeps a distilled record of every transaction on your Web site. When you create a usage report in a SharePoint Team Services–compatible Web page editor, such as Microsoft FrontPage 2002, the data from the log file is arranged into a readable format.

Note that usage data is collected for an entire virtual server at a time, even if you have separate Web sites on a virtual server.

Although you view these reports from within a SharePoint Team Services–compatible Web page editor, you administer the settings for processing the usage log with SharePoint Team Services and FrontPage 2002 Server Extensions commands. Using HTML Administration pages and command-line tools, you can control:

• Whether the usage data is generated.

  If you do not want to use the usage analysis features, you can turn off the usage analysis log to conserve hard-disk space (although the log files themselves are not large). If you decide that you do want to use these features, you can turn the log processing on again. Note that each time the log file is processed, the data is appended to the original log. If you want to conserve disk space, you should turn off usage analysis before it has been run the first time. This option is available only on the Microsoft Windows platform.

• When the usage log is processed.

  You can schedule the usage log to be generated at a convenient downtime for your Web site (by using the command line or HTML Administration pages on Windows or the cron  facility on UNIX). If internal employees primarily use the Web site, for example, you might schedule the log to be processed at night, when demand on the site is lower than during working hours. You can schedule the log to be processed on a daily, weekly, or monthly basis, depending on how closely you want to track your site's usage. If you only run usage reports once a month, you can process the log once a month. If you want to know how your site is being used every day, you can set the log to be processed daily. Scheduling usage log processing is not available for FrontPage 2002 Server Extensions on the Windows 2003 Server family platform.

• How long usage data is stored.

  You can store usage data indefinitely. However, unless you are running long-term usage studies, you probably do not need to store that much data. You can specify that usage analysis data be automatically deleted after a certain number of months.

• Whether to use 24-hour increments in the log file.

  If you use this option, data is accumulated for full days only. For example, if you process the usage log at 12:35 A.M. on Friday, the data is processed through midnight on Thursday. If you turn off this option, all hits are processed right up to the time set for processing. In this case, usage processing contains all hits up until 12:35 A.M. on Friday. The data in the log file determines the data used for the usage reports. Choose the 24-hour increments option if you prefer a cleaner usage report, with data for full days only.

Analyzing Web Site Usage on Windows

On the Windows platform, you have the choice of using HTML Administration pages or the command-line tools to administer usage analysis settings. On the Windows platform, the usage analysis processes rely on the SharePoint Timer Service for scheduling, and scheduling the Timer Service can be done with either HTML Administration pages or the command-line tools. For more information about the SharePoint Timer Service, see Scheduling Timed Jobs.

Note   The scheduling feature for usage analysis log processing is not available for FrontPage 2002 Server Extensions on the Windows Server 2003 family platform. Use the command line tools and the usage operation to process the log instead. The scheduling feature is available on the Windows 2000 and Windows NT 4.0 platforms. For more information about the usage operation, see Command-line Operations.

Using HTML Administration Pages to Administer Usage Settings

You can control the usage analysis log settings by using HTML Administration pages. By using the Site Administration pages, you can turn usage analysis on or off, set the schedule when to process the usage log, and specify when to delete stored usage data.

To view the Site Administration page 

• If you are a server administrator, on the server computer click Start, point to Programs, point to Administrative Tools, and then click Microsoft SharePoint Administrator, and then on the Server Administration page, click the name of the site you want to manage.

• If you are a site administrator, on your Web site, click Site Settings, and then under Web Administration, click Go to Site Administration.

Usage analysis is enabled by default. If you do not want to use the usage analysis features for your Web site, you can disable the usage analysis logging process.

To turn off usage analysis log processing 

• Go to the Site Administration page for the virtual server you want to change.

• Under Configure Usage Analysis Settings, click Change usage analysis settings.

• In the Usage Analysis section, select Off.

  If you later decide that you do want usage analysis, you can turn the log processing on again by selecting On in the Usage Analysis section.

• Click Submit.

If you choose to use the usage analysis features, you can control how often and when the usage log is processed. You can set the log to be generated on a daily, weekly, or monthly basis, and then set the time of day to process the log. Scheduling usage log processing is not available for FrontPage 2002 Server Extensions on the Windows Server 2003 family platform.

To specify when to process the usage analysis log 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Recurrence settings area, do one of the following:

   • To set a daily recurrence to run the log, select Daily, then select the time.

   • Or, to set a weekly recurrence, select Weekly, then select time and day of the week.

   • Or, to set a monthly recurrence, select Monthly, then select a day of the month and a time.

4. Click Submit.

You can store usage data indefinitely, but that can lead to a cluttered usage report. Because you most likely will not need old usage data, you can set the stored data to be deleted automatically. You specify how long to keep usage data in monthly increments (specify 0 to keep data only for the current month, 1 to keep data from the previous month, and so on). By default, usage data is stored for 12 months.

To set expiration for stored usage data 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings area, select the Automatically delete stored usage data after ___ months check box.

4. In the months box, type the number of months you want to store usage analysis data.

5. Click Submit.

When you process the usage log, you can specify whether to use full-day increments, or continue collecting data up until the time for log processing.

To track usage data by full days only 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings section, select the Process log file data for full days only check box.

4. Click Submit.

You can also specify an administrator e-mail name so that you can be notified when the automatic usage analysis processes have been run.

To specify an administrator e-mail name 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings section, in the Administrator E-mail Address(es) box, type the full e-mail address for the administrator to notify.

   If you want to include multiple e-mail addresses, separate each e-mail address with a comma. For example, "someone@microsoft.com,someone@microsoft.com".

4. Click Submit.

If you no longer want separate usage analysis settings for your Web site, you can reset the usage analysis settings to match the global usage analysis settings for your server. When you restore the defaults for usage analysis, the per-Web site or per-virtual server settings are deleted, and the global settings are applied to your Web site. For example, if the global usage analysis settings specified that usage data is kept for one month, and the log file data is processed in 24-hour increments, when you click Restore Defaults, these settings are applied to your Web site.

To restore the default usage analysis settings 

1. Open the Configure Usage Analysis administration page.

2. Click Restore Defaults.

Using the Command Line to Administer Usage Settings

You can also use the command-line tools, Owsadm and Owsrmadm, to control the usage analysis settings. To change the usage analysis settings, you use the usage operation.

Scheduling Usage Log Processing

The usage operation takes the schedule parameter and controls all of the usage analysis settings. Usage analysis is enabled by default, and the log is set to process every Sunday morning. To turn usage analysis off, you set the schedule parameter to off. The following example syntax shows how to turn off usage analysis for port 80:

owsadm.exe –o usage –schedule off –p 80

Note   The scheduling feature for usage analysis log processing is not available for FrontPage 2002 Server Extensions on the Windows Server 2003 family platform. To process the usage analysis log on this platform, use the usage operation without the schedule parameter. When you run the usage operation without the schedule parameter, the usage log processing begins immediately, rather than waiting for a scheduled time. For example, to begin running usage analysis log processing immediately for port 80, you would type:

owsadm.exe –o usage –p 80

You can include this command-line string in a script or batch file to automate log processing.

To turn usage analysis on and set the usage log file to be processed at a particular time, you use the usage operation and the schedule parameter set to a particular time. The following example syntax shows how to set the usage analysis job to be run every Monday at 11:59 P.M. for port 80.

owsadm.exe –o usage –schedule "weekly at Mon 23:59" –p 80

You can schedule the usage log to run daily, weekly, or monthly simply by specifying a day, a week, or a month in the schedule parameter. To set the log to be processed daily, use the format "daily at <time>" in the schedule parameter, for example "daily at 23:59". To specify monthly processing, use the format "monthly at <day> <time>". For example, use "monthly at 12 23:59" to process the log on the twelfth day of every month at 11:59 P.M. For more information about the values to use with the schedule parameter, see Scheduling Timed Jobs.

If you want to start the usage log to process now rather than at the next scheduled time, you can use the usage operation without the schedule parameter. Using the operation this way gives you an instant build of the usage analysis log.

Setting Other Usage Analysis Properties

To set the expiration time for usage data, you use the usageanalysislogexpiry property with the setproperty operation. For example, to set the usage analysis log to expire in six months, you would use the following syntax:

owsadm.exe –o setproperty –pn usageanalysislogexpiry –pv 6

To set the usage log to store information for full days only, you use the usagefulldaysonly property with the values true or false.

For more information about setting properties by using the command-line tools, see Setting Configuration Properties.

Choosing a Log Format

There are several standard log formats in use on the Windows and UNIX platforms. For Windows, it is recommended that you use the W3C Extended Log File Format for best results with usage analysis. You can, if you prefer, use the NCSA common log format or the IIS format, but less data is accumulated when you use these formats. On the Windows platform, usage analysis uses the following logging properties: Date, Time, Client IP Address, User Name, Method, URI Stem, Http Status, Bytes Sent, User Agent, and Referrer. The Method, URI Stem, and Http Status properties are required in order for usage analysis to run properly. The Date and Time properties are also required if they cannot be determined from the log file name.

For UNIX platforms, the preferred format is the NCSA's extended/combined log format. This format logs all of the information that the usage reports need. If you prefer to use a different log format for UNIX, be sure that the log format follows these guidelines:

• Each directive (such as %t or %u) must be separated by a comma, a space, or some other standard separator.

• Usage analysis processing uses the following directives for UNIX: Date/Time: %t; Client IP Address: %a; User Name: %u; Method/URI Stem: %r; Http Status: %s; Bytes Sent: %b; User Agent: %{User-agent}I; Referer: %{Referer}. The Http Status and URI Stem directives are required. It is recommended that you include the %t directive if the date and time cannot be determined from the log file name.

• If you do not want to use the %r directive, then you must include the %U (file name) directive.

• The Date/Time directive must appear as %t. Usage analysis does not recognize the %{format}t directive.

• You can log data from multiple virtual servers in a single file if the %v (full host name) directive is included in the log format.

Changing Logging Settings on Windows

Usage reports are generated from log files created by the Web server your Web site is hosted on (either Internet Information Services or FrontPage-patched Apache Web server). The usage data you have to work with depends on the logging settings specified for your Web server. If you want the full capabilities of the usage analysis features in SharePoint Team Services and FrontPage 2002 Server Extensions, you must make sure that the logging settings for your Web server are configured to produce the data you want.

For IIS 5.0, the default settings generate most of the usage analysis data you need (with the exception of referral data). For IIS 4.0, you must change the logging settings to generate the complete usage analysis data.

To change the usage logging settings for IIS 4.0 

1. On your server computer, click Start, point to Programs, point to Windows NT 4.0 Option Pack, point to Microsoft Internet Information Server, and then click Internet Services Manager.

2. Click the plus sign (+) next to Internet Information Server.

3. Right-click your Web server name, and then select Properties.

4. In the Master Properties box, select WWW Service, and then click Edit.

5. On the Web Site tab, in the Active log format box, select W3C Extended Log File Format, and then click Properties.

6. In the Extended Logging Properties box, on the Extended Properties tab, select the following check boxes: Date, Time, Client IP Address, User Name, Method, URI Stem, Http Status, Bytes Sent, User Agent, Referrer.

   Note that the Method, URI Stem, and Http Status check boxes must be selected (and Date and Time are recommended) in order for usage analysis to run properly. Also note that you can select as many logging check boxes as you like, but usage analysis only uses data from the check boxes listed in this step.

7. Click OK to close the Extended Logging Properties box.

8. Click OK to close the WWW Service Master Properties box.

9. Click OK to close the Properties box.

By default, all of the data you need for usage reports is collected by IIS 5.0, with the exception of referral data. If you want to know which site referred a user to your site, you must add the referral data to your log files for IIS 5.0.

To add referral data to IIS 5.0 log files 

1. Click Start, point to Programs, point to Administrative Tools, and then click Internet Services Manager.

2. Right-click your Web server name, and then select Properties.

3. In the Master Properties box, select WWW Service, and then click Edit.

4. On the Web Site tab, in the Active log format box, select W3C Extended Log File Format, and then click Properties.

5. In the Extended Logging Properties box, on the Extended Properties tab, select the Referer check box.

   Note that the Method, URI Stem, and Http Status check boxes must also be selected for usage analysis to run properly.

6. Click OK to close the Extended Logging Properties box.

7. Click OK to close the WWW Service Master Properties box.

8. Click OK to close the Properties box.

Analyzing Web Site Usage on UNIX

On the UNIX platform, you can use HTML Administration pages to set the expiration for stored usage data or to specify whether to track usage data by full days only. If you have a mail server available, you can also specify an e-mail name to receive notification that the usage log processing has been run. To schedule usage analysis log processing, however, you must use the cron facility.

Using HTML Administration Pages to Administer Usage Settings on UNIX

You can set stored usage data to be deleted automatically by specifying how long to keep the data in monthly increments (specify 0 to keep data only for the current month, 1 to keep data from the previous month, and so on). By default, usage data is stored for 12 months.

To set expiration for stored usage data 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings area, select the Automatically delete stored usage data after ___ months check box.

4. In the months box, type the number of months you want to store usage analysis data.

5. Click Submit.

When you process the usage log, you can specify whether to use full-day increments, or continue collecting data up until the time for log processing.

To track usage data by full days only 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings section, select the Process log file data for full days only check box.

4. Click Submit.

You can also specify an administrator e-mail name so that you can be notified when the automatic usage analysis processes have been run.

To specify an administrator e-mail name 

1. Go to the Site Administration page for the virtual server you want to change.

2. Under Configure Usage Analysis Settings, click Change usage analysis settings.

3. In the Additional Usage Analysis Settings section, in the Administrator E-mail Address(es) box, type the full e-mail address for the administrator to notify.

   If you want to include multiple e-mail addresses, separate each e-mail address with a comma. For example, "someone@microsoft.com,someone@microsoft.com".

4. Click Submit.

If you no longer want separate usage analysis settings for your Web site, you can reset the usage analysis settings to match the global usage analysis settings for your server. When you restore the defaults for usage analysis, the per-Web site or per-virtual server settings are deleted, and the global settings are applied to your Web site. For example, if the global usage analysis settings specified that usage data is kept for one month, and the log file data is processed in 24-hour increments, when you click Restore Defaults, these settings are applied to your Web site.

To restore the default usage analysis settings 

1. Open the Configure Usage Analysis administration page.

2. Click Restore Defaults.

Using the Command Line to Administer Usage Settings on UNIX

On the UNIX platform, you must use the command-line tools, Owsadm and Owsrmadm, to administer usage analysis settings. This is because on UNIX, the usage analysis processes are scheduled by using the cron facility. The cron facility is available only from the command line, so HTML Administration pages cannot be used for scheduling usage analysis tasks on UNIX. For more information about using cron and crontab, see the manual (man) pages for your UNIX system.

Running Usage Log Processing on UNIX

If you want regular and consistent usage analysis reports, you must process the FrontPage-patched Apache log files on a regular basis. To process the log files, you use the usage operation with Owsadm or Owsrmadm. You can use the same parameters as you use on the Windows platform to run the log processing. To schedule the log to be processed automatically, however, you use the cron facility rather than the usage operation with the schedule parameter. For example, to run usage processing for all virtual servers on the current server, you would use the following syntax with the cron facility:

/usr/local/frontpage/version5.0/bin/owsadm.exe –o usage [–p <port> -m <hostname>]

Running Remote Usage Processing on the UNIX Platform

On the UNIX platform, you have the ability to run usage analysis processing from a remote computer. You can set up a computer to read the content from the Web server and do the usage analysis, without running the usage analysis process on the actual Web server. When you do usage processing remotely, you do not have to worry about interrupting or slowing service on your main Web server while the usage analysis is running.

For example, say you have a Web server named ServerA. It hosts the content for your site and performs all of the FrontPage 2002 Server Extensions processes.

1. Using network file system (NFS), you create a share on ServerA that has all of the Web site content in it.

2. You then create another share that has the server root in it.

3. On ComputerB, install FrontPage 2002 Server Extensions in exactly the same position in its file system (so the two systems mirror each other — for example, all content on both computers is in \usr\www\htdocs\webcontent).

4. Copy the we*.cnf files from \usr\local\FrontPage\version5.0 from ServerA to ComputerB.

5. Run the usage operation by running the following on ComputerB:

   owsadm.exe –o usage

At the end of this process, the configuration files copied from ServerA are read on ComputerB, the usage analysis processing is performed, and all of the data is written back into the content that was shared out from ServerA.

Specifying Log File Locations

You must specify where the Web server log files are stored when you run usage processing on the UNIX platform. You use the LogFileLocation, LogRollover, and LogFileFormat properties to specify this information. The following table describes these properties.

For example, to use log files created every week on Sunday at midnight, the LogFileLocation value would be /usr/local/www/logs/access_log.%m-%d-%Y, and the LogRollover value would be weekly at Sun 00:00. The LogRollover value must be a valid SharePoint Timer Service value, such as "Mon at 03:00" or "Weekly at Mon 1:21". For more information about the values you can use with the Timer Service, see Scheduling Timed Jobs.

Note   For a complete list of the properties you can set from the command line, see Command-line Properties.

To get an actual path to a log file, you combine the LogFileLocation string with a time and date, which is derived from the LogRollover setting. The following table lists the format specifiers used in the LogFileLocation path:

Note   Any of the format specifiers with a numeric value can be preceded by an integer (for example, "%2n") to specify the exact number of digits to use. Otherwise, for all specifiers except %y and %Y, the minimum number of digits will be used.

The following format specifiers are not supported: hour in 12-hour format (%I), AM/PM indicator (%p), minute as decimal number (%M), seconds (%S), long-format day names (%A), and long-format month names (%B).

To specify the LogFileLocation, you include both the path and the format specifiers to use. For example, the following string gives you a log file path to files named with the month and day of the month in decimal format:

/usr/local/www/vhost1/host%m%d.log

As another example, if you wanted to rotate your log files at 3:00 A.M. every Saturday, you could copy the log files to a file with the absolute path /usr/local/www/t1234/logs/MMDD.log, where MMDD is the month and day of the month when the log files are copied. You would then set the LogRollover property to "weekly at Sat 03:00" and set the LogFileLocation property to /usr/local/www/t1234/logs/%2m%2d.log.

Specifying a Domain Name for Usage Log Processing

Usage processing uses the domain name for your Web site to determine how many visits your site has received. With some Web server configurations, the domain name cannot be determined accurately. In this case, you can use the UsageServerUrls property to specify the domain name or names for your Web site. You must specify the domain name for your Web site if your Web site is configured in any of the following ways:

• Your Web site accepts requests for multiple domain names (for example, https://www.mycompany.com and https://mycompany.com)

• Your Web site is set up for multihosting through hardware, rather than software (you only specified the IP address in the server configuration).

• No domain name is specified in the configuration file and the Web server domain name is different from the machine name.

Set the UsageServerUrls property to a comma-separated list of all the domain names that can be used to gain access to your Web site. Include the https:// prefix at the beginning of each entry in the list. For example, in the first scenario, you would specify https://www.mycompany.com,https://mycompany.com as the value for the UsageServerUrls property. If your Web site uses Secure Sockets Layer (SSL), you must also include the https:// prefix in the list.

The following example syntax sets the UsageServerUrls property to recognize both www.mycompany.com and mycompany.com:

owsadm.exe –o setproperty –pn usageserverurls 
–pv "https://www.mycompany.com,https://mycompany.com" –p 80

Related Links

For more information about the SharePoint Timer Service, see Scheduling Timed Jobs.

For more information about setting the usage analysis properties on the command line, see Setting Configuration Properties.