Searching SharePoint Team Web Sites and FrontPage-Based Web Sites

Updated : April 1, 2004

SharePoint™ Team Services from Microsoft® and FrontPage® 2002 Server Extensions from Microsoft include full-text search functionality that allows your users to find documents on your Web sites. The search functionality is implemented differently depending on the version of SharePoint Team Services or FrontPage 2002 Server Extensions that you are using. New versions of SharePoint Team Services, included with Microsoft Office XP Service Pack 3 (SP3), and FrontPage 2002 Server Extensions, included with Microsoft Windows Server™ 2003 Service Pack 1 (SP1), provide Integrated Indexing Server Search, a more scalable implementation of searching than the original versions of these technologies.

The following table lists the platforms, versions, and search implementations for SharePoint Team Services and FrontPage 2002 Server Extensions.

Note: To install or upgrade programs on the server running SharePoint Team Services or FrontPage Server Extensions, you must be a member of the local Administrators group. To use the server administration tools for SharePoint Team Services and FrontPage 2002 Server Extensions, you must be a member of the local Administrators group.

On This Page

Using the Search Proxy and Search Bot Methods
Using WAIS
Using Integrated Indexing Server Search

Using the Search Proxy and Search Bot Methods

When the original versions of SharePoint Team Services or FrontPage 2002 Server Extensions are installed on the Windows platform, the search features use the document index and search component of Index Server. Index Server is a content-indexing and search component included with Microsoft Internet Information Server (IIS) 3.0 and later. The Index Server documentation provides a full discussion on such features as creating custom searches, and provides more information on installing Index Server, setting up the search catalog, associating it with a virtual server, and setting administration options.

Index Server uses catalogs to index the content of your Web site. The index catalogs for searching are stored on the file system of the Web server computer. If you are running Windows 2000 Server, you have a dedicated catalog for SharePoint Team Services. If you are running Windows NT 4.0 Workstation (FrontPage 2002 Server Extensions only), or Windows 2000 Professional, Web site content is indexed in the default catalog for your Web server. If you are running Windows NT 4.0 Server (FrontPage 2002 Server Extensions only), Web site content is indexed in separate catalogs for each Web site on the Web server.

Note: The search features in SharePoint Team Services and FrontPage 2002 Server Extensions are turned on automatically when you install SharePoint Team Services. There are no administrative controls within SharePoint Team Services or FrontPage 2002 Server Extensions to disable searching. You can turn off searching by turning off Index Server, but the search controls in the SharePoint team Web site user interface are not disabled.

About the Search Proxy Method for SharePoint Team Services on Windows 2000 Server

When you add a Search box to a SharePoint team Web site running on Windows 2000 Server, you use the search proxy method for searching. Using this method, SharePoint team Web sites are indexed in a dedicated search catalog used only by SharePoint Team Services. This catalog is created automatically when SharePoint Team Services is installed and appears as a virtual server called OWS_Search_Proxy on the root web of your Web server. When you add virtual servers to your Web server and extend them with SharePoint Team Services, each virtual server is automatically added to the catalog as a virtual directory. For example, if you have Web server X with virtual servers a, b, and c, you will have virtual directories named OWS_VDIR_A, OWS_VDIR_B, and so on within the OWS_Search_Proxy virtual server. These virtual directories point back to the content on the individual virtual servers. Segmenting the catalog into virtual directories allows you to search within a single virtual server, rather than across all virtual servers on the server computer.

Because the catalog is owned by SharePoint Team Services, it is updated automatically when you uninstall SharePoint Team Services. If you uninstall from a single virtual server, the virtual directory for that virtual server is automatically removed from the OWS_Search_Proxy virtual server. If you uninstall completely, the OWS_Search_Proxy virtual server is removed as well.

Note: The search proxy method is only used for Search boxes in SharePoint team Web sites. If you add a Search Web component (or Search bot) to a team Web site, the Search bot method is used for that component.

About the Search Bot Method on Windows NT 4.0 or Windows 2000 Professional

When you add a Search Web component (also known as a Search bot) to a page on a SharePoint team Web site or FrontPage-extended Web site, that Web site’s index is saved in an Index Server catalog on the Web server computer. If you have multiple Web sites (as you can with Windows NT 4.0 Server), you end up with multiple catalogs. This method does not scale well, because performance decreases quickly when you have several search catalogs on a Web server.

When you install Office XP SP3 or Windows Server 2003 SP1, this search method is replaced with the new method, Integrated Indexing Server Search.

Note: You must upgrade existing virtual servers and recalculate the Web site before Integrated Indexing Server Search can be used. Integrated Indexing Server Search is automatically used for any new virtual servers you extend after you have installed one of the service packs. You can upgrade your virtual server by using the Upgrade virtual server command on the Virtual Server Administration page or the upgrade operation for Owsadm.exe. You can recalculate a Web site by using the Recalculate the Web command (on the Site Administration page), the recalc operation (for Owsadm.exe or Owsrmadm.exe), or the Recalculate Hyperlinks command (Tools menu in FrontPage).

Using WAIS

On the UNIX platform, the FrontPage 2002 Server Extensions use the WAIS search engine. With the WAIS engine, a full-text index is created in the _vti_txt directory under the Web's top-level directory. The index is updated whenever a page is saved or whenever the Web is recalculated. New content is automatically added to the index as pages are saved. However, deleted content is not removed from the index until the Web site is recalculated using the Recalculate the Web command (on the Site Administration page), the recalc operation (for Owsadm.exe or Owsrmadm.exe), or the Recalculate Hyperlinks command (Tools menu in FrontPage).

On the Windows platform, you can also use the WAIS search engine with the downloadable version of FrontPage 2002 Server Extensions (not the version included with the Windows Server 2003 family). To use WAIS, you must set the NoIndexServer registry key for each port.

Note: WAIS is not available in Windows Server 2003. If you install Office XP SP3 or later or Windows Server 2003 SP1 or later, Integrated Indexing Server Search is available. If you use Windows 2003 Server without Office XP SP3 or Windows Server 2003 SP1, you must use Indexing Service (search proxy and search bot methods). After you install one of the service packs, any existing Search bots are broken until you upgrade the virtual servers. You must also delete the NoIndexServer registry key for all ports and for any specific ports that used the WAIS search method. You can upgrade your virtual server by using the Upgrade virtual server command on the Virtual Server Administration page or the upgrade operation for Owsadm.exe.

Using Integrated Indexing Server Search

Integrated Indexing Server Search provides a more scalable way to search Web site content for SharePoint Team Services and FrontPage 2002 Server Extensions. When you install Office XP SP3 or Windows Server 2003 SP1, Integrated Indexing Server Search is applied to the server. Any new virtual servers that you extend use the new search method. To use the new method with existing virtual servers, you must manually upgrade each existing virtual server, and then recalculate each Web site.

Note: You can upgrade your virtual server by using the Upgrade virtual server command on the Virtual Server Administration page or the upgrade operation for Owsadm.exe. You can recalculate a Web site by using the Recalculate the Web command (on the Site Administration page), the recalc operation (for Owsadm.exe or Owsrmadm.exe), or the Recalculate Hyperlinks command (Tools menu in FrontPage).

With the Search bot method of searching, the search catalogs are stored on the Web server computer, with one catalog per Web site on that server. Because Indexing Service runs into performance issues when there are too many search catalogs, this method does not scale. Also, because of the way Index Server or Indexing Service interacted with FrontPage 2002 Server Extensions, if you search a Web site that contains content stored on a computer other than the Web server computer (for example, if you wanted to use a UNC Web, with Web site content stored on a networked computer), the content is not included in the search and an error is generated.

In contrast, with Integrated Indexing Server Search, catalogs are stored on the content server, and Indexing Service is run on the computer that contains the content. By default, there is only one catalog per content server computer, so neither the content server nor the Web server’s performance is impacted by the number of catalogs. FrontPage 2002 Server Extensions and SharePoint Team Services simply maintain a registry list of the content and catalogs that belong to a site, no matter where the content is stored. So, you can search content from a UNC or Distributed File System (DFS) path and return the results just like any other type of search.

When a user searches a site using Integrated Indexing Server Search, the search process goes through the following steps:

1. The Internet Explorer client sends the search request to the Web server running FrontPage 2002 Server Extensions or SharePoint Team Services.

2. The Web server forwards the search query to Indexing Service running on the computer that contains the site content.

3. The Indexing Service returns the search results to the Web server.

4. The Web server returns a search results page to the client computer.

These steps are the same whether the content is on the same computer as the Web server, or on a separate computer. The Indexing Service always runs at the content location, whether that is the Web server computer, or a UNC or DFS content location.

The change in how searching is performed does not affect how the user interacts with the Search bots on the Web pages. The user simply types the search text in the Search box and clicks Go. Results are returned and listed on a page. If you want to customize the Search bot, you can do so by using FrontPage, just like any other Web component.

Note: The result set for a search run using Integrated Indexing Server Search is limited to 250 records.

About UNC, DFS, and Integrated Indexing Server Search

If you are using UNC or DFS storage of Web site content, keep in mind the following points:

• UNC and DFS storage of Web site content is only available for FrontPage 2002 Server Extensions running on the Windows Server 2003 family.

• For DFS, all backup computers in the distributed file system must have the same catalog location and name.

• For UNC and DFS, you must verify that Indexing Service is enabled on the computers storing the Web content. If you do not turn Indexing Service on for the content computers, searching will not work.

If you are using Windows Integrated authentication, you must also enable Kerberos delegation between servers in order to allow the Web server computers to query the content computers.

For more information about UNC, see Connecting Web Sites to UNC Network Shares.

About Security and Integrated Indexing Server Search

If you are using Integrated Indexing Server Search in a large-scale environment, it is recommended that you use the following methods to create a more secure environment:

• Use separate application pool identities for each site. For more information, see Internet Information Services Help.

• Use a separate user account for each virtual server’s anonymous account in IIS.

• Use the user isolation feature to ensure that anonymous users of one site cannot browse another. For more information about separate authentication, see Authenticating Users Separately for Each Virtual Server.

These steps prevent a user from accessing the search catalog directly (for example by using an ASP page) and accessing data from more than one virtual server. Alternatively, you can also use a single application pool identity for all sites, with the ASP.NET Trust level set to Medium, to prevent this type of access, but this method is not as secure as the combination of separate application pool identities, separate anonymous accounts, and the user isolation feature. For more information about setting application pool identities, see Internet Information Services Help.

Installing, Upgrading, and Integrated Indexing Server Search

If you install a clean version of FrontPage 2002 Server Extensions with Windows Server 2003 SP1, Indexing Service is configured with Integrated Indexing Server Search and is started automatically. Also, if you install SharePoint Team Services on a server running Windows Server 2003 SP1, Integrated Indexing Server Search is automatically configured and started.

Note: If you do not want to enable searching, you can set the NoIndexServer registry key.

If you are applying the service packs to FrontPage 2002 Server Extensions or SharePoint Team Services, and you were using Indexing Service to search your sites, the existing catalogs remain in use for each virtual server until you upgrade the virtual servers. You can upgrade your virtual server by using the Upgrade virtual server command on the Virtual Server Administration page or the upgrade operation for Owsadm.exe. After you upgrade a virtual server, you must recalculate the Web sites before Integrated Indexing Server Search is used. You can recalculate a Web site by using the Recalculate the Web command (on the Site Administration page), the recalc operation (for Owsadm.exe or Owsrmadm.exe), or the Recalculate Hyperlinks command (Tools menu in FrontPage).

Note: If you were using the Search bot search method, and you want to remove the old catalogs, you can do so by using the management console for Indexing Service after you have upgraded your virtual server and recalculated your Web sites.

Virtual Directories and Integrated Indexing Server Search

Non-local virtual directories (including those used with UNC or DFS) are not supported with Integrated Indexing Server Search. Local virtual directories are supported, but you must manually define the scope for the virtual directory to make searching work correctly. To define the scope, you create a string value entry for the virtual directory under the following registry path:

HKEY_Local_Machine\System\CurrentControlSet\Control\ContentIndex\Catalogs\<catalog name>\Scopes\

The default catalog name is FPSE_Search. If you have created your own catalog, you should add the entry under the name of your catalog. The name of the new value entry is the path you want to include or exclude from search. The entry takes the string (REG_SZ) format, and the value data for including a path is “,,5”. To exclude a path, you use the value data “,,4”. For example, if you have a virtual directory at d:\inetpub\wwwroot on your content server, you can add a string value with the name “d:\inetpub\wwwroot” and the data “,,5”. For example, to index all of d:\inetpub\wwwroot except for anything in the _vti folders, you would use the name “d:\inetpub\wwwroot\*_vti*” with the data “,,4”.

Caution: Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Identifying Content for Integrated Indexing Server Search

When you extend a virtual server with FrontPage 2002 Server Extensions (in Windows Server 2003 SP1) or SharePoint Team Services (in Office XP SP3), a registry key (searchcatalogname) is added to track the Indexing Service catalog name for the site content for that virtual server. By default, the search catalogs are stored under %contentdrive%:\FPSECatalog with the name FPSE_search. If you change the name of the catalog, you must be sure to update this setting. For example, if you have created and scoped your own catalog for a site, you can specify that your catalog be used for search, rather than the default.

You can update the catalog name in the registry by running the following operation:

Owsadm.exe –o setproperty –p <port> -pn searchcatalogname –pv <catalog name>

For example, to set the searchcatalogname for port 80 to Site1_search, you would use the following syntax:

Owsadm.exe –o setproperty –p 80 -pn searchcatalogname –pv Site1_search

You can also set the searchcatalogname registry key directly in the registry, but you must do so before you extend the virtual server, and you must stop and restart Indexing Server after setting the key.

Note: The value you enter for searchcatalogname cannot contain characters that are invalid characters for folder names, including the following: \ / : * ? " < > |. Also, the scope for your catalog cannot be longer than 244 characters.

If you are using a UNC or DFS path for the site content, the underlying machine path to the root of the Web site, such as e:\content\root\, is stored in a registry key (uncrootwebpath). This registry key is set automatically to the underlying machine path for your site content when you extend or upgrade the virtual server.

Excluding Content from Integrated Indexing Server Search

Content is excluded from searching when it meets either of the following criteria:

• The content is stored in a folder that has been set to noRead in IIS. This setting is only respected for directories, not for individual files.

• The content is stored in a folder that has been set to noIndex in the file system. If you set noIndex for just the folder, all files and subfolders are still indexed. You must set noIndex for the folder, subfolders, and files to exclude all of the folder content.

  Note: If content is set to noIndex in IIS, but is not set to noIndex in the file system, it will still be indexed. If you do not want the content to be searched by the FrontPage Search bot, you must set noIndex on the file system.

Performance and Integrated Indexing Server Search

When you first turn Indexing Service on, it crawls all of the content that the scope identifies. This takes some time, and decreases the performance on the content computer. Be sure that you are ready for a decrease in performance the first time you enable Indexing Service.

Indexing Service can potentially crawl and index all content on the file system and on all Web sites stored on the content computer. To minimize the performance impact of Indexing Service, it is recommended that you turn off system and Web indexing for all content except the content you want to be searchable on the computer. For more information about configuring Indexing Service, see the Help system for Windows Server 2003.

Uninstalling and Integrated Indexing Server Search

When you remove FrontPage 2002 Server Extensions or SharePoint Team Services from a virtual server, the scopes for the content on that virtual server are removed from the registry. If there are no scopes remaining, the search catalog is removed. If you have manually created a scope, the scope cannot be removed automatically, and the catalog will remain.

When you uninstall FrontPage 2002 Server Extensions or SharePoint Team Services, all virtual servers are unextended, and the catalogs are removed, unless you have manually created a scope.