Word Breakers and Stemmers

Word Breaker Registration

For the word breakers of a language to be used, they must be registered. For registered word breakers, associated linguistic resources—stemmers, noise words (stopwords), and thesaurus files—also become available to full-text indexing and querying operations. To view a list of the languages whose word breakers are currently registered with SQL Server, use the following Transact-SQL statement:

SELECT * FROM sys.fulltext_languages

If you add, remove, or alter a word breaker, you need to refresh the list of Microsoft Windows locale identifiers (LCIDs) that are supported for full-text indexing and querying. For more information, see How to: Alter the List of Registered Word Breakers and Filters (Transact-SQL).

Several licensed third-party word breakers are shipped with SQL Server 2008. You can manually load additional third-party word breakers (and stemmers) for several languages (Danish, Polish, and Turkish). For more information, see How to: Load Licensed Third-Party Word Breakers.

Full-Text Language Option

For a localized version of SQL Server, SQL Server Setup sets the default full-text language option to the language of the server if an appropriate match exists. For a non-localized version of SQL Server, the default full-text language option is English.

When creating or altering a full-text index, you can specify a different language for each full-text indexed column. If no language is specified for a column, the default is the value of the configuration option default full-text language.

For more information, see default full-text language Option.

Choosing a Language When Full-Text Indexing a Column

When creating a full-text index, we recommend that you specify a language for each indexed column. If a language is not specified for a column, the system default language is used. The language of a column determines which word breaker and stemmer are used for indexing that column. Also, the thesaurus file of that language will be used by full-text queries on the column.

There are a couple of things to consider when choosing the column language for creating a full-text index. These considerations relate to how your text is tokenized and then indexed by Full-Text Engine. For more information, see Best Practices for Choosing a Language When Creating a Full-Text Index.

To view the word breaker language of a column

• How to: View or Change the Properties of a Full-Text Index (SQL Server Management Studio)

• sys.fulltext_index_columns (Transact-SQL)

  SELECT 'language_id' AS "LCID" FROM sys.fulltext_index_columns;
  

Impact of New Word Breakers in SQL Server 2008

SQL Server 2008 includes word breakers for more than 50 diverse languages, of which 23 also exist in SQL Server 2005. Only the word breakers for English, Korean, Thai, and Chinese (all forms) remain the same. For other languages, SQL Server 2008 introduces a new generation of word breakers that have better linguistic rules and are more accurate than earlier word breakers. Potentially, the new word breakers might behave slightly differently from the word breakers in imported SQL Server 2005 full-text indexes. This is significant if a full-text catalog was imported when a SQL Server 2005 database was upgraded to SQL Server 2008. One or more languages used by the full-text indexes in the full-text catalog might now be associated with new word breakers. For more information, see Full-Text Search Upgrade.

Word Breaker Versions for Languages Supported in SQL Server 2005

Only the word breakers for English, Korean, Thai, and Chinese (all forms) remain the same. The following table lists the word breakers that existed in SQL Server 2005 and indicates whether they have been updated in SQL Server 2008. For a complete list of all the SQL Server 2008 word breakers, see sys.fulltext_languages (Transact-SQL).

For a complete list of supported languages, see sys.fulltext_languages (Transact-SQL).

Word-Breaking Time-out Errors

A word-breaking time-out error might occur in a variety of situations. For information about these situations and how to respond in each situation, see MSSQLSERVER_30053.

Obtaining Information About Word Breakers

Viewing the Tokenization Result of A Word Breaker, Thesaurus, and Stoplist Combination

• sys.dm_fts_parser (Transact-SQL).

To return information about the registered word breakers

• sp_help_fulltext_system_components (Transact-SQL)

See Also

Reference

Concepts

Change History