Indexing stall >1 hr for 30 min.

Details

Explanation

This alert indicates that a performance or corruption issue may affect the content indexing catalogs on a passive mailbox database copy in a particular Microsoft Exchange Server 2010 database availability group (DAG).

A database availability group (DAG) is a set of Mailbox servers that provide automatic recovery from failures that affect individual databases. Any server in a DAG can host a copy of a mailbox database from any other server in the DAG. When a server is added to a DAG, it works together with the other servers in the DAG to provide automatic recovery from failures that affect mailbox databases.

The MS Search (Exchange) and Microsoft Exchange Search Indexer services provide content indexing functionality for the Exchange databases.

System Center Operations Manager monitors the indexing process to determine whether an indexing backlog occurs. An alert is generated when more than one hour has passed since the last notification event was indexed. In this scenario, if the passive mailbox copy becomes the active copy, users may be unable to search on or locate items in Exchange.

User Action

To resolve this error, do one or more of the following:

• Review the Application log and System log on your Exchange 2010 servers for related events. For example, events that occur immediately before and after this event may provide more information about the root cause of this error.

• Review the Operations Console in Operations Manager for detailed information about the cause of this problem. For more information, see the "Introduction" section in this article.

• Run the Troubleshoot-CI.ps1 PowerShell script to determine whether an issue affects content indexing on the server. This script is located in the Scripts folder under the Exchange installation folder. In this scenario, check for the following items:

  • Corruption - This is indicated by a CatalogHealth error code of CatalogNeedReset.

  • Stalled indexing - This is indicated by a The Time Since the Last Notification was indexed value of more than two hours.

  • Deadlock issues - This is indicated by a CatalogHealth time stamp that is more than two hours old for each catalog entry.

  • Backlogged indexing - This is indicated when the Age of the Last Notification Indexed counter is more than 48 hours old.

• For deadlock issues and for stalled indexing issues, restart the MS Search (Exchange) and Microsoft Exchange Search Indexer services.

• For corruption issues, reset the search index, and then reseed the indexing catalog. To do this, follow these steps:

  1. Start the Exchange Management Shell, and then run the cd $exscripts command to change to the Scripts directory.

  2. Run the ResetSearchIndex.ps1 script together with the name of the Exchange mailbox database. For example, .\ResetSearchIndex.ps1 DB1. The following appear:

     WARNING: Waiting for service 'Microsoft Exchange Search Indexer (MSExchangeSearch)' to finish stopping... MSExchangeSearchService stopped Deleting catalog for DB1 Removing: C:\ProgramFiles\Microsoft\ExchangeServer\V14\Mailbox\DBX1\CatalogData-<GUID> Control-C to quit or Enter to continue: MSExchangeSearchService Started.

  3. Allow sufficient time for the new search catalog to become up-to-date. To determine whether the catalog is up-to-date, examine the value of the Full Crawl Mode Status performance counter. This counter is located in the MSExchange Search Indices performance object. When you configure this counter, select the database on which the catalog has been created. While the database is being crawled, Full Crawl Mode Status has a value of 1. When the full crawl has completed successfully, Full Crawl Mode Status has a value of 0 (zero).

  4. To reseed the search catalog on other nodes in the DAG, use the following command:

      Update-MailboxDatabaseCopy -Identity DB1\MBX1 -SourceServer MBX2 -CatalogOnly
     

     In this command, the SourceServer value represents the server on which you ran the ResetSearchIndex.ps1 script. For more information, see Update a Mailbox Database Copy.

• Examine the current replication status for each replica database. To do this, use the Get-MailboxDatabaseCopyStatus cmdlet. This cmdlet returns information about all copies of a particular database, information about a specific copy of a database on a specific server, or information about all database copies on a server.

For more information, see the following topics in Exchange 2010 Help:

• Understanding Mailbox Database Copies

• Understanding Database Availability Groups

For More Information

If you are not already doing so, consider running the Exchange tools, which have been created to help you analyze and troubleshoot your Exchange environment. These tools can help make sure that your configuration aligns with Microsoft best practices. They can also help you identify and resolve performance issues and improve mail flow. To run these tools, go to the Toolbox node of the Exchange Management Console. To learn more about these tools, see Managing Tools in the Toolbox.