Troubleshooting the Volume Shadow Copy Service

 

Microsoft Exchange-aware Volume Shadow Copy Service (VSS) backups are supported for both active storage groups and databases and for passive storage groups and databases. The passive copy backup solution that Microsoft provides is a VSS-only solution. It is implemented by the Exchange Replica VSS Writer, which is part of the Replication service. Streaming backups are supported only from the active storage groups. You cannot use streaming backup APIs to back up a replica database. A VSS backup program must be used together with a VSS Requestor for the Exchange writer.

In Microsoft Exchange Server 2003, you can run two backups at the same time against two different storage groups by using streaming backups. However, you cannot use VSS for this. For example, by using VSS in Exchange 2003, you cannot back up Storage Group Two until the backup on Storage Group One is finished.

By using Microsoft Exchange Server 2007, you can run two separate VSS backup jobs against the same Exchange server. The Exchange 2007 Writer also lets you restore Exchange data to an alternative location. The Exchange 2003 Writer does not let you target VSS restore operations to a location other than the original path.

VSS backups cannot be restored to the replica location by using the Exchange Writer. However, they can be restored to an alternative location, such as the replica copy directory.

In Exchange 2007, the Store.exe VSS writer enables restore operations to any storage group. This includes the recovery storage group (RSG). The Store.exe VSS writer also enables the restoration of databases to a folder location that is not associated with a storage group. If you do this, you can use the Jet database engine to replay the restored transaction logs to the database to bring the database to a consistent and mountable state. Then, you can move the database to another location.

Troubleshooting the Volume Shadow Copy Service

By default, VSS is installed on a Windows Server 2003 server. It is set to start manually, and it runs if a requester (a backup program) can use the writers in the system. Windows Server 2003 Service Pack 2 (SP2) addresses some issues that affect the performance of VSS.

For a list of the issues that are fixed in Windows Server 2003 SP2, see Microsoft Knowledge Base article 914962, List of updates in Windows Server 2003 Service Pack 2.

Although you can install only the updates for VSS, we recommend that you install the full service pack. Additionally, the Microsoft Knowledge Base article for the relevant update rollup package can help resolve some VSS snapshot issues in Windows Server 2003.

For more information, see Knowledge Base article 940349, Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues

The following hotfixes are available to resolve VSS issues that you may experience.

KBID Number Title

833167

A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003

915331

The backup process may fail and a time-out error may occur in Volume Shadow Copy Service writers

930800

Event ID 9840 or 9607 is logged when a VSS backup operation fails in Exchange 2007 or in Exchange 2003

910250

A snapshot backup of an Exchange Server 2003 database fails, and event ID 9607 is logged

923628

The backup application stops responding when you perform a backup that uses a volume snapshot in Windows Server 2003 with SP1

933779

A snapshot copy backup of a storage group fails and Event IDs are logged in the Application log in Exchange Server 2003

Before a backup operation runs, the Exchange writer must be in a stable state, and must experience no errors. You can determine the state of the Exchange writer by running the following command:

VSSadmin list writers

The output from this command lists all writers that are registered by using VSS. In the command results, examine the State field of the writers. Notice whether the State field displays a state other than Stable. The following example output shows the Exchange writer in a failed state.

Writer name: 'Microsoft Exchange Writer'

Writer Id: {GUID}

Writer Instance Id: {GUD}

State: [14] Failed

Last error: Retryable error

If the Exchange writer is in a failed state, restart the Microsoft Exchange Information Store service, and then run the VSSadmin list writers command to determine whether the writer state returns to a stable state. The following example output shows the Exchange writer in a stable state.

Writer name: 'Microsoft Exchange Writer'

Writer Id: {GUID}

Writer Instance Id: {GUD}

State: [1] Stable

Last error: No error

If the output from the VSSadmin list writers command is blank, there may be a registry error that prevents the operating system from detecting and displaying the appropriate writer information. For more information about how to fix this issue and about how to re-create the list of VSS writers, see Microsoft Knowledge Base article 940184, Error message when you run the "vssadmin list writers" command on a Windows Server 2003-based computer: "Error: 0x8000FFFF".

The VSSadmin list providers command lists the registered Volume Shadow Copy providers on the computer. Typically, when you run this command, "Microsoft Software Shadow Copy provider 1.0" is listed as one of the providers or as the only provider. Other third-party backup programs may use their own providers. The following example output shows the results from the VSSadmin list providers command.

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'

Provider type: System

Provider Id: {GUID}

Version: 1.0.0.7

For a successful backup, make sure that the provider or providers are listed and that the Exchange writer is in a stable state. For more information about other tests that you can run, type VSSadmin /? at a command prompt.

Tracing the Volume Shadow Copy Service

If you still experience backup issues, you should enable VSS tracing. For more information about how to enable tracing, see Microsoft Knowledge Base article 887013 How to enable the Volume Shadow Copy service's debug tracing features in Microsoft Windows Server 2003.

You can create a batch file to set the registry entries that are described in article 887013. To do this, type or copy the following text into a text file. Then, save the file as "EnableVSSTracing.bat." This batch file saves tracing information in a file that is named C:\trace.txt.

net stop swprv 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFile /t REG_SZ /d C:\trace.txt /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceLevel /t REG_DWORD /d 0xffffffff /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceEnterExit /t REG_DWORD /d 1 /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToFile /t REG_DWORD /d 1 /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToDebugger /t REG_DWORD /d 1 /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFileLineInfo /t REG_DWORD /d 1 /f 
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceForceFlush /t REG_DWORD /d 0 /f

To trace VSS, follow these steps:

  1. Modify the Windows registry to enable tracing. To do this, follow the steps in article 887013, or run the EnableVSSTracing.bat file that you created previously.

  2. Reproduce the VSS issue that you experience.

  3. After you receive the VSS error that corresponds to the issue that you want to troubleshoot, stop the VSS services. To do this, run the following commands:

    net stop VSS
    net stop SWPRV
    
  4. Disable VSS tracing. To do this, run the following command:

    reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /f
    
  5. Review the trace file that was generated.

Tracing the Exchange Writer

The Exchange Writer is the built-in VSS writer that is provided together with the Microsoft Exchange information store. The writer runs in the Store.exe process. The Exchange Writer is used when the VSS requester (VSS backup program) requests a VSS backup from the active database. Like all the other components in Exchange Server 2007, the tracing is generated in an event trace log (ETL) file.

The information store provides the following tag for the Exchange writer.

Tag Description

TagVSS

In Exchange Server 2007, the only tag that is available for the Exchange writer is TagVSS. This tag combines all the VSS tags that are provided in Exchange 2003, including the following:

  • Exchange VSS Snapshot Initialize

  • Exchange VSS Snapshot OnIdentify

  • Exchange VSS Snapshot OnPrepareBackup

  • Exchange VSS Snapshot OnThaw

  • Exchange VSS Snapshot OnPostSnapshot

  • Exchange VSS Snapshot OnBackupComplete

  • Exchange VSS Snapshot OnPreRestore

  • Exchange VSS Snapshot EcAddComponents

  • Exchange VSS EcPrepareJetInitBackup

  • Exchange VSS Snapshot EcBuildSGArrayFromWriteComp

Note

The default freeze time-out for the Exchange writer is 20 seconds.

Tracing the Replica Writer

The Exchange Cluster Replica Writer is the built-in VSS writer that is provided together with the replication service. This writer runs in the M.E.Cluster.Replay.exe process. The Exchange Cluster Replica Writer is used when the VSS requester (VSS backup program) requests a VSS backup from the copy database. Like other components in Exchange Server 2007, the tracing is generated in ETL format.

The Replica Service provides the following tag for the Exchange Cluster Replica Writer.

Tag Description

ReplicaVssWriterInterop

  • This trace provides the interactions between the VSS writer and the replication service.

Use Extra.exe to trace the Replica Writer

You can also use the Microsoft Exchange Troubleshooting Assistant (Extra.exe) to trace the Exchange VSS writer. To do this, follow these steps:

  1. Run the following command from the Exchange Management Shell:

    Set-EventLogLevel "MSExchange Repl\Exchange VSS Writer" -level expert
    
  2. Click Start, click Run, type extra.exe, and then click OK.

  3. Click Go to the Welcome Screen, and then click Select a Task.

  4. Click Trace Control, and then click OK if you receive a message that states that Exchange does not have a module to interpret traces.

  5. Specify a location to save the trace file, and then set the trace file size to 200 megabytes (MB).

  6. Click Set manual trace tags, and then click to select the following trace type check boxes:

    • PFD

    • Fatal

    • Error

    • Warning

    • Info

    • Debug

  7. In the Components to trace list, click to select the Cluster.Replay check box.

    Note

    This check box is only applicable for a clustered Exchange environment.

  8. In the Trace Tags list, click to select the following check boxes:

    • ReplicaInstance

    • ReplicaVssWriterInterop

  9. In the Components to trace list, click to select the Store check box.

  10. In the Trace Tags list, click to select the following check boxes:

    • tagVSS

    • tagMostError

  11. Click Start Tracing.

  12. Start an Exchange VSS backup. When the backup fails, click Stop tracing now in the Exchange Troubleshooting Assistant.

Running the BETest tool

BETest is a VSS requester that tests advanced backup and restore operations. This tool can be used to test an application's use of complex VSS features, such as the following:

  • Incremental and differential backup

  • Complex restore options, such as authoritative restore

  • Rollforward options

You can use the BETest tool to take a VSS backup of an Exchange storage group and then test the Exchange VSS writer. BETest can perform most of the operations that a VSS requestor can perform. You can also use BETest to take a VSS Snapshot of the active and replica databases on an Exchange 2007 server.

BETest is available in the VSS SDK 7.2. To obtain this SDK, see Microsoft Download Center article Volume Shadow Copy Service SDK 7.2.

The version of BETest that is available in the SDK uses a text-based version of the Components file (Components.txt). A version of BETest that uses an XML-based version of the Components.txt file is available in the Windows SDK. For more information about how to obtain and use the Windows SDK version of BETest, see BETest Tool.

Note

The steps in this topic use the text-based version of the Components file in BETest.

To use the BETest tool, follow these steps:

  1. Obtain and install the BETest tool. By default, the i386 version of the tool is installed in the following folder:

    C:\Program Files(x86)\Microsoft\VSSSDK72\TestApps\betest\obj\i386

    Note

    An AMD64-based version of BETest is also available. Always change to the directory of the appropriate operating system version before you run BETest.

  2. Run the following command to obtain a list of available writers:

    betest.exe > AvailableWriters.txt
    
  3. Create a Components.txt file to specify the VSS components for BETest. The Components file uses the following format to specify options:

    "<WriterId>": "<component-logical-path>" {"target" # "new target", ...}, ..."<component-logical-path>" : '"<subcomponent-logical-path>,...";
    
  4. Obtain the value for the WriterId entry for use with the Components.txt file. To do this, follow these steps:

    1. Open the AvailableWriters.txt file by using a text editor, such as Notepad.

    2. Search for the following text:

      Microsoft Exchange Writer

    3. Note the value for the WriterId field. For example, 76fe1ac4-15f7-4bcd-987e-8e1acb462fb7.

  5. Determine the path for the <component-logical-path> placeholder in the Components.txt file. This path may be specified in one of the following formats:

    • <logical-path>

    • <logical-path>\<component-name>

    • <component-name> (if no logical path exists)

    For example, Microsoft Exchange Server\Microsoft Information Store\<ServerName>\<StorageGroupGUID>.

  6. Add the values to the Components.txt file, and then save the changes to the file. The following is example content of a Components.txt file.

  1. "{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\<Server Name>\68e5b198-468e-4328-8e8d-8d5276b8e355 ";

In this example, the first GUID represents the Exchange writer. The second GUID represents a specific storage group. You can choose to run the BETest tool against only one storage group if you have multiple storage groups. To obtain the GUID for a particular storage group, you can use the LDP.exe tool or the Get-StorageGroup cmdlet. If you use the LDP tool, locate the following object, and then note the value for objectGUID in the details pane.

CN=First Storage Group,CN=InformationStore,CN=<ServerName>,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=First Organization,CN=Organizations,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=fourthcoffee,DC=com

Note

This path uses the default names for the organization and administrative group. Modify this path as appropriate for your organization.

To obtain the storage group GUID from the command-line, run the following command at the Exchange Management Shell:

Get-StorageGroup |fl Guid

Streaming backups are only supported from active storage groups. If you want to back up a passive database copy, you must use a VSS backup. If the database is a Continuous Cluster Replication database, or if you use Local Continuous Replication, the Components.txt file will resemble one of the following.

For a CCR replica copy

  1. "{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\Replica\<ClusteredMailboxServerName>\<StorageGroupGUID> ";

For an LCR replica copy

  1. "{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}":"Microsoft Exchange Server\Microsoft Information Store\Replica\<Server Name>\68e5b198-468e-4328-8e8d-8d5276b8e355 ";

Note

The success or failure of a streaming backup does not indicate the health of the Exchange VSS writer.

  1. After the Components.txt file has been saved, run the following command to start BETest:

    BETEST.exe /B /E /T 1 /S output.XML /C components.txt /D c:\betest > Output.txt
    

    This command creates a backup in the C:\BETest directory.

    Note

    You can also run this command without the /E option.

  2. If the backup is unsuccessful, examine the contents of the Output.txt file to review any error messages that have been logged.

For More Information

Exchange Server 2007 Service Pack 1 (SP1) and several Exchange Update Rollup packages include fixes for VSS issues. We recommend that you install the latest Exchange 2007 service pack together with the latest Update Rollup package for Exchange. For more information, see How to Obtain the Latest Service Pack or Update Rollup for Exchange 2007.