Transaction Log Replay from Offline Backup
Topic Last Modified: 2006-03-14
By Nino Bilic
This article covers a subject that is often misunderstood, and can cause loss of data in your disaster recovery scenarios.
After a system failure, the checkpoint is consulted when the database next starts, and transactions that were not yet written to the database files are applied before the database is again made available to clients.
But in a typical offline backup, the database did not fail. It was dismounted normally, and there are no unfinished transactions that need to be placed into it. So why would you need to do transaction log replay when restoring an offline backup? The reason is that you want to roll forward with transaction log files created after the time of backup.
A side effect of the disaster recovery capability offered by transaction logging is the ability to roll the database forward from backup. Restoring a Microsoft® Exchange backup does not mean losing all data since the backup was taken, if you have the succeeding transaction logs. As shown previously, it is possible to re-create a database from nothing if you have the transaction logs.
|Exchange allows you to enable circular logging. When circular logging is enabled, transaction logs older than the current checkpoint are immediately and automatically deleted. This setting reduces the amount of disk space required for logging. However, you lose the ability to roll the database forward from backup. In Exchange Server version 5.5, circular logging was the default. In Exchange 2000 Server, the default was changed and log files are no longer automatically deleted. If you fail to do online backups or reduce the log files manually, the log drive will eventually fill up. In all versions of Exchange, the administrator can enable or disable circular logging. The Microsoft Exchange Information Store service for Exchange Server 2003 and Exchange 2000 Server or the Information Store for Exchange Server 5.5 must be stopped and restarted to change the circular logging status in any storage group.|
The following sections describe the requirements for successful transaction log replay:
Database and transaction logs must match
All transaction logs needed by the database must be present
Checkpoint file must be valid
The Log Signature value recorded in the database header must match the Signature value in each transaction log. Also, the Signature value for the database attachment entry in the log file header must match the DB Signature in the database.
Although comparing log file and database headers will give you some confidence about the likely success of transaction log replay, the comparison may not be conclusive. This is because databases can be attached and detached with different log file streams, or even in the middle of a single log file.
A log file header lists only the databases that were attached at the moment the previous log file rolled over. Theoretically, a database could be created, attached, and detached in the middle of a transaction log file and no evidence of its existence would be found in any log file header, because the database came and went in the middle of a log file.
More frequently, you may find that a log file you need for replay does not list the involved database in its header. But the log file may still be appropriate to the database. The log file signature, in this case, will be found inside the database, but you cannot make a conclusive match in the other direction. This circumstance is most likely to occur when a new log file sequence has been generated. No databases will be listed in the first generation of a new set of log files, because the databases are not attached to the log files until after the header has been created.
You cannot replay only some of the transaction logs needed. It is important to understand the difference needed and optional transaction log files:
If the database files are in a Clean Shutdown (Consistent) state, the database does not need any transaction log file replay to start. The database can attach itself to any available transaction log set or it can even cause a new set of transaction logs to be generated if no existing logs are available.
If the database files are in Dirty Shutdown (Inconsistent) state, at least one transaction log file will be needed for replay to recover the database. Typically, there will be several log files needed, including all of the log files from the checkpoint to the current log file at the time the database was stopped. Optionally, additional transaction log files can also be replayed into the database to roll it forward.
Starting in Exchange 2000 Server, the Log Required field in each database header lists the lGenerations of the transaction log files needed to recover the database. If the Log Required field is 0-0, no transaction logs are needed. But if a range of files is listed, and you cannot locate these files, you cannot recover the database except by restoring it from backup or by repairing it.
The Log Required field lists only the decimal generation of the log files needed. It does not list the log file prefix. In most cases, it will be obvious which log files are referred to by the Log Required field. But in cases where multiple log streams exist, or you suspect that events have happened recently that may have altered the database signatures involved, matching log and database headers against each other will be an essential skill.
The wrong checkpoint file can cause transaction log replay to fail. Worse, in some cases, the wrong checkpoint file can cause transaction log replay to succeed erroneously.
Suppose you have an offline backup of a database. You restore this database and you want to roll it forward with transaction logs created since the backup was taken. But you leave the current checkpoint file intact.
Remember that the checkpoint file defines the first log that Exchange will scan for transaction log replay. If the checkpoint is far ahead of the first log that you want to replay into this database, many of the files you want to replay will be skipped.
The outcome of leaving the wrong checkpoint file intact will depend on what is in the log files. In most cases, all transaction log replay for this database will be skipped during recovery, and the database will be attached to the logs at the end of recovery, with only the data that was in it originally.
After the recovery sequence has finished, and this database has been attached to the logs again, you cannot stop the database, delete the checkpoint file, and start replay again. Attaching the database will change it sufficiently so that replay of the skipped log files will no longer be possible. You must restore the database again, putting it back in its original state, if you want to have another chance at transaction log replay.
The following sections describe the requirements for successful transaction log replay from offline backups:
Last log file must be a valid current log file
Database files must be restored to the previous file path
When transaction logs are filled, they are renamed from Enn.log, where Enn stands for the log file prefix for the storage group, to Ennxxxxx.log, where xxxxx is a hexadecimal number corresponding to the lGeneration value in the log file header. Thus, a current log file named E02.log with an lGeneration value in it of 0x6C will be renamed when it is full to E020006C.log.
When the first database in a storage group is mounted, Exchange expects to find either no log files, in which case a new Enn.log will be created, with an lGeneration of 1, or Exchange expects to find a set of log files composed of at least one log file and with the last file in the set named Enn.log.
Sometimes, when restoring an offline backup, you may not have access to the current log file. This could happen, for example, in a situation where the last log file in the set was damaged or destroyed in a system failure. In this case, you must rename the highest numbered available log file to Enn.log before recovery can start.
Exchange administrators should be aware that renaming log files can be dangerous, and can leave a database in an unrecoverable and inconsistent state. In Exchange 2000 Server, the replay fail-safe keeps you from doing harm to a database by renaming and removing log files, but you will still not be successful in completing recovery if you rename log files except under the following conditions. You can safely and successfully rename log files if:
The log file is the last log listed in the Log Required sequence.
The log file was created after the database was last mounted.
You cannot rename a log file in the middle of a sequence of needed log files to try to truncate the sequence and do partial replay. In Exchange Server 5.5, doing so will leave your previously consistent database in an inconsistent state, and in need of repair or a second restoration. In Exchange 2000 Server, Exchange will not allow replay to begin.
If you examine the attachment record for a database in the header of a transaction log file, you will notice that the file path to the database is listed as part of the record. When transaction logs are replayed during database mount, the database must be in this exact path. This is true both for recovery after a system failure and from an offline backup. If the database is not in this path, replay will fail, with a -1811 error (JET_errFileNotFound).
If you have moved a database to a different path since an offline backup was taken, there will be problems with transaction log file replay. In Exchange 2000 Server and Exchange Server 5.5, replay would succeed up to the point in the logs where the database was moved, and then all data past that point would be discarded, leaving the database recovered with only partial data. In Exchange Server 2003, new functionality has been added to Exchange Server Database Utilities (Eseutil) to solve this problem.
In general, the simplest way to initiate transaction log replay is to mount any one database in a storage group. This causes recovery to run for every database in the storage group, mounted or not. Recall that all databases in a storage group share a single set of transaction logs. Transactions for all the databases are interleaved in the log files. Thus, recovery occurs at the level of the entire storage group, and not just the individual database.
Since Exchange 4.0, Eseutil /R has been capable of running soft recovery without mounting a database. But if you use Eseutil to do this, you must manually define paths to the log and checkpoint files at the command prompt. If you make a mistake or fail to use all necessary command-line switches, your recovery can fail. If no transaction logs are found in the current directory or in the path pointed to in the Eseutil command line, Exchange will assume no logs exist, and will try to do recovery by creating a new Enn.log. This recovery will probably fail, but it will leave the Enn.log file on your disk, which may cause confusion later.
In most cases, mounting a database will accomplish the same thing as using Eseutil. If you mount the database to cause recovery to run, you do not have to worry about making a mistake in a complex command.
|In Exchange Server 5.5 and previous versions, using Eseutil /R or any mode of Isinteg on a database newly restored from online backup would damage the database. The only safe way in Exchange Server 5.5 to recover a database restored from online backup is to mount the database. In Exchange 2000 Server, safeguards were introduced to stop this problem from happening.|
Even though using Eseutil to recover a database can be risky, it has always been useful in some special scenarios. For example, you can recover a database and replay transaction log files on a computer that does not have Exchange installed.
|To run Eseutil on a computer that does not have Exchange installed, you must copy these four files from a functioning Exchange server: Eseutil.exe, Ese.dll, Jcb.dll, and Exosal.dll. For Exchange Server 5.5 and previous versions, you need only Eseutil.exe and Ese.dll. Be careful about mixing different versions of these files on an Exchange server. Ese.dll is the core Extensible Storage Engine (ESE) database engine file, and mixing versions of it on a running server could lead to serious trouble. The recommended way to use multiple versions of these files is to copy them to a network share that is not in the system path, rather than copying them directly to a running Exchange server.|
In Exchange Server 2003, new functionality has been added to Eseutil that greatly simplifies transaction log file replay with offline backups. The Exchange Server 2003 version of Eseutil allows out of place replay of transaction log files into an offline backup. You no longer have to put databases in the exact paths defined in the log files.
In Exchange 2000 Server, to roll an offline backup forward, it was first necessary to dismount all databases in a storage group. This had to be done so that the checkpoint file could be removed. If the checkpoint file was left intact, it would determine the point from which replay starts. If you delete the checkpoint file, but leave other databases running in the storage group, the checkpoint file will be re-created within 30 seconds.
Along with stopping the entire storage group, you also had to restore the databases to their previous file paths. This was sometimes a problem in disaster recoveries where the previous drive configuration of the server had not been restored.
|When restoring online backups, these restrictions do not apply. The online backup API ignores the checkpoint file during recovery, and also overrides the database paths in the log files as necessary. The new out of place functionality in Eseutil helps bring offline backup transaction log replay up to functional parity with online backups.|
Eseutil for Exchange Server 2003 now allows you to define a database path at the command prompt, along with a system (checkpoint) and transaction log path. If you define a database path, it will override the path in the transaction log header for that database.
In practical terms, this means you can recover a database and replay transaction logs without interfering with the current functioning of the rest of the storage group. The command for recovering a database out of place with Eseutil is:
ESEUTIL /R Enn /I /D[database path] /S[chk file path] /L[log file path]
In this command, the /R switch sets Eseutil to recovery mode. Enn stands for the log file prefix for the transaction log files to be replayed. The /I switch means to ignore mismatched/missing database attachments. For more information, see the More About Eseutil /R /I section. The last three parameters define file paths to the various files needed for recovery.
If you omit the /S and /L switches, the paths default to the current directory from which Eseutil is running. If you omit the /D parameter, the database path does not default to the current directory, but to the database path written in the log files. If you use the /D parameter with no path after it, the database path is the current directory.
You can simplify the command by running Eseutil from the correct directory.
To do this, you should copy the database files and transaction log files to a separate temporary directory. You will not use a checkpoint file in this scenario. Rather, you will have only the log files present that you want to replay, and will start replay with the first log file in the temporary directory.
From a command prompt in the temporary directory, you can recover the database with:
ESEUTIL /R Enn /I /D
This command causes Eseutil to look only in the current directory for all files to be recovered. All transaction logs will be played forward as far as possible, and then the database will be made consistent and detached from the logs. After this, you can move the database files into place in the destination storage group. You can then mount the database and attach it to the current set of transaction logs, with all recovery accomplished before the database is returned to the original storage group.
The process of recovering an offline backup out of place is detailed in the following:
Restore the offline backup database files to a temporary directory on the same logical drive as the current path for the database, but not to the current database path.
Determine which transaction logs should be replayed with the database.
Because the database files are in Clean Shutdown state, the Log Required field will read 0-0 and will not tell you which log file must be the first to be replayed. But the Last Consistent field of the database header can be used to determine the anchor log, the first log file needed to begin rolling forward.
You can copy additional log files from other backup sets, and even from the current log file directory for the storage group, as long as you can keep going with an unbroken sequence of log files.
In many cases, you will not need the Enn.log file from the running storage group. It will depend on whether the current log has transactions for the database you are about to recover. If the database has not been running for a while, but other databases in the storage group are mounted, it is likely the log file has rolled over since the last time the recovery database was attached to the log stream.
If you do want to replay this log file into the database, and other databases in the storage group are currently running, the log file will be held open and will not be able to be copied. There are two ways you can get around this:
You can dismount the running databases for a few minutes and copy the log.
You can perform a differential online backup, which causes a new current log file to be generated and will force the current log file to close whether it is full or not.
Verify that the anchor log matches the database by comparing signatures in the database header to the log file signature. This check may seem redundant, because you will probably already have a good idea of which logs belong with which databases, but you should do it if you are at all uncertain about the history of the database and log files.
Test the transaction logs in the temporary directory using the command:
Eseutil /ML Enn
When you run Eseutil /ML with the log prefix as a parameter instead of with a specific log filename, Eseutil checks every transaction log with that prefix in the current directory for signature matches, sequence gaps, and internal file damage.
Prior to Exchange 2000 Server, if you were unsure of the status of your log files, you had to examine every log header individually to see if the logs all matched and were in series. And, there was no way to verify that all the log files were undamaged. Exchange Server 5.5 log files did not have internal checksums.
For replay to succeed, there must be no gaps found in the sequence of transaction logs, and the signatures of all log files must be identical. Do not leave log files in place that do not match the signature of the anchor log. If log files are not all in contiguous sequence, you must remove log files past the break in the sequence before beginning soft recovery. A gap in sequence will definitely cause recovery to fail.
Rename the highest log to Enn.log.
After you are satisfied that you have all required log files, and that all log files match each other and match the database files, run this command from the temporary folder:
ESEUTIL /R Enn /I /D
Verify that replay completed successfully by:
Examining the Application log for events listing successful replay of each log file. Investigate any errors or warnings and determine the causes.
Examining the headers of the database files to ensure the files are in Clean Shutdown state.
Move the database files, the .edb and .stm files, into place in the paths defined for them in the storage group.
In Exchange System Manager, select the This database can be overwritten by a restore check box on the Database properties page for the database. If you forget this step, the database will fail to mount, and you will receive an error reminding you to complete it.
Mount the database and verify that it is accessible to clients and that all expected data is in it.
In the procedure outlined in the previous section, the /I switch was used with Eseutil to recover the database.
The /I switch for the Eseutil recovery function was added with Exchange 2000 Server. It means to ignore missing or mismatched database attachments. There was no need for this switch in Exchange Server 5.5, because it was impossible to have one database running while another was dismounted.
In Exchange Server 5.5, the public and private databases were automatically mounted and dismounted together by starting and stopping the Microsoft Exchange Information Store service. You could not have one of them running without the other one.
The ability to independently mount databases in Exchange 2000 Server introduces several complexities with regard to backup and recovery. In Exchange Server 5.5, you did not have to deal with one database in a storage group running while another database was being restored.
Suppose that a drive fails for a single database in a storage group. In most cases, such a failure will temporarily bring down the entire storage group. All the database files are likely to be in Dirty Shutdown state.
When you try to remount other databases in the storage group, Exchange expects to find all databases available. If one of them is missing, perhaps because of a drive failure, none of the databases will be mountable. You will receive a -1216 error (JET_errAttachedDatabaseMismatch).
This error occurs because Exchange detects that database files that were running at the moment of the failure are now missing. It is possible that running recovery without the missing database will put the storage group into a state where it may be impossible to recover the missing database later.
This does not mean you would not be able to recover from a previous backup of the database. But it may be impossible to recover the failed version of the database if you recover the rest of the databases in the storage group without it being present. Changes may be made to the log files by this recovery that prevent soft recovery from working.
If you try to recover the storage group with Eseutil /R, this will also fail with the same -1216 error, unless you add the /I switch. This switch instructs recovery to ignore the missing database and continue.
If you know that the database has been destroyed, and you must restore it from backup, you can run this command to recover the rest of the databases so they are again mountable:
ESEUTIL /R Enn /I /S[chk file path] /L[log file path]
|If there is any chance that you may be able to recover the failed database, you should make a copy of the current Log Required logs for the database before forcing recovery to complete on the other databases. If you save these logs, you will be able to use Eseutil to recover the missing database out of place when it is again available. This will work because you have saved a copy of the transaction logs before recovery runs for the other databases and alters them. If you do not save a copy of the current log before forcing recovery on the other databases in the storage group, recovery may change the current log so that it is impossible to later recover the missing database.|
For more information, see the following Exchange Server resources: