Reanimating Active Directory Tombstone Objects
At a Glance:
- How deleted objects are stored in Active Directory
- Using LDP and ADRESTORE to find and restore tombstones
- Object attributes and systemFlags
While it may not be your favorite topic to think about, accidental data loss is a fact of life and, as an IT professional, you have to be prepared for all sorts of recovery scenarios. In the April 2007 issue of TechNet Magazine, I discussed the importance of having a plan in
place for recovering Active Directory®
users and groups. In that article, available at technetmagazine.com/issues/2007/04/ADRecovery
, I covered the mechanics of object linking, replication, deletion, and authoritative restores. Unfortunately, the authoritative restore function requires that you start a domain controller (DC) in Directory Service Restore Mode (DSRM). Doing this renders the DC unavailable for the duration of the restore operation.
There is another approach you should be aware of. Tombstone reanimation (which has nothing to do with zombies) provides the only way to recover deleted objects without taking a DC offline, and it's the only way to recover a deleted object's identity information, such as its objectGUID and objectSid attributes. It neatly solves the problem of recreating a deleted user or group and having to fix up all the old access control list (ACL) references, which contain the objectSid of the deleted object. Just keep in mind that tombstone reanimation does have its own limitations, which I will discuss, so you'll still want to keep authoritative restores in your box of tricks.
Tombstone reanimation was introduced in Windows Server® 2003 Active Directory to simplify certain data recovery scenarios. This feature takes advantage of the fact that Active Directory keeps deleted objects in the database for a period of time before physically removing them. In this article I'll give you a detailed description of how Active Directory deletes and reanimates objects, and I'll show how you can recover deleted objects using standard Microsoft tools.
What Is a Tombstone?
When Active Directory deletes an object from the directory, it does not physically remove the object from the database. Instead, Active Directory marks the object as deleted by setting the object's isDeleted attribute to TRUE, stripping most of the attributes from the object, renaming the object, and then moving the object to a special container in the object's naming context (NC) named CN=Deleted Objects. The object, now called a tombstone, is invisible to normal directory operations. It does not show up in any Microsoft® Management Console (MMC) snap-ins, and most Lightweight Directory Access Protocol (LDAP) utilities are blissfully unaware of the tombstone's existence. The tombstone is, for all intents and purposes, gone. The data, however, is still there—it's just invisible. So why does Active Directory keep tombstones, otherwise deleted objects, in the database?
While invisible to other processes, a tombstone is visible to the Active Directory replication process. In order to make sure the deletion is performed on all the DCs that host the object being deleted, Active Directory replicates the tombstone to the other DCs. Thus the tombstone is used to replicate the deletion throughout the Active Directory environment.
The Lifetime of a Tombstone
Active Directory typically deletes an object from the directory in response to an LDAP delete protocol operation or a Security Accounts Manager (SAM) delete operation. This operation is called the originating delete and is distinct from the delete operations performed by the Active Directory replication process. Note that this discussion does not concern dynamic objects, which have a different delete mechanism and are directly deleted by Active Directory when their time to live (TTL) expires.
When Active Directory receives the delete operation, it first runs through a checklist to make sure the object is allowed to be deleted. This process is surprisingly involved as it makes sure the following criteria are all true:
- The isDeleted attribute of the object is not already set to TRUE. (You can't delete an already deleted object!)
- The "private object" resource-specific security descriptor control flag is not set in the object's security descriptor. (This appears to be an undocumented "feature." The private object flag is bit 1 of the Sbz1 byte of the security descriptor structure.)
- The "disallow delete" bit (0x80000000) is not set in the systemFlags attribute of the object.
- The isCriticalSystemObject attribute of the object is not set to TRUE.
- The object's security descriptor grants the appropriate access rights to the user to delete the object. (More specifically, the user is allowed to delete the object itself and is allowed to delete a child from the parent object.)
- The object is not a cross-reference object (objectClass = crossRef) for an existing naming context.
- The object does not have any subordinate objects. (If the LDAP delete operation includes the LDAP "tree delete" control object id, Active Directory will automatically delete subordinate objects as well, unless they have the isCriticalSystemObject attribute set to TRUE. This prevents critical system objects from being deleted inadvertently as part of a tree deletion operation. You can of course delete these objects individually.)
- The object is not a critical internal object (for instance, it isn't the nTDSDSA object for the DC or placeholder objects for ancestors of the NC heads).
In addition to these criteria being true, Active Directory must also be in an appropriate state to process the delete operation. For example, if Active Directory is in the process of renaming a domain, it will not delete a domain trust account or a crossRef object.
If it is determined that the object can in fact be deleted, Active Directory proceeds to turn the object into a tombstone. Active Directory first strips unnecessary attributes from the object, leaving those specified in Figure 1 and those defined in the schema to be retained in the tombstone. (See the section "Recovering Object Attributes" for more information on this.) It then changes the relative distinguished name (RDN) of the object to CN=<old RDN>\0ADEL:<objectGUID>, where \0A indicates the ASCII linefeed character and <objectGUID> is the objectGUID expressed as a string. The lastKnownParent attribute is then set to the distinguished name (DN) of the parent container of the object and the isDeleted attribute is set to TRUE. Active Directory then removes all forward and backward link attributes from the object. Finally, if the systemFlag attribute of the object does not have the "disallow move on delete" bit set, Active Directory moves the object to the CN=Deleted Objects container for the NC. (See the "systemFlags Attribute and Object Behavior" sidebar for more information on this topic.)
Figure 1 Attributes saved in a tombstone
|Hard-Coded to Be Saved
|Saved Due to searchFlags Setting
You should note that the CN=Deleted Objects folder is flat and has no object hierarchy. You might think this would cause name conflicts if you deleted two different objects with the same CN. This isn't the case, though. Since the objectGUID is incorporated into each tombstone's RDN, each tombstone's RDN is unique within the CN=Deleted Objects container.
Obviously, objects don't remain in the CN=Deleted Objects container forever. The default tombstone lifetime is 60 days for forests initially built using Windows® 2000 and Windows Server 2003, and 180 days for forests that were initially built with Windows Server 2003 SP1. You can change the tombstone lifetime by setting the tombstoneLifetime attribute of the CN=Directory Service,CN=Windows NT,
CN=Services,CN=Configuration, DC=<root domain> object.
Every 12 hours, each domain controller starts a garbage collection process. (This can be changed by setting a new value for the garbageCollPeriod attribute of the CN=Directory Service,CN=Windows NT,
CN=Services,CN=Configuration,DC=<root domain> object.) This garbage collection scans all of the tombstones on the DC and physically deletes any that are older than the tombstone lifetime.
Using LDP to Find Tombstones
LDP is a Windows Explorer-like utility for working with Active Directory. LDP was originally designed by the Active Directory development team to test its LDAP code while Active Directory was under development. Now part of the Windows Server 2003 Support tools, LDP has evolved into a robust tool for working with Active Directory.
Even though tombstones are invisible to normal directory operations, you can find tombstone objects in Active Directory using LDAP search operations and special LDAP extensions called controls. Controls are a mechanism, defined in the LDAP standard, used to extend the LDAP protocol to provide additional functionality beyond what is defined in the LDAP standard while remaining compatible with other LDAP-compliant software. Active Directory supports 22 controls, including the Return Deleted Objects control. When used to extend an LDAP search operation, this control retrieves deleted objects that would otherwise be invisible.
To demonstrate how this works, I log into the foo.local domain with Enterprise Administrator credentials and then use the Active Directory Users and Computers (ADUC) MMC snap-in to create a user object (CN=John Smith,CN=Users,DC=foo,
DC=local), as shown in Figure 2. I then delete the user object, again using ADUC.
Figure 2 Using ADUC to create a user (Click the image for a larger view)
Now I can run the LDP program and use it to display the tombstone for the deleted user. The first step is to connect LDP to a specific DC and authenticate using the appropriate credentials. To connect to the DC, I simply select the Connect option from the Connection menu. Since I am running LDP on the DC, I just press OK to connect to the local DC. If you are running LDP remotely, you must specify the name of the DC you want to connect to. After I successfully connect, LDP displays the attributes of rootDSE—this contains attributes describing the state and configuration of the DC itself (see Figure 3).
Figure 3 LDP connected to a DC (Click the image for a larger view)
Now, to authenticate to the DC using domain or enterprise administrator credentials, I select Bind from the Connection menu. Because I'm already logged in as an Enterprise Administrator for the forest (Domain Administrators and Enterprise Administrators by default have the rights to list and reanimate objects in the CN=Deleted Objects container), I simply leave the Username and Password in the Bind dialog empty and press OK. Otherwise, I could enter the appropriate credentials into the Bind dialog.
Next I want to list the contents of the CN=Deleted Objects,DC=foo=DC,local con-
tainer. You will normally never see the CN=Deleted Objects container because the container itself is marked as deleted. The only way you can search the CN=Deleted Objects container is to use the Return Deleted Objects LDAP control.
To use this control with LDP, go to the Browse menu and select Search. The search dialog shown in Figure 4 will appear. The Base Dn field lets you specify where in the directory tree to start searching. I enter the DN of the Deleted Objects container for the domain–in this example, the DN is CN=Deleted Objects,DC=foo,DC=local.
Figure 4 LDP Search dialog
In the Filter field, I specify the search criteria LDP will use. The default value is (objectClass=*), which instructs the search to return all objects that have a value for the objectClass attribute. Because even deleted objects in Active Directory have a value for objectClass, the default filter will return every object within the search scope. In this example, I change the filter to (objectClass=user), limiting the search to user objects. This will make it easier to find the deleted John Smith user among the possibly thousands of tombstones in the CN=Deleted Objects container.
You may find that there are more objects in the CN=Deleted Objects container than Active Directory will return in a single search operation. To address this problem, you would typically use a paged LDAP search, which returns the results a group at a time. However, LDP does not let you specify both a paged and extended search. So you'll have to refine your LDAP filter to locate the object you want.
The Scope radio buttons let you specify how much of the directory tree to search. A base search will only return the object specified by the Base Dn field. A one-level search will return objects that are immediately subordinate to the Base Dn object. And a subtree search will return all objects subordinate to the Base Dn object. Because the CN=Deleted Objects folder is flat, I use a one-level search to retrieve all tombstones.
So far, I have merely outlined a conventional LDAP search. If I were to run it now, LDP would display an error indicating that CN=Deleted Objects,DC=foo,DC=com does not exist since it's marked deleted. I still have to include the Return Deleted Objects LDAP control with the search operation. To do so, I click the Options button on the Search dialog and select Extended for the Search Call Type, as shown in Figure 5. This option allows me to specify controls for the search operation. Here, I also change the Attributes list to *. This causes LDP to display all of the normal attributes for each object it retrieves, instead of LDP's default set of attributes.
Figure 5 LDP Search
Now I press the Controls button to display the Controls dialog. LDP's Controls dialog is a little awkward, so when you do this, be sure you carefully follow these steps for adding the Return Deleted Objects control.
Open the Load Predefined dropdown list, select Return deleted objects, and press the Check in button. This adds the object identifier (OID) for the Return Deleted Objects control (1.2.840.1135220.127.116.117) to the Active Controls list, as shown in Figure 6. LDP then adds the control to all subsequent extended search operations. Press OK to save the control settings and press OK again to close the Search Options dialog.
Figure 6 Adding the Return Deleted Objects control
The Controls dialog has some peculiar behavior. Specifically, even though an OID appears in the Active Controls list, this doesn't necessarily mean that the control is actually in effect. Sometimes you have to check out a control and then check it back in to ensure that the control will be used in the next LDAP operation.
Now I'm ready to run my search. I press the OK button on the LDP Search dialog and LDP retrieves all the user objects from the CN=Deleted Objects container for the domain. Figure 7 shows my results.
Figure 7 Results from the CN=Deleted Objects container (Click the image for a larger view)
Examining the LDP Results
My search returned two tombstone objects. First, you can see the DN of the first tombstone: CN=John Smith\0ADEL:41800281-6bc4-42c3-a99b-b283022b3af8,CN=Deleted Objects,DC=foo,DC=local. The objectGUID of the deleted object is represented as a string (41800281-6bc4-42c3-a99b-b283022b3af8). Beneath the DN is a list of attributes showing the values for objectClass, cn, distinguishedName, and so on. You can see that the value of the isDeleted attribute is TRUE, meaning the object has in fact been deleted. You'll also see that the objectSid has been preserved in the tombstone—this is important for recovering user and group tombstones, as I'll discuss later. The lastKnownParent attribute, which represents the DN of the object that contained the deleted object before it became a tombstone, is also extremely important when recovering tombstones.
In my sample, LDP displayed two tombstone objects, both of which were user objects named CN=John Smith that originally came from the CN=Users,dc=foo,dc=local container. How can two separate objects with the same RDN come from the same container? The explanation is pretty simple. Before running LDP to search for tombstones, I created the CN=John Smith user object in the CN=Users,DC=foo,DC=local container and deleted it. Then I created another user object with all the same attributes and deleted that one as well. Because I had already deleted the first CN=John Smith user object, it was perfectly reasonable to create the second one, even though it had the same name. But these are separate, unique objects, distinguished by their objectGUIDs. Because Active Directory incorporates the objectGUID into the tombstone's DN, they appear as separate objects in the CN=Deleted Objects container.
How a Tombstone Is Reanimated
Active Directory provides a mechanism for restoring a tombstone back into a normal object. This is effectively an undelete function for deleted objects. The function is a specially formed LDAP modify operation that must include two specific attribute modifications: it must remove the isDeleted attribute (not just set it to FALSE) and it must move the object to another container by changing the object's distinguishedName. The new distinguishedName typically (but not necessarily) uses the lastKnownParent attribute as the container and keeps the same RDN minus the \0ADEL:<objectGUID> component that Active Directory added when it created the tombstone.
Before reanimating the tombstone, Active Directory checks to ensure certain necessary conditions are all true. The user must have the reanimate tombstone extended access right defined on the NC head of the NC containing the tombstone. The user cannot be reanimating his own object. The isDeleted attribute of the tombstone must be set to TRUE. And the owner Security Identifier (SID) of the object, as defined in the security descriptor, must have a SID from one of the domains in the forest or a trusted domain.
If the object in question is in the Configuration or Schema NCs, the FLAG_CONFIG_ALLOW_RENAME and FLAG_
CONFIG_ALLOW_MOVE bits must be set in the tombstone's systemFlags attribute. If the object is in the Configuration or Schema NCs and the FLAG_CONFIG_ALLOW_LIMITED_MOVE flag is set, the object is only able to be moved to a container that has the same grandparent—in other words, the object must retain its great-grandparent after the move. And if the object is in a domain NC or an application partition, then the FLAG_DOMAIN_DISALLOW_RENAME and FLAG_DOMAIN_DISALLOW_MOVE bits must not be set in the tombstone's systemFlags attribute.
The user must have the rights, as defined in the security descriptor, to modify the RDN (usually CN, but not always) and to add the object to the new parent container. And the new parent container must be a possible superior for the tombstone's objectClass, as defined by the schema.
Reanimating a schema object is never allowed. This, however, is not really an issue since you can't legitimately delete an object from the Schema NC.
If all of the checks are successful and the necessary requirements are met, Active Directory performs the following steps on the tombstone to reanimate the object:
- The isDeleted attribute is removed.
- If not otherwise prescribed in the modify operation, the objectCategory is set to the most specific objectClass indicated in the tombstone.
- The RDN is changed and the object is written to the new parent container.
After reanimation, the object has the same objectGUID and objectSid attributes it originally had. This means that external references to the object, for instance in ACLs, don't have to be reset as they do if you recreate a deleted object. The reanimated object looks and acts just as it did before it was deleted. There is one important difference, though: the restored object is missing any attributes that were not saved in the tombstone.
Using LDP to Reanimate Tombstones
Now that I've shown you how the nuts and bolts of tombstone reanimation work, I want to demonstrate how I can use LDP to restore the CN=John Smith user I deleted. First I connect to a DC and authenticate. Now I can perform a modify operation using LDP. In the modify operation parameters, I can remove the isDeleted attribute and specify a new DN at the same time.
Because I am operating on a tombstone, I have to specify the Return Deleted Objects LDAP control. To set this control for the modify operation in LDP, select Controls from the options menu, select Return deleted objects from the Load Predefined dropdown list, press the Check in button, and press OK to save the control settings.
Now I select Modify from the Browse menu to open the Modify dialog and enter the DN of the tombstone object I want to restore. The easiest way to do this is to copy and paste the DN from the results of the search operation I performed earlier.
Next, I have to enter a list of attribute operations to perform. Reanimating a tombstone requires two attribute operations—delete the isDeleted attribute and replace the distinguishedName attribute with the desired DN of the reanimated tombstone. So I type isDeleted into the Attribute field and select the Delete radio button. When I press the Enter button, the attribute operation is added to the Entry List, as shown in Figure 8.
Figure 8 Specifying attribute operations to be performed
Then I enter distinguishedName into the Attribute field, select the Replace radio button, and enter the new DN of the object in the Values field. In this case, I use the original distuiguishedName of the user, CN=John Smith,CN=Users,DC=foo,DC=local. When I press the Enter button, the modify operation is added to the Entry List.
Because I have to include the Return Deleted objects LDAP control with the modify operation, I have to use an extended LDAP modify. To do this, I check the Extended checkbox. Once everything is set, as shown in Figure 9, I press the Run button to make the changes. LDP displays the results in a large text window.
Figure 9 Change the distinguishedName
When I go back to the Active Directory Users and Computers MMC snap-in and look in the CN=Users container, I'll find the once-deleted user object back where it came from. But the object will have some important differences from its original state, which I'll look at in a moment.
Using ADRESTORE to Reanimate Tombstones
Once you figure out how to use LDP, reanimating a tombstone is not terribly difficult. But it's not very convenient, either. Fortunately, the good folks at Sysinternals (a company that is now part of Microsoft) developed a command-line tool to simplify the reanimation process. This tool, called ADRESTORE, is available from the Microsoft Web site at microsoft.com/technet/
sysinternals/utilities/AdRestore.mspx. Installation is simple. Just copy the executable to an appropriate directory on your machine, for instance the C:\WINDOWS\SYSTEM32 directory.
ADRESTORE runs in two modes. If you run it with no parameters, it will list all the tombstones in the CN=Deleted Objects container of the default domain. You can add a search string to the command line to select the objects to display, for instance:
This will display all the tombstone objects in the CN=Deleted Objects container that contain the string "John" in their CN or OU attribute—it uses the LDAP search filters cn=*John* and ou=*John*. This isn't exactly the most flexible way to search tombstone objects, but it is workable for most situations. Figure 10 shows the output my search returned in ADRESTORE.
Figure 10 Attributes saved in a tombstone (Click the image for a larger view)
If you want to reanimate a tombstone, not just find it, then you need to specify the
–r switch, along with an optional string, such as the following:
This command will prompt you to reanimate each matching tombstone object. ADRESTORE always reanimates the object into the container given by the tombstone's lastKnownParent attribute—there is no way to specify a different container.
ADRESTORE provides a convenient command-line interface for using the Active Directory reanimation functionality. Though not very flexible, it is quite a bit easier to use than LDP. Still, ADRESTORE doesn't avoid two significant problems with reanimating tombstones: recovering the attributes stripped from the object when it was first deleted, and recovering objects deleted from the Configuration NC.
Recovering Object Attributes
In terms of data recovery, tombstone reanimation has a great advantage over performing an authoritative restore: tombstone reanimation doesn't require the DC to be taken offline. And reanimating tombstones is much better than simply recreating a new version of a deleted object. If you recreate an object, it will always get new objectGUID and objectSid (if it's a security principal such as a user) attributes. As a result, any external references to the object, such as ACLs, will have to be updated to reflect the new object's identity. This can be a major headache.
Some attributes are stripped when an object is deleted, and these attributes aren't recovered with tombstone reanimation. But Active Directory saves some key attributes in the tombstone, the most important of which are identity-related attributes such as objectGUID and objectSid. It saves many other attributes by default as well (see Figure 1). Most of the attributes that are hardcoded to be saved aren't very interesting, particularly in a discussion about recovering deleted user objects. But you can specify additional attributes to be saved in the tombstone by setting bit 3 (0x00000008) of the searchFlags attribute of the corresponding attributeSchema object in the Active Directory schema. Some attributes already have this bit set in the schema in Windows Server 2003 R2 (these are also shown in Figure 1).
Installing certain applications may change bit 3 of the searchFlags of existing attributes or even add new attributes that have bit 3 set. Exchange Server, for example, configures several additional attributes to be saved in the tombstone.
Active Directory will not save forward-linked or backward-linked attributes in the tombstone, even if you specify to do so in the searchFlags attribute of the schema object. In particular, Active Directory doesn't save things such as the member attribute of a group or the memberOf attribute of a user. Thus tombstone reanimation doesn't provide a solution for recovering group memberships in Active Directory for this, see my "Disaster Recovery: Active Directory Users and Groups" article at technetmagazine.com/issues/2007/04/ADRecovery
. So you will still have to provide an alternate mechanism for recovering this information when reanimating tombstones.
If you want to use tombstone reanimation as part of your data recovery strategy, either you'll need to make sure that the important attributes are saved in the tombstone or you'll need to provide another way to save and recover important attributes, such as using LDIFDE or another backup and restore scheme. Other options include third-party Active Directory data recovery products that provide an automated mechanism for backing up and restoring attributes that aren't saved in and recovered from the tombstone.
Reanimating Objects from the Configuration Naming Context
One other substantial limitation that tombstone reanimation suffers is that, generally speaking, you cannot reanimate objects that were deleted from the Configuration NC. Those objects are subject to special rules with respect to moving and renaming, as defined by the object's systemFlags attribute. Unless otherwise specified by the systemFlags attribute, objects in the Configuration NC cannot be moved or renamed, which means their tombstones cannot be reanimated. Active Directory enforces certain values for the systemFlags attribute when it creates objects of specific classes, as defined in Figure 11. Note that Active Directory applies the bit-wise logical OR operation to these values in combination with any systemFlags value specified in the add operation. This way even if the application specifies a different value for the systemFlags attribute, the necessary bits will be properly set.
Figure 11 Default systemFlags settings
The only objects from the Configuration NC that you can reanimate under normal circumstances are server objects. Interestingly, when Active Directory deletes a server object, the tombstone is not moved to the CN=Deleted Objects container for the Configuration NC; the tombstone simply remains where the object is. This allows the Knowledge Consistency Checker (KCC), which is the process responsible for calculating the Active Directory replication topology, to automatically recover from certain scenarios where deleted server objects might inhibit proper replication. So if you find yourself trying to reanimate a server object, remember that you won't find it in the CN=Deleted Objects container.
Reanimation in Your Recovery Plan
Tombstone reanimation should be a key part of your data recovery toolkit. This mechanism is the only way to recover deleted objects without taking a DC offline and, equally important, it's the only way to recover a deleted object's identity information. This neatly solves the problem of recreating a deleted user or group and having to fix up all the old ACL references.
But tombstone reanimation is an incomplete solution. You have to provide your own mechanism for recovering the attributes that Active Directory doesn't retain in tombstone objects, and your ability to recover deleted objects from the Configuration NC is limited. Remember that if you do elect to reanimate a deleted user or group, you will still have to recover the group memberships and any other linked attributes of which you might be in need.
Gil Kirkpatrick is the CTO at NetPro and has been developing software for Active Directory since 1996. Along with Guido Grillenmeier from HP, he delivers the popular Active Directory Disaster Recovery workshops. Gil is also the founder of the Directory Experts Conference (www.dec2007.com).
© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited