Post aVirtualiptica: Addressing Incidents in a Virtual Environment
Published: September 15, 2010
Author: Bruce Cowper, Communications and Response, Microsoft Trustworthy Computing
Ever been in the situation where you have or think you may have had an incident in a virtual world? By incident I'm referring to an occurrence that could be the relatively simple, though unexpected, event of a virtual machine starting and stopping, or something truly bad, such as a suspected or actual security breach.
Often, the first instinct in the heat of the moment is to fix the problem and move on. In this article, I will walk you through some of the procedures you can use not only to get your system back up and running, but also to understand better what actually led to the event. This article is not a forensics guide to the Microsoft Server Virtualization platform; it is simply a set of ideas, based on my experience, which you can adapt for your own use. I have linked to Virtual PC Guy's (Ben Armstrong) blog for many of these examples. (Armstrong's blog is a great reference, with many useful articles.)
It is vital to understand the security architecture of your chosen virtualization infrastructure and to map the system so you know how it is configured and interconnected, and what is attached (physical and logical drives and devices). While much of this information can be gleaned from sources once an issue has occurred, it is much easier to obtain before something goes awry. So, ideally, the first thing to do is map your system. A good start would be to have a look at Ben's post here on file locations.
The virtual machine (VM) configuration (.xml for Hyper-V and .vmc for Virtual Server or Virtual PC) files contain some very useful information, as shown in the examples below.
Hyper-V Devices and Drives
<pathname type="string"> C:\Virtual Machines\Server 2008 R2 Ent\Server 2008 R2 Ent.vhd</pathname>
<snapshots> [by default these are stored in the same place as the virtual hard disks (VHDs)]
Virtual Server/Virtual PC Devices and Drives
<absolute type="string">C:\Virtual Machines\SEA-DC-01.vhd</absolute> (absolute path and name of VHD; there is often also a relative path available)
<undo_pathname> (if an Undo disk is assigned, this will tell you where it is stored)
<absolute type="string">C:\Virtual Machines\VirtualPCUndo_Server 2008 Demo DC_0_0_18114910022010.vud</absolute> (note: the .vud name includes a time stamp)
<saved_state> (the virtual save file is created when the machine is put into a saved state)
<absolute type="string">C:\Virtual Machines\Server 2008 Demo DC.vsv</absolute>
<FriendlyName type="string">Network Adapter</FriendlyName> (NIC name)
<IsConnected type="bool">True</IsConnected> (NIC state)
<PortName type="string">12722a73-a2a6-4f39-a87b-af9f87d66a97</PortName> (NIC ID)
<SwitchName type="string">90a36b11-3b68-4695-bc81-8ff900091cf</SwitchName> (virtual net ID)
Virtual Server/Virtual PC Networking
<controller_count type="integer">1</controller_count> (there is one virtual network card)
<ethernet_controller id="0"> (we are looking at the first and only network card)
<id type="bytes">43671231F4124A50AF9752C36F9E85B1</id> (virtual net ID)
<name type="string">Intel(R) Gigabit Ethernet Adaptor</name> (NIC name)
There is other information to be gleaned from VM configuration files, but this will give you a good start toward identifying the files and interconnections associated with your virtual machine. Information on devices, drives, and networking may help you not only in pulling all of the files together, but also in recovering files and rebuilding an infrastructure after a failure. Even if your next step is to simply restart or restore the virtual machine from a backup, it is a good idea to make a copy of the virtual machine from which to do your analysis or recovery.
Next, we can look at the log files on the host server to understand what has been happening with the virtual machines. These can be useful in understanding many of the basics, such as the likely state of the virtual machines at a given time and what may have happened leading up to the issue. Here is a short list of the common event IDs:
There are obviously more events that can occur. It is important to keep in mind that, with Visual Server, some events may be impacted by whether logging is turned on or off for the virtual machine. In the case of Hyper-V, the logging is slightly different but the principles are the same. The content of the events will contain information about which virtual machine is impacted, but remember that with Hyper-V each VM has a unique identifier or GUID. You can obtain the GUID of a VM through either the name of the.xml file or through WMI and Windows PowerShell. There is a useful set of scripts here to perform the necessary queries.
So what can you do with the information you have gathered? Since you now understand how the virtual machine is configured and have a copy of the necessary files, the next step is often to re-create the virtual machine in a test environment. There are some considerations you should be aware of when you do this. For example, when you move a virtual machine from one physical server to another on Hyper-V, you will need to perform an export / import. When this is done, a common challenge is that if you are using a different type of processor on the new server, or it doesn’t have the necessary resources (CPU / memory), you may need to adjust the settings for these components in Hyper-V Manager or System Center Virtual Machine Manager. In some cases, starting from a saved state or snapshot is not possible due to these changes, but simply booting from fresh may be.
If you are unable to get the virtual machine up and running, don’t panic. There a number of ways to access virtual hard disks (VHDs) without having to boot up the virtual machine. For example, you can mount a VHD on a Windows Server 2008, Windows Server 2008 R2, Windows Vista, or Windows 7–based system directly from the Disk Management tool. You can also use a myriad of free tools and your favorite forensics kit as many of them will mount the virtual disks directly.
Interestingly enough, many of these tools (and others) will also enable to you convert VHDs between formats for different virtualization platforms, and allow you to import or migrate them into a new configuration. A useful reference can be found here.
You should also know that VHDs and the associated files are simply that: If you are looking to recover these files, you don’t need to look any further than your backup in the Volume Shadow Copy Service. This is often a very effective way of creating a disaster recovery backup, and many of the tools on the market work directly with Volume Shadow Copy Service for just this purpose. If all else fails, you can always look in the slack space for old copies of VHD files, configuration files, snapshots, undo disks, virtual save files, and other files. In practice, I have had mixed success with this process because the files are so large that they frequently get overwritten.
While I have only scratched the surface of the tools, techniques, and processes involved in obtaining information about your virtual machine and addressing an incident, hopefully I have given you some idea where to start looking for information. I recommend that you always have your systems well-documented, backed up, and managed; however, if this is not the case and something goes wrong, it’s my experience that, most often, not all is lost.