The Desktop FilesInside the Windows Imaging Format

Wes Miller

This column is based on prerelease versions of Windows deployment tools. All information herein is subject to change.

With the release of Windows Vista comes a significant change in how Windows is installed and deployed. The setup engine and the entire suite of deployment tools for Windows Vista (as well as the forthcoming version of Windows Server, code-named "Longhorn") have been rewritten. I discussed these changes and the work that

the Windows® deployment team put into making these changes in recent TechNet Magazine articles. But there is a lot to discuss on this topic, and so I will be covering Windows PE, Windows installation and deployment, and other Windows management and security topics in this new regular column.

In this first installment, I'll discuss the new Windows Imaging format (WIM). If you aren't already familiar with the details of imaging, be sure to read the sidebar "The Typical Imaging Process" for an overview. In my next column I'll discuss the Windows Automated Installation Kit—a new suite of tools that includes ImageX (the command-line tool used to create and manipulate WIM images) and Windows Deployment Services (WDS), which replaces Remote Installation Services (RIS)—looking at how these tools enable easier OS deployments.

The Turning Point

Year after year, the Windows deployment team has heard the same feedback from OEMs and enterprise customers. They want better support for imaging and they want Microsoft to provide unified imaging tools. These users don't want the task of selecting a third-party vendor and learning the intricacies of their product. Moreover, we had frequently heard the complaint that these OEMs and enterprise customers don't think they should have to pay for imaging tools used to deploy Windows; instead, they want Microsoft to provide these tools.

Early on in the development of Windows Vista™, the Windows deployment team made a very big gamble. We decided to completely discard the entire setup engine—many components of which had existed for nearly a decade. Setup had always been fragile and had far too many steps. We wanted to speed up the process, remove unnecessary steps, and improve the reliability of Windows setup overall. To do this, we would shift the entire setup process from the hand-stitching engine we had used for some time, and replace it with an entirely image-based setup process.

Designing a New Imaging Format

Early in the process of redesigning the setup, we set out to define what kind of image we should use. The classic argument has been sector versus file. That is, should we use an image format that stores a copy of a volume as a sector-by-sector duplicate or one that stores every single file on an individual basis. The benefit that most people expect is speed (with a sector approach) versus compression (with a file-based approach). In the end, we wound up with a format that I feel is a great compromise between the two. It delivers very good performance, but also provides for great compression.

To begin, we defined several key objectives we had to achieve to make an imaging format that meets the needs of OEM and enterprise users. Our list was heavily slanted towards a file-based architecture. I think we all saw the benefits of this approach, and therefore we first set off in that direction. And soon WIM files were born. (And in case you're wondering, the "M" in WIM doesn't stand for anything. We named it WIM based on Windows team events that recur near shipping milestones—and *.WI seemed like an odd abbreviation.) So let's take a look at the set of goals we came up with for the new image format.

Windows File System Format Neutral The image format had to function across both file system architectures that Windows supports. Though almost all systems use NTFS when running, many OEMs and enterprises use FAT file systems as a stopgap to allow them to use MS-DOS® to deploy Windows. They then convert to NTFS later on. We had to continue supporting this for Windows Vista.

Volume Geometry Neutral We had to get past the historical problem of capturing a large volume and applying it to a smaller one. Even when the imaging tool can get past this through partition manipulation tricks, it's unreliable. In a file-based architecture, the image only takes up as much space as the files that comprise it. Since the file system is restored independently, there is no reliance on volume size.

The Typical Imaging Process

When an OEM duplicates large numbers of systems or a large organization needs to deploy Windows to numerous machines, they usually use imaging technology. The premise is to take the host system, build it exactly as you want, and then duplicate it onto the target systems. In case you aren't familiar with the imaging process, here is a quick overview.

1. The administrator installs Windows on a system and configures it as desired. This involves adding and removing components, installing commonly used apps, and tweaking the UI and other settings. Let's call this the host system.
2. The administrator then runs Sysprep (provided by Microsoft in the form of the Windows deployment tools), which performs the heavy lifting of cleaning up the attributes that make a Windows installation unique (such as the computer name, the Windows Security Identifier (SID), and other unique identifiers across the system). They are reset when the second half of Sysprep runs.
3. Next, the admin shuts down the computer and reboots into Windows PE, MS-DOS, or another OS that allows him to run the imaging app.
4. Now the admin runs the imaging tool, creating an image of the reference system. This image is a file (or collection of files) that contains an exact replica of the reference disk or partition.
5. The system is shut down and the admin archives the image, which can be updated (adding applications, hotfixes, and service packs) as needed.
6. The image is replicated to one or more computers, and Sysprep completes its work, running through an abbreviated version of setup (called Mini-Setup or Windows Welcome, depending on what version of Windows you are running Sysprep on and how Sysprep is configured). This stage relies on a tiny native-mode executable called setupcl.exe.

System Architecture Neutral The image architecture had to work with any architecture that Windows might support. At the time, this meant supporting the very different x86 and IA64 (Intel Itanium) architectures. Today, this also includes the x64 (AMD64 and EM64T) architecture.

Must Be Fast to Capture and Apply The image had to be reasonably fast to capture, given compression. More critical, however, it had to be very fast at image application.

Support Non-Destructive Image Application Since we were replacing the upgrade option (which replaces Windows without harming user data or applications), as well as deriving a tool that could be used for system recovery, we wanted it to be able to image a system without destroying everything already on the volume. Sector-based formats, without considerable work, don't let a user do this—in general, you restore an entire volume or nothing at all.

Support Significant Compression When Windows Vista first started development we were still focused on shipping on CD. Considering an image of Windows XP is nearly 1GB, we decided it was important to compress the image as tightly as possible. There are tricks you can perform with a sector-based image to do this, but the moving parts in such an implementation concerned us. Many sector-based formats omit whitespace on disk to save space. (Why record a 0 to an image if it's always a 0?) However, by using a file-based approach, we handled this inherently, since we weren't picking up whitespace.

Support Single Instancing of Files This is the same as the previous issue. Windows is well-known for storing multiple copies of the same file in a few different locations. The file-based approach made it easy to store only one instance of each file and use the file as necessary.

Support Media (Disc) Spanning Even in the early days of WIM, when we could fit an image on a single CD, we knew that we had to support multiple pieces of media. Even if we shipped Windows on one CD, once users added applications, service packs, and so on, their images would likely grow beyond the size of a single CD or DVD. So we had to let them span across multiple pieces of media for their own installation and recovery solutions.

Support Multiple Volume Images in One File Blob One image rarely works for everything. When we engineered "single instancing," we also included the ability to pick up more than one volume—and the same compression and single instancing works across each. So if you pick up an image of Windows XP Professional, and then append a new image of Windows XP Tablet PC Edition (which is a superset of Windows XP Professional), the image will only increase by the files that are explicitly different between the two versions.

As a result, an OEM or enterprise user can append many derivative images to each other without an exponential increase in size. This ability would not have been easy to achieve with sector-based formats.

How WIM Works

The WIM format was designed to be as simple as possible, while meeting our guiding principles. The file-based approach allowed us to do this. But how exactly does it work?

In a nutshell, you can think of a WIM file as a CAB file (this is similar to the ZIP format, but Microsoft designed and owns the CAB format). In fact, I once heard WIM referred to as "son of CAB." The key difference between a WIM file and a CAB file is that, in addition to capturing and compressing the file itself, a WIM image stores metadata that applies to the files and directories that make up the volume captured in a given volume image. So it's sort of like an archive file with all the meta information needed to restore the volume to the way it was when the image was created—access control lists (ACLs), short/long file names, attributes, and so on.

Note that at no point during the capture process is partition information (size or type) gathered. Nor does the apply process partition the system. Unlike most imaging tools, ImageX, being partition neutral, requires that the partition be created and formatted prior to application. To automate the process before applying your image, you should use the diskpart and format command-line tools.

Nothing limits what can be stored in a volume image. An image can contain two volumes, the same volume captured on a Tuesday and a Thursday, an image with a service pack and one without, or anything else that meets your needs. The main thing to understand is that files that are identical will be captured only once. The less that two volume images have in common, the more space will be taken up.

I should also point out that there is nothing designed into the WIM format that precludes it from being used with earlier versions of Windows. It will work just as well with Windows 2000 as it will with Windows Vista. Now let's take a closer look at what exactly happens when images are captured and applied.

Capturing an Image

As an image of a volume is captured, the following steps take place:

Volume Metadata is Captured ImageX gathers data about file names, NTFS ACLs, and other file system attributes across the volume. (You can opt to exclude certain files with a script file.)

File Data is Captured File is loaded by ImageX. Each file is loaded on the volume and prepares to collect certain data about it.

File Hash is Generated A cryptographic hash of the file based on the file itself is generated. This hash becomes the unique identifier of the files in the image.

Duplicates are Checked for and Removed If another file with the same hash already exists in the image, it is assumed that this is the same file and the new file is referenced by pointing to the already existing file.

Unique File Data is Compressed Unlike in a CAB or many other archive formats, each file in a WIM file is compressed individually rather than as a collection of files grouped together as one data stream. You can also opt not to compress at all, in which case this step is skipped.

Volume Metadata is Compressed and Stored Once all of the files have been archived, a metadata entry for the volume is created. This lists each of the files that were added to the image for the volume.

Image XML Data is Generated and Stored This is a reference for each volume image that is captured. If you run ImageX /info, for example, this XML data is what you'll see.

Cached WIM Data Index is Written Finally, the master data index (along with the cached data) is written. This is the master file table for the entire WIM; it includes one entry per hash.

As you can see, a WIM is created on a file-by-file basis. Furthermore, you have one WIM file, regardless of the number of volumes that you append to the image.

Image Overload

There were a number of teams developing similar solutions when we started work on WIM. The Automated Deployment Services team was developing an imaging solution for a different customer segment. Their approach had some very interesting low-level implementation details, but their imaging engine, like most at the time, was sector-based. The Windows XP Embedded team also built a sector-based imaging engine to allow embedded independent hardware vendors (IHVs) to deploy their own images at system manufacturing time. And when the Virtual PC team was acquired by Microsoft in 2003, they brought along another file format for Virtual Hard Disk (VHD) files.

All of these formats were at the core sector-based. They met the needs of each of their customer segments, but these solutions wouldn't meet our guidelines for WIM. As a result, we continued building the new WIM format, and the ImageX command-line tool to capture images. Eventually, a number of other teams adopted our format well before it was seen publicly as part of the Windows Vista beta. The SMS team, for example, delivered an early implementation of WIM in their Operating System Deployment Feature Pack.

Applying an Image

Now let's look at the counter process: applying an image.

Cached WIM Data Index Loaded This correlates to the final step in the previous procedure. Since this is the master index, it is the first thing to be loaded from the WIM after you've specified which volume image to apply.

Image Metadata is Retrieved and Loaded ImageX loads the metadata specific to each volume image.

Directory Structure is Created To ensure that every file has a location to be restored to, the volume's entire directory tree is created.

File Data is Extracted The files are extracted from the WIM, in the exact same sequential order in which they were archived.

File Data is Applied In this step, files are loaded, decompressed (if the volume image was compressed in the WIM), and copied to each referenced location. Finally, all metadata for each file is applied.

All Directory Metadata is Applied It is important that this is the last step since there is a potential that an ACL on a directory could preclude the image from being applied correctly.

Flexible Compression

One of our key goals with the WIM imaging format was to provide flexible compression options. So we provided the ability to capture a volume image with compression when necessary, while also allowing for image capture without compression. The latter option is good for scenarios where the image doesn't need to be aggressively compressed or when capturing at high speed is critical.

There are a few key tenets to bear in mind when it comes to compression. Compression always takes longer than decompression. The more aggressive the compression algorithm, the longer it will take to compress a given set of data. And the more data you feed to a compression algorithm, the better it will compress.

WIM images can be compressed using XPress or LZX compression—and, of course, no compression is an option. XPress is a good compromise for when you need some compression, but you don't want to dedicate too much time to the capture process. LZX provides the optimal compression ratio—but, as you can guess, this method takes considerably longer.

Using no compression results in the fastest image capture, but it also results in the largest image. The compression type is a per-WIM option. Once you've captured one volume image into a WIM, all future volume images must be captured with the same compression setting.

Note that because WIM images compress on a file-by-file basis, the compression isn't as complete as it could be. This was a conscious decision—we did this so that the image would be more easily editable. The ability to edit images after capture is an important option. For a file to be edited after the image has been captured, the file data and its metadata must be replaced. If we had compressed the entire WIM all at once, support for editing files would have been incredibly complicated for us to engineer (if it were possible at all) and replacing a file in the image would take a considerable amount of time. With the compromise we made, we came up with a format that is well compressed and can still be edited using our tools.

I hope this gives you a good understanding of how the imaging capabilities we designed for Windows Vista work. In my next column, I'll discuss some of the tools you can use to manage and deploy WIM images, taking a closer look at ImageX and Windows Deployment Services.

Wes Miller is a Development Manager at Pluck (www.pluck.com) in Austin, Texas. Previously, Wes has worked at Winternals Software in Austin, and at Microsoft as a Program Manager and Product Manager for Windows. You can get in touch with Wes at technet@getwired.com. Wes would like to express his gratitude to John Macintyre, the current WIM/ImageX Program Manager at Microsoft, for his assistance with this column.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.