Manage shims in an enterprise
Applies to: Windows 7, Windows 8, Windows 8.1
When addressing application compatibility issues in preparation for a deployment of Windows, among the most flexible and powerful tools available are application compatibility fixes, commonly called shims. However, many organizations do not leverage shims to the extent that they could, either because they don’t understand the underlying technology or because they do not have a process in place to manage shims over the remaining lifetime of the applications they are used to remediate.
This article shares best practices for addressing application compatibility issues using shims. It details how shims work, when to consider applying shims, and how to manage the shims you do apply.
In this article:
- Understanding shims
- Deciding when to use shims as a compatibility mitigation
- Custom shim database management strategies
- Merging custom shim databases
- Custom shim database deployment
How shims work
Application compatibility in Microsoft Windows operating systems is one of the fundamental pillars of its development, alongside performance, reliability, and manageability. To reduce deployment costs and accelerate adoption, Microsoft invests in deep technical solutions to ensure broad compatibility of existing software, driving compatibility into the engineering and release process.
The Microsoft Windows Application Compatibility Infrastructure (Shim Infrastructure) is one such powerful technical solution. As the Windows operating system evolves – changing to support new technology, incorporate bug fixes, and/or implement a modification in strategy – changes to the implementation of some Windows Application Programming Interface (API) functions may affect applications that use them. Because the nature of the dependencies on particular implementations vary so much, it may not be possible to create a single implementation that works for all existing software, while simultaneously providing the desired new behavior. In the past, we would address such compatibility issues by placing branches within the source code for Windows, but doing so presents a long-term challenge for the serviceability and reliability of the Windows operating system. It also means that we have to be aware of the issue before we ship Windows – and that is certainly not always true for enterprise software! Using the Shim Infrastructure, however, you can target a specific application fix specifically towards a particular application (and typically, for particular versions of that application), with these fixes housed outside the core Windows functions and maintained separately.
The Shim Infrastructure implements a form of API hooking. Specifically, it leverages the implementation of dynamic linking to redirect API calls from Windows itself to alternative code – the shim itself. The Windows Portable Executable (PE) and Common Object File Format (COFF) Specification includes several headers, and the data directories in this header provide a layer of indirection between the application and the linked file. Calls to external binary files take place through the Import Address Table (IAT). Consequently, a call into Windows looks like Figure 1 to the system.
Figure 1. Application calling into Windows through the IAT
Specifically, you can modify the address of the Windows function resolved in the import table, and then replace it with a pointer to a function in the alternate shim code, as shown in Figure 2.
Figure 2. Application redirected to the shim prior to calling Windows
This indirection happens for statically linked dll files when the shimmed application is first loaded. Windows also shims dynamically linked dll files by hooking the GetProcAddress API.
Design implications for the shim infrastructure
You may find certain consequences of the Shim Infrastructure design relevant when determining your shim strategy.
First, as shown in Figure 2, the code that runs inside a shim still sits outside Windows. Consequently, Windows holds shim code to the same security restrictions as the application code itself. In fact, to Windows, the shim code appears to be application code. As a result, you cannot use shims to bypass any security mechanisms present in Windows. For example, no shim is available to bypass the User Account Control (UAC) prompts yet still run the application with elevated permissions (a common request from enterprise customers first experimenting with shims in an attempt to remove administrator rights from more users). You can shim the application not to require administrator rights, or you can shim it to prompt the user to provide administrator rights, but in order to receive administrator rights with UAC enabled, the user will have to approve the elevation. The same is true for code that you write yourself.
Therefore, when evaluating the security implications of using shims in your enterprise, you are not opening any additional security vulnerability. In fact, using shims to avoid having to loosen security descriptors or make security policy more lax can frequently be the more secure choice. For example, without shims, you may be able to mitigate a compatibility issue by loosening the ACLs on a particular directory, but this decision has an effect on the entire system. Using shims, you may be able to redirect the file access to a per-user location for that application. As another example, an application may be explicitly checking for administrator rights. Without shims, you may have to grant the application administrator rights to pass this check. If the check is an unnecessary one, however, a shim could simply lie about whether the current user has administrator rights, allowing the check to succeed without exposing additional security surface area.
Because the Shim Infrastructure, in essence, injects additional code into the application before it calls into Windows, any mitigation you can use a shim to accomplish can be done by modifying the application code itself. At a minimum, the application could include code similar to what Windows implements in the shim immediately prior to calling into Windows APIs. As a result, even when shims are not used as the final remediation, they can often be very instructive during troubleshooting.
The performance impact of shims will vary, depending on what the shim is doing. Let’s consider some examples. A version lie shim, which simply pretends to be an earlier version of Windows, performs about the same– it doesn’t take any more work to return a value of 5.1 than it does to return a value of 6.2. The ForceAdminAccess shim, as another example, performs much faster than the APIs it intercepts – since it always says “yes, the user is an admin” without looking, it avoids the performance impact of actually determining if the user is an administrator! As a final example, the CorrectFilePaths shim inspects the path to determine if it should be modified, so you add some performance overhead to do the string comparison for all intercepted file system calls, plus the additional overhead of replacing the string argument when a match is found. As you can see, there is not one easy answer, so it’s best to test when you have performance requirements. What most customers find, however, is that the applications that require shims for remediation are often older, and as a result still meet their performance criteria by virtue of the fact that today’s hardware is much faster than the hardware it ran on years ago when they acquired the software.
Finally, because shims run as user-mode code inside a user-mode application process, you cannot use a shim to fix kernel-mode code. For example, you cannot use shims to resolve compatibility issues with device drivers or with other kernel-mode code. (For example, some antivirus, firewall, and antispyware code runs in kernel mode.)
Back to top
Deciding when to use shims as a compatibility mitigation
When Microsoft consultants work with customers on using shims to resolve compatibility issues with Windows, they begin the conversation with an overview of how shims work, ensuring that people at various levels in the organization understand the technical implications of using shims for mitigations. However, the decision is more than a technical one.
When you are considering whether shims are appropriate, your first question should be whether your business will require a valid support agreement for the application on the version of Windows you are seeking to deploy. If the answer is yes, then your first check should be whether you have a version that is supported by the vendor. If you have a vendor supported application that fails, you don’t need to use shims – you just need to contact the vendor and open a support case to get it fixed!
It is when vendor support is not required that most customers consider using shims. You may be taking a version that is not supported on the version of Windows you are trying to deploy, and which has a failure, and attempting to get it working again. But keep in mind that shimming the application to get it working doesn’t change the support state – it will remain unsupported by the vendor even after you have gotten it working.
Scenarios in which customers decide to use shims
The scenarios in which Microsoft consultants have assisted customers in using shims to mitigate application issues include:
You acquired the application from a vendor that is no longer in business.
Several applications Microsoft consultants have worked with have been from vendors that have since gone out of business; so clearly, support is no longer an option. The application is nice to have if the consultants can keep it working, because it prevents you from having to implement the software yourself (as frequently there are no vendors offering a competing offering). However, because the source code is not available, shimming is the only option for compatibility mitigation. When this is the case, we recommend noting the risk of long-term use of an application where support is no longer available.
You developed the application internally.
While most customers would prefer to fix all their applications to be natively compatible, there are some scenarios in which the timing does not allow for this. The team may not be able to fix all of them prior to the planned deployment of Windows, so they may choose to shim the applications that can be shimmed and modify the code on the ones where shims are insufficient to resolve the compatibility issue. Alternately, you may already be at work on a long-term project to replace the application with a new one, and does not wish to invest further in the existing version, instead using shims to resolve compatibility issues until the new application is completed. Then, the customer deploys the natively compatible version upon completion, without having to delay the deployment of Windows.
You acquired the application from a vendor that will eventually be releasing a compatible version, but support is not critical.
When an off-the-shelf application is neither business critical nor important, some customers use shims as a bridge solution. Users could theoretically wait until a compatible version is available, and its absence would not block the deployment, but being able to provide users with a shimmed and functional version can bridge that gap until a compatible version is available (or budget to acquire it becomes available).
In general, Microsoft consultants have found that good communication and collaboration among technology owners (“do the shims succeed in making the application sufficiently compatible”) and business owners (“can I accept the support terms of using a shimmed version of this incompatible application”) helps the decision process.
Deciding which versions of an application to shim
While a thorough description of applying a shim to a particular application is outside the scope of this white paper, one aspect that is important to note is that shims can be applied to particular versions of applications, either as “up to or including” or just a particular version. Either one ensures that the next version of the application that is released will no longer have the shim applied.
This is important to many customers. They want to ensure that they can fix the incompatible application for a particular version but still encourage the developer (whether an internal development team or a vendor) to fix the application by having the shim no longer apply the next time the version number is incremented.
Support for shims
While support policies for applications made compatible with Windows using shims is up to each software vendor, another frequent question is, how is the code for the shims themselves supported?
Shims ship as part of Windows and are updated through Windows Update. Consequently, they fall under the same support terms as the rest of the Windows operating system.
You can apply shims that the Windows product team creates to your own applications, but Microsoft does not provide the necessary tools for you to create your own custom shims leveraging the shim infrastructure.
Back to top
Custom shim database management strategies
If you have decided to use shims as part of your application compatibility mitigation strategy (for certain classifications of applications), the next question is, which strategy should you employ for managing custom shim databases? The customers Microsoft consultants have worked with have tended to select one of two approaches to managing their custom shim databases: deploying fixes as part of the application package or managing a centralized custom shim database.
Regardless of the approach your organization chooses, here are general recommendations for improving the management of custom shim databases:
Define standards for when to apply shims.
You also want to define the scenarios in which shims are appropriate to use, as discussed earlier, based on the specific business and technology needs of your organization.
Define standards for custom shim databases.
You may want to define standards for how to map shims to particular applications. For example, you may want to ensure that shims always include a version check so that the shim stops being applied to subsequent versions of the applications.
Define a resource responsible for addressing questions and enforcing standards.
Having an individual or a team responsible for being familiar with the technology and standards around the use of shims has consistently been an important predictor of success. Many customers ramp up on shims and mitigations in general in response to an operating system migration but soon forget the details when the migration is over. As the databases are managed over time, you will want to ensure that some resource continues to remain familiar with these details.
Avoid over-engineering your shim management solution.
While we do recommend having a strategy, several customers have engineered a process of monthly review and deployment of shim updates. However, in practice, customers who use shims to resolve application issues tend to find nearly all of them prior to the initial deployment, and only occasionally find new shims they want to apply afterwards. Having a sophisticated process for ongoing governance assuming constant churn in this space generally results in wasted effort.
Deploying fixes as part of an application package
One strategy for deploying application fixes is to include the custom shim database—containing a single entry for the application the package is installing—directly into the installation package. During the early phases of compatibility testing, this can seem like the easiest approach. However, over time this approach can grow more complex. Microsoft consultants recommend evaluating the following considerations prior to selecting this approach.
How many applications will you end up shimming?
The thing to keep in mind is that custom shim databases are still databases. Consequently, if you were to have 1,000 shimmed applications, it takes longer to open and query 1,000 different one-row databases looking for a match to a given application than it would to open a single database and query against 1,000 rows. Note that this performance penalty is something you pay every time a new process is created on Windows. While not generally significant enough for end users to notice, depending on the environment (because of the law of large numbers) this can eventually average out to an enterprise-wide increase of higher power consumption that is no longer trivial in nature.
Can you track which applications you have deployed to which computers?
It is possible that you will eventually find that the shims assigned to resolve a set of compatibility issues in an application are not comprehensive and that later you will need to deploy an updated version of the custom shim database that resolves the additional issues your organization later discovered. If you deployed the original custom shim database as part of the installation package, you will need to locate each client that has installed this application and the original custom shim database for it to replace it with the new version. While this strategy can be a good approach if you are shimming only a few applications, when the number of applications grows beyond a few, most customers eventually opt against using it.
Managing a centralized custom shim database
An alternate strategy most customers consider (and most end up using) is to manage either a single custom shim database or several custom shim databases for large subsets of the organization. Doing so makes it easier to enforce policy and provide consistent updates to application mitigations you discover that you need to support your migration to Windows 7. Microsoft consultants recommend evaluating the following considerations prior to selecting this approach.
Do I have deployment tools to deploy and update custom shim databases to all the target computers?
If you are planning to manage a centralized custom shim database, ensure that the tools are in place to deploy and update this custom shim database across all the computers in your organization that require this. As additional applications have shims applied to them, ensure that the target computers have the updated shim database installed prior to using the application.
Do I have centralized resources in place I can dedicate to managing and updating the centralized custom shim database?
If you are taking the centralized approach, make sure that you have identified appropriate owners and that the application owners and testers have a clear path for escalating a request for a shim to result in deployment of an update to target computers.
Microsoft consultants have found that this strategy tends to be the best approach, when you have a solid deployment infrastructure in place and centralized ownership of the process. The primary advantages have been accountability and simplifying support (as the deployment of a particular version of a shim is more consistent across the organization).
Back to top
Merging custom shim databases
Customers who have selected a centralized custom shim database approach benefit from the improved performance of searching a single database to determine whether Windows should apply a shim to a particular executable file. A frequent question Microsoft consultants receive is how to merge custom shim databases to create a single custom shim database. Customers have generally taken the following approach:
- Application compatibility testers generally run on a computer containing the latest version of the organization’s custom shim database (which still may be a preliminary version).
- If an application requires an additional shim, the tester will create a second custom shim database containing the shims required for that application, which the tester uses to verify the fixes, and through customer acceptance testing.
- If the application passes all the functionality and integration tests, the single-application custom shim database is forwarded to the team that manages the custom shim database.
- The central team opens the master copy of the organization’s custom shim database. This step is important, because the database contains a globally unique identifier (GUID) that makes updating the database easier (installing a new version of a database with the same GUID as an existing database installed on the computer uninstalls the old version).
- The central team can then copy and paste the shims that were applied in the new custom shim database into the master shim database for the organization. (These are options on the right-click menu of Compatibility Administrator.)
- The central team then redeploys the new version of the custom shim database containing the additional application fix to all users.
Custom shim database deployment
Deploying a custom shim database to users requires the following two actions:
- Placing the custom shim database (*.sdb file) in a location to which the user’s computer has access (either locally or on the network).
- Calling the sdbinst.exe command-line utility to install the custom shim database locally – sdbinst.exe is located in the system32 directory, which is in the path on nearly all Windows configurations.
While any approach that completes these two actions will work, customers commonly use one of the following two approaches:
- Packaging the *.sdb file and a script in an .msi file and then deploying the .msi file, making sure to mark the custom action not to impersonate the calling user. For example, if using Microsoft Visual Basic® Scripting Edition (VBScript) script, the custom action type would be msidbCustomActionTypeVBScript + msidbCustomActionTypeInScript + msidbCustomActionTypeNoImpersonate = 0x0006 + 0x0400 + 0x0800 = 0x0C06 = 3078 decimal.
- Placing the *.sdb file on a network share, and then calling a script on target computers, making sure to call the script at a time when it will receive elevated rights (for example, from a computer start-up script instead of a user log-in script).
Note that you must ensure that the installation of the custom shim database executes with administrative rights.
This example script is taken from the custom action of a Windows Installer (MSI-based) installation of a custom shim database:
Initial deployment and updates
Because testing and mitigation of application compatibility issues typically happens prior to the deployment of a new version of Windows, a common approach is to include the custom shim database containing all known issues at the time of deployment with the corporate image. Then, as you need to update your custom shim database, you could provide these updates using one of the mechanisms described above. This is the methodology that Microsoft uses to manage the System shim database. The initial version was released with the Release to Manufacturing (RTM) version of Windows, and updates are provided with Windows Update. When you use this approach, you are using a methodology proven at a very large scale.
Back to top
Shims are a powerful tool for mitigating application compatibility issues and moving forward with your Windows 7 deployment. However, understanding when to shim and how to manage shims within an organization can be a challenge.
This white paper has reviewed the approaches that other customers have used to leverage this powerful technology while minimizing risk and costs. Understanding how shims work is often the most important step in adopting the technology. You have seen how to set criteria for selecting the applications for which using this tool is appropriate after the decision has been made to adopt the technology. Finally, this paper discussed the approaches that most customers use to manage and deploy a custom shim database.
Back to top
- Understanding application compatibility
- Five things every IT professional should know about application compatibility
- Five steps to Windows 8.1 application readiness
- Get started with application compatibility in a Windows deployment
- Windows Assessment and Deployment Kit
- Windows 8.1 compatibility cookbook
- Windows 8.1 compatibility center
About the author
Chris "The App Compat Guy" Jackson is a Principal Consultant and the worldwide lead for application compatibility at Microsoft, specializing in Windows, Internet Explorer, and Office internals. Jackson is a widely recognized expert in the field of application compatibility, creating technical documentation, training, and service offerings used inside and outside of Microsoft and based on years of real-world experience with enterprise customers and independent software vendors. Author or co-author of numerous technical papers and articles, he is also a featured speaker at major industry conferences around the world and publishes a popular blog.