Application Compatibility

Inside the Application Compatibility Toolkit 5.5

Chris Corio and Chris Jackson

 

At a Glance:

• Application Compatibility Manager and the Agent Framework
• Compatibility Evaluators
• Storing and analyzing the data
• The Standard User Analyzer
• Internet Explorer Compatibility Test Tool

Contents

Application Compatibility Manager
Compatibility Evaluators
ACT Database
Analysis
Online Sync
Standard User Analyzer
Internet Explorer Compatibility Test Tool
Conclusion

The Application Compatibility Toolkit (ACT) helps you manage the process to determine whether your applications are compatible with a new version of Windows before you deploy them. It can also help you determine how an operating system update will impact these applications. There have been a number of articles and white papers written on how to use the ACT 5.5. What we want to do in this article is go under the hood and explore how each of the tools in this toolkit works. We'll start with the tool most people associate with the toolkit: the Application Compatibility Manager.

Application Compatibility Manager

The Application Compatibility Manager provides a framework for discovering information about your production environment, organizing your analysis of this data, and helping to drive the testing process. Let's dig into this tool to understand how it works.

Evaluator deployment packages (MSIs) You deploy ACT 5.5 data collection packages as MSIs, but if you dissect the MSI, you'll find that it doesn't actually do much work. Instead, it extracts an executable file that performs the setup. Figure 1 shows the process tree for a typical run of the setup process.

Figure 1 Agent Framework Installation Process Tree

The third instance of msiexec.exe, shown in red, is an interesting one: it's a call to uninstall the original MSI. Essentially, the MSI extracts an .exe (shown as a .tmp file), and then uninstalls the outer MSI. Later, a second MSI (called by msiexec) is installed. In one test, the deployment MSI finished installing at 1:19:46, and it finished uninstalling at 1:19:50. The MSI appears installed for only 4 seconds! Consequently, when deploying a data collection package, don't look for the installation of the MSI to determine if you ought to install it to a target computer, because you're not likely to find it. You need to look for other evidence that you have installed agents to each workstation.

For example, while the MSI you deploy will have a new generated product code each time, the inner MSI will always have the same product code (DC93B45B-D4F5-4FFE-9B47-042BD6FA8CC5), and this can be used as evidence that an ACT data collection package is installed (although you won't be certain which one, as all ACT data collection package have this footprint). Note that we do not recommend uninstalling a data collection package using this MSI entry; we discuss the recommended way to uninstall an ACT data collection package next.

Afsetup.exe performs the bulk of the work, and you'll want to be aware of this binary in the event you have any issues with your agent deployment. You can uninstall the agent framework using the following command:

%program files%\Microsoft Agent Framework\Agent Framework\afsetup.exe /uninstall

Agent Framework The ACT 5.5 Agent Framework provides a structure for gathering data from a large collection of computers in an enterprise. A design goal of agents is to be unnoticed and suitable for deployment into a production environment. Consequently, the work of the individual agents is heavily performance-optimized (which you'll notice in some of the design decisions made in the agents).

The framework itself is straightforward. When the Agent Framework is installed, you can find its files in %program files%\Microsoft Agent Framework. It runs as a service (actdcsvc.exe) that allows the scheduling and launching of agents. The XML found in the Agent Framework\Data subdirectory configures the agents to launch according to a specified schedule. You can find the agent executable files under Agent Framework\Agents.

Agents are nothing more than executable files—the Agent Framework service simply manages the schedule and launches the appropriate executable files as specified in the installation and configuration XML. Figure 2 shows an example of the process tree for the first 5 minutes of a typical Agent Framework install.

Figure 2 Agent Framework Process Tree

What we have here is a straightforward service that simply schedules and executes agents. The agents do the work, so let's explore them.

The ACT 5.5 Inventory Agent, collect.exe, is one of the most significant. Essentially, this agent searches client systems and compiles an inventory of applications. In terms of the data required for planning an application compatibility project (an application name aligning to a vender, version, and eventually a support statement), Inventory Agent consistently performs extremely well compared to other software inventory tools.

What's particularly helpful is not just how well the agent scours the system looking for evidence (detailed below), but how well it then cleans up that data before sending it along. The tool creates a series of "buckets" and then consolidates all of the evidence found in the searches detailed below, flattening, removing duplicates, and categorizing it. The agent looks for evidence in a number of locations to find applications.

The MSI Database search uses the MsiEnumComponents API to enumerate all applications installed using Windows Installer.

The add/remove programs search opens both HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Current Version\Uninstall and HKEY_CURRENT_USER\Software\Microsoft\Windows\Current Version\Uninstall and collects data from each subkey found here.

The Windows Shell search works by enumerating the contents of HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders, as well as HKEY_USERS\<…>\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders for each user with a profile on the computer. The agent then searches for all of the files and links in these directories or their subdirectories (specifically excluding desktops, which tend to be "parking lots" for executables downloaded from the Web, to reduce noise). This search captures software that, for example, does not register itself in Add/Remove Programs but does add a Start menu shortcut.

The app paths search enumerates all of the subkeys in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\App Paths registry key, picking up any applications that have registered themselves to receive a program-specific application path.

The path environment variable search looks for executable files in each directory (but not subdirectories) pointed to in any user's path environment variable. It first looks for executable files in paths returned from a call to ExpandEnvironmentStrings (which returns both system environment variables and the current user's environment variables) plus the contents of HKEY_USERS\<…>\Environment for all other users. This enables the inventory collector to find applications (typically command-line applications) installed by simply placing the file in the file system and then appending to the path environment variable.

The file-extension handlers search looks at each registered file extension and captures programs registered to handle those file extensions. It first opens HKEY_LOCAL_MACHINE\Software\Classes and enumerates each entry that begins with a period (.) as these are file extensions. For each entry that's found, the agent enumerates the program IDs and locates the shell\open\command subkeys of each program ID to find executable files. It then does the same for each user, enumerating the contents of HKEY_USERS\<…>\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts. This allows the agent to find software where the installation simply dropped binaries and then registered itself as a handler for a file extension.

The registry run/runonce search looks at the following registry keys to locate executable files that may represent applications:

• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run
• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce
• HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVerison\RunOnceEx

Examining these registry keys allows the agent to find software that runs with each startup but that may not leave other evidence for the collector to find.

The Service Control Manager search queries the Service Control Manager (using the EnumServiceStatus API) and then queries each service for details (using the QueryServiceConfig API). This allows the agent to find software whose installer manually places executable files on the file system and sets the registry keys to register the service.

The Windows Components search queries the optional components installed for Windows, to capture these applications as well. It does this by querying sysocmgr.inf (in %windir%\inf) and enumerating the optional components selected. This allows the agent to find software that is likely not to exist in a standard installation of Windows.

Compatibility Evaluators

While knowing what software you have installed on computers across the enterprise is, typically, the most critical data for running an application compatibility project, the Agent Framework also lets you gather additional data from the computers with agents installed. The compatibility evaluator agents provided with ACT are performance-optimized for production use, and consequently collect a limited amount of data. Review the data collected and run a proof-of-concept to measure the value of the data before investing heavily in a massive compatibility evaluator deployment.

UAC Compatibility Evaluator The UAC Compatibility Evaluator (uacce.exe) installs a custom shim database (uacce.sdb) and custom shims (uacdetct.dll) in order to determine whether an application can run as a standard user (or as a protected administrator) on Windows Vista or Windows 7. (Shims are small pieces of application code that are inserted between the application and Windows, which are typically used in Windows to resolve application compatibility issues but here are used to detect them.) It applies these shims to processes running on the system by setting a compatibility layer for explorer.exe; it sets the __COMPAT_LAYER environment variable to include UACCEDetection, a layer defined in the custom SDB. Because child processes inherit layers, any process created using Explorer will have these shims applied.

What do the shims do? The FileOperations shim intercepts calls to the _lcreat, CopyFile, CopyFileEx, CreateFile, DeleteFile, MoveFile, MoveFileEx, MoveFileWithProgress, ReplaceFile, LZOpenFile, EncryptFile, DecryptFile, and DuplicateEncryptionInfoFile APIs. For each call, it checks to see if the ACL restricts access or if the file lives in a special path (in case you have loosened the ACLs).

The ACL check is determined by taking the existing ACL, removing "administrator" ACEs and privileges, and then calling AccessCheck on the resource to determine if it succeeds for a non-administrator user. The special path check looks to see if the path is in the root directory, %program files% directory, %system% directory, or %windows% directory. Any failures are recorded.

The RegistryOperations shim intercepts calls to the RegCreateKey, RegCreateKeyEx, RegDeleteKey, RegDeleteKeyEx, RegDeleteValue, RegOpenCurrentUser, RegOpenKey, RegOpenKeyEx, RegCloseKey, RegOpenUserClassesRoot, RegReplaceKey, RegRestoreKey, RegSetValue, RegSetValueEx, and RegUnloadKey APIs. For each call to an intercepted API, the shim checks to see if the ACL would deny access to a standard user. This ACL check is the same one used for file operations—stripping down the token and calling AccessCheck. Failures are recorded.

The ProfileOperations shim intercepts calls to the WritePrivateProfileSection, WritePrivateProfileString, WritePrivateProfileStruct, WriteProfileSection, and WriteProfileString APIs. For each call to these APIs, the shim checks to see if the ini file is mapped using an ini file mapping. If not, the shim checks the permissions on the file, as indicated in the FileOperations shim (since, if not mapped, an ini file is just another file), and failures are recorded.

The RestrictedNamespace shim intercepts calls to the CreateFileMapping API. If the name of the object includes either the Global\ or Session\ namespace, then a failure is recorded.

The ElevatedRunLevel shim intercepts calls to the CreateProcess, CreateProcessAsUser, CreateProcessWithLogon, and CreateProcessWithToken APIs. All intercepted APIs check the target executable to determine if the installer detection feature of Windows will trigger an elevation event (which returns ERROR_ELEVATION_REQUIRED when using the CreateProcess APIs). Furthermore, the shim checks the target identity for APIs that change the user credentials to determine if they are administrator accounts.

If you look closely at how detection works, you'll see that it is assumed that you are running elevated while the agents are working. Since the agents were designed for use on production systems, the assumption is that the application is already working. And if it's working and would have a problem with UAC, then you're probably running as an administrator!

Windows Vista/Windows 7 Compatibility Evaluator The Windows Vista/Windows 7 Compatibility Evaluator is actually split into two separate agent executables, since one unit of work can be detected statically, while the other is monitored and analyzed at run time.

The GINA Session 0 agent (ginasession0.exe) performs a straightforward registry check. It looks at HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\GinaDLL to determine if you have installed a custom GINA. It also enumerates services, looking at the type value in the registry to see if the 0x100 bit is set (the SERVICE_INTERACTIVE_PROCESS flag). Any failures are recorded.

Much like the UAC Compatibility Evaluator, the Deprecation Agent (dep.exe) installs a custom shim database (dep.sdb, or win2kagents.sdb on Windows 2000) and custom shims (depdetct.dll), setting the __COMPAT_LAYER environment variable for Explorer.exe to include DeprecationAgentLayer, which ensures that any processes launched through Explorer are shimmed. (Note that the shim database also excludes specific binaries from the shim that are known to fail if they detect that they have been shimmed.)

What are the deprecation agents looking for? Let's examine the shims it uses.

The DllLoadOperations shim intercepts LoadLibrary and LoadLibraryEx and compares the library name passed to the list of deprecated DLLs. This list is stored in DepManifest.csv, which you can find in %program files%\Microsoft Agent Framework\Agent Framework\Agents\DEP. The entries where the first column is DllType are used.

The ExeLoadOperations shim intercepts the CreateProcess, CreateProcessAsUser, CreateProcessWithLogon, CreateProcessWithToken, ShellExecute, ShellExecuteEx, and WinExec APIs, comparing the executable name passed to the list of deprecated EXEs, also stored in DepManifest.csv where the first column is ExeType.

The RegistryLoadOperations shim intercepts the RegCreateKey, RegCreateKeyEx, RegCloseKey, RegDeleteKey, RegDeleteKeyEx, RegDeleteValue, RegOpenCurrentUser, RegOpenKey, RegOpenKeyEx, RegOpenUserClassesRoot, RegReplaceKey, RegRestoreKey, RegSetValue, RegSetValueEx, and RegUnloadKey APIs, comparing the registry key passed to the list stored (again) in DepManifest.csv—this time with a first column of RegType.

The ApiLoadOperations shim directly shims specific APIs that have been deprecated, noting that an application called it. For Windows Vista, there is only one: StiCreateInstanceA in sti.dll. (And there's even a story behind this; see https://blogs.msdn.com/tomarcher/archive/2006/03/22/windows-vista-sti-and-a-story-about-customer-service.aspx.)

One thing you'll notice about this agent is that it depends on running down level: the fact that you called something is recorded only if you succeed in calling it.

Update Compatibility Evaluator The Update Compatibility Evaluator is useful in determining which Windows binaries a particular application is using, which enables you to target your testing more precisely when Windows Updates are released. The Update Compatibility Evaluator works by installing a kernel-mode driver (fdrtrace.sys) that implements both a File System Filter Driver and registry hooks to record the file system and registry activity generated by applications. It leverages a service (uiaservice.exe) and an executable file (uiaconvert.exe) to interpret the data and align it to applications and access to Windows binaries.

Bucketizer Agent The Bucketizer agent maps the data collected from compatibility evaluator agents against the computer's inventory in order to categorize any issues detected into buckets for each application.

Compressor Agent Just like the name sounds, the purpose of the Compressor agent (compressor.exe) is to compress the files to be uploaded to the specified file share into a CAB file to save network bandwidth.

Uploader Agent The Uploader agent is responsible for uploading the data collected from other agents to the specified file share. The work of this agent is relatively straightforward—copying data collected locally to the indicated location. If the location is inaccessible, the agent waits 5 seconds before retrying. The agent will retry three times before giving up.

An important new feature in ACT 5.5 is that Data Collection Packages can be "tagged"—configured so that all logs uploaded by a particular data collection package will have a particular tag applied to them. This allows you to more effectively address some scenarios, such as thoroughly understanding which software is being used for each group inventoried (previously this was hard to measure for overlapping applications) or for consolidating data from several groups or organizations.

Log Processing Service The Log Processing Service runs actdcsvc.exe as service. If you think that sounds familiar, you're exactly right—this is the same agent framework service discussed earlier! In this case, it runs the following three agents:

• The Decompressor agent (decompressor.exe) is the companion to the Compressor agent used on the workstations you collect from; it extracts the XML files generated on the client from the CAB files used to upload them. Once extracted, the Listener agent detects the change and can proceed.
• The Listener agent (listener.exe) monitors the specified directory for changes. It waits for changes to take place in the monitored directory, then processes them by sending an RPC call to the Queuing agent.
• The Queuer agent (queuer.exe) sets up ports to listen for RPC calls from the Listener agent. When it receives these calls, it processes the file and uploads them to the database. The Queuer agent moves completed XML files from the root or uncompressed folder to the Processed folder when it succeeds. If it fails, it moves the files to the Failed folder. Quite often, if a log fails to process, you can simply move it back to the root directory of the monitored share to try again, and often this will succeed.

A question that comes up quite frequently is whether you need to save these processed files. Since the data is already in the database, you technically do not need to save them, and most people can safely delete them. However, these files can come in handy to rebuild the database or to consolidate two databases, so it is worth considering your future need for regenerating the database before deleting this data.

ACT Database

The Application Compatibility Manager may be where most of the magic happens, but the ACT database is where all the data lives. There are a number of tables in the ACT database. While the schema is undocumented, it is very straightforward ( but, alas, subject to changes with almost every new release of the ACT ). Important tables include Applications, Machines, Devices, and Issues, and they are used to organize all of the information returned by the various agents and processed by the Log Processing service.

Applications are tied to data with a unique Application ID, generated using the name, version, vendor, and language (NVVL) of the application.

You can host the ACT database on Microsoft SQL Server 2005 or later, including the Express versions. (SQL Server 2000 is no longer supported.) The Application Compatibility Manager creates the database during the initial configuration phase of ACT's Enterprise Configuration. The script used to create the database resides in %allusersprofile%\Microsoft\Application Compatibility Toolkit 5\CreateDB.sql.

A key aspect of the ACM is the ability to create reports. You create reports by filtering the ACT database using a query builder control in ACM, which is shown when you click Toggle Filter. You can filter based on criteria related to entities in the ACT database and link these clauses to create AND and OR relationships. You can save your report as an .adq file—an .xml file that defines the filters used. The .adq file is a portable format that neither contains data nor depends on a particular set of data.

Online Sync

When performing an online sync, ACT uses a public Web service. In previous versions of ACT, unique Application IDs for all of your software was sent, but you could hide your compatibility assessment from the community if you wanted. With ACT 5.5, when you decide not to share information about an application, ACT never sends the Application ID (which also means you don't receive data in return).

The community data comes from the Microsoft Compatibility Exchange database and includes logo certification data and community data (other users' votes). A separate system today powers windows.com/compatibility, and this data adds the outcome of manual research into the compatibility state of commercially available software; ACT 5.5 will be the first release of ACT to incorporate this data , which provides far more information about the known state of compatibility than was previously available.

Standard User Analyzer

The Standard User Analyzer (SUA) consists of several binaries to monitor and analyze application execution and to communicate concerns that arise during application testing. The main SUA UI allows you to target an application to analyze and select the launch conditions for this application. SUA will then employ Application Verifier as its underlying platform to monitor applications. After monitoring the application's execution, SUA presents the acquired data in the SUA UI where the user can work to understand the data and even create mitigation packages that plug into the Windows application compatibility infrastructure.

Application Verifier Application Verifier is a tool that offers developers a way to monitor application execution. SUA uses Application Verifier to track application behavior that would require administrator privileges by hooking specific APIs and watching their usage.

Application Verifier implements a plug-in model (which, as of this writing, is not yet documented for public use) where specific dlls can be registered and employed to monitor targeted applications. (Plug-ins are configured using the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\{ApplicationVerifierGlobalSettings}.) The dll that SUA uses to monitor applications is vfLuaPriv2.dll, which is installed into the %windir%\System32 directory during installation. The vfLuaPriv2.dll binary supersedes vfLuaPriv.dll installed by Application Verifier.

When SUA launches an application for testing, it initially checks to ensure Application Verifier is installed on the system and prompts the user to install it if necessary. Next, SUA uses the Application Verifier command line interface to register an application; for example, if notepad.exe is targeted for monitoring, the command line SUA calls would be the following:

appverif.exe -enable luapriv -for "notepad.exe"

Application Verifier matches the luapriv string with the name specified in vfLuaPriv2.dll.

When SUA configures an application, it also tells Application Verifier to monitor any child processes created by that application. Application Verifier creates a separate log file for each process, but SUA combines these log files to create a single, consolidated report. SUA can also save the collection of logs in a .cab file for viewing on another computer.

Standard User Analyzer UI SUA can test an application in one of three different contexts: launching the application with the current user's token, with the current user's token disabling file and registry virtualization, or with elevated permissions. In each case, SUA will launch an elevated process, SUAnalyzerSrv.exe, since Application Verifier requires elevated permissions to configure. Figure 3 shows a typical process tree when testing notepad.exe.

Figure 3 Testing Notepad.exe

SUAnalyzerSrv.exe and AppVerif.exe will always run with administrator privileges in order to prepare the environment for the target application to be tested. SUA will then launch notepad.exe under the specified security context and wait for the application and all child processes to finish executing. SUA then aggregates and processes the Application Verifier logs and displays the results in the UI.

The SUA results are broken down into categories of issues including File, Registry, INI, Token, Privilege, Name Space, Other Objects, and Process. These categories correspond to a tab at the top of the SUA UI and allow you to focus on different issues as you test the application.

The File and Registry tabs show you file system or registry operations that would require Administrator privileges. The luapriv Application Verifier plug-in monitors the behavior of relevant APIs such as CreateFile or CreateRegistryKey for cases where their use would require Administrator privileges. In some cases, the UAC File and Registry Redirection feature would allow these operations to succeed on Windows Vista.

Each tab shown in the UI generally results from luapriv monitoring the behavior of a set of APIs. For example, the INI tab focuses on finding issues related to the use of the WriteProfile APIs, originally used in 16-bit applications. A discussion of each tab is available in the SUAnalyzer.rtf documentation included as part of the SUA installation. You can expand the events under each tab to view detailed information about the event, including a stack trace for where in the application the event occurred.

After reviewing an application's test results, SUA allows you to test and create mitigation packages for an application. A mitigation package is an MSI that registers an application compatibility database on a system and can use loosen.exe, a utility that ships with SUA, to adjust file or registry ACLs. The mitigation packages can be deployed throughout your environment.

Internet Explorer Compatibility Test Tool

The Internet Explorer Compatibility Test Tool works by enabling the Internet Explorer Compatibility Evaluator (iece.exe). In ACT 5.0, you could deploy the evaluator as part of a data collection package, but since the collection required that you already have the target browser installed, this was removed for ACT 5.5. (After all, discovering that something doesn't work after it is installed is suboptimal; most people would prefer to discover any issues before they deploy the new browser!)

The Internet Explorer Compatibility Evaluator is fairly simple—the only work it has to do is to modify the registry so that Internet Explorer enables logging. Because the browser itself provides the detection, the fidelity of the data is extremely high. For details on all of the events that Internet Explorer can detect (along with remediation recommendations for each), see Internet Explorer Application Compatibility.

Internet Explorer logs information to the Event Log, and the test tool collects information from the registry and consolidates it. You can analyze the information directly in the tool, or you can upload events to a consolidated ACT database. If you choose to upload them to an ACT database, the Internet Explorer Compatibility Evaluator runs agents just like the agents run in a data collection package, first running collect.exe, then bucketizer.exe, and finally compressor.exe, after which it prompts you for a location to save the CAB file.

Conclusion

Well, that concludes our under-the-covers tour of the Application Compatibility Toolkit 5.5. The product provides a number of useful tools to support your migration to Windows Vista or Windows 7, and by understanding more about what each one does, you'll be better able to evaluate the potential of each tool and invest in them appropriately.

Chris Jackson is the technical lead for the Windows Application Experience SWAT team at Microsoft. He has worked with enterprise customers around the world to help them investigate and mitigate application compatibility issues, as well as providing instructional training about Windows application compatibility for numerous industry events. Chris can be reached at https://blogs.msdn.com/cjacks.

Chris Corio was a member of the Windows Security team at Microsoft for more than five years. His primary focus at Microsoft was application security technologies and management technologies for securing Windows. You can reach Chris at winsecurity@chriscorio.com.