Testing Web Application and Add-on Compatibility
Updated: August 25, 2010
Applies To: Windows 7
When faced with a large catalog of software, it is tempting to look for ways to reduce the amount of resources required to test them. One possibility is using automated tools, but no tool has yet emerged (either from Microsoft or a third party) which is able to provide automated testing that detects an adequate number of issues to be predictive, yet is able to effectively filter them to only those issues which are likely to cause an application to fail. (Most err on the side of generating too much information, making it difficult to focus on the issues that would disrupt business.)
Because there is no single tool to discover or test for compatibility problems of web applications, organizations might run into errors in reviewing of applications. The flowchart below illustrates the recommended methodology for testing large-scale web applications in an organization.
For example, in some applications, changes in the rendering model may offset a given object by a few pixels. This example illustrates an important concern for an organization’s testing effort: defining criteria for what degradation in functionality must be fixed. In some businesses, a change such as the one just described has no impact on productivity whatsoever, and may even go unnoticed. If testing and remediating of applications is outsourced, the team should work with the vendor to ensure that issues such as these are analyzed to determine whether a fix is necessary. In other cases, a minor change of a few pixels may completely diminish the usefulness of the application and must be fixed. Managing when to remediate an application can drastically change the resources required for fixing your applications portfolio.
Another strategy to reduce resource consumption is to put a limit on the set of applications that will be tested. For example, an organization may decide to manually test only 10 percent of its application catalog. These applications have the highest priority and overall business impact. The test team will perform scenario-based “smoke” testing against business objectives to ensure that the web application still helps achieve business goals successfully. This testing is done by those who sign off on whether or not an application is compatible with the organization’s internal systems. By testing the applications before deploying them to end users, companies enable employees be more productive during the rollout, because they can focus on doing their work rather than having them test the compatibility of business applications.
After the smoke testing phase is complete, organizations can feel confident to have employees put the application and browser compatibility to the final test. To make this process as simple as possible, IT departments can use a lab environment, remote desktop technology, or other virtualization technologies to create a testing environment for users.
For applications that failed the smoke test, IT pros should work with application owners to perform additional formal testing to capture and classify issues found, and apply appropriate remediation.
Case Study: How the IT Team at Microsoft Tests Compatibility with Internet Explorer Releases
During the release cycle for Internet Explorer 8, Microsoft began web application testing to be ready for an Internet Explorer 8 deployment when the final version was released. While the pre-release/beta versions were not production ready, many of the design changes were present in early application builds and could help highlight areas where web applications would need to be fixed to support the new version. Given the size and number of systems in use at Microsoft, it was prohibitively expensive to test everything; it would have cost more to manage the risk than simply to accept it.
The Microsoft Information Technology team tests approximately 5 percent of websites and web applications before rolling out a new browser. They test using the “sampling” methodology to determine if any existing websites or web applications are incompatible prior to deployment. For more information, see Inventory and Prioritization of Critical Web Sites and Add-ons earlier in this document.
To read more about internal compatibility testing procedures, see:
Application Compatibility Testing Using Virtual PC Images
Microsoft recognizes that many organizations need to establish some kind of testing environment in order to get started with web application compatibility testing. The process to build and configure the computers needed for this task is time consuming and often requires additional expenditures. To facilitate web application compatibility testing, Microsoft created a set of pre-built application compatibility Virtual PC (VPC) images available for free downloads. VPC images are pre-configured virtual hard drive images designed to be loaded onto existing computers running the Virtual PC software. These images allow IT professionals to thoroughly test various versions of Windows and Internet Explorer with existing applications in a short amount of time. The images are available for all currently supported versions of Windows, and contain many compatibility testing resources and applications.
For more information, see Virtual PC software.
For testing images, see Internet Explorer Application Compatibility VPC Image web page.