Using the Windows Applications Exploratory Test Methodology

  • Article

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

The Windows Applications Exploratory Test Methodology defines a process for learning about an application so that you can identify the most important aspects to test when no one who is knowledgeable about the application is available for testing.

When you use the exploratory test methodology, you do not define the test cases in advance. Rather, you define the test cases and execute the tests as you explore and learn about the application.

Exploratory Testing Overview

The purpose of exploratory testing is to determine, in a limited amount of time, whether an application works well on the new version of Windows for the majority of users. Exploratory testing addresses both application functionality and stability. This method is not a comprehensive test method, and it is not a substitute for a more formal method if application experts are available.

Exploratory testing is an interactive process during which you concurrently explore the application, design the tests, and execute the tests. Although the method is freeform in many respects, the process includes specific tasks, objectives, and deliverables, making it a systematic process with auditable results.

This method is based on the General Functionality and Stability Test Procedure developed by James Bach. For more information about the exploratory test methodology, see the "Windows Applications Exploratory Test Procedure" white paper in the Application Compatibility Toolkit.

Exploratory Testing Process

The exploratory testing process consists of five tasks:

  1. Identify the purpose of the application.

  2. Identify the functions that the application supports.

  3. Identify potential areas of application instability.

  4. Document issues and questions that arise about the application during the exploration process.

  5. Create a test outline, and use it to execute the tests.

As you identify the purpose, functions, and areas of potential instability, you create a test case outline and use it to test each function and to record problems. The results of the exploratory testing process are notes and documented application failures.

Because it is not possible to test all functions of an application in a limited amount of time, you can simplify the testing process by making risk-based decisions about how much attention to give each function. To help you make these decisions, the methodology categorizes functions as primary or contributing.

To determine which functions are primary and which are contributing, you must understand the purpose of the application. A primary function is associated with the purpose of the application and is essential to that purpose. Any function that contributes to the utility of the application, but is not essential to it, is a contributing function. Typically, primary functions warrant the most testing. Sometimes a group of contributing functions might be considered a single primary function, or a single primary function might be separated into primary and contributing subfunctions. For more information about how to classify a function, see the "Windows Applications Exploratory Test Procedure" white paper in the Application Compatibility Toolkit.

After identifying and categorizing the functions, test all of the primary functions that you can reasonably test in the time available. Then test a sample of significant contributing functions. Many contributing functions probably will be tested as you test the primary functions.

Also test areas where application instability is most likely to occur, especially in primary functions. Functions that are potentially unstable include:

  • Functions that interoperate with other applications.

  • Functions that handle events that are external to the application, for example, waking up a sleeping computer when a fax arrives.

  • Functions that make intensive use of memory.

  • Functions that interact extensively with the operating system.

  • Functions of unusual complexity.

  • Functions that change operating parameters, such as preference settings.

  • Functions that modify the operating system configuration.

  • Functions that intercept or recover from errors.

  • Functions that replace basic operating system functions.

  • Any function or set of functions that involves multiple simultaneous processes.

  • Functions that manipulate multiple files simultaneously.

  • Functions that open files over a network.