Site Server - Guidelines for Testing Your Site Server - Commerce Edition Web Site
August 1999
Introduction
Testing is a key part of the software development process, and is iterative. This document describes:
Types of tests
Test documents
Test error ("bug") reports
The appendix contains a sample outline for each type of testing document.
Types of Tests
There are five levels of testing:
Test Level |
Focuses on… |
|---|---|
Build Verification Testing (BVT) |
Testing the stability of the software build. |
Functionality |
Testing program functions—a program can have several levels of functionality, depending on its complexity. |
Ad hoc ("guerilla") |
Unstructured, random testing by casual users. |
Integration |
Testing the interaction between modules or program components—testing ways in which the program works with other products or platforms (both hardware and software). |
Stress (also known as Load or Performance tests) |
Testing the number of transactions or level of use the program can consistently sustain. |
The types of testing to be done can vary from program to program, but should always include BVT, functionality, and ad hoc testing. There are two types of test methods that can be applied to each type of test:
Method |
Focuses on… |
|---|---|
Black box |
Testing the standard user interface (such as black-box functionality testing). Test cases use the same standard interfaces that customers are expected to use. |
White box |
Testing application code (such as white-box stress testing). Test cases use batch files, SQL queries, and similar methods to interact directly with program code. |
Test Documents
There are three types of documents to create before testing begins:
Test plan
Test specification
Test cases
The test plan must be in place before test specifications can be written; test specifications must be written before test cases can be created; test cases must be created before testing can begin.
The Test Plan
Test plans should describe in detail everything that the test team, the program management team, and the development team need to know about the testing to be done. A test plan should contain the following information:
Content |
Describes…. |
|---|---|
What is to be tested |
The scope of the testing—the functionality that is to be tested; test criteria, such as the following: |
When it is going to be tested (schedule) |
Beginning date, end date, and dates for all significant milestones. |
Where it is to be tested |
What computers, networks, and other hardware are required to perform the tests; where they will be located. |
How it is to be tested |
What types of testing will be done: |
Who is going to do the testing |
The names of the test team and each tester's responsibilities |
Assumptions and risks |
List of conditions that must be met for testing to be successful—probable outcome if the necessary conditions aren't met |
Test planning can be done for the entire project, or on a function-by-function basis for each of the program sub-functions.
The Test Specification
The test specification conveys the entire scope of testing required for a set of functionality and defines individual test cases sufficiently that testers can use it as a basis for creating the test cases. Format of the test specification can vary, depending on project context. Some common formats include the following:
Format |
Defines… |
|---|---|
Matrix |
Program functions and function variables |
Written description or scenario |
Process |
Screen print or format |
Fields |
Database schema |
Fields, tables, stored procedures |
Graphic |
GUI form or window |
Outline |
The functions to be tested |
A test specification can combine any of these formats, as needed.
Test Cases
There are four basic types of test cases:
Type |
Focuses on… |
|---|---|
Functionality |
Testing what the software is supposed to do |
Boundary |
Testing the defined values and defaults |
Positive (or Valid) |
Proving that the software does what it should do |
Negative (or Invalid) |
Proving that the software doesn't do what it shouldn't do |
Test cases can be organized in a variety of ways, depending on the size, type, and complexity of the software being tested. For example, tests can be organized by type of test, by product feature, or by a combination of the two.
Test cases organized by test type might be organized as follows:
BVT tests
Functional tests
Integration tests
Load tests
Stress tests
Test cases organized by product feature might be organized as follows:
Primary-level features or functions
Secondary-level features or functions
Tertiary-level features or functions, and so forth
Depending on complexity of the software, you might organize test cases using a combination of methods, such as the following:
Test Error Reports
An error, or "bug," occurs when the software does something it shouldn't, or won't do something it should. A Test Error Report should contain the following information:
Section |
Describes… |
|---|---|
Description |
What went wrong. This section should be descriptive enough that the problem and its location are obvious. |
Reproduction steps |
The detailed procedure for reproducing the bug. This might include platform or installation details. |
Tracking data |
Who found the error, who's fixing it, and other pertinent data. |
Appendix
This section contains a sample outline for each type of testing document.
Sample Outline for a Test Plan
Test plans should contain the following information:
Section |
Contains… |
|---|---|
Introduction/Purpose |
Description of program, features. |
Risks and assumptions |
List of conditions that must be met for testing to be successful; probable outcome if the necessary conditions aren't met. |
Scope |
Test methodology, approach; detailed description of what is in or out of scope. |
Feature/Program-specific function/Area |
Testing type, deliverables. |
Test environment |
Platform (operating system); platform dependencies. |
Test criteria |
Acceptance, pass, suspension of test criteria. |
Bug tracking |
Location of bug database, key bug data format. |
Test case management |
Location of test cases, test case management structure. |
Test team |
Members of test team, escalation path. |
Test platform (hardware) |
Office-based test hardware, lab-based test hardware. |
Test schedule |
Beginning and end dates, key milestones. |
Appendix |
Most test plans should contain the following appendices: |
Test Specification
The format and content of test specifications vary, depending on the project. However, test specifications should always define the scope of the entire testing effort and provide sufficient detail to enable testers to construct test cases.
Sample Outline for Test Cases
Test cases should contain the following information:
Section |
Contains… |
|---|---|
Title |
Name of the test—a one-line description of what is being tested (ideally implying the expected result). |
Steps |
How to do the test—a list of the discrete tasks required to complete the test case. Form names, menu items, field names, and input data should all be specifically identified in each step. |
Expected results |
Description of what should happen (system behavior and/or output) after each step of the test. |
Assumptions |
Notes about specialized platform requirements, key functionality, steps needed to set up the test case. |