Microsoft Commerce Server 2000: Site Testing

Commerce Server 2000

This chapter provides a general methodology for testing software, and then provides more specific information about how to test your Microsoft Commerce Server 2000 site. Testing is essential for any Commerce Server implementation to ensure software reliability and system performance and capacity.

Testing Methodology

Cc936694.spacer(en-US,CS.10).gif Cc936694.spacer(en-US,CS.10).gif

Testing is a key part of the software development process. This section describes the types of testing you should do and the documents you must create to define your testing process.

Types of Tests

The following table lists the five types of tests you should use in your testing process.

Test type


Build Verification Testing (BVT)

Stability of the software build.


Program functions. A program can have several levels of functionality, depending on its complexity.

Ad hoc ("guerilla")

Various site components. This is unstructured, random testing by casual users.


Interaction between modules or program components and the ways in which the program works with other products or platforms (both hardware and software).

Stress (also known as load or performance testing)

Number of transactions or levels of use that an application can consistently sustain.

Tests can be manual, scripted, or automated. The types of testing you do can vary, but should always include BVT, functionality, and ad hoc tests. Performance testing is especially important for an e-commerce application. The following table lists the two types of test methods that you can apply to each type of test.



Black box

Standard user interface. Test cases should be constructed so that they test the same standard interfaces that customers are expected to use.

White box

Application code. Test cases use batch files, SQL queries, and similar methods to interact directly with program code.

Types of Test Documents

There are three types of documents to create before you begin testing your site:

  • Test plan: Strategy and schedule for testing your Commerce Server site.

  • Test cases: Detailed, step-by-step descriptions of each test.

  • Test error reports: Reports that describe errors encountered during testing.

Test Plan

Developing a good test plan is key to successful testing. In fact, it can help define the application. To create an effective test plan, you must first understand your company's strategic vision for the site, and then define your test strategy accordingly.

A test plan includes the strategy and scheduling information for the entire project—everything except descriptions of the test cases. The test plan lists the features that need testing, the required resources, the risks you need to consider, and the testing schedule. The results of your test-planning process should be a stable specification, an accurate schedule, and a test plan containing an effective test strategy.

The test plan should reiterate the goals of the site and divide the site into testing areas. In addition, the plan should include inter-group procedure information, such as how bugs will be tracked and resolved, how test releases will be handled, how shared components will be tested, and criteria for determining whether a build is accepted or rejected for testing. Finally, the test plan must identify which testing tasks will be automated and which testing tasks will be done manually.

Test plans should describe in detail everything that the test team, the program management team, and the development team need to know about the testing to be done.

The following table lists the sections of a test plan.




The site and its features, followed by the test methodology and approach.


What will be tested and what won't be tested.
The types of tests to be conducted, such as:
Black box and/or white box
Manual, scripted, and/or automated
BVT, functionality, integration, stress, and so on
Deliverables, including a list of reports to describe test results and bugs to be tracked.


Functionality to be tested, on a feature-by-feature basis.

Test environment

Platform (operating system) and platform dependencies.

Test criteria

Acceptance criteria, pass criteria, and suspension of test criteria.

Bug tracking

Location of the bug database and the bug data format.

Test case management

How to manage test cases and where they will be located.

Test team

Members of the test team and each tester's responsibilities; escalation path for resolving issues.

Test platform (hardware)

The computers, networks, and other hardware required to perform the tests, both for office-based test hardware and lab-based test hardware. If you are setting up a test lab, it should be described in this section.

Test schedule

The beginning and ending dates for the project, the release date for the site, and the key dates for all milestones.

Assumptions and risks

Conditions that must be met for testing to be successful and the probable outcome if the necessary conditions aren't met.


The following information (as appropriate):
High-level test specification matrix
Test error report template
Sign-off page

Test Cases

A test case is a detailed, step-by-step description of a test. The following table lists four basic types of test cases.

Type of test case



What the site is supposed to do


Defined values and defaults

Positive (or Valid)

That the site does what it should do

Negative (or Invalid)

That the site doesn't do what it shouldn't do

You can organize test cases in a variety of ways, depending on the size, type, and complexity of the site you are testing. For example, you can organize test cases by type of test, by site feature, or by a combination of the two.

You can organize test cases by test type, such as the following:

  • BVT tests

  • Functional tests

  • Integration tests

  • Stress tests

Alternatively, you can organize test cases by product feature, such as the following:

  • Primary-level features or functions

  • Secondary-level features or functions

  • Tertiary-level features or functions

Depending on the complexity of the site, you might organize test cases using a combination of methods, such as the following:

  • Primary level:

    • BVT tests

    • Functional tests

    • Integration tests

  • Secondary level:

    • BVT tests

    • Functional tests

    • Integration tests

  • Tertiary level:

    • BVT tests

    • Functional tests

    • Integration tests

    • Stress tests

The following table lists the sections of a test case.




The test. The title should be a one-line description of what is being tested (ideally implying the expected result).


How to do the test, including a list of the discrete tasks required to complete the test. You should specifically identify the form names, menu items, field names, and input data in each step.

Expected results

What should happen (system behavior and/or output) after each step of the test.


Any special platform requirements, key functionality, and the steps needed to set up the test case.

Test Error Reports

A test error report is a report that describes errors encountered during testing. An error, or bug, occurs when the site doesn't perform the way it was designed to perform. For example, if you attempt to add an item to your shopping basket, but the application doesn't add the item, an error has occurred. The following table lists the information that a test error report should contain.




The operation that did not work correctly. This section should describe in detail what the error is and where it is located.

Reproduction steps

Detailed steps for reproducing the error. This might include platform or installation details, if necessary.

Tracking data

The person who found the error, the person who is responsible for fixing it, and any other pertinent data.

Testing a Commerce Server Site

Cc936694.spacer(en-US,CS.10).gif Cc936694.spacer(en-US,CS.10).gif

Before you begin testing, you must set up your test environment.

Important Create a separate test environment in which to do your testing. Do not install new content or software in your production environment before it has been thoroughly tested.

For best results, you should create a test environment that exactly duplicates your production environment. Testing each tier of the architecture separately, as shown in the following list, also helps to reduce the complexity of the testing task. You must also test the security of each tier. The following is a sample tiered architecture:

  • User interface layer (top tier)

    • Visual appearance and usability

    • Links

    • Browser compatibility

  • Business logic layer (middle tier)

    • Software performance testing (business logic, tax, shipping calculations)

    • Server load

    • Stress

    • Gateways

  • Database layer (bottom tier)

    • Search results

    • Query response time

    • Data integrity

    • Data validity

    • Recovery

User Interface Layer (Top Tier)

To be sure that your site is attractive and user-friendly, you should test the following elements in the user interface layer (top tier):

  • Visual appearance and usability

  • Links

  • Browser compatibility

Visual Appearance and Usability

The visual appearance and usability of a Web site, including its catalog pages, is important because a visually pleasing and easy-to-use site enhances the customer experience and encourages repeat visits. The following table lists the elements that you should test in the user interface layer.


Test for

Text font style

Compatibility with different browsers. Many available fonts don't display well on all browsers, especially older browser versions. In some cases, fonts can even be displayed as unreadable characters.

Font size consistency

Consistency of font size throughout the Web site. A body text font size of 10 to 14 points, and a heading font size of 18 to 24 points are generally acceptable sizes.

Use of colors

User-friendly combinations of foreground and background colors throughout the site. Avoid combinations that make the site difficult to use. For example, it is difficult to read yellow text on a white background. You also should not use red and green in close proximity, because many people are red/green colorblind.


Speed of download. Balance the use of images to get a reasonable download speed. The fewer images you use, the faster your site can download. To increase download speed, for example, consider replacing full-size images with thumbnails.

Spelling and grammar

Correct spelling and grammar. Use the spelling checker to check the spelling throughout the site, and then proofread the site for errors the spelling checker didn't catch, such as the different uses for "there" and "their."
Proofread the entire site for correct grammar.
Note Be sure to give your home page special attention, because it is the first page that a visitor to your site sees.

Reliable and consistent information

Information accuracy. Verify all facts and figures that relate to products and services with the legal, marketing, and business teams.

The following list describes some other tests you should perform. Test to be sure that:

  • Each toolbar and menu item can be used to navigate correctly using either the mouse or the keyboard.

  • It's possible to navigate correctly through all pages using either the mouse or the keyboard.

  • You have used proper format masks. For example, all drop-down lists should be sorted alphabetically. The date entry should also be properly formatted.

  • Colors, fonts, and font widths are standard for the field prompts and displayed text.

  • Vertical scroll bars or horizontal scroll bars do not appear unless needed.

  • The window can be resized.

  • All of the text displayed in each window is spelled correctly, including the window caption, status bar options, field prompts, pop-up text, and error messages.

  • All character or alphanumeric fields are left-aligned (depending on the language).

  • All numeric fields are right-aligned.

  • Defaults are displayed, if there are any.

  • All windows have the same look and feel.

  • The tab order is from top left to bottom right, avoiding read-only or unavailable fields in the tab sequence.

  • The cursor is positioned on the first input field when a window is opened.

  • Each control behaves correctly, including push buttons, radio buttons, list boxes, and so on.

  • The command buttons appear dimmed when not in use.


Hyperlinks in Web sites can be broken, missing, or improperly assigned, making it impossible for a site visitor to navigate to the appropriate Web page. Make sure that all links work properly. You can use third-party software to search for broken and incorrect links; however, you must search for missing links manually.

The following table describes the types of links you should test.

Type of link

Test for

Broken links

A break in a link between the current page and a linked page or graphic. For example, if a developer changes the name of a products page from "Product.htm" to "Products.htm," the link between the home page and the Products page will no longer work. When users click this type of link, they go to an error page instead of to the correct page.

Missing links

Links that have not yet been created. For example, a developer might forget to create a link between the Products button on the home page and the Products page. If this happens, site visitors can't use the Products button to go to the Products page from the home page.

Incorrect links

Links that take a site visitor to the wrong page.

The following Web sites offer online testing for broken and incorrect links at a nominal fee:

Browser Compatibility

Text, graphics, or colors can appear differently, depending on the browser used to look at them. To ensure that your site appears the way you want it to look on all major browsers, you should use development software that is compatible with most of the popular browsers, such as Microsoft Internet Explorer, Netscape Navigator, America Online (AOL), and Microsoft WebTV.

The following table compares the compatibility of browsers with common Web site components.

Web site components

Internet Explorer 4.0 and later

Internet Explorer 3.0

Netscape Navigator4.0 and later

Microsoft ActiveX controls




Microsoft Visual Basic Scripting Edition (VBScript)








Java applets




Dynamic HTML








Business Logic Layer (Middle Tier)

Middle-tier testing primarily tests software performance and load. You test software performance to ensure that the software performs in accordance with operational specifications for response time, processing costs, storage use, and printed output, regardless of the volume of traffic on your site.

If your servers are overloaded during peak traffic volumes, you might have to invest in additional Web servers to prevent downtime and to enable you to offload traffic from an overloaded server during peak times. The ability of a Web server to handle a heavy load at peak hours depends on network speed and the server's processing power, memory, and storage space.

You should gather data on software performance during:

  • Current and expected normal transaction volumes

  • Current and expected peak transaction volumes

  • Minimal transaction volumes

You should test all components fully, including facilities and equipment, and make sure that your local area networks (LANs) and wide area networks (WANs) are performing satisfactorily.

The following table lists the elements you should test in the middle tier.


Test that

Server load

Servers continue to function during heavy traffic volume. Identify the optimum number of simultaneous users that the server can handle before performance begins to degrade (capacity). Then, use the results of load tests to decide whether or not to add additional servers to manage peak traffic loads.
Most load-testing software simulates multiple logons, after which it calculates the optimum load factor for the server. Software is then configured using this test data so that the server will stop accepting further requests from online users if the traffic increases beyond its load capacity.
The Microsoft Web Application Stress (WAS) tool simulates multiple browsers requesting pages from a Web site. This tool can realistically simulate many requests with relatively few client computers. For more information about Web server load testing, see .


The server can process the expected load. For example, a bank transaction processing system might be designed to process up to 100 transactions per second and an operating system might be designed to handle up to 200 separate terminals.
Stress tests steadily increase the load on the system beyond the maximum design load until the system fails. This type of testing has the following functions:
To test the failure behavior of the system so that if system load should exceed the maximum-anticipated load, you can see whether you lose data or services.
To discover system defects that would not be detectable under normal circumstances.

Payment gateways

Payment gateways are reliable and compatible. The payment gateway is software that facilitates payment transactions by taking credit card details from customers, and then validating them with a transaction clearinghouse.

Database Layer (Bottom Tier)

The database for a Commerce Server site typically contains data for catalogs, shopping baskets, user profiles, and orders. For this reason, it's extremely important to test the database layer to ensure the integrity of your data. Database testing is an ongoing process because no database is static. When you create a database, you should also create a copy and store it either on the same computer or on another computer. Run your tests on the copy of the database, not the original. When your tests are successful, you can install the necessary changes in the original database.

The database does not have to be, and usually isn't, located on the same server that hosts your Web site's storefront. The database server should be separated from the Web server by a firewall, which adds complexity to the testing processes.

The following table lists the elements that you should test in the database layer.


Test that

Search results

Search results are relevant, provide direct links to the requested pages, and do not result in "wild goose chases." The Search option is one of the most frequently used functions of online databases.
Testers can assume the role of the online user and try out random Search options with different keywords. Search results should be recorded by the percentage of relevance to the keyword.

Query response time

The turnaround time for responding to database queries is within specifications. These test results can help identify problems such as bottlenecks in the network, problems with specific types of queries, problems with database structure, or problems with the hardware.

Data integrity

Important data, including catalog, pricing, shipping table, tax table, order, and customer data are correct. Testing should be repeated on a regular basis, because data changes over time.

Data validity

Data is valid. Errors caused by incorrect data entry are probably the most common data-related errors, and also the most difficult to detect. For example, $67 can be entered mistakenly as $76.
To find this type of error, you can sometimes use queries to validate data fields. For example, you can write a query that compares the sum of the numbers in the database data field to the sum of the numbers from the data source. A difference between the sums indicates an error in at least one data element.
Another way to reduce data validity errors is to use data validation rules in data fields. For example, if the date field uses the MM/DD/YYYY format, you can incorporate a data validation rule, such that MM does not exceed 12, DD does not exceed 31, and so on.


The system can recover from faults and resume processing within a predefined period of time.
The system is fault-tolerant; processing faults don't halt system functioning.
Data recovery and restart are correct (in case of auto-recovery). If recovery requires manual intervention, then the mean time to repair the database should be within predefined, acceptable limits.
Recovery testing forces the system to fail in a variety of ways to ensure that you can recover from system failures.

The following list describes some tests you should do to ensure data integrity. Make sure that:

  • Data creation, modification, and deletion in data tables work as specified.

  • Sets of radio buttons represent a fixed set of values. Test what happens when a blank value is retrieved from the database.

  • Each value is saved fully when a particular set of data is saved to the database. Strings should not be truncated and numbers should not be rounded off.

  • Default values are saved in the database if a user doesn't enter other values.

  • New data is compatible with existing data, hardware, versions of the operating system, and interfaces with other software.


You must test security on all parts of your site because ensuring the security of transactions over the Internet is important to gaining customer confidence. Gaining and keeping the confidence of online customers is extremely important to the success of your site.

The main technique for testing security is to attempt to violate built-in security controls. You must make sure that system protection mechanisms can prevent unauthorized access. You should then attempt to overwhelm the system by continuous requests, thereby denying service to others. You might purposely cause system errors during recovery or browse through unsecured data to find any possible keys to system entry.

The following table lists types of security breaches that you should test for.

Security breach



There are two types of secrecy breaches: the ability to read a file to which you have not been granted access, and the ability to read data while in transit between two computers.
Secrecy breaches differ from authentication and spoofing security breaches in that the perpetrators access the information directly rather than by having to imitate a legitimate user.
Spoofing a user's identity means breaching the user's authentication information. In this case, a hacker obtains a user's personal information or the necessary information to replay the authentication procedure.


Sending a network ID and password copied from an authorized user. An authentication breach is often caused by spoofing.


An unknown user, who cannot be traced, performing an illegal operation.

Integrity control (data tampering)

Modifying system or user data with or without detection, such as an unauthorized change to stored or in-transit data
Formatting a hard disk
Introducing an undetectable network packet in a communication
Making an undetectable change to a sensitive file

For more information about best practices for setting up security on your site, see Designing Secure Web-Based Applications for Microsoft Windows 2000 by Michael Howard (published by Microsoft Press). There are two primary areas of concern in e-commerce security: network security and payment transaction security.


Unauthorized users, or hackers, can wreak havoc on your Web site by accessing confidential information or by damaging data. To prevent unauthorized access, you can configure the network operating system and the firewall to manage network security.

You must configure the network operating system to allow only authentic users to access the network. You must also install and configure firewalls to ensure that data transfer is restricted to only one point on the network. This effectively prevents hackers from accessing data.

For example, if hackers access an unsecured FTP port (such as Port 25) on a Web server, they could use it as an entry point to the network and access data on the server. The hackers might also be able to access any computer connected to this server. You should design your security tests to reveal vulnerable areas and highlight the network settings that need to be reconfigured to enhance security.

You can use third-party programs to test for the following:

  • User rights

  • Removable disk locations

  • Strong password policies

  • Use of logon scripts and password expiration dates

  • Passwords stored in clear text or encrypted form

For more information about testing network security, see .

Payment Transactions

When customers purchase goods and services over the Internet, they can be apprehensive about providing their credit card information. To make your site secure for your customers, you must ensure that the following two conditions are met:

  • Credit card information is transmitted and stored securely.

  • Strong encryption software is used to store credit card information, and only limited, authorized access to this information is allowed .

For more information about secure electronic transactions, see the following third-party sites: