Planning

Microsoft Deployment Test Feature Team Guide

Planning

Published: November 09, 2007

Figure 3 shows the primary activities that occur during the Planning Phase. These activities and deliverables are crucial to the success of the testing process, in addition to the validation of the Microsoft Deployment implementation in the test lab environment. The Test feature team plans its test activities and determines its strategies in part by examining the interaction among and operational effectiveness of the organization, its users, and internal IT processes in addition to the business goals defined for the Microsoft Deployment implementation.

Figure 3. Activities during the Planning Phase

Planning Checklist

The Test feature team must determine specific aspects of the test process. These include the test strategy, the required tests for their environment, creating a test plan, and preparing a test schedule to keep things in line with the overall timeline of the project. Table 2 shows details of the Planning Phase.

Table 2. Planning Phase Checklist

	High-level steps in the test Planning Phase
	Determine the test strategy. Create detailed, step-by-step directions that outline how to perform an activity for the preparation of each test type.
	Identify required test types. Determine which test types are required. The recommended test types are unit testing, functional testing, integration testing, stage testing, and pilot and production testing.
	Define the test plan. Determine which test types are required for the different stages. Determine the responsibilities of the Test feature team, the strategy for each test type, and prerequisites for each team member.
	Prepare the test schedule. This step includes the following tasks: Test environment setup Documentation review Preparation of high-level test scenarios Detailed test case preparation Test execution Number and duration of test cycles

The Test feature team works closely with the various feature teams to provide objective validation of the Microsoft Deployment implementation and its components. The primary deliverable for the Test feature team during the Planning Phase is the test plan, which interacts with and affects three key elements of the Microsoft Deployment implementation:

The test plan influences the overall Microsoft Deployment project plan.
The test schedule, which is included in the test plan, affects the Microsoft Deployment project schedule.
The test lab requirements, which are included in the test plan, factor into the hardware and facilities plan; these requirements also influence the Microsoft Deployment budget.

The various components that make up the test plan are not prepared in isolation. When developing the test plan, the Test feature team must collaborate with other Microsoft Deployment implementation teams, and use information gained from multiple sources, including the:

Documentation provided with Microsoft Deployment.
Development plan for the Microsoft Deployment project.
Functional specification for the Microsoft Deployment project.

The Test feature team focuses on testing how the process and technology components operate as a whole in the test lab environment. This approach considers the broad business context while testing the Microsoft Deployment implementation’s build process, technical components, features, and functionality before deployment into the production environment.

Top of Page

Test Plan Considerations

When preparing the test plan, Test feature team members must consider several aspects of the implementation process, including:

Which test types are required for the different stages of project implementation and, specifically, what are the responsibilities of the Test feature team in each test type.
The strategy used for testing each test type.
The various roles for the Test feature team members.
The required prerequisite activities for each team member.

Each aspect is covered in more detail in the following sections. Once again, this information is provided as a guide to be incorporated into the organization’s existing testing practices.

Testing Types

The preparation of complex technological solutions involves several types of tests, and each type has its own requirements and testing strategy. The deployment project becomes more refined as it progresses from test type to test type until it is has been thoroughly tested and is ready for introduction into the production environment.

The recommended test types for Microsoft Deployment include:

Unit testing. The first test type focuses on the analysis of a single deployment component. As they prepare to build the overall deployment project, team members from each feature team begin analyzing the components for which they are responsible. At this point, team members often install these components in isolated environments to validate the components’ capabilities. This test type is often performed by individuals on a single computer.
Functional testing. After the individual teams become more familiar with the technological components for which they are responsible, they move on to functional testing—validation that products and components work as designed. Guidance for this test type is derived from the functional specifications of the overall project.
Integration testing. The next test type integrates each of the various components that make the deployment project into one cohesive whole. This testing is performed in the integration or test lab by the Test feature team itself.
Staging testing. Staging testing is an optional test type that provides a final validation for the procedures used to implement the deployment project. Although there could be a margin of error in the preceding test types, there must be no errors here. When a deployment project is introduced into the production environment, it must be completely error-free to ensure its longevity and long-term supportability.
Pilot and production testing. The final stage of testing often involves the same components used in production. When the teams implement the deployment project in an actual production environment, they begin with a pilot deployment—targeting a small, representative population of users in the production environment to perform a final validation of all of the implementation strategies and procedures before deploying the system to the entire production environment. This test type focuses more on training, communications, and support strategies than on actual technological strategies, although these are also validated. Pilot testing is linked to production testing, because if all goes well, the technologies and components introduced for the pilot program become the components used in full production.

Figure 4 shows the different test types and illustrates the cyclical graduation process from test type to test type.

Figure 4. The five test types

Testing Strategy

The instruction sets—detailed step-by-step directions that outline how to perform an activity—for the preparation of each test type are derived from the information found in each feature team guide in Microsoft Deployment. Each feature team must document these instructions and update them each time issues or errors are corrected. For this process to succeed, the different teams must run test cycles. At various stages in a test cycle, different sets of tests are executed for each of the five test types identified above. This iterative approach to testing provides the means to refine all processes and procedures before introducing them into the production network.

Test Cycles

Testing with multiple test cycles ensures that issues found in test cycle N are resolved in regressive test cycle N + 1. This process ensures a high-quality deployment project and is the justification for the multiple test types that the Test feature team must use.

Each test type begins with the production of baseline images of the test computers. Instructions for creating these baseline computers are found in the various feature team guides in Microsoft Deployment. As tests are performed and processes graduate from one test type to the next, these baseline images are updated with the findings of the previous type. These updates mean that baseline images improve with each test type. At the end of the entire test cycle, the deployment project reaches a stable state.

Pass or Fail Criteria

Before the first test execution cycle, the following criteria must be defined to ensure defect prevention and bug resolution:

All test cases must pass with expected results as outlined in the Test Cases Workbook.
A test case is considered to have passed if the actual result matches the expected result documented for the test case. An actual result that does not match the expected result must be treated as a failed test case, and a bug must be created with an assigned severity score and priority.
If a test case fails, the deployment guidance is not necessarily defective. For example, misinterpretation of product documentation, incomplete documentation, or inaccurate documentation could cause failures. Each failure must be analyzed to discover its cause based on actual results, and the results must be described in project documentation, and be escalated to the correct feature team.

These criteria must be part of the testing plan. They can be supplemented with other criteria customized to meet the organization’s specific needs.

Identify Test Case Types

Finally, the test plan must include different types of test cases. Various tests are possible, but in the context of this deployment project, the following types are often the most meaningful:

Installation testing. These types of tests are used to verify that deployment project components are installed correctly. The deployment project components consist of all the code and tools provided with Microsoft Deployment, along with the code and tools already used within the organization. This test case also covers elements such as performance and bandwidth use during installation.
Configuration testing. These tests are used to verify that all deployment project configuration options are present and correctly execute when invoked. For instance, during the deployment of a computer, post-build processes must operate properly. These processes include the application of additional software components as well as the restoration of a user’s personal settings through the Windows User State Migration Tool (USMT). Because the deployment project consists of multiple configurations, ensure that each configuration works as expected.
Functionality testing. This testing provides basic verification of operating system and application functionality from the development and test perspective.
User acceptance testing. This testing ensures that the system as a whole performs as it should under typical user operation. User acceptance testing must also be performed on each application package that the Application Management feature team creates.
Security testing. This testing ensures that the security aspects of the deployment project are maintained throughout the testing cycle. Security tests include items such as the verification that passwords are not displayed in plain text in any component of the deployment project (or credentials that are displayed in plain text have single-purpose rights and permissions), logs do not capture credential information, and each team role in the deployment project, including users, has proper access rights to all the shared folders that make up the deployment project.

Prepare to Test the Microsoft Deployment Implementation

Team members are selected during the Envisioning Phase. During the Planning Phase, team members must begin preparing for Microsoft Deployment testing. This testing involves the review of three key documentation sets. The review of these documentation sets also assists in the preparation of the test plan. The documentation sets are:

The Microsoft Deployment documentation. Specifically, this feature team guide, the Deployment Customization Guide , Image Customization Guide , Image Customization Desktop Samples , and the Image Customization Server Samples. Team members can review the other feature team guides as well, but these guides are the most crucial.
The development plan for the Microsoft Deployment implementation. This plan must be reviewed in depth.
The functional specification that establishes a baseline for the Microsoft Deployment features and functionalities to be deployed into the production environment. The deployment project architect who belongs to the Development Role Cluster prepares this document in the Planning Phase. Members of other role clusters provide input in the preparation of this document.

The Test feature team relies on this documentation to:

Assess test requirements and define prerequisites for undertaking testing.
Estimate the time and resources required to test all the features being developed.
Provide feedback to developers about any specifications in the document that are unclear, ambiguous, or contradictory to prevent incorrect implementations resulting from misunderstanding.
Determine whether any specific deployment project features would be difficult or impossible to test.

In situations in which a particular feature cannot be tested, the Test feature team’s escalation process must include passing the issue to the core team. The core team then decides whether the feature must be redeveloped or retained and deployed with a release note informing IT Operations that the feature could not be tested. Untested elements must be kept to a strict minimum.

At this stage, functional specification review and sign-off by the Test feature team confirms the team’s ability to test the deployment project. This sign-off is also a Planning Phase deliverable for the core team. (See the Microsoft Deployment Planning Guide for more information about functional specifications.)

Top of Page

Prepare the Test Plan

The Test feature team is responsible for developing the test plan for the Microsoft Deployment project. This test plan consists of several components. A sample test plan template is provided in the Microsoft Deployment documentation set and can be found in C:\Program Files\Microsoft Deployment\Documentation\Job Aids\Test Plan.doc. The components most relevant to the Test feature team are found in the following sections.

Test Scope

The scope for the Test feature team’s activities is determined by the deployment project requirements and the functional specification. The test scope defines the range and type of testing activities used to validate Microsoft Deployment technology and processes. In the case of Microsoft Deployment, technology denotes the scripts and tools that the Development feature team created, while processes refers to the methodologies, enabling technologies, and practices that the other Microsoft Deployment teams use. The test scope must also contain the types included in the test plan.

Lab Requirements for Each Test Type

The testing goal is to obtain certification for a product to be deployed into the production environment. Certification involves a comprehensive verification of all of the components that make up the product—software packages and operating system images, in addition to all the components that support deployment of the project—according to predefined test parameters. When the test lab mirrors the production environment, Microsoft Deployment systems and applications can be certified with confidence, because the lab test result represents what is to be expected in the production environment.

The Test feature team is responsible for designing and ensuring that the Microsoft Deployment test lab accurately represents the production environment. The development plan and the Microsoft Deployment Infrastructure Remediation Feature Team Guide assist the team in determining the hardware, software, infrastructure, and facilities required in the test lab environment. Test lab requirements vary according to the types of tests performed.

For example, if the team is conducting feature testing only, it might consider limiting the amount and type of hardware requested for the lab. If, however, the team also plans to conduct stress testing, the lab’s hardware requirements might significantly increase. VMs are especially beneficial when testing, and they can be used in most test types. For example, unit testing is often performed on the user’s own computer using one or more VMs that provide all the functionality required for the test. Functional testing can use the same mechanism or use a dedicated piece of hardware running multiple VMs. Similarly, the integration and staging test types can also rely on VMs for most of their tests. VMs can simulate servers, but deployed images must include at least one physical example of each computer type targeted for the production deployment. This combination ensures that deployed images include all the correct drivers. For more information about using running VMs using Virtual Server 2005, see Microsoft Virtual Server 2005 R2 at http://www.microsoft.com/windowsserversystem/virtualserver.

Bug Rating, Reporting, and Tracking

The Test feature team develops the method for reporting and tracking bugs. The mechanism for bug reporting and tracking must include features that allow the Test feature team to assign bugs to the right person or team, prioritize bugs, assign severity numbers to them, reopen closed bugs, link bugs, generate different views of the same bug, and create reports. The Test feature team is also responsible for developing a process for bug triage and creating a schedule to track the status of all test bugs. For more information about bug tracking, see the section, “Appendix A: Finalize the Test Lab,” later in this guide.

Change Control

Change control ensures that the core team is aware of and agrees to any changes to lab hardware or software. The Test feature team follows the change control process established by the lead team and that is used throughout the Microsoft Deployment project life cycle. The test lead must post the status of hardware and software in the lab, in addition to the testing schedules, so that the various feature teams are aware of all lab activity. The test lead must also have procedures in place to restore the lab to its original state at the completion of the project.

Test Schedules

The test schedule is affected to a large extent by the Microsoft Deployment development schedule. The Test feature team might carry out unit tests of deployment project modules as the teams release them, or it might conduct only complete system tests after the Image Engineering feature team has completed the entire build process document and has created complete deployment images.

Based on its experience, the Microsoft Deployment Test feature team at Microsoft recommends conducting unofficial unit testing while the build process document is in development. This approach gives the Test feature team enough time to create relevant test cases and to become familiar with the deployment project during the early stages of the Stabilizing Phase. In addition, these unit tests can be used for base functionality testing.

The test schedule must include, at minimum, the following tasks:

Test environment setup
Documentation review
Preparation of high-level test scenarios
Detailed test case preparation
Test execution
The number and duration of testing cycles

Risks and Dependencies

The Test feature team typically looks for and assesses the following types of risks, and then factors them into the test plan and test schedule:

Lab requirements, based on the development plan, might exceed the Test feature team’s budget and time allocations.
Test cases can be completed in parallel with input from the Development feature team and functional specifications.
The test lab build might not have been completed by the start of the Stabilizing Phase because of an inability to procure and install all of the lab equipment.
The test lab does not properly reflect the production environment. For example, it might not include proper firewall configurations or even all of the Group Policy objects (GPOs) found in production. The goal is to make the test lab as similar to production as possible.

Mitigation plans for each of these risks must be part of the test plan. The risks detailed in the previous paragraph are examples of the most common risks, but, depending on the organization, others could exist and might require contingency plans.

Testing Tools

The test plan must include a description of the tools testers will use. This tools list must include the various test scenarios, the test types, and the test cases to be used. The list must reference supporting tools, such as the bug tracking system, the change control system, and any documentation system to be used, in addition to the scheduling tools the team lead will use to control testing schedules.

Top of Page

Milestone: Test Plan Developed and Accepted

Milestones are synchronization points for the overall deployment project. For more information, see the Microsoft Deployment Planning Guide.

At this milestone, shown in Table 3, a test plan is in place that includes a detailed understanding of network bandwidth and server capacity requirements.

Table 3. Planning Phase Test Plan Milestone and Deliverable