Chapter 5 - Stabilizing Phase

Article
12/05/2007

Goals for the Stabilizing Phase

Figure 5.1: The Stabilizing Phase in the MSF Process Model

The primary goal of the Stabilizing Phase is to improve solution quality to meet the acceptance criteria for release to production. The team conducts testing on a UNIX migration solution whose features are complete. During this phase, the team completes tasks and creates deliverables that transition the feature-complete build into a state where the defined quality level is reached and the solution is ready for full production deployment. Testing during this phase extends testing that was conducted during development by emphasizing usage and operation under realistic environment conditions. The team focuses on resolving and triaging (prioritizing) bugs and preparing the solution for release.

During the early stages of this phase it is common for testing to report bugs at a rate faster than developers can fix them. There is no way to tell how many bugs there will be or how long it will take to fix them. However, two statistical signposts known as bug convergence and zero bug bounce can help the team estimate when the solution will reach stability.

Note: The terms alpha and beta are widely used to describe the state of IT projects. These terms can lead to confusion because they are interpreted in many different ways. If you use them, be sure to define them clearly and make sure that the team, customer, and stakeholders all understand the given definitions.

Once a build has been deemed stable enough to be a release candidate, the solution is deployed to a pilot group. This phase culminates in the Release Readiness Approved Milestone, indicating team and customer agreement that all outstanding issues have been addressed.

The tasks summarized in Table 5.1 should be completed during the Stabilizing Phase. This project guide describes the processes and roles required to accomplish them. Detailed information specific to each migration project about each task, especially technical information, is provided in the migration guide for that project.

Table 5.1 Major Stabilizing Phase Tasks and Owners

Major Tasks	Owners
Testing the solution The team executes the test plans created during the Planning Phase, which were enhanced and tested during the Developing Phase.	Test
Resolving defects The team corrects defects identified through testing and from other sources. New tests are developed to reproduce issues reported from other sources and are integrated into the test suite.	Development, Test
Conducting the pilot The team moves a solution pilot from development to a staging area in order to test the solution with actual users and real scenarios. The pilot is conducted prior to starting the Deploying Phase.	Release Management
Closing the Stabilizing Phase The team documents the results of completing the tasks performed in this phase and seeks management approval at the Release Readiness Approved Milestone meeting.	Project team

Team Focus during the Stabilizing Phase

Table 5.2 addresses the tasks described previously, but considers them from the perspective of the team roles.

The primary team roles driving the Stabilizing Phase are Test and Release Management.

Table 5.2 Role Cluster Focuses and Responsibilities in Stabilizing Phase

Role Cluster	Focus and Responsibility
Product Management	Communications plan execution; launch planning
Program Management	Project tracking; bug triage
Development	Bug resolution; code optimization; hardware or service reconfiguration
User Experience	Stabilization of user documentation materials; training materials
Test	Testing; bug reporting and status; configuration testing
Release Management	Pilot setup and support, deployment planning; operations and support training

Testing the Solution

In the Stabilizing Phase, testing is performed not only on individual components of the solution, but on the solution as a whole, because all features and functions of the solution are now complete, and all solution elements have been built.

The testing that was begun during Developing, according to the test plan created during Planning, continues along with the tracking, documentation, and reporting activities.

User Acceptance Testing

Although user testing and usability studies began during the Developing Phase, they receive more emphasis during Stabilizing. These are conducted to ensure that the new system is able to successfully meet user and business needs. This is not to be confused with customer acceptance, which occurs at the end of the project.

User acceptance testing is performed on a collection of business functions in a production environment after the completion of functional testing. This is the final stage in the testing process before the system is accepted for operational use. It involves testing the system with data supplied by the actual user or customer rather than the simulated data developed as part of the testing process.

User acceptance testing often reveals errors and omissions in the system requirements definition. The requirements may not reflect the actual facilities and performance required by the user. User acceptance testing may demonstrate that the system does not exhibit the anticipated performance and functionality.

The result of this test is an answer to the question of whether or not the solution conforms to the overall user requirements, which determines if the system is ready for production.

Running a pilot for a select set of users helps in user acceptance testing. Surveys and their results from these users about different aspects of the solution (user friendliness, convenience, visual appeal, relevance, and responsiveness) are keys to reaching final user acceptance criteria.

User acceptance testing also gives support personnel and users the opportunity to understand and practice the new technology through hands-on training. The process helps to identify areas where users have trouble understanding, learning, and using the solution. Release testing also gives Release Management the opportunity to identify issues that could prevent successful implementation.

Regression Testing

Regression testing refers to retesting previously tested components and functionality of the system to ensure that they function properly even after a change has been made to parts of the system. For migration projects, this is the most important class of tests.

As defects are discovered in a component, modifications should be made to correct them. This may require retesting of other components in the testing process.

Component system errors can present themselves later in the testing process. The process is iterative because information is fed back from later stages to earlier parts of the process. Repairing program defects may introduce new defects. Therefore, the testing process should be repeated after the system is modified.

Here are some guidelines for regression testing:

Test any modifications to the solution to ensure that no new problems are introduced and that the operational performance is not degraded due to the modifications.
Any changes to the solution after the completion of any phase of testing or after the final testing of the system must be subjected to a thorough regression test. This is to ensure that the effects of the changes are transparent to other areas of the system and other systems that interface with the system.
Modifications to the solution components may also require modifications to the testing cases. The project team must create test data based on predefined specifications. The original test data should come from other levels of testing and then be modified along with the test cases.

Test Tracking and Reporting

Test tracking and reporting occurs at frequent intervals during the Developing and Stabilizing phases. During the Stabilizing Phase, this reporting is driven by the bug count. Regular communication of test status to the team and other key stakeholders ensures a well-informed project.

Bug Convergence

Bug convergence is the point at which the team makes visible progress against the active bug count. At bug convergence, the rate of bugs resolved exceeds the rate of bugs found; thus the actual number of active bugs decreases. Figure 5.2 illustrates bug convergence.

Because the bug count will still go up and down, even after it starts its overall decline, bug convergence usually manifests itself as a trend rather than a fixed point in time. After bug convergence, the number of bugs should continue to decrease until zero bug bounce.

Figure 5.2: Bug convergence is the point where the new bug rate drops below the bug resolution rate

Interim Milestone: Bug Convergence

Bug convergence tells the team that the end is actually within reach.

Zero Bug Bounce

Zero bug bounce is the point in the project when development finally catches up to testing and there are no active bugs for the moment. Figure 5.3 illustrates a zero bug bounce.

After zero bug bounce, the bug peaks should become noticeably smaller and should continue to decrease until the product is stable enough for the team to build the first release candidate.

Figure 5.3: Zero bug bounce

Careful bug triaging is vital because every bug that is fixed creates the risk of creating a new bug or regression issue. Achieving zero bug bounce is a clear sign that the team is nearing a stable release candidate.

Note that new bugs will certainly be found after this milestone is reached. But it does mark the first time when the team can honestly report that that there are no active bugs even if it is only for the moment and it focuses the team on working to stay at that point.

Interim Milestone: Zero Bug Bounce

After zero bug bounce, the bug peaks should become smaller and should continue to decrease until the solution remains stable enough for the team to build the first release candidate.

Release Candidates

After the first achievement of zero bug bounce, a series of release candidates are prepared for release to the pilot group. Each release is marked as an interim milestone.

Guidelines for declaring a build a release candidate include the following:

Each release candidate has all the required elements to qualify for release to production.
The test period that follows determines whether a release candidate is ready to release to production or the team must generate a new release candidate with the appropriate fixes.
Testing the release candidates, done internally by the team, requires highly focused and intensive efforts, and focuses heavily on flushing out critical bugs.
Testing requires a triage process for resolving any newly discovered bugs.
It is unlikely that the first release candidate will be the one that is actually released, because, typically, critical bugs will be found during its intensive testing.

Interim Milestone: Release Candidates

As each new release candidate is built, there should be fewer bugs reported, triaged, and resolved. Each release candidate marks significant progress in the team's approach toward deployment. With each new candidate, the team must focus on maintaining tight control over quality.

Interim Milestone: Pre-Production Testing Complete

Eventually, a release candidate is built that contains no defects whose resolution requires a change. Once that has occurred, no more defects will be found within an isolated staging environment; all testing that can be done before putting the migrated component into production has been completed.

Conducting the Pilot

During the pilot, the team tests as much of the entire solution, in a true production environment, as possible. A pilot release is a deployment to a subset of the live production environment or user group. Depending on the context of the project, the pilot can take various forms:

In an enterprise, a pilot can be a group of users or a set of servers in a data center.
For migration projects, a pilot might involve testing the most demanding application or database that is being migrated with a sophisticated group of users that can provide helpful feedback.
Commercial software vendors, such as Microsoft, often release products to a special group of early adopters prior to final release.

The common element in all these forms of piloting is testing under live conditions. The pilot is not complete until the team ensures that the solution is viable in the production environment and every component is ready for deployment.

Following Best Practices

Prior to beginning a pilot, the team and the pilot participants must clearly identify and agree upon the success criteria of the pilot. These should map back to the success criteria for the development effort.
Any issues identified during the pilot must be resolved either by further development, by documenting resolutions and workarounds for the installation teams and production support staff, or by incorporating them as supplemental material in training or help materials.
Before the pilot is started, a support structure and issue-resolution process must be in place. This may require that support staff be trained. The procedures used for issue resolution during a pilot may vary significantly from those used during deployment and when in full production.
In order to determine any issues and confirm that the deployment process will work, it is necessary to implement a trial run or a rehearsal of all the elements of the deployment prior to the actual deployment.

Deciding Next Steps

Once enough pilot data has been collected and evaluated, the team is at a point of decision. One of several strategies must be selected:

Stagger forward. Deploy a new release to the pilot group.
Roll back. Execute the rollback plan and revert the pilot group back to the previous configuration state they had before the pilot (as closely as feasible). Then try again with a more stable release.
Suspend. Suspend the entire pilot.
Fix and continue. Issue a fix to the existing code to the pilot group.
Proceed. Advance to the Deploying Phase.

Creating Pilot Test Reports

As cycles of the pilot tests are completed, the team prepares reports detailing each lesson learned and how new information is incorporated and issues are resolved. The following may result from performing the pilot testing:

The identification of additional risks.
The identification of frequently asked questions for training purposes.
The identification of user errors.
The ability to secure buy-in and support from pilot users.
Documentation of concerns and issue resolutions.
Updates to documentation, particularly help files and deployment plans.
Determination of whether or not all success criteria were met.

Interim Milestone: Pilot Complete

This milestone signifies that the pilot (or pilots) has been successfully completed and that the team is ready to proceed to Deploying.

Closing the Stabilizing Phase

Closing the Stabilizing Phase requires completing a milestone approval process. The team needs to document the results of the different tasks it has performed in this phase in order to submit the project to management for approval.

Key Deliverables from the Stabilizing Phase

A deliverables checklist for the Stabilizing Phase includes:

Final release
Release notes
End-user help and training materials
Testing and bug reports
Test tools
Traceability audit
Source documentation and executables
Project documents
Updated risk management document
Milestone review report
- Team member project progress report
- Team lead project progress report

Key Milestone: Release Readiness Approved

The Stabilizing Phase culminates in the Release Readiness Approved Milestone. This milestone occurs at the point when the team has addressed all outstanding issues and has released the solution and made it available for full deployment. It is the opportunity for customers and users, operations and support personnel, and key project stakeholders to evaluate the solution and identify any remaining issues they need to address before release.

Project teams usually mark the completion of a milestone with a formal sign-off. Key stakeholders, typically representatives of each team role and any important customer representatives who are not on the project team, signal their approval of the milestone by signing or initialing a document stating that the milestone is complete. The sign-off document becomes a project deliverable and is archived for future reference.

As the team transitions to the Deploying Phase, responsibility for ongoing management and support of the solution officially transfers from the project team to the operations and support teams.