Windows Confidential: Testing, Testing
Whatever happened to the classic software testing cycle of alpha, beta and, finally, release candidate? The Internet has changed all that.
Raymond Chen
The Internet has changed language—certainly the language of technology. The devaluation of terms like “beta testing” is further along than you think.
It used to be that a product went through an alpha and beta test phase. Then you started cranking out release candidates, or RCs, and shipping the version that passed all release criteria. Alpha builds never left the building. Beta builds went to a small number of trusted testers.
Some time ago, the term “release candidate” began meaning what would otherwise have been called a beta. This seemed to be done in order to make people start paying attention. Recently, though, the term “beta” itself has changed because people’s expectations have changed.
First of all, beta testers used to be a select group of people chosen for their ability to test the product under a wide range of conditions. They’d be expected to write up highly detailed bug reports so that the software developers could address any issues they found. The title of Beta Tester was fairly prestigious. It meant that you were so valuable that a company was willing to share highly sensitive information with you in order to secure your assistance.
Bella Beta
Nowadays, the secretive Beta Tester is an endangered species. Everybody expects a public beta. When a public beta is announced with limited participation, competition for those coveted beta keys becomes intense. For example, all 75,000 beta slots for Microsoft Security Essentials were filled within the first 24 hours.
How many of those people downloaded the program because they intended to install it on a wide variety of systems and take the time to write up quality bug reports? How many of them downloaded the software just for the cachet of having a copy of a pre-release product? I bet a lot more fell into the latter category. The kid who dies with the most software wins.
Mind you, there’s a lot of polishing that happens before a beta release. Another consequence of a widespread public beta releases is that it’s usually the first glimpse people have of your software. You never get a second chance to make a first impression. Even if people know, “This is just a beta—the final product will be better,” their subconscious will say, “I remember that product. It sucked.”
Operating a public beta is like posting a public invitation to your birthday party. Sure, more people will show up, but will your party be any better? Bug reports submitted by closed beta testers tend to be higher quality because the testers understand that if they don’t submit quality bug reports, they won’t be invited to participate in the next beta. On the other hand, if you open a public beta, you’ll be deluged with half a million comments in six weeks. You’ll have to sift through them all to look for the useful ones.
In one article, a public beta tester complained that he submitted about 25 bugs, and only three of them were fixed. Let’s consider this in context. There were half a million comments, and from those, nearly 2,000 fixes were developed. That represents a fix rate of 0.4 percent. On the other hand, the fix rate for the beta tester in question was three out of 25—or 12 percent. That’s 30 times the average fix rate, and he still wasn’t happy.
Now the pendulum has swung back the other way. We used to accelerate the perceived progress of a product, calling it an RC even though it was really just a beta. Now we intentionally under-report any progress. The prevalence of “perpetual beta” on the Internet means people expect something released with the beta label to be fully functional, or nearly so.
The product teams are well aware of the shifting sentiment toward software labeled beta. Now, when Windows releases something called a beta, it’s really something that has already passed the point where any significant architectural changes can be made. They’re just looking for bugs.
.jpg "Raymond Chen")
Raymond Chen’s Web site, The Old New Thing*, and his book of the same title (Addison-Wesley, 2007) deals with Windows history, Win32 programming and getting lost in Beijing.*