Windows ConfidentialLast Check-In Chicken

Raymond Chen

As a project winds down, the team often plays an informal game known as Last Check-In Chicken. In this game, teams "compete" in order to avoid being responsible for the last change to the product because, in principle, if it weren't for your bug, the project could have shipped a day earlier. (This is rarely actually true, but it is, nonetheless, the motivation for the game.) The competition is purely virtual—nobody really keeps score, and the "winner" is soon forgotten.

Unless you're that winner.

One of the senior members of the Windows team admitted to me that he won the game twice in a row back in the earlier days of Windows. The second time was particularly noteworthy because the change was made after manufacturing had already begun. The entire manufacturing run had to be recalled and restarted.

An attempt to make it three in a row was thwarted, however. The third attempt ended up being the second-to-last check-in rather than the last. The final check-in was made by another one of my colleagues who not only made the final check-in to Windows 2000 but then started a new dynasty by also making the final check-in for Windows XP.

One of the ironies of the game of Last Check-In Chicken is that it is often the people who have their act together (and, therefore, the people whom you'd least expect) who end up winning. This isn't because their code is inferior, mind you. By virtue of being well prepared, they're in a much better position to make that final check-in, even though it's not in a component for which they normally have responsibility.

Making a late check-in is a major undertaking because the insulation that normally separates the developer from the master tree is no longer there. Under normal circumstances, a check-in goes like this:

• You make your check-in to your subproject.
• Your subproject's build lab churns on the change overnight and produces a build in the morning.
• Your subproject's test team pores over the code for a week or more before giving the OK for the check-in to be released to the trunk.

Doing it in this way, it is much more likely that if you have made any mistakes, they will get caught before they become an official part of the project. Some subprojects are so large that they apply this principle recursively, creating sub-subprojects that verify their changes independently before they are released to the larger subproject to which they belong.

But when you're in the project endgame and the release management team decides to take your fix, the clock is ticking and there's no time for the leisurely multi-week change verification process. Scattered throughout the project are developers who regularly build the entire project (not just their subproject), from the make clean all the way to creating disk images on a private release server and verifying that build constraints are satisfied. (For example, one of the things they would do is make sure the CD image isn't too big to fit on a CD!)

If a team finds itself in a situation where it needs to make a last-minute check-in, it will typically look to whichever team member is well prepared—if there isn't such a person, the team will then enlist the assistance of someone from a nearby team who is well prepared.

Now a little game of Cyrano takes place. The team provides the change to the well-prepared developer, and the developer builds the disk images, verifies the build constraints, and then points the test team to the private release server. If the test team happens to find a problem, the development team comes up with a new fix and hands it to the well-prepared developer, who restarts the cycle.

In other words, the well-prepared developer acts as a one-man virtual build lab for that team. Once everybody is satisfied, the well-prepared developer makes the check-in. This is why those who are well-prepared end up winning the game of Last Check-In Chicken. It's not because they are on the hook for a problem but because they are ready to help those who are on the hook.

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history, Win32 programming, and exploding coffee machines.