Failures That Could Never Happen
Here are two "impossible" stories from my career. These really happened; they are not friend-of-a-friend stories. (The data centers will remain unnamed.) The moral, of course, is that messes happen with even the best planning, but that good practices keep messes from becoming disasters.
Back in the big-iron days (this is a long time ago), I worked on a mini that would invariably fault around 4AM one day a week. Nothing in the logs gave us a clue; it simply faulted and restarted, whether or not it was running a job. After some weeks of this, we resolved to stay all night to see if we could figure it out. Around 4AM, the cleaning person came in with a big vacuum cleaner. The only outlet conveniently located in the middle of the room had plugged into it some critical component, now long forgotten. Of course, she unplugged it, plugged in the vacuum, cleaned up, and then replaced the original plug. And we were laughing too hard to stop her.
More recently, workers had a tile out in a raised-floor data center. Around the corner came someone carrying an armful of papers. There was no warning or rail around the missing tile, and he couldn't easily see down through the papers. So he stepped in the hole. As he pitched forward, he dropped the papers and stuck out his hands to catch himself on the nearby wall. On the wall was the emergency power shutoff button for the entire data center, and for some reason it had no protective "cage" around it. Sure enough, the one part of the wall our falling friend could reach was the shutoff button.
In the first case, we found a way to secure the plug the next day; in the second, a cage was procured that afternoon. In neither case was Allen Funt around, unfortunately.
The point is not that you can, or can't, prepare for everything. Stuff will happen. What you must be prepared for is rapid recovery around whatever unfathomable event happens.