Trustworthy Administrators

By Steve Riley
Sr. Program Manager
Security Business and Technology Unit

See other Security Management columns.

Do you trust your administrators? That seemingly innocent question creates a serious dilemma in the minds of a lot of people. While we all know what we’d like the answer to be, the disappointing fact is that, increasingly, the true answer is the opposite. This became apparent in discussions I had with many attendees at TechEd US in May—there is genuine concern about the trustworthiness of administrators.

Consider the following example. A small high-tech organization based in southern California suddenly realizes that most of its intellectual property, stored on a SharePoint cluster, is gone. Not just erased: it’s as if the information had never existed. After some forensic analysis they discover that a logic bomb programmed to destroy data had been placed in the SharePoint cluster by none other than the disgruntled enterprise administrator. Why? This particular administrator learned, by reading confidential emails, that he was going to get laid off. So in retaliation he cobbled up the disastrous logic bomb, put it on the cluster, and walked out the front door (probably with a stupid grin on his face that no one gave a second thought about). The bomb dutifully followed its devastating instructions and began systematically wiping out disk clusters late at night on a Friday. And, of course, the victimized organization had no backups. It was only through recovering archived copies of documents in the email inboxes of several employees that the organization was able to reconstruct the bulk of its intellectual property—and avoid being driven completely out of business.

Do you trust your administrators? It’s a serious question, and it deserves serious thought. I asked this question in a packed seminar room of nearly 1000 attendees listening to my presentation on security policies and no one—no one—raised a hand. That frightened me. If we can’t trust the very people we hire to build and manage our mission-critical networks upon which our business successes depend, we might as well unplug it all and revert to the days of stone knives and bearskins.

What’s your process for interviewing, investigating, hiring, monitoring, and terminating your administrators? “Hah, process he says!” Yes, process. Depending on what it is they administer, these people have nearly or absolutely unfettered access to everything in your network. Ponder, for a moment, the kinds of access a typical administrator has:

• Physical access into the building

• Physical access into the computer room

• Physical access to the computers themselves, their storage devices, and their input/output mechanisms

• Electronic access to the information stored in the computers

• Remote access into the network, possibly from anywhere in the world

• Capability and authorization to create accounts and to modify data access controls

• Access to logs and systems management tools

That’s a lot of power concentrated in one person, power that can be used for good and abused for bad. What are you doing to help ensure that the people you entrust with such power will only use it for good?

These days, loyalty has evaporated. It’s rare indeed to find someone, especially among post-Boomers, who feels any loyalty to the organization that employs them. Yes, I’ll admit that rings with the sound of a crass generalization, but my observations and research tend to support the notion (I’ll avoid discussions of the political and economic situations that have given rise to the trend). Such lack of loyalty means that the likelihood of a powerful administrator abusing his or her power for revenge or personal gain is greater now than it’s ever been. I never used to believe the statistics claiming that most attacks originated from the inside. Now, I do.

Of course, not all inside attacks stem from revenge or greed. Administrators are especially vulnerable to social engineering. Not all administrators are easily duped, although some are—another uncomfortable fact is that many technical people don’t have finely-honed social skills (“I got into IT because I wanted to avoid people,” goes the common saying). Social engineering attackers know this and will play games to motivate people into doing what the attackers want. “Hey, I’ve seen you at work, wow are you cute…” can melt the resolve of many an admin, especially when the attacker uses a voice changer to acquire the sultry tones of an attractive woman (administrators are typically male). Computers and networks are becoming more difficult to penetrate; why bother hacking the system when you can hack the sysadmin instead?

To put it boldly: you must trust your administrators. If you can’t trust the crop you’ve got now, then it’s time to find replacements. If this describes your situation, please don’t put it off: make it one of your highest priorities.

I offer now some useful bits of advice regarding administrators. Consider all of them; implement what makes sense to you.

Eliminate the dreaded “Administrator” account. These accounts, whether local or domain, allow administrators to anonymously run amok across your network. You see something suspicious in a log and you know only that “Administrator” made some change, not who actually did the work. (It also indicates an unskilled attacker, someone who forgot to eliminate evidence from the log.) Remove potential anonymous attack vectors by doing this with all your various “Administrator” accounts:

1. Line up all your administrators in front of a computer. Have a corporate auditor join the line at the end.

2. Open the account properties and start to change the password.

3. Tell each of your administrators to enter four or five characters, thus contributing a portion of the password.

4. After every administrator has entered a portion, have the auditor contribute the last set of characters and save this new password.

5. Disable the account.

With this procedure you’ve practically eliminated the ability for a rogue administrator to act anonymously. You don’t need this account for anything, and now you’ve contrived a situation which requires devious administrators plus a corrupt auditor to collude in an attack. The chance of that happening is slim indeed; if it were, I’d guess that at least one of them is playing a double-agent. If you can’t trust your administrators, it’s certain that they themselves have little, if any, trust of each other!

Implement background investigations for all newly-hired administrators. Earlier I mentioned the notion that it’s plausibly easier to hack the sysadmin than the system. Becoming a sysadmin is a loftier goal, a goal entirely with in the abilities of talented attackers. What do you really know about that new sysadmin you just hired?

Some large organizations already conduct background checks (Microsoft does). Such behaviors might not be part of your corporate culture—but don’t use that as an excuse not to implement investigations. Given that these are the people directly charged with the critical responsibility of protecting your crucial information assets, you want some assurance that they’ve demonstrated honest and trustworthy behavior in former jobs and in their general modes of living. Make it very plain during the interview process that background checks are part of the hiring process.

It’s politically more challenging to conduct background investigations of the administrators you’ve already hired. But if there’s a way you could do this within the culture that exists at your workplace, I think it’s just as important.

Carefully consider the details of your termination procedures. I relate a somber tale to illustrate the point. An organization faced with dwindling profits and ballooning expenses made the difficult decision to terminate several administrators. Managers decided to inform them together in the conference room one afternoon. “You’ve got two hours to save any personal information off your computers and to pack up your belongings.” Must I finish the tale? As you’ve surely surmised, in those two hours the network was demolished; the former administrators vanished before anyone realized what happened.

Don’t conduct personnel conversations over cleartext electronic mail—encrypt it if you can, or hold all human resources conversations in person. When the time comes to inform administrators of job loss, disable all their access before you talk to them. Call the administrator into your office and have someone from your auditing or corporate security department accompany the terminated administrator back to his desk to collect his possessions and exit the facility. Don’t permit any contact with computers.

Don’t fear the pink slip. These are difficult yet essential topics to discuss. The information in your computer systems is as critical to the success of your business as are the products or services you provide. Even though your administrators’ overarching goal must be to protect that information from harm, they also must simultaneously provide users with the access they need to do their jobs—admittedly, some serious cognitive dissonance exists here. Your administrators must be made aware—and regularly reminded—of the expectations and outcomes of their jobs. Administrators who willfully violate policy should be terminated; these are not the people you want to keep on staff. A signed acceptable use policy helps support your decision, especially if the former employee attempts to sue you for wrongful termination.

Let your policies be your guide. A robust security policy includes detailed descriptions of the duties and procedures of administrators, precisely enumerating allowed and prohibited behavior. Administrators who know what’s expected of them, and the penalties associated with violations, tend to be administrators whom you can trust. Better, of course, is individual and collective behavior—indeed a pervasive workplace culture—that demonstrates how the value of their individual talents and efforts contributes to the success of the organization. Most people are good people. Reward good people for doing the right things and allow innovation to flourish. I’m guessing there isn’t much to worry about when your administrators feel genuinely appreciated.