Help: I Got Hacked. Now What Do I Do? Part II

  • Article

Security Management

Jesper M. Johansson, Ph.D., CISSP, MCSE, MCP+I
Security Program Manager, Microsoft Corporation

See other Security Management columns.

On This Page

There Is More to Information Security Than Viruses There Is More to Information Security Than Viruses
Salvaging Data in the Absence of Backups Salvaging Data in the Absence of Backups
Recovering After an Attack – What Tools Will Help? Recovering After an Attack – What Tools Will Help?
Conclusion Conclusion

The last few weeks have been very interesting. A lot of people have read the previous article (https://www.microsoft.com/technet/community/columns/secmgmt/sm0504.mspx), and I think almost all of them wrote me to comment on it. In addition, folks from various internal teams at Microsoft have been contacting me about it. There was even a discussion about the article on the “Full Disclosure” mailing list, although, as usual for that list, it quickly deteriorated into a “why I do not like Microsoft” discussion.

The feedback has been great. So great, in fact, that I decided to write a follow-up to the article to further elaborate on some points. The feedback has largely been divided into three categories, which, paraphrased a bit, are the following:

  • Finally someone is speaking the truth: There is more to the world than viruses.

  • That’s ridiculous: We don’t have enough backups to flatten hacked systems.

  • Here’s a product that will help (largely from various product teams here at Microsoft).

Each category merits a bit more elaboration, which is the purpose of this article. The first category of comments is the easiest, and most interesting, to answer. The genuine truth is that there is a lot more to our world than the run-of-the-mill e-mail viruses.

There Is More to Information Security Than Viruses

If you read the first article (https://www.microsoft.com/technet/community/columns/secmgmt/sm0104.mspx) you probably already know that I do not believe a system can ever be completely secure—at least, not if you are interested in actually using it. One additional piece of basic lemma (https://www.britannica.com/ebc/article?eu=405886&query=lemma&ct=) information I accept is that there is a lot more to information security than worms and viruses (while there are differences between worms and viruses, and the experts in the field are still debating exactly what these differences are, we are primarily concerned with general malicious code here, and henceforth I will treat worms and viruses as one type of attack and refer to them collectively as “worms”). Yet we tend to always focus on worms, and why not? Sasser/Blaster/Lion/Trino/Ramen/Slapper and other worms are very disruptive and cost tremendous amounts of money to get rid of. With a few exceptions (such as the Linux Ramen worm and Code Red), however, a worm itself is not destructive. That is not to say that worms cannot be destructive, but that at least when you are dealing with a worm, you basically know what you get. A lot of people got the same worm and at least some of them were able to dissect it, saving you the trouble.

However, some of these worms put back doors into the system—back doors that truly evil people can then use to do much worse things to your system than the worm did. Nimda did this. So did Slapper (another Linux worm). Once a worm puts a back door on your system, the system can be controlled by bad guys across the Internet. In addition, none of these worms created any kind of log file that would tell you what was done through the back door. Once a system is affected with one of these worms, you can no longer trust anything on that system. In fact, if the worm even manages to get on the system it should really be treated as a symptom that the system is untrustworthy. A different attack may very well have already used the same vector (method of attack) as the worm, and may have done much worse things to the system than the worm itself. The key point here is that we need to stop focusing so much on worms and focus more on the vulnerabilities that the worms exploited.

E-mail worms are a slightly different problem; essentially, they are a layer 8 problem—a problem exploiting users, not technology. That means we need to treat e-mail worms differently—if users stopped double-clicking on e-mail attachments, these problems would go away. Conversely, if management would let us just block all e-mail attachments, at least for those users who won’t learn not to double-click on suspicious attachments, the problems would also go away. You can do this pretty easily with any relatively recent version of Microsoft Exchange and Outlook. Of course, any of those users would also double-click on a malicious Trojan, but that is an orthogonal issue. The mere fact that an e-mail worm got on the system is not a strong indication that the system is compromised by an active attacker in the same way it is when a network worm, like Sasser, gets on the system.

At the end of the day, I am considerably more worried about active attackers than I am about worms—and active attackers can use a lot of different vectors to get into your systems and your network. The worm problem can be solved with some relatively simple (to enumerate, not necessarily to implement) steps:

  • Make sure all the relevant patches are deployed as soon as they are released.

  • Use a firewall.

  • Use an antivirus program. If you do not have one, go to https://www.microsoft.com/protect and you can get one for free. For more information on how to deploy antivirus solutions across a larger environment, see the new Microsoft Antivirus Defense-in-Depth Guide at https://go.microsoft.com/fwlink/?LinkId=28734.

  • Teach users not to double-click unsolicited attachments without checking first whether they are legitimate, or block them from doing so.

Preventing active, determined attackers from getting into the system is not always that simple. In addition, as some readers pointed out, we do not always have enough backups to reliably recover a system. Therefore, it becomes very important to detect whether a system has been compromised, and, if so, what was done to it. There are some ways to do that.

Salvaging Data in the Absence of Backups

One of the things that keeps me awake at night (other than small children and engine noise from the airplane I’m on) is not the possibility that the barbarians are at the gate but that they are already inside the gates and we do not know it. We know there are a lot of bad people out there. During Microsoft’s recent Tech-Ed conference some criminal group put a “bounty” out, awarding $50,000 to whoever could destroy the network at the conference. How can we tell whether these people are inside the system? If they are not very good, they will leave tracks, such as new accounts, strange files, and potentially unstable systems. The majority of attackers today probably fall into that category; at least I hope they do. Then there are the really good ones. These are the ones who disappear into the OS as soon as they get on the box. They install a rootkit on the system that makes the system no longer trustworthy. Windows Explorer and the command line will no longer show you the files that are actually on the system. The registry editor is now lying. Account manager tools will not show you all the users. At this stage of an intrusion, you can no longer trust the system to tell you about itself. That’s where you get into a flatten and rebuild (some people call it “nuke and pave”) scenario. The system is now completely compromised. Can you detect that this has happened, and what was done?

There are some tricks for detecting this type of intrusion. One is to use a network-based intrusion detection system (IDS), which will track traffic coming into and going out of your network. A network-based IDS is neutral and can, assuming it has not been compromised, give you a good idea of what is going into and out of a suspect system. A thorough discussion of IDS systems is beyond the scope of this article, and, for most of us, IDS systems are beyond our immediate need anyway. In many cases, we would get more value from actually securing the network than from spending the time it takes to implement an IDS. Most of our networks can use a lot of additional security work, and if we do not do that first, we will simply ensure that we get a lot of interesting IDS logs.

You can also detect that a system has been compromised by analyzing the system itself, but it involves some in-depth forensics. For example, since you cannot trust the system itself, you must boot it to neutral media, preferably read-only media. One option is to boot to Windows PE, which is a command-line only, CD-bootable version of Windows XP or Windows Server 2003. Windows PE is not generally available, however. Another option is to get a copy of Winternals ERD Commander or System Restore (https://www.winternals.com). Both tools are based on WinPE. ERD Commander is essentially a GUI on top of WinPE that gives you a great set of tools for resurrecting a crashed system. Since it is a neutral installation, you can trust the commands on that disk to tell you what is actually going on with the suspect machine. System Restore is a superset of ERD Commander that also includes the ability to check a system against a baseline. For example, let’s say you build a Web server. As soon as it is built, you create a baseline for what the system looks like. At some point after you enter the system into production, you suspect it has been hacked. You may, for instance, detect odd network traffic emanating from the system and decide to analyze it. You can take the system offline, boot it to a recovery disk, and run a comparison against the latest snapshot. This will tell you exactly what has changed. Whether those changes actually reflect an intrusion is not clear, though. You have to make that call.

One final note on forensic work: If you really do believe an intrusion has happened and you want to take legal action, you probably should not do forensics on your own. Disconnect the system to prevent the compromise from spreading, but after that, call in a forensics expert. The risk is simply too great that you will destroy the evidence and make it inadmissible in court. If you have a need to preserve the evidence, use a professional to gather it.

Recovering After an Attack – What Tools Will Help?

Finally, the most important step: a system has been hacked, how do you get service up and running? It depends on the type of system. First, I do not believe in backing up clients. Clients in a network should be storing their data on servers; then we back up servers. If the client gets compromised, we rebuild it. It is complicated enough to back up servers. We do not need to make it much worse by including clients in the backup plan.

Obviously, however, this statement is not true in a home environment or on a very small network. In those environments I use a few built-in tools to generate minimal backups of the data (not the programs). Windows XP and Windows 2000 have a decent backup tool built in. You can use it to generate backups of the system to arbitrary media. A somewhat simpler option, in my opinion, is to use the Files and Settings Transfer Wizard. Every few weeks, I run the Files and Settings Transfer Wizard in Windows XP to create a copy of all my data and my entire profile. The backup then gets burned to CD or copied to another hard drive. If the system should fail, it is a matter of a few hours to rebuild it, reinstall all the patches, and then restore the data and profile using the Files and Settings Transfer Wizard. As an additional safety measure, I also configure my home systems with roaming profiles. The profile is only roaming to another hard drive in the same system, but at least this protects me in the case of hard drive failure. But if all you have is a run-of-the-mill virus, restoring a system from backup may be overkill. If you are simply trying to clean off a virus, refer to chapter 4 of the Microsoft Antivirus Defense-in-Depth Guide (https://go.microsoft.com/fwlink/?LinkId=28734) for details on how to proceed.

On a larger network, we need server backups. The complete backup plan and process is beyond the scope of this article. However, as many of the comments from the previous article mentioned, no matter how we plan our backups we often do not have enough backups and therefore have to try to reconstruct data from the compromised system. There are some tricks for doing this. First, identify the last trusted backup and restore the data from that to an isolated replacement system. Then copy the data off the compromised system to the isolated replacement system. Next, use a differencing tool, such as Windiff, to run a diff of the backup comparing it to what you found on the compromised system. Keep in mind that this must be done with the compromised system booted to neutral media, as was explained earlier, otherwise you run the risk of compromising the backup as well. For each item that has changed, identify the data owner. Now task each data owner with certifying the differences. If the differences are accepted, reverse integrate them into the trusted backup. This is a complicated and time-consuming process, but it is the only way to be sure that you are getting what you should back to the new system and are not simply restoring compromised information to a new system. If you just restore from a compromised backup, at best, you will have untrustworthy data. At worst, you will just have created another compromised system.

Conclusion

The process I have outlined above is definitely painful. No doubt about it. The only thing I can say is “don’t shoot the messenger.” This is why we spend so much effort on figuring out how to prevent getting your systems hacked. Getting back in service is painful. By comparison, preventing yourself from getting hacked is a lot less painful in the long run, and a lot less likely to turn into a resume-generating event. In future columns, we will return to more prescriptive guidance to help you minimize your dependency on recovery.

As always, this column is for you. Let us know if there is something you want to discuss, or if there is a better way we can help you secure your systems. Just click the “Comments” link below, and send us a note.