Securing Windows 2000 Server

Chapter 10: Responding to Incidents

Published: November 17, 2004 | Updated : May 31, 2006

Note: Welcome to the TechNet Archive. We've created this Archive area so that we can continue to make available older content that is still of interest to some of our users. This allows us to streamline the content offerings on the site and keep it focused on the newest, most relevant content.

How prepared is your information technology (IT) department to handle a security incident? Many organizations only learn how to respond to a security incident after suffering an attack. By this time, the incident may have proved much more costly than is necessary. Proper incident response should be an integral part of your overall security policy and risk mitigation strategy.

There are clearly direct benefits in responding to security incidents. However, there may also be indirect financial benefits. For example, your insurance company may offer discounts if you can demonstrate that your organization is typically able to quickly and cost-effectively handle attacks. Or, if you are a service provider, a formal incident response plan might help win business, because it shows that you take seriously the process of good information security.

This chapter will provide you with a process and procedures to use when responding to intrusions identified in a Microsoft® Windows® 2000 Server-based environment. The value of forming a security incident response team with explicit team member roles is explained, as well as how to define a security incident response plan. A case study also provides the context to help illustrate how to best apply this process and the related procedures to your organization.

On This Page

Minimizing the Number and Severity of Security Incidents
Defining an Incident Response Plan
Scenario—Contoso Incident Handling
Summary

Minimizing the Number and Severity of Security Incidents

In most areas of life, prevention is better than cure, and security is no exception. Wherever possible, you will want to prevent security incidents from happening in the first place, as discussed in chapters 5 through 8. However, it is impossible to prevent all security incidents, so when a security incident does happen, you will need to ensure that its impact is minimized. There are prudent measures that you can take to minimize the number and impact of security incidents, including both preventative and responsive measures:

• Clearly establishing and enforcing all policies and procedures. Many security incidents are accidentally created by IT personnel who have not followed or understood change management procedures or have improperly configured security devices, such as firewalls and authentication systems. Your policies and procedures should be thoroughly tested to ensure that they are practical and clear and provide the appropriate level of security.

• Gaining management support for security policies and incident handling.

• Routinely assessing for vulnerabilities in your environment. This assessment should be done by a security specialist with special clearance to perform these actions.

• Routinely checking servers to ensure that they have all of the latest patches installed.

• Establishing security training programs for both IT staff and end users. The largest vulnerability in any computer is the naïve user—the ILOVEYOU worm effectively exploited that vulnerability among IT staff and end users.

• Posting security banners that remind users of their responsibilities and restrictions, along with a warning of potential prosecution for violation. Without such banners it may be difficult or impossible to prosecute attackers. You should obtain legal advice to ensure that the wording of your security banners is appropriate.

• Developing, implementing, and enforcing a policy requiring complex passwords. Information on complex password security options and best practices is detailed in Chapter 5, "Securing the Domain Infrastructure."

• Routinely monitoring and analyzing network traffic and system performance.

• Routinely checking all logs and logging mechanisms. These checks would include operating system event logs, application specific logs and intrusion detection system logs.

• Verifying your back up and restore procedures. You should be aware of where backups are maintained, who can access them, and your procedures for data restoration and system recovery. Make sure that you regularly verify backups and media by selectively restoring data.

• Creating a Computer Security Incident Response Team (CSIRT). This team is a group of people with responsibilities for dealing with any security incident. Your CSIRT should consist of members whose duties are clearly defined to ensure that no area is left uncovered in your response (more details on assembling a CSIRT can be found later in this chapter).

• Training members of your CSIRT on proper use and location of critical security tools. You should also consider providing portable computers that are preconfigured with these tools to ensure that no time is wasted installing and configuring tools to respond to an incident. These computers and the associated tools must be properly protected when not in use.

• Assembling all relevant communication information. You should ensure that you have contact names and phone numbers for people within your organization who need to be notified (including members of the CSIRT, those responsible for supporting all of your systems, and those in charge of media relations). You will also need details for your Internet service provider (ISP) and local and national law enforcement agencies. Consider contacting local law enforcement before an incident happens to help you ensure that you understand proper procedures for communicating incidents and collecting evidence.

• Placing all emergency system information in a central, offline location, such as a physical notebook or an offline computer. This emergency information includes passwords to computers, Internet Protocol (IP) addresses, router configuration information, firewall rule set lists, copies of certification authority keys, contact names and phone numbers, escalation procedures, and so on. This information must both be kept extremely physically secure and be readily available. One method of securing and making this information readily available is to encrypt it on a dedicated security portable computer that is placed in a secure vault and limit access to the vault to authorized individuals such as the CSIRT leader and the CIO or CTO.

Assembling the Core Computer Security Incident Response Team

The CSIRT is the focal point for dealing with computer security incidents in your environment. Its responsibilities include:

• Monitoring systems for security breaches.

• Serving as a central communication point, both to receive reports of security incidents and to disseminate vital information to appropriate entities about the incident.

• Documenting and cataloging security incidents.

• Promoting security awareness within the company to help prevent incidents from occurring in your organization.

• Supporting computer and network auditing through processes such as vulnerability assessment and penetration testing.

• Keeping up with new vulnerabilities and attack strategies employed by attackers.

• Keeping up with new software patches.

• Analyzing and developing new technologies for minimizing security vulnerabilities and risks.

• Providing security consulting services.

• Continually honing and updating current systems and procedures.

The ideal CSIRT membership and structure depends on the type of your organization and your risk management strategy. However, the CSIRT should generally form part or all of your organization's security team. Inside the core team are security professionals responsible for coordinating a response to any incident. The number of members in the CSIRT will typically depend on the size and complexity of your organization. However, you should ensure that there are enough members to adequately cover all of the duties of the team at any time.

CSIRT Team Leader

It is important that the CSIRT has an individual responsible for its activities. The CSIRT Team Leader will generally be responsible for the activities of the CSIRT and will coordinate reviews of its actions. These reviews may lead to changes in polices and procedures for dealing with future incidents.

CSIRT Incident Lead

In the event of an incident, there should be one individual responsible for coordinating the response. The CSIRT Incident Lead has ownership of the particular incident or set of related security incidents. All communication about the event is coordinated through the Incident Lead, and when speaking with those outside the CSIRT, he or she represents the entire CSIRT. The Incident Lead may vary depending on the nature of the incident and is often different from the CSIRT Team Leader.

CSIRT Associate Members

Besides your core CSIRT team, you should have a number of specific individuals who handle and respond to particular incidents. Associate members will come from a variety of different departments in your organization, and they should specialize in areas that are affected by security incidents but that are not dealt with directly by the core CSIRT. Associate members may either be directly involved in an incident or serve as entry points to delegate responsibility to a more appropriate individual within their departments.

The following table shows some suggested CSIRT associate members and their roles.

Table 10.1  CSIRT Associate Members

How the CSIRT Responds to an Incident

In the event of an incident, the CSIRT will coordinate a response from the core CSIRT and will communicate with the associate members of the CSIRT. The following table shows the responsibilities of these individuals during the incident response process.

Table 10.2  Responsibilities of CSIRT During the Incident Response Process

Defining an Incident Response Plan

All members of your IT environment should be aware of what to do in the event of an incident. Although the CSIRT will perform most actions in response to an incident, all levels of your IT staff should be aware of how to report incidents internally. End users should report suspicious activity to the IT staff directly or through a help desk rather than directly to the CSIRT.

The incident response plan should be reviewed in detail by all team members and easily accessible to all IT staff. This review will help to ensure that when an incident does occur, the right procedures are followed.

Your incident response plan should include these steps:

• Making an initial assessment

• Communicating the incident

• Containing the damage and minimizing the risk

• Identifying the type and severity of the compromise

• Protecting evidence

• Notifying external agencies

• Recovering systems

• Compiling and organizing incident documentation

• Assessing incident damage and cost

• Reviewing the response and updating policies

Note The Word document “The Incident Response Quick Reference Card” can be used during an incident as a checklist to ensure that all phases are properly executed. This document's file name is JA1001.doc and it is included in the self-extracting executable file called SecWin2k.exe that is included with this guide.

These steps are not purely sequential. Rather, they happen throughout the incident. For example, documentation starts at the very beginning and continues throughout the entire life cycle of the incident; communication also happens throughout the entire incident.

Other aspects of the process will work alongside each other. For example, as part of your initial assessment, you will gain an idea of the general nature of the attack. It is important to use this information to contain the damage and minimize risk as soon as possible. If you act quickly, you can help to save time and money, and your organization's reputation.

However, until you understand in more detail the type and severity of the compromise, you will not be able to be really effective in containing the damage and minimizing the risk. An overzealous response could even cause more damage than the initial attack. By working these steps alongside each other, you will get the best compromise between swift and effective action.

Note It is very important that you thoroughly test your incident response process before an incident occurs. Without thorough testing, you cannot be confident that the measures that you have in place will be effective in responding to incidents.

Making an Initial Assessment

Many activities could indicate a possible attack on your organization. For example, a network administrator performing legitimate computer maintenance will appear similar to someone launching some form of attack. In other cases, a badly configured computer may lead to a number of false positives in an intrusion detection system, making it more difficult to spot genuine incidents.

As part of your initial assessment, you should:

• Take initial steps to determine whether you are dealing with an actual incident or a false positive.

• Gain a general idea of the type and severity of attack. This assessment should provide at least enough information to begin communicating it for additional research and to begin containing the damage and minimizing the risk.

• Record your actions thoroughly. These records will later be used for documenting the incident (whether actual or false).

Note Although you would like to avoid false positives as much as possible, it is always better to act on a false positive than fail to act on a genuine incident. Your initial assessment should therefore be as brief as possible, yet still eliminate obvious false positives.

Communicate the Incident

When you suspect that there is a security incident, you should quickly communicate the breach to the rest of the core CSIRT. The incident lead, along with the rest of the team, should quickly identify who needs to be contacted outside of the core CSIRT. This approach will help to ensure that appropriate control and coordination of the incident can be maintained, while minimizing the extent of the damage.

Be aware that damage can come in many forms, and that a headline in the newspaper describing a security breach can be much more destructive than many system intrusions. For this reason, and to prevent an attacker from being tipped off, only those playing a role in the incident response should be informed until the incident is properly controlled. Based on the unique situation, your team will later determine who needs to be informed of the incident—anyone from specific individuals up to the entire company and external customers.

Contain the Damage and Minimize the Risk

By acting quickly to reduce the actual and potential effects of an attack, you can make the difference between a minor and a major one. RFC 2196 defines a series of priorities for containing damage in your environment. The exact response will depend on your organization and the nature of the attack that you face. However, the following priorities are suggested as a starting point:

1. Protect human life and people's safety. This priority should always be your first priority.

2. Protect classified and sensitive data. As part of your planning for incident response, you should clearly define which data is classified and which is sensitive. This information will allow you to prioritize your responses in protecting the data.

3. Protect other data , including proprietary , scientific , and managerial data. Other data in your environment may still be of great value. You should act to protect the most valuable data first before moving on to other, less useful data.

4. Protect hardware and software against attack. This priority includes protecting against loss or alteration of system files and physical damage to hardware. Damage to computers can result in costly downtime.

5. Minimize disruption of computing resources (including processes). Although uptime is very important in most environments, keeping computers up during an attack may result in greater problems later on. For this reason, minimizing disruption of computing resources should generally be a relatively low priority.

There are a number of measures that you can take to contain the damage and minimize the risk to your environment. As a minimum, you should:

• Try to avoid letting attackers know that you are aware of their activities. This measure can be difficult to accomplish, because some essential responses may alert attackers. For example, if there is an emergency meeting of the CSIRT, or you require an immediate change of all passwords, any internal attackers may know that you are aware of an incident.

• Compare the cost of taking the compromised and related computers offline against the risk of continuing operations. In the vast majority of cases, you should immediately take the computer off the network. However, there may be service agreements in place that may require keeping computers available, even if there is the possibility of additional damage. Under these circumstances, you may choose to keep a computer online with limited connectivity to gather additional evidence during an ongoing attack.

  In some cases, the damage and scope of an incident may be so extensive that you may have to take action that invokes the penalty clauses specified in your service level agreements. In any case, it is very important that the actions that you will take in the event of an incident are discussed in advance and outlined in your response plan so that immediate action can be taken when an attack occurs.

• Determine the access point or points used by the attacker and implement measures to prevent future access. Measures may include disabling a modem, adding access control entries to a router or firewall, or increasing physical security measures.

• Consider rebuilding a fresh computer with new hard disks (the existing hard disks should be removed and put in storage because they may be used as evidence if you decide to prosecute attackers). Ensure that you change any local passwords. You should also change administrative and service account passwords elsewhere in your environment.

Identify the Severity of the Compromise

To be able to recover effectively from an attack, you need to determine how seriously your computers have been compromised. This assessment will determine how to contain and minimize the risk, how to recover, how quickly and to whom you communicate the incident, and whether to seek legal redress.

You should attempt to:

• Determine the nature of the attack (this determination may be different from what the initial assessment suggests).

• Determine the attack point of origin.

• Determine the intent of the attack. Was the attack specifically directed at your organization to acquire specific information, or was it a random attack?

• Identify the computers that have been compromised.

• Identify the files that have been accessed and determine the sensitivity of those files.

By performing these actions, you will be able to determine the appropriate responses for your environment. A good incident response plan will outline specific procedures to follow as you learn more about the attack. Generally, the nature of the attack symptoms will determine the order in which you follow the procedures defined in your plan. As time is crucial, less time consuming procedures should generally be followed before more lengthy ones. To help determine the severity of the compromise, you should:

• Contact other members of the response team to inform them of your findings, have them verify your results, determine whether they are aware of related or other potential attack activity, and help identify whether the incident is a false positive. In some cases, what appeared to be a genuine incident on initial assessment will prove to be a false positive.

• Determine whether unauthorized hardware has been attached to the network or whether there are any signs of unauthorized access through the compromise of physical security controls.

• Examine key groups (Domain Administrators, Administrators, and so on) for unauthorized entries.

• Search for security assessment or exploitation software. Cracking utilities are often found on compromised computers during evidence gathering.

• Look for unauthorized processes or applications currently running or configured to run using the startup folders or registry entries.

• Search for gaps in, or absence of, system logs.

• Review intrusion detection system logs for signs of intrusion, which computers may have been affected, methods of attack, time and length of attack, and the overall extent of potential damage.

• Examine other log files for the following: unusual connections; security audit failures; unusual security audit successes; failed logon attempts; attempts to log on to default accounts; activity during nonworking hours; file, directory, and share permission changes; and elevated or changed user permissions.

• Compare computers to previously conducted file/system integrity checks. This comparison allows you to identify additions, deletions, modifications, and permission and control modifications to the file system and registry. A great deal of time can be saved when responding to incidents if you identify exactly what has been compromised and what areas need to be recovered.

• Search for sensitive data such as credit card numbers and employee or customer data that may have been moved or hidden for future retrieval or modifications. Computers may also need to be checked for nonbusiness data, illegal copies of software, and e-mail or other records that may assist in an investigation. If there is a possibility of violating privacy or other laws by searching on a computer for investigative purposes, you should contact your legal department before you proceed.

• Match the performance of suspected computers against their baseline performance levels. Of course, this task presupposes that baselines have been created and properly updated. For more information about creating a performance baseline, see Chapter 27 of the Windows 2000 Professional Resource Kit, "Overview of Performance Monitoring" (Microsoft Press, ISBN: 1-57231-808-2).

When determining which computers have been compromised and how, you will generally be comparing your computers against a previously recorded baseline of the same computer before it was compromised. Assuming that a recent computer snapshot is sufficient for comparison may put you in a difficult situation if the previous snapshot comes from a computer that has already been attacked.

Note Tools such as EventCombMT, DumpEL, and Microsoft Operations Manager (MOM) can help you determine how much a computer has been attacked. Third-party intrusion detection systems give advance warning of attacks, and other tools will show file changes on your computers. For more information about these tools, see Chapter 9, "Auditing and Intrusion Detection."

Protect Evidence

In many cases, if your environment has been deliberately attacked, you will want to take legal action against the perpetrators. If you are going to take such action, you will need to gather evidence that can be used against them. It is extremely important to back up the compromised computers as soon as possible, and prior to performing any actions that could affect data integrity on the original media.

You should get someone skilled in computer forensics to make at least two complete bit-for-bit backups of the entire computer using new, never-before-used media. At least one backup should be on a write-once, read-many media such as a CD-R or DVD-R. This backup should only be used for prosecution of the offender and should be physically secured until needed.

The other backup may be used for data recovery. These backups should not be accessed except for legal purposes, so you should physically secure them. You will also need to document information about the backups, such as who backed up the computers, at what time, how they were secured, and anyone who had access to them.

After the backups are performed, you should remove the original hard disks and store them in a physically secure location. These hard disks can be used as forensic evidence in the event of a prosecution. New hard disks should be used to restore the computer.

In some cases, the benefit of preserving data may not equal the cost of delaying the response and recovery of the computer. The costs and benefits of preserving data should be compared to those of faster recovery for each event.

For extremely large networks, comprehensive backups of all compromised computers may not be feasible. Instead you should back up all logs and selected, breached portions of the network.

If possible, back up system state data, as well. It may be months or years until prosecution takes place, so it is important to have as much detail of the incident archived for future use.

Often the most difficult legal aspect of prosecuting a cyber crime is collecting evidence in a manner acceptable to the particular jurisdiction’s laws of evidence submission. Hence, the most critical component to the forensic process is detailed and complete documentation of how computers were handled, by whom, and when, to demonstrate reliable evidence. Sign and date every page of the documentation.

After you have working, verified backups, you can wipe the infected computers and rebuild them. This action will get you back up and running quickly. It is the backups that provide the critical, untainted evidence required for prosecution. A different backup than the forensic backup should be used to restore data.

Notify External Agencies

After the incident has been contained and data preserved for potential prosecution, you will need to start notifying appropriate external entities. Potential agencies include local and national law enforcement, external security agencies, and virus experts. External agencies can provide technical assistance, offering faster resolution and providing information learned from similar incidents to help you fully recover from the incident and prevent it from occurring in the future.

For particular industries and types of breaches, it may be necessary to specifically notify customers and the general public, particularly if customers may be affected directly by the incident.

If the event caused substantial financial impact, you may need to report the incident to law enforcement agencies.

For higher profile companies and incidents, there may be media involvement. Although media attention to a security incident is never desirable, it is often unavoidable and allows the organization to take a proactive stance in communicating the incident. At a minimum, the incident response procedures should clearly define the individuals authorized to speak to media representatives.

Typically, these will be people in the public relations group within your organization. You should not attempt to deny to the media that an incident has occurred, because doing so is likely to damage your reputation more than proactive admission and visible responses ever will. However, the media does not need to be notified for each and every incident regardless of its nature or severity. You should assess the appropriate media response on a case-by-case basis.

Recover Systems

How you recover your system will generally depend on the extent of the security breach. You will need to determine whether you can restore the existing computer while leaving intact as much as possible, or if it is necessary to completely rebuild the computer.

Restoring data presumes, of course, that you have clean backups—backups made before the incident occurred. File integrity software can help pinpoint the first occurrence of damage. If the software alerts you to a changed file, then you know that the backup you made just before the alert is a good one and should be preserved for use when rebuilding the compromised computer.

An incident could potentially corrupt data for many months prior to discovery. It is therefore very important that as part of your incident response process, you determine the duration of the incident. (File/system integrity software and intrusion detection systems can assist you in this determination.) In some cases, the latest or even several prior backups may not be long enough to get to a clean state, so you would be advised to regularly archive data backups in a secure off-site location.

Compile and Organize Incident Evidence

The CSIRT should thoroughly document all processes when dealing with any incident. This documentation should include a description of the breach and details of each action taken (who took the action, when they took it, and the reasoning behind it). All people involved with access must be noted throughout the response process.

Afterwards, the documentation should be chronologically organized, checked for completeness, and signed and reviewed with management and legal representatives. You will also need to safeguard the evidence collected in the protect evidence phase. You should consider having two people present during all phases who can sign off on each step, which will help reduce the likelihood of evidence being inadmissible and the possibility of evidence being modified after the fact.

Remember that the offender may be an employee, contractor, temporary employee, or other insider within your organization. Without thorough, detailed documentation, identifying an inside offender will be very difficult. Proper documentation also gives you the best chance of prosecuting offenders.

Assess Incident Damage and Cost

When determining the damage to your organization, you should consider both direct and indirect costs. These costs could include:

• Costs due to the loss of competitive edge from the release of proprietary or sensitive information.

• Legal costs.

• Labor costs to analyze the breaches, reinstall software, and recover data.

• Costs relating to computer downtime (for example, lost employee productivity, lost sales, replacement of hardware, software, and other property).

• Costs relating to repairing and possibly updating damaged or ineffective physical security measures (locks, walls, cages, and so on).

• Other consequential damages such as loss of reputation or customer trust.

Review Response and Update Policies

When the documentation and recovery phases are complete, you should review the process thoroughly. Determine with your team the steps that were executed successfully and what mistakes were made. In almost all cases, you will find that your processes need to be modified to allow you to handle incidents better in the future.

You will find weaknesses in your incident response plan, which is the point of this post-mortem exercise—you are looking for opportunities for improvement, which should initiate a whole new round of the incident response planning process.

Scenario—Contoso Incident Handling

To see how the different stages of incident response should work when dealing with an attack, the following case study has been designed, showing the response of the Contoso CSIRT team to an infection of the Code Red II virus. Although this case study is fictional, the measures taken closely mirror those taken by real organizations in attacks.

Table 10.3  Contoso Case Study

Summary

Much of this chapter has dealt with measures that you can take to minimize the risk of being attacked. However, organizations have the most success at reaching their security goals when they do everything that they can to minimize their chances of attack, and then plan what they will do when they are attacked. Part of this process is to audit carefully for attack, which is covered in Chapter 9, "Auditing and Intrusion Detection." Another equally important part is to have a defined, well rehearsed set of responses that you can put into place if a successful attack does occur.

Related Topics

Hacking Exposed Windows 2000 by Joel Scambray and Stuart McClure (McGraw-Hill Professional Publishing, ISBN: 0072192623).

The Computer Security Institute (www.gocsi.com) releases an annual study called the Computer Crime and Security Survey.

More Information

For information about incident response, see the Forum of Incident Response and Security Teams (FIRST) Web site at www.first.org.

Incident Response: Investigating Computer Crime by Chris Prosise and Kevin Mandia (McGraw-Hill Professional Publishing, ISBN: 0072131829).

The Internet Security Guidebook: From Planning to Deployment by Juanita Ellis, Tim Speed, William P. Crowell (Academic Pr, ISBN: 0122374711).

For more information about developing computer security policies and procedures for sites that have computers on the Internet, see RFC 2196, at www.ietf.org/rfc/rfc2196.txt?number=2196.

For more information about performance monitoring, see Chapter 27, "Overview of Performance Monitoring," in the Windows 2000 Professional Resource Kit at www.microsoft.com/resources/documentation/
windows/2000/professional/reskit/en-us/part6/proch27.mspx.

For more information about CERT, see the CERT Coordination Center at www.cert.org.

Download

Get the Securing Windows 2000 Server

Solution Accelerator Notifications

Sign up to stay informed

Feedback

Send us your comments or suggestions