Oh Patch How I Hate Thee; Let Me Count the Ways

Jesper M. Johansson, Ph.D., CISSP, MCSE, MCP+I

Security Program Manager
Microsoft Corporation

See other Security Management columns .

Welcome back to the article series on security management. In this article we will discuss patches, dispel some of the mystery about them, and tell you how they will impact your security. When I first started writing this column I did not intend to talk about patches, but when I asked around, everyone I spoke to said I have to have one column on it. It is such an important topic that I could not ignore it. To that end, this is the patch column. It is a bit longer than what I would normally write because I am hoping I will not have to write another one. Unless something significant changes in the patch management world, I plan not to. Therefore, this is a rather lengthy column. Future columns will likely be shorter. As always, however, this column is for you, not me. If there is something you want me to cover, click the “Comments” link at the bottom of the page and tell me.

On This Page

Introduction
What Is A Patch?
Security Update Testing
Tools to Manage Security Updates
Advanced Tips and Tricks
Conclusion

Introduction

If the thing security administrators hate most is criminal hackers, the thing they hate second-most has to be patches. (Henceforth, when I say “patch” I primarily refer to security patches.) Nothing rouses emotions quite so much as patches. Everyone dislikes them, from the developers that have to create them, to the software vendors that have to release them, to the administrators that have to deploy them. Quite possibly, there are only three groups that like patches: the software vendors that sell patch management solutions, the consultants who deploy them, and the security researchers who use vulnerability discoveries as marketing tools.

I hate patches too. Patches disrupt the normal work flow and make us do maintenance work. Nobody likes fixing things, particularly not when it is not evident that they are broken. The problem is that if we do not patch our networks now, attackers will often demonstrate with ample clarity just how broken they in fact are.

A perfect patch would be one that is applied completely silently, that does not cause a moment’s down time, and that does not break any of your applications. At this point, it is also just a dream. Patches are issued to resolve particular vulnerabilities that attackers could exploit. As such, they are typically issued on a much tighter schedule than a typical product release. There simply is no time for a beta program or extensive compatibility testing. Patches cannot be as thoroughly tested as a full product release. This means that occasionally, patches can cause problems. Software is by far the most complex thing mankind has ever created. There is nothing else we have ever done that has such interrelationships and far reaching consequences. A small, seemingly innocuous, change in one component, could very well introduce problems for some other piece of software. This sometimes happens with patches just like with any other software upgrade. It is unfortunate, but the truth.

The real solution is in preventing having to issue patches in the first place. This involves both you as the security administrator, and your software vendor. In future columns, we will look at what you can do to protect your network and hopefully have to install fewer patches. On the vendor side, the major thrust of Microsoft’s Trustworthy Computing Initiative is directed toward improving software quality and enabling simpler, protected implementation. These efforts are actually paying off. Windows Server 2003 has far fewer patches associated with it than Windows 2000, and they are lower in severity. In the first 180 days after the release of Windows 2000, 32 patches were released, whereof 5 were of critical severity. Windows Server 2003 had 13 patches in the same period, with only two critical. However, this does not help you if you cannot yet implement Windows Server 2003 for some reason, and besides, even one patch is one too many. Even one patch can cause instability and disrupt your work.

Because of the potential for patches to cause problems, I often hear people say that they will not install patches because they cause instability. Well, keep in mind why patches are issued in the first place – to prevent exploitation of some security issue. If you think patching makes a system unstable, what do you think happens when that system gets hacked? Most criminal hackers are not very good system administrators. Relying on them to keep your systems stable is a losing proposition from the start. That’s not to mention what it takes to clean them out of your system. In the next article, we will look at what you need to do if you think you have been hacked.

What Is A Patch?

What is a patch? Well, actually, it could mean a number of things. Microsoft has a very specific nomenclature around patches. For more information, see Microsoft Knowledge Base article 824684. When security administrators refer to a “patch” what we usually mean is actually a “security update”. There are other types of patches as well, that have nothing to do with security. However, as this is a security column, we are primarily interested in security updates.

The official definition of a security update is a broadly released fix for some kind of security issue that can be used in an exploit of a system – a security issue that has been found to be used in an exploit, has already been made public, or is about to be made public shortly. The first category of security updates is the worst. These are so-called “zero-day” vulnerabilities of vulnerabilities where the vendor was never notified of the problem before an attacker started using it. Fortunately, these are rare, rarer in the Windows world than in the Unix/Linux world, but rare nevertheless. To my knowledge, there have only ever been less than a handful of these on the Windows platform. The second category comprises problems that were posted to some public forum without first notifying the vendor. In a few instances, such problems have been exploited by would-be criminals before the vendor gets a chance to address them. This has given rise to a call for “responsible disclosure,” whereby the vendor gets a chance

to address the problem before the finder makes the findings public. Unfortunately, some application security analysts, for whatever reason, decide to play outside these proposed societal norms of good behavior; putting customers at risk and making the vendor rush out security updates which may not be of the same quality as a carefully developed and tested service pack. Most of the security updates which have been recalled and released have fallen into this category. By contrast, the third category of security updates is where the finder reports the problem to the vendor and then gives the vendor sufficient time to address the issue properly before reporting the findings publicly.

One variant conspicuously absent from this list is fixes for vulnerabilities found by the vendor. In rare instances, where such a problem is believed to be externally known, or where the likelihood of discovery is high, a vendor may release a security update for an internally found security issue. Often these internally discovered vulnerabilities are discovered while producing a security update for an externally discovered security issue in the same component. In these cases, the security update will usually fix all the problems. However, for some internally found vulnerabilities the fix is held for a service pack. Service packs are generally much better tested than security updates, and are much less likely to cause stability problems. In addition, uptake on service packs is generally much higher than for security updates, as many customers only install service packs. As long as the risk to customers is deemed low, it is usually desirable to hold the fix until a service pack. Many times, I have had customers argue that vendors should make public all the information about a security update, and then let the customer make the case for whether to apply the fix or not. However, this argument does not hold water once you consider that the Heisenberg Uncertainty Principle applies here. By measuring something, in this case, risk, you change it. What is risk? Risk is a relationship:

Risk = Damage Potential * Likelihood of Damage

In order for customers to be able to ascertain the risk, the vendor has to disclose information about the problem. Since this information is then made public the people who would use that vulnerability to attack systems also get it. This increases the likelihood that they will discover exactly how to exploit the problem, thus increasing the risk. As I mentioned before, to date only an extremely small fraction of vulnerabilities on the Windows platform have been exploited before the vendor released a security update. This lends credence to the argument that the more information is available about a security issue, the higher the likelihood that potential criminals will be able to exploit the problem. This may change going forward, but to date, it still holds.

It is important to keep in mind that vendors do not issue security updates to make your life difficult. While it costs customers significant amounts of money to manage security updates, it also costs the vendor a lot, both in terms of money and in terms of credibility and trust. This means that the release of a security update is not to be taken lightly. Any time a security update is released, we need to immediately start evaluating whether to apply it and when. This is just a part of our job, and if you are facing difficulties getting resources to do that, you need to work on convincing your management that this is a fact of life in security management. In a future column, I plan to cover how to “sell” security to management. If you have great suggestions on how to do that, please let me know. You can give feedback by clicking the “Contact Us” link at the bottom of this page and then send feedback to TechNet, including a link to this page in the message. As a field, we are not great at selling security to management, and if you know a way to do it and do it well, please let me know.

To assist you in determining how soon you need to deploy a security update, some vendors, including Microsoft, apply a risk rating to individual vulnerabilities in a security update. Microsoft’s ratings, and the associated recommendations for how fast to deploy them, are shown in this table.

There are three ways security updates are shipped. The first is with a Security Bulletin as a fix to one or more vulnerabilities in a single component. Security Bulletins are posted at https://www.microsoft.com/technet/security/current.aspx. If you have not already done so, you should register to be notified of security bulletins at https://www.microsoft.com/technet/security/bulletin/notify.mspx.

The second way to receive a security update is through an update rollup. An update rollup is simply a collection of security updates, possibly with other updates included. Often the update published with a Security Bulletin is actually an update rollup. For example, all security updates for Internet Information Services (IIS) are now rollups, including fixes for all previous problems as well.

The third way to get security updates is with a Service Pack. A service pack is a tested, cumulative set of all hotfixes, security updates, critical updates, and other fixes. A service pack may also include security fixes for issues found internally to Microsoft during the production of the service pack. When a service pack is published, all the fixes included, including ones found internally to Microsoft, are documented in a Knowledge Base article.

All things being equal, service packs are preferable to security updates since they are better tested and have gone through a wider beta program. However, security updates also go through significant testing.

Security Update Testing

Every time someone tells us to install security updates, they will also end with “and be sure to test this in your environment prior to rolling it out in production.” Security updates have a bad reputation, not entirely deservedly so, for breaking things. The fact of the matter is that the vast majority of installations see no stability issues with the majority of security updates. However, for the few that do, the impact can be disastrous. Let’s do some math. If an update is installed on 20 million computers (not a very high number on a widely distributed platform like Windows) and 0.01% of those computers had problems with the update, we have 2,000 machines that had problems. That number explains why we often hear about problems with security updates. Even if a problem only affects a very small fraction of systems, it is enough to generate media attention.

Occasionally, there are intrinsic problems in security updates but these are relatively rare. The majority of problems experienced with security updates are due to third-party applications or modifications to default configuration settings. This is the reason behind the “test in your environment” comment. Given the extremely heterogeneous environments computer software is deployed in, it is impossible to include all permutations in a test pass. For example, just because a security update did not cause problems in testing, does that mean it will not cause problems when applied to a gas pump running Windows NT 4.0 Workstation? How about a cash register running Windows XP Professional? It is impossible to tell. However, there are some general rules of thumb that can help reduce your test pass.

It is a good chance that all security updates are tested with common sets of configurations. For example, it is usually safe to assume that Windows operating system (OS) security updates, and service packs in particular, are tested with supported versions of Microsoft Office and vice versa. For application security updates, it is safe to assume that it has been tested on all the platforms the application runs on. It is a good bet too, that the update has been tested and will work correctly with hardware listed on the Windows Hardware Compatibility List – hardware that displays the Windows logo for the OS the update applies to.

However, what about other, less widely used, third-party software? Are SQL Server security updates tested on a system that also runs SAP, PeopleSoft, or Siebel? Are IIS security updates tested with that really cool shareware doodad that makes your server play Yankee-Doodle Dandy through the PC speaker when it is rebooted? Generally speaking, it is up to the third-party vendor (ISV) to test security updates with their product. In some cases, the vendor has a specific patch policy that states that they will only support their product if it is running on a platform configured a particular way, including service pack level. In this latter case, you risk voiding your support contract if you install patches. What to do?

The course of action would depend on the security update in question. For a critical security update, you have four options:

1. Follow the ISV’s recommendation and wait for them to accredit the update with their product, possibly running the risk of getting your network hacked in the meantime

2. Perform your own testing and if everything seems OK, install the update anyway, running the risk of losing your support options

3. Just install the update, with the same consequences as option 2

4. Press the ISV to speed up testing and accreditation, which runs the same risks as option 1, but may expose you for a shorter time window depending on the vendor’s response

The answer is not as clear as it may seem. Follow step 1, and you run a high probability of getting yourself hacked. Option 2 is a bit risky, just in case you missed anything. However, it is really the best option, particularly if you combine it with option 4. There are many ISVs who lag significantly behind in security updates. Quite frankly, if you were to only run some products on the platforms they are accredited for your operation will probably be taken down by hackers in short order.

There is a neat new way to get a near-perfect replica of a production system that you can use for testing. Microsoft Virtual PC 2004 includes the ability to make a new virtual disk based on an existing hard disk. Using this functionality you can install Virtual PC on a production system (mind you, probably not one that is currently serving clients) and make a replica of its disk. Then you can copy that disk image off to another computer. Now install Virtual PC on that other computer and create a new virtual machine pointing to the disk image you just created. On the first boot, you will need to remove Virtual PC from the virtual machine, and possibly install a few drivers. However, you can now set up the disk in undoable mode and have a copy of a production system on which you can test new security updates. Obviously, this does not work for all types of testing, such as testing with particular hardware which is not emulated by Virtual PC, but it works pretty well for testing interoperability with software. Cost-wise it is far more efficient than purchasing and managing additional hardware since Virtual PC only costs about $150.

Tools to Manage Security Updates

I cannot write a column on patch management without pointing out the tools available to you to help with patch management. Generally speaking, Microsoft does not charge for patch management tools, so except for where noted, everything mentioned here is free.

MBSA

The Microsoft Baseline Security Analyzer (MBSA) is Microsoft’s free solution for scanning systems for missing security updates and other potential vulnerabilities. It is available at https://www.microsoft.com/mbsa.

MBSA is a patch scanner, not a vulnerability scanner in the true sense of the word. The way I distinguish between the two is that a patch scanner focuses primarily on whether a security update is installed or not, not necessarily on whether a missing security update represents a real security issue. For example, if you have a web server running on Windows Server 2003, you may be notified that a security update is missing for a security issue in NetBIOS that could lead to information disclosure. However, since your web server sits behind a firewall and does not use NetBIOS, the fact that this security update is missing does not represent much of a problem. Beyond this distinction, a patch scanner also requires administrative privileges as well as particular services to be running on the target you are scanning. For example, MBSA requires the Server service running on the target you are scanning.

A true vulnerability scanner, such as the ISS Internet Scanner, focuses on actual security issues that an attacker could use to compromise a system. To that end, a vulnerability scanner typically does not require privileges on the target system. Rather, it evaluates that system from the attacker’s point of view.

MBSA version 1.2 was released on January 15, 2004, and adds the following new features to the previous version:

• Scans for additional products, such as Office

• Scans for Internet Connection Firewall state

• Scans for Automatic Update state

• Checks IE zone configuration

• Localized into French, German, and Japanese

End-user solution: Automatic Updates

For end users, the best security update is one they do not have to think about. The Automatic Update (AU) service of Windows Update (WU) was designed for that environment. While AU can be used to notify users that new Windows security updates are available, it can also be configured to automatically download and install those updates. This latter mode is the preferred solution as it also works when no administrators are logged on. Keep in mind, however, that if a security update requires a reboot, AU will automatically reboot the system. This quality makes AU singularly unsuitable for servers.

Small and Medium Network Solution: Software Update Services and AU

For really small networks, your best bet is probably to just configure AU on all the clients and then manually update the server(s). However, what if you have custom or rare applications that you need to test the updates against before you roll them out to the entire network? Software Update Services (SUS) is for you. SUS is a free download from Microsoft that basically gives you your own WU server. The difference is that while WU will present your users with all the available security updates, SUS will only give them the ones that you have blessed. AU normally scans a system against the security updates published on WU. However, if you have a SUS server you can configure AU to scan against it instead, ensuring that all your clients have all the OS updates you want and none other.

Enterprise Solution: Systems Management Server

If you already have an Enterprise Management System (EMS), such as Microsoft Systems Management Server (SMS), Tivoli, CA Unicenter, etc, in place, use that to distribute your security updates for you. There is a free feature pack for SMS that gives you the same flexibility with SMS that you have with SUS, with the added power of an industrial-strength EMS. Obviously, SMS still costs money, but once you have that infrastructure, the feature pack makes using it to distribute security updates much easier.

Which one to pick?

Generally speaking, if you have no other management solution in place, and you have less than about 500 clients to update, SUS and Auto Update is probably your best solution. If you have more than 500 clients, you should consider an enterprise management solution, such as SMS, as this will do more things for you than just updates. If you already have an EMS, investigate if you can’t use it to apply updates. If you have a very small network, with no centralized management need, consider using Auto Update configured to query Windows Update, which is the default. There is more information on how to pick a patch management solution at the Microsoft Web site. For more technical and in-depth guidance, refer to the Microsoft Solution for Patch Management.

Don’t forget your applications

SUS, AU and WU currently distribute operating system security updates. What about your applications? Microsoft is working on integrating the update distribution solutions, but at this time, updates for applications are distributed separately. Here are some of the places to look:

• Microsoft Office – https://officeupdate.microsoft.com

• Other Microsoft applications – Security Bulletin Search at https://www.microsoft.com/technet/security/current.aspx

• Third party applications – Look for the support site on their respective web sites

Advanced Tips and Tricks

So, what about your servers? All the solutions mentioned above are basically for clients. How do you update your servers? The first step is to find the updates. You can find all Microsoft security updates at the TechNet web site. Next, however, you need to apply them. There are some tricks for that. Please – keep in mind that these are advanced techniques. They should only be used by experts, and only after careful study and practice with the technique, the target application, and the update. These techniques are not for use by end-users.

Microsoft is working hard to build better update installers. One new (to Microsoft) technique is hot-patching, which updates running code in memory. Hot patching first replaces the actual executable on disk and then tries to update the running image of that executable. If this works, then no reboot of the system is necessary. Once the image is restarted, the updated on-disk version is used, with no reboot needed. This technique requires the user that installs the update to have the “Debug Programs” privilege. If you have removed that privilege from Administrators, for example, you need to grant it back at least to the accounts that will install updates.

Another technique being built into the new installers is warm patching. In this case, the update installer attempts to shut down the application(s) that is using the files in the update prior to installing it. This is important because often the reason a system has to be rebooted after patching is because the update updates files that are in use. By shutting down the service prior to patching it, you can avoid having to reboot the system after the update is installed. This is obviously faster than a full system reboot. However, warm and hot-patching is not available for all security updates, and does not work for some types of security updates. You may be able to manually recreate the same effect though.

Minimizing Reboots by Careful Inspection

One way to manually recreate the effect of a warm patch is to perform some careful investigation of the security update and then turn off the items that are using the files in that are being updated. For example, let us say you need to roll out an IIS security update on a large server farm. The first step is to find out which files the update updates. In most cases, the affected files are listed in the patch info section of the security bulletin. If they are not, you can find out by extracting the update and looking at it. You can do that by running the update executable with the /x switch. While you can also use a tool such as WinZip to unpack the update it will not work on some newer updates. Some recent packages have a special binary format and if you extract them using WinZip you will see some strange files, none of which look like a real binary. To be safe, always use the /x switch. Now you have a directory with the files in the update. Depending on the update type, you can ignore update.exe, update.inf and most of the rest of the files in the root directory of the update. Those are part of the update installer, and are not actually installed. The next step is to find out which applications are holding these files open. Start out by listing the files. Next download a copy of System Internals excellent Process Explorer (https://www.sysinternals.com). Process Explorer is much like the Windows Task Manager, on serious steroids! Process Explorer not only shows you the executables that are running. It shows you the handles they have open. A “handle” here means that the program holds some object open. For example, an IIS security update may include w3svc.dll. Click Find:Find DLL… and then type in w3svc.dll and hit enter. You will see inetinfo.exe listed. Now, double-click inetinfo.exe in the results window. That switches the main window into DLL view (you can manually change into DLL view by hitting CTRL+D), and shows you not only w3svc.dll, but all the DLLs loaded into the inetinfo.exe process. Double-click any of the DLLs and you get an enhanced properties dialog for it, which gives you a wealth of information about the file. When you are done playing with Process Explorer, write down all the processes that hold the binaries in the update open.

If you would rather have a command line tool for this, the built-in tasklist tool takes a /m switch which will render output with all modules loaded into all processes. Unfortunately, it does not print the modules for each process on separate lines so it is not easy to find all the processes that have a particular DLL loaded.

Now you need to decide whether you can avoid a reboot or not. Frankly, there is no easy answer here, but if you have a lot of servers to patch, it is well worth some testing. Generally speaking, if you can shut down any processes that use the files in the update, you can avoid the reboot. However, keep in mind that some services, such as Windows Management Instrumentation (WMI) will automatically restart if you try to just terminate them. Thus, any custom scripts you use to install the updates will have to either perform a graceful shutdown, which the service may prevent, or disable it first, and then terminate it. You can use the sc.exe command line tool to manage services in a script. Once you have figured out which services need to be stopped and how all that is left is to write a script that does some or all of the following:

• Disable service

• Stop the service

• Install the update, and do not forget to parse the return value of the update installer to ensure that it installed properly

• Re-enable the service

• Start the service

While this will still result in some amount of downtime, it is highly preferable to a full reboot.

In load-balanced environments you can usually take boxes off line one at a time during non-peak hours, patch them, and then re-enter them into service. In a cluster environment you can accomplish this by failing over services to other systems in the cluster temporarily. If you cannot do this for some reason, using the same technique outlined above will still help you minimize the time you are running without a full complement of systems though. Notice also that using clusters and load balancing to minimize downtime is relatively complex and needs to be carefully planned before execution.

You can also batch update installation so that you can install several updates at one time. This also helps avoid multiple reboots, and can minimize downtime. Most Windows security updates, and some other updates, take various command line switches that you can use in batch files. For example, the /z switch of a typical Windows security update means “do not reboot the system after patching.” This means that you can create a batch file that installs multiple security updates, all with the /z switch, and then reboot afterward.

Note that not all updates use this installer. For example, Windows Media security updates and some Internet Explorer security updates use a different syntax that does not implement the equivalent of the /z switch. If in doubt, run the update with the /? switch to check the syntax. The installer for Windows updates is also changing and adding functionality, so be sure to check the switches for each update before you include it in a batch file.

In some cases, several security updates update the same files. In this case, it is important to ensure that the installation order is correct. You can do that with the qchain tool. The tool is discussed in Knowledge Base article 296861. Generally speaking, however, qchain is not needed in fixes released after December 2002 since they include the functionality in qchain already. See the Knowledge Base article for more details.

Last, you can build custom installation sources that already have the security updates and service packs. This process, known as “slipstreaming” is not for the faint of heart and beyond the scope of this already long article, but more details are available in several Knowledge Base articles as well as the resource kit.

Conclusion

Patches are a necessary evil for system administrators. All systems require security updates to some extent, and managing security updates is a necessity. That does not mean that managing patches is a simple and pleasant experience. In this article, we cover basics of patch management, as well as some advanced techniques for minimizing downtime in a datacenter environment. The techniques you use for managing security updates are environment dependent, but the fact still remains that you must consider how to do so.

There are two parting thoughts I want to leave you with. The first is that everything in this article is subject to your organization’s information security policy. You cannot create a patch management strategy without having an information security policy to back it up. This is particularly important when it comes to protecting yourself after you had to take the servers down for patching, or if something went wrong. Having an information security policy signed by the CEO to point to in that case can be the difference between merely having to do extra work and having to start working on your resume instead.

Lastly, consider the impact of patch management on your security. Will having all the security updates make you secure? No. Security is ongoing process. Not installing security updates may significantly increase your chances of getting hacked. However, keeping up-to-date with security updates allows you to focus more on the security issues that are not due to vulnerabilities in the products you use. Think of it this way, once you have the security updates installed you can focus on the interesting and complex problems of managing the operational security of a network, at all the wonderful levels of complexity of security management. In the next set of articles, we will look at various aspects of this process.