IT Value Card

Published: October 26, 2004

This card describes how Microsoft Information Technology (Microsoft IT) uses Microsoft® Systems Management Server (SMS) 2003 to manage the process of distributing applications, to manage hardware and software assets, and to manage the deployment of security updates and other software updates across the enterprise.

Executive Overview

Microsoft IT uses SMS to manage 165,000 computers worldwide. Microsoft IT ensures that nearly 90,000 users in more than 400 Microsoft locations around the world are able to access the corporate network 24 hours a day, seven days a week.

Challenges in patch management for the desktop computer environment at Microsoft that are common to many organizations are:

• The number of software updates has increased as efforts to maintain security have increased.

• Efficient enterprise patch management requires automation.

• There is a need to reduce the amount of employee downtime caused by installing software updates.

• Inventories of software updates must be thorough and efficient.

There are aspects of the Microsoft work environment that are not typical of a large-scale enterprise. Some of the unique aspects of the desktop environment at Microsoft include:

• Users maintain administrative rights, have diverse desktop implementations, and may have multiple desktop computers.

• A diverse mix of approved software versions exists because groups do not unilaterally install software immediately. Groups have much autonomy to choose the software versions that best fit their work models.

• To test software functionality, many employees frequently rebuild their systems completely, sometimes daily.

• Microsoft operates in an extremely active and challenging security environment. Challenges include:

  • Each month, Microsoft experiences approximately 100,000 intrusion attempts.

  • Each month, Microsoft probes, scans, and quarantines more than 125,000 virus-infected e-mail messages.

Benefits

• More efficient deployments of software updates. Elimination of custom scripting has shortened the packaging of updates from 5-10 business days to no more than four business days—and usually within one day. Elimination of custom scripting not only allows for more efficient deployments, but also saves work hours of testers and packagers.

• Reduction of unplanned downtime. Users can choose the best time to install a software update within a set grace period.

• Improved inventory capability. Knowing what systems exist is a proactive approach to providing security for the corporate network.

SMS Desktop Patch Management Architecture

All Microsoft IT SMS clients run SMS 2003 Advanced Client and all SMS site servers use Advanced Security. Advanced Security means that the SMS service on the SMS servers runs in the local system context rather than using a service account. In addition, the SMS service communicates from server to server by using the computer account rather than a connection account, making SMS more secure. Advanced Security also makes SMS easier to administer because no account maintenance is required.

Microsoft IT implemented SMS 2003 on Windows Server 2003 and SQL Server 2000 SP3a with the following configuration:

• One central site server. This is a high capacity, high-throughput server with 8 processors and 8 gigabytes (GB) of random access memory (RAM).

• 22 primary site servers. These are four-processor, 4-GB servers. In the headquarters data center, there is one cluster balanced through Network Load Balancing.

• 30 dedicated secondary site servers. These are two-processor, 2-GB servers.

• 100 shared secondary site servers. SMS 2003 runs on approximately 100 secondary site servers that are shared with other services, such as file and print services.

Microsoft IT examined patching requirements at Microsoft and decided to create separate SMS 2003 infrastructures—one to update servers, and one to update desktop computers. Microsoft IT based this decision on the following factors.

• Security updates are more critical for servers than for desktop computers because servers affect the security and workflow of large groups of workers. Microsoft IT determined that it could more easily meet the short time frame for updating servers if it did not have to share the infrastructure for updating servers with the resources and sustainer functions that are regularly running for managing desktop computers.

• The software platform baseline for servers at Microsoft is uniform and unilaterally enforced, whereas desktop computers run a wide variety of software versions and service pack levels.

Patch Deployment

Roles

The Microsoft IT patching process is a collaborative effort that involves several types of Microsoft personnel:

• Corporate Security analyst. Reviews the bulletins posted by the Microsoft Security Response Center (MSRC), analyzes the information supplied by the Corporate Security team, recommends the enforcement dates, and facilitates the flow of information between Corporate Security and the desktop patching team.

• Client Support security program manager. Relays information from the security analyst to the SMS patch administrator and tracks the progress of deployment.

• SMS patch administrator. Creates and prepares the deployment package for a software update, and then distributes the update. Other duties include testing updates, rescanning the environment to assess success, and providing custom reports to interested parties.

Risk Management Approach

To determine the need to deploy security updates and Microsoft Office updates to desktop computers throughout the organization, Microsoft IT must first determine the risks that vulnerabilities pose.

By using a combination of MBSA 1.2—included in SMS 2003—and an internally developed tool, Corporate Security continuously scans the entire desktop computer environment to monitor and ensure consistent and timely installation of software updates for operating systems and applications. Measuring the environment allows Microsoft IT to create and maintain a baseline of systems in the environment.

If Corporate Security identifies a vulnerability, it assesses the risk of the issue that the software update is intended to correct. The MSRC security bulletin assigns a priority rating based on the average needs of Microsoft customers. The Corporate Security analyst evaluates the update according to the Microsoft environment and adjusts the priority level, which then determines how quickly an update will pass through the change process.

Microsoft IT uses the criteria shown the following table to assess the priority of the software-update request for the Microsoft desktop environment relative to the update's original risk assessment from the MSRC.

The Corporate Security analyst sends an e-mail message to the Client Support security program manager with a recommendation of an enforcement date, based on the priority level. The security program manager then submits a request to the SMS patch administrator, stating the updates required, the clients and operating systems affected, and the completed enforcement dates.

Process of Deploying Software Updates

1. Windows Update pushes the update to client desktops, and users of those clients are requested to install the update.

2. E-mail and Web notifications are sent to clients.

3. After a particular deadline, SMS forces the installation of the update.

4. An internal tool scans clients, and the network ports of noncompliant computers are disabled.

Users install updates on 70 percent of computers before SMS forces the installation. There is a 1 percent client vulnerability rate two weeks after deployment of an update.

Lessons Learned and Best Practices

SMS 2003 offers IT administrators a powerful toolset to manage the deployment of software updates to desktop computers more efficiently for both Office and Windows. Microsoft IT realized, however, that effective and efficient patch management requires a balanced interdependence among three primary elements: processes, people, and technology.

Processes

• Formulate and enforce security guidelines

• Formulate an overall patch management process

• Baseline the environment

• Test the impact of the update

• Manage the impact of deploying updates to clients

• Consolidate Critical Updates That Have Passed Their Enforcement Periods

• Deploy updates in a logical order

People

• Staff appropriately

• Appoint person or committee to prioritize software updates

Technology

• Use software update features for troubleshooting

• Use the Security Update Inventory Tool and scanning tools

• Use the Office Update Inventory Tool

• Use Source Path Update Management for Office Updates

• Slipstream and tailgate routine updates into new computer builds

• Consolidate updates into service packs

• Implement SMS 2003 Advanced Client throughout the enterprise

• Streamline the production environment

Global Microsoft IT Environment

The Microsoft enterprise is large, complex, and constantly changing. The mission of the Microsoft IT group is fairly unique. In addition to running a world-class utility that keeps the business productive, its primary mission is to be Microsoft's first and best customer. This involves testing all enterprise software in the early stages of beta development by deploying it throughout the company, providing valuable feedback to product groups to ensure predictable and trustworthy services for customers, clients, and partners. The following data gives some idea of the environment in which this all occurs (numbers are approximate):

• Nearly 90,000 users of IT

• More than 300,000 computers and devices

• More than 400 sites supported worldwide

• Global line-of-business (LOB) applications (for example, Siebel, Clarify, MS Sales, and World-Wide Sales and Marketing Database)

• Global virtual Help Desk

• Seven sites running Microsoft Exchange globally

• 110 servers running Exchange

• 38 mailbox servers

• More than 3 million internal e-mail messages per day

• More than 8.8 million external e-mail messages per day

• More than 6.8 million e-mail messages blocked per day

• More than 7.5 million remote connections per month

For more information on Systems Management Server 2003, go to http://www.microsoft.com/smserver/default.mspx