Implementing System Center Operations Manager 2007 at Microsoft

Technical White Paper

Published: March 2, 2007
Updated: February 13, 2008

On This Page

	Executive Summary
	Introduction
	Microsoft IT Monitoring Solution
	Best Practices
	Conclusion
	For More Information

Executive Summary

Microsoft Information Technology (Microsoft IT) is responsible for monitoring the health of more than 10,500 servers that are running more than 2,300 corporate applications. Until the development of Microsoft® System Center Operations Manager 2007, Microsoft IT used several individual installments of Microsoft Operations Manager 2005 for monitoring. Using multiple installations of Operations Manager 2005 sometimes resulted in duplication of effort and redundant data. This technical white paper describes how Microsoft IT created a centralized monitoring solution by deploying Operations Manager 2007 in its corporate network environment.

This paper assumes that readers are technical decision makers and IT professionals who are already familiar with the Windows Server® operating system and Operations Manager 2005. It describes the planning, deployment, and operations of Operations Manager 2007 at Microsoft. Though this paper provides guidance through best practices based on Microsoft IT's early-adopter experiences, it is not intended to serve as an instructional guide. Each environment is unique, and each organization should consider its situation and needs before planning to deploy any product or solution.

Introduction

The Microsoft server environment consists of roughly 10,550 production servers, 3,990 pre-production servers, and another 50 development, staging, and test servers. Microsoft IT is responsible for monitoring all of them and the more than 1,500 line-of-business applications and 2,300 total applications that the servers host, many of which are critical to the operations of Microsoft. The previous solution, Operations Manager 2005, was a compilation of many autonomous management groups that were managed by several different internal support groups because of its lack of granular rights delegation. In addition, many application support teams and lab managers deployed their own Operations Manager 2005 infrastructures to monitor their applications and lab server hardware. At one point, more than 160 active Operations Manager 2005 management groups existed in the Microsoft IT environment.

To streamline operations and the infrastructure, Microsoft IT needed a centralized monitoring solution. With the deployment of Operations Manager 2007, Microsoft IT made a fundamental shift toward centralizing the way it provided monitoring services to organizational groups.

Challenges of Using Operations Manager 2005 for Monitoring

Although the Operations Manager 2005 implementation gave the individual teams substantial control and agility over their monitoring, it also presented several problems that Microsoft IT needed to consider during the design of the Operations Manager 2007 environment:

• There was a significant duplication of effort to provide fault monitoring. Each management group required both hardware and people to maintain the infrastructure. Multiple management groups often monitored the same asset. This meant that one failure could engage multiple teams.
• Even with all the autonomous management groups, gaps existed in the service offering. Most lab environments lacked hardware monitoring, and several applications were not being monitored.
• There was no comprehensive, holistic view of the health of the Microsoft IT network environment. Network, hardware, Microsoft SQL Server®, and application monitoring existed in separate systems.
• Many smaller groups depended on a single expert to manage their Operations Manager 2005 implementation. If that person left, the team often lost its monitoring expertise.

Benefits of Using System Center Operations Manager 2007 for Monitoring

The Operations Manager 2007 deployment provided several benefits for Microsoft IT:

• The introduction of role-based security and improvements to both scalability and Run As accounts removed the roadblocks that Microsoft IT previously had in using Operations Manager 2005 to deploy a centralized monitoring platform to be used by many teams within Microsoft. The centralized design enabled Microsoft IT to eliminate monitoring fragmentation and to better utilize resources.
• By using the Distributed Application Management Wizard, Microsoft IT has been able to rapidly develop management packs for custom applications.
• Microsoft IT invested a substantial amount of time and energy into creating and tuning custom management packs for Operations Manager 2005. Use of the management pack conversion tools in Operations Manager 2007 enabled Microsoft IT to retain and carry forward the work that the team had previously invested in its management packs.
• Operations Manager 2007 management packs are based on Service Modeling Language (SML), which is a part of the Dynamic Systems Initiative (DSI). As a result, Microsoft IT has been able to use existing management packs and develop new ones based on health models. Health models in turn use classes to provide a more dynamic, relational representation of the state of a system, service, or application.
• Operations Manager 2007 provides a more robust monitoring service offering due to improvements in product scale and support of clustering key infrastructure roles, as well as log shipping for disaster recovery of the Operational database.
• Operations Manager 2007 significantly reduced agent deployment efforts by using Microsoft Systems Management Server (SMS) packages. Agents can discover their configuration automatically from the Active Directory® directory service.
• The introduction of Operations Manager 2007 Audit Collection provided an efficient means of collecting and storing security event logs.
• Operations Manager 2007 provides a full software development kit (SDK) that provides Microsoft IT a complete interface to develop automation of administrative tasks.

Microsoft IT Monitoring Solution

Operations Manager 2007 gave Microsoft IT an opportunity to reassess its service offering and to completely rebuild its monitoring infrastructure. Based on the needs of the organization and the existing customers of the monitoring solution, Microsoft IT developed a new offering, running on a new infrastructure, that provides parity for the earlier service that was based on Operations Manager 2005.

The application has undergone significant improvements to many of the features introduced in previous versions of Operations Manager, and it offers a number of new features that improve its capabilities in areas such as security auditing, event log collection, and desktop monitoring.

To build and deploy the new monitoring solution running on Operations Manager 2007, Microsoft IT used the following phases:

1. Planning the Operations Manager 2007 environment
2. Designing the Operations Manager 2007 infrastructure
3. Deploying Operations Manager 2007

Planning the System Center Operations Manager 2007 Environment

Prior to any planning work, the engineers on the monitoring team within Microsoft IT held a series of meetings with representatives from their existing customer base as well as a set of prospective monitoring customers to discuss their requirements. The purpose of these meetings was to understand what each of these groups needed from the monitoring solution. The result of these meetings was a detailed list of requirements ranging from the functionality that absolutely must exist to those that were optional. From this list, Microsoft IT was able to begin the process of planning the deployment, as well as the aspects of the service offering, targeted toward meeting the requirements.

Informed by the requirements gathered, Microsoft IT approached the design of the Operations Manager 2007 environment with three main objectives:

• Provide a complete monitoring service offering.
• Provide a centralized and holistic monitoring platform.
• Reduce resources devoted to monitoring.

Provide a Complete Monitoring Service Offering

With the Operations Manager 2005 deployment, Microsoft IT had deployed monitoring agents only to production servers. Although this sufficed for the purposes of just monitoring the production aspects of the data centers, Microsoft IT's customers had become increasingly interested in expanding their monitoring to non-production systems. Because of that interest, Microsoft IT opted to expand the agent base with Operations Manager 2007 to all servers in Microsoft data centers. This expansion fulfilled customer requirements and served as the foundation for providing a complete monitoring service for the future.

Provide a Centralized and Holistic Monitoring Platform

With Operations Manager 2005, a number of different groups within Microsoft opted to deploy their own management groups to monitor their specific servers or applications. Although this gave individual support groups a substantial amount of control and agility over their monitoring needs, it required more resources than were necessary for monitoring. More importantly, it prevented Microsoft IT from viewing the health of the entire environment in a comprehensive manner.

With Operations Manager 2007, Microsoft IT planned, from the beginning, to design an infrastructure that would be capable of the necessary scale while at the same time providing flexibility and control to each support group. An additional benefit to this design is that all management packs would function together in the same management group, giving all users a holistic view of devices, applications, or services from hardware to software.

Reduce Resources Devoted to Monitoring

With the expansion of the scale and the comprehensiveness of the monitoring deployment, the Microsoft IT goal was to reduce the resources required to provide monitoring overall. The resource savings were made possible through hardware consolidation and centralized monitoring, which improved the productivity of the monitoring team.

The key focus of reducing hardware costs for Microsoft IT came in the form of consolidation. Early in the planning phases, through conversations with other teams within Microsoft, Microsoft IT determined that if it could provide an infrastructure capable of scaling appropriately and a service offering that was complete and flexible enough, the other teams running their own infrastructures would abandon their systems in favor of running their monitoring on a centrally managed platform.

In the area of reducing people resources, the focus was twofold:

• Freeing people from maintaining their own monitoring solution
• Using new Operations Manager 2007 features to reduce existing support costs

The first objective is really a byproduct of the consolidation. As support teams merge their monitoring into Microsoft IT's centralized monitoring platform, the people on those teams that have traditionally focused on monitoring will be able to spend more of their time focusing on their team's normal workload. Some examples of where the second objective has been possible are:

• Coupled with using SMS for deploying agents, Microsoft IT uses Active Directory integration so that agents can automatically discover their monitoring configuration when the agent health service starts.
• Microsoft IT is using an agent health remediation management pack to automate repairs on unhealthy agents.
• Microsoft IT is using the Operations Manager 2007 SDK as an interface for other forms of custom-built automation.

Designing the System Center Operations Manager 2007 Infrastructure

As with previous Operations Manager deployments, Microsoft IT built two distinct infrastructures, one for the pre-production environment and another for the production environment. The following sections describe the server roles, the details about each environment, the hardware specification that Microsoft IT developed for each server role, and the measures that Microsoft IT has implemented for redundancy and disaster recovery.

Server Roles in System Center Operations Manager 2007

The various Operations Manager 2007 server roles are:

• Operational database server. The Operational database is a Structured Query Language (SQL) database hosted on a Microsoft SQL Server 2005 Service Pack 1 (SP1) server. This database houses all of the configuration data for a management group and contains all data generated by the systems in the management group. The database must be online at all times for the management group to function.
• Root management server. The root management server is a new server role in the Operations Manager 2007 infrastructure. The root management server is responsible for hosting the SDK and configuration services and is the central coordinator of health monitoring. The SDK service is the single point in a management group that responds to SDK requests, including all operations consoles and connectors. The Config service is the source of authority for all configurations in a management group. All Operations Manager 2007 servers and agents ultimately get their configuration from the Config service.
• Management server. The management server role has not changed significantly from Operations Manager 2005. Management servers are responsible for managing agents, which includes sending configuration data to the agents, receiving operational data from the agents, and forwarding that data to the Operational database. In some instances, the management servers are also responsible for writing data to a data warehouse.
• Gateway server. The gateway server is a new role in Operations Manager 2007 that functions as a proxy between one or more management servers and a set of agents. In the presence of a firewall or the lack of a two-way trust between a set of agents and one or more management servers, a gateway server can be deployed to act as an intermediary.
• Audit collection server. Operations Manager 2007 introduces the Audit Collection feature set. These features focus specifically on collecting security events from monitored computers and storing that data in a central repository for alerting and forensics purposes. Three distinct roles are related to Operations Manager 2007 Audit Collection: the forwarder, the collector, and the database. The forwarder, a component of the Operations Manager 2007 agent, provides real-time monitoring of the security log on managed computers. The forwarder encrypts and transmits security events to the collector, a component of the Operations Manager 2007 management server. The collector makes security events available to Windows® Management Instrumentation (WMI) subscribers in real time and maintains that data in the audit collection database.
• Data warehouse and reporting server. The data warehouse and reporting server hosts a database that the reporting component of Operations Manager 2007 uses.

Pre-production Environment

The purpose of Microsoft IT's pre-production environment is to act as a complete (as possible), yet scaled-down version of the production environment. Microsoft IT uses the pre-production environment primarily for evaluation purposes. Evaluations range from working with the Operations Manager product group for testing new builds or features, to stressing product scale, to staging a proposed change prior to its official release into production. Figure 1 shows the design of a pre-production management group for Microsoft IT.

Figure 1. Management group in pre-production environment

The key aspects of the pre-production deployment are:

• The deployment consists of two management groups to enable Microsoft IT to set up a tiered infrastructure.
• The mid-tier management group manages all agents (approximately 3,500 agents at the time of this writing).
• The Operations Manager 2007 data warehouse and reporting are installed in the mid-tier management group.
• The agent base spans multiple Active Directory domains and forests.
• The majority of the agents managed in pre-production are production servers within Microsoft data centers.
• Agents are multi-homed to this management group and the production management group to enable them to send monitoring data to multiple locations.

Production Environment

As the name implies, the production environment provides live monitoring of Microsoft data-center servers. As shown in Figure 2, the Microsoft IT production environment consists of three management groups. Two of the management groups (shown on the left side of the figure) exist in the corporate intranet. The third management group (shown on the right side of the figure) exists to provide monitoring for systems in networks and domains on the perimeter network (also known as DMZ, demilitarized zone, and screened subnet).

Figure 2. System Center Operations Manager 2007 production environment

The key design aspects of the production environment are as follows:

• The environment consists of three management groups to allow for scalability and to span across diverse network and domain spaces.
• The management groups have been tiered so that all users can view alert data from a single console. The first management group is connected to the other two management groups to form these tiered relationships.
• Each management group manages a similar number of agents (totaling more than 10,500 agents at the time of this writing).
• Microsoft IT has deployed a single data warehouse, which receives operational data from all three management groups.
• The agent base spans multiple Active Directory domains and forests.
• Microsoft IT is using two 2-node clusters in each management group to host the roles of root management server and Operational database server—a cluster for each role. Clustering provides redundancy for these important server roles.
• Microsoft IT has deployed a failover database server into each management group. This SQL Server-based server is located remotely from the rest of the management group and contains the up-to-date contents of the Operational database shipped to it via log shipping.
• Each management group contains at least two management servers dedicated to agent management.
• In the third management group, gateway servers were deployed into remote network locations to manage agents in those spaces. Because the gateway servers communicate back to management servers in separate networks and separate forests, certificate-based authentication is used between the management servers and the gateway servers.

Hardware Specification for Infrastructure Servers

Table 1 lists the platform specifications that Microsoft IT is using for each server role. To determine hardware specifications, Microsoft IT started by studying the resource utilization of the Operations Manager 2005 servers in the production environment. Microsoft IT used the CPU, memory, and disk utilization statistics as a baseline and compared that baseline to the current hardware profiles that were available. After purchasing the appropriate hardware, Microsoft IT verified performance during beta testing. Microsoft IT's experiences required minor revisions to some disk configurations, but Microsoft IT discovered that, for the most part, the hardware originally procured was more than sufficient for Operations Manager 2007.

Table 1. Microsoft IT Hardware Specification for System Center Operations Manager 2007 Infrastructure

Server role	Number of servers per unit	Processor	Memory	Operating system	Disks*
Operational database server	Two per management group—Windows Clustering cluster	Two dual core	8 gigabytes (GB)	Windows Server 2003 with SP1, 64 bit	Storage area network (SAN) Windows Clustering Quorum: 5 GB SQL data: 130 GB Redundant array of independent disks (RAID) 10 SQL log: 10 GB RAID 5 TempDB: 10 GB RAID 5
Root management server	Two per management group—Windows Clustering cluster	Two dual core	16 GB	Windows Server 2003 with SP1, 64 bit	SAN Windows Clustering Root management server state drive: 5 GB
Failover database server	One per management group	One dual core	8 GB	Windows Server 2003 with SP1, 64 bit	SAN SQL data: 130 GB RAID 10 SQL log: 10 GB RAID 5 TempDB: 10 GB RAID 5
Operations Manager 2007 Audit Collection database	One per Operations Manager 2007 Audit Collection deployment	Two dual core	8 GB	Windows Server 2003 with SP1, 64 bit	SAN SQL data: 1 terabyte RAID 10 SQL log: 50 GB RAID 5 TempDB
Management server	Two per management group	One dual core	4 GB	Windows Server 2003 with SP1, 64 bit	Not applicable
Management Server and Operations Manager 2007 Audit Collection	One per Operations Manager 2007 Audit Collection deployment	One dual core	4 GB	Windows Server 2003 with SP1, 64 bit	Not applicable
Data warehouse	One per the entire deployment	Four dual core	16 GB	Windows Server 2003 with SP1, 64 bit	SAN SQL data: 400 GB RAID 10 SQL log: 30 GB RAID 10 TempDB: 30 GB RAID 10
Gateway server	Two per security-enhanced environment	Two dual core	4 GB	Windows Server 2003 with SP1, 64 bit	Not applicable

*All systems contain the following physical disks: operating system, 50 GB; Operations Manager installation drive, 18 GB.

Redundancy and Disaster Recovery

Microsoft IT included measures in the deployment design to provide tolerance for failures of either individual resources or major sections of the infrastructure. For the purposes of this white paper, redundancy and high availability relate to the measures taken for a single resource.

Disaster recovery relates to measures taken to ensure that operations can be resumed in the event of a catastrophic failure (for example, loss of the entire data center that hosts the primary infrastructure). Table 2 describes the approaches that Microsoft IT used for each significant role in the Operations Manager 2007 infrastructure.

Table 2. Microsoft IT's Redundancy and Disaster Recovery Approaches

Role	Redundancy/high availability	Disaster recovery
Management server	Approach: Deploy multiple management servers for agent failover. Description: In a management group, if one management server goes offline, the agents and gateway systems reporting to it will fail over to another management server in the management group. In Microsoft IT, agent failover behavior is defined in Active Directory, where agents can automatically discover it.	Approach: Deploy management servers in multiple locations. Description: If one site goes offline, the agent will fail over to the management server in another site, assuming that the site's failover configuration allows this. Microsoft IT is considering having management servers hosted on virtual machines in the failover location, as opposed to procuring more dedicated hardware.
Root management server	Approach: Cluster the root management server. Description: By using Windows Clustering, Microsoft IT has deployed the root management server role onto a two-node cluster to allow the underlying servers to go offline without having to take the role itself offline for a prolonged amount of time.	Approach: Promote a remote management server to root management server. Description: If the root manager server cluster becomes unavailable, Microsoft IT will promote one of its remaining management servers into the root management server role. Management servers are deployed in remote locations to account for a site-wide outage.
Gateway server	Approach: Deploy multiple gateway servers per location Description: The approach is the same as management servers, with two notable exceptions. First, multiple gateway servers need to be deployed in each distinct network/Active Directory space where the gateway's agents are located. Second, gateway servers need to have multiple management servers that they can communicate with to ensure that they themselves can fail over if necessary.	None: Because the gateway servers are located in the same physical segment as the agents, if the location is offline, the agents will be offline as well.
Operational database server	Approach: Cluster the SQL Server-based server that the Operational database is hosted on. Description: By using Windows Clustering, Microsoft IT has deployed the SQL Server-based server that the Operational database role is running on onto a two-node cluster to allow the underlying servers to go offline without having to take the role itself offline for a prolonged amount of time.	Approach: Implement log shipping to a remote SQL Server-based server. Description: A stand-alone SQL Server-based server has been deployed in a remote location and log shipping was set up to replicate data from the primary Operational database to the failover SQL Server-based server. If a disaster occurs, the management group can be manually pointed to use the failover database.
Operations Manager 2007 Audit Collection collector	None.	None.
Operations Manager 2007 Audit Collection database	None.	None.
Data warehouse	None.	Approach: Use database backups. Description: Scheduled periodic database backups with off-site storage are implemented. If a disaster occurs, the latest backup sequence is restored to a new server.

Deploying System Center Operations Manager 2007

Microsoft IT deployed Operations Manager 2007 in the following phases:

1. Conducting lab validation
2. Deploying the infrastructure
3. Deploying the data warehouse and reporting servers
4. Deploying management and gateway servers
5. Deploying Operations Manager 2007 Audit Collection components
6. Running workflows to create necessary objects in Active Directory for agent deployment
7. Deploying agents
8. Deploying management packs
9. Deploying the user interfaces (UIs)
10. Deploying additional integration

Conducting Lab Validation

Prior to deployment of any technology into the pre-production or production environment, Microsoft IT tests the technology for feature readiness and performance impact. The goals of lab validation for Operations Manager 2007 were to:

• Perform rudimentary tests against the application.
• Verify that the server and client components of this application would not adversely consume resources, such as CPU and memory, on test server systems.
• Verify that the build was suitable for production deployment (server and client components); Microsoft IT observed resource utilization with the application installed over at least one contiguous 72-hour period.
• Deploy a basic set of management packs and confirm that they did not adversely affect client or server resource utilization.

Deploying the Infrastructure

After the lab validation was completed, Microsoft IT set up the Operations Manager 2007 infrastructure based on the designs discussed earlier in this document. The following subsections discuss the steps taken to deploy the infrastructure.

Procure Hardware

The first step toward deploying the infrastructure was the procurement of the necessary hardware. Microsoft IT started by studying the resource utilization of the servers in the production Microsoft Operations Manager 2005 SP1 infrastructure. It then reviewed the results of that study to determine the appropriate hardware to order for the new infrastructure.

Implement Necessary ACLs and Firewall Policies

After the hardware was in place, Microsoft IT identified the places in its deployment where changes to access control lists (ACLs) on routers would be required to allow communications to function. Microsoft IT then worked to have those changes approved and implemented. An additional point of consideration was that in each case where an ACL was to be implemented, the server or servers involved in the ACL needed to issue a static Internet Protocol (IP) address to ensure that the ACL could be implemented as specifically as possible.

Deploy Operational Database and Root Management Server Cluster

To ensure high availability of both roles, Microsoft IT decided to run the Operational database and root management servers as clustered resources. For each management group, prior to installing any Operations Manager 2007 components, Microsoft IT clustered two servers into a Windows Clustering cluster for each role. Within each cluster, Microsoft IT created two resource groups, each requiring its own distinct network name and static IP address:

• Root management server cluster:
  • Quorum resource group. Microsoft IT created this resource group during the setup of Windows Clustering. The clustering services use it as a repository and to facilitate failover.
  • Root management server resource group. Microsoft IT created this resource group manually. It hosts the services and one shared drive used for the root management server.
• Operational database cluster:
  • Quorum resource group. Microsoft IT created this resource group during the setup of Windows Clustering. The clustering services use it as a repository and to facilitate failover.
  • SQL Server resource group. Microsoft IT created this resource group during the installation of SQL Server 2005 SP1. It hosts the services and the shared drives used for the clustered instance of SQL Server.

After setting up the cluster for each management group, Microsoft IT began the process of building the management group by installing the Operational database role on the clustered instance of SQL Server. After it installed the Operational database, Microsoft IT took the following steps to set up and initialize log shipping to the failover SQL Server-based server that would be used for disaster recovery:

1. Change the recovery mode for the Operational database from Simple to Full.
2. Use the log shipping settings on the properties of the Operational database to set up the remote copy of the database on the failover server and to configure the log shipping setup.

After Microsoft IT configured the Operational database and its failover copy, the next step was to install the clustered root management server. For the installation, Microsoft IT installed the management server components onto all relevant nodes in the cluster and then made some configuration changes to merge those management servers into one clustered root management server.

Deploying the Data Warehouse and Reporting Servers

After Microsoft IT deployed the core components of each management group, it installed the data warehouse.

To prepare for the warehouse installation, Microsoft IT:

1. Installed and configured all prerequisite software on the data warehouse and reporting servers.
2. Determined the service account to be used for the Write Action and Data Reader accounts.
3. Configured service account security for the Write Action and Data Reader accounts on the root management server and in the Operations Manager 2007 console.
4. Obtained all needed information for the installation (root management server name, data warehouse instance name, data warehouse database name, SQL Server Reporting Services instance name for each reporting server, Write Action account, Data Reader account).
5. Set up appropriate ACLs to allow access through the firewall.
6. Ran the prerequisite checker to ensure that the required software components were installed.

Microsoft IT installed the data warehouse on the data warehouse servers and installed reporting on the reporting server for the management group. For multiple management groups, reporting should be installed on each management group's reporting server. Microsoft IT has three management groups and three reporting servers.

After Microsoft IT deployed several management packs, it waited 30 minutes to validate that the installation was successful by determining whether data was moving into the data warehouse and verifying that reports were working in the console.

Deploying Management and Gateway Servers

The next step in deploying the infrastructure was to deploy the management servers and gateway servers. Microsoft IT installed two management servers per management group and then set up the gateway servers.

The first step in setting up the gateway servers was to get certificates issues for each server that would be involved in the management server/gateway communication channel. Microsoft IT maintains its own certification authority (CA); it requested one certificate for each management server and gateway server and imported these into the local certificate store on the system for which they were issued. After Microsoft IT obtained the certificates, it followed the setup steps provided in the Operations Manager 2007 deployment guides for gateway servers. For every network or Active Directory space that required gateway servers, Microsoft IT opted to deploy two gateway servers, and set them up so that either gateway server would be capable of communicating with multiple management servers, to allow for failover.

Deploying Audit Collection Components

Operations Manager 2007 Audit Collection is a Windows platform service for the forwarding, collection, storage, and analysis of security event data. Operations Manager 2007 Audit Collection marks the first step in the separation of auditor and administrator roles by providing security-enhanced, real-time export of security events. The Operations Manager 2007 Audit Collection collector monitors continuous connections from computers running the Operations Manager 2007 Audit Collection agent and loads reported events into a relational SQL database.

Operations Manager 2007 Audit Collection focuses heavily on security (end-to-end authentication and integrity of the event data) and on analysis. Because Operations Manager 2007 Audit Collection stores the collected information in the SQL database in a relational, well-indexed fashion, it provides analysts the ability to access security event information without requiring full-text queries.

While designing the Operations Manager 2007 Audit Collection infrastructure, Microsoft IT addressed several considerations:

• Scale (the number and types of systems to be monitored)
• Event filtering (what events will be logged in the database)
• Data retention requirements (storage)
• Agent deployment mechanism
• Audit policy
• Agent configuration

Scale

Microsoft IT determined that the number of servers to monitor on a single collector ultimately depends on several factors in the environment where Operations Manager 2007 Audit Collection is running. One factor is the role of the server (or desktop computer) being monitored. Usually, it is a safe assumption that a domain controller will produce more security-related events per second than a member server will, unless that member server is a very large file server with a verbose audit policy applied.

The current scope of the Microsoft IT deployment of Operations Manager 2007 Audit Collection is approximately 135 domain controllers and 3,500 member servers. Currently, in the highest-traffic domain, one domain controller produces approximately 100 times the number of events that one of the member servers produces over a specified period.

Event Filtering

Event filtering (registry key: DbQueueQuery) is a feature of Operations Manager 2007 Audit Collection that enables Microsoft IT to filter out the events that the team does not want to write to the database. This filter touches two aspects of the Operations Manager 2007 Audit Collection design: data retention requirements and scale. Microsoft IT determined that if it was not selective about which events it discarded and used a less stringent filter, the storage requirements increased and data retention (configurable in days) suffered. Due to bottlenecks in the throughput of DB loader inserts to the SQL Server-based server back end, nonrestrictive filters limit the number of agents that can be pointed to a single Operations Manager 2007 Audit Collection collector.

Data Retention Requirements

The data retention goal for Microsoft IT's Operations Manager 2007 Audit Collection-related data is 30 days. In higher-volume environments, Microsoft IT was achieving only 24-28 days by using 1 terabyte of SAN storage. The higher-volume environments consist of two Operations Manager 2007 Audit Collection deployments that have 13 to 15 domain controllers forwarding security events. Each deployment averages approximately 2,500-4,000 events per second. Randomly picking any weekday from the database, there are several million rows of data consuming approximately 50 GB of storage for that day.

Agent Deployment Mechanism

Microsoft IT used SMS as the deployment mechanism for deploying Operations Manager 2007 Audit Collection. The Operations Manager 2007 Audit Collection collector has a one-to-one relationship with an Operations Manager 2007 Audit Collection database server, meaning that only one Operations Manager 2007 Audit Collection collector can point to a specified Operations Manager 2007 Audit Collection database server. At the time of this writing, not all Operations Manager 2007 Audit Collection deployment scenarios have been tested—mainly, the ability to run an Operations Manager 2007 Audit Collection collector on an Operations Manager 2007 gateway server. However, Operations Manager 2007 supports this scenario.

Audit Policy

Microsoft IT had to consider its audit policies and plan for scalability when designing its Operations Manager 2007 Audit Collection implementation. Because the filtering of security events occurs at the collector, every security event that is logged on a system that is running an Operations Manager 2007 Audit Collection agent will be forwarded to the collector and then filtered.

Operations Manager 2007 Audit Collection comes with a comprehensive collection of performance monitors. The DB loader counts and the number of incoming events per second indicate how hard an Operations Manager 2007 Audit Collection deployment is working to examine scale.

Operations Manager 2007 Audit Collection also features AdtAdmin.exe, a command-line tool that can be useful for several different tasks. One of its more useful parameters is the -stats parameter. It provides the detail needed to troubleshoot and ensure that the agent base is forwarding events and the volume of events on an agent-by-agent basis. The output that is supplied can also be used to calculate network utilization through the Received Packet Count and Received Packet Size counters.

Agent Configuration

Microsoft IT uses Domain Name System (DNS) service records to point agents to collectors by using a registry key (instance) to provide a unique string that translates to the DNS service record name, allowing multiple Operations Manager 2007 Audit Collection instances to exist in the same DNS space.

Running Workflows to Create Necessary Objects in Active Directory for Agent Deployment

From this point in the process, the entire Operations Manager 2007 infrastructure was set up. Because Microsoft IT opted to use Active Directory integration features to disseminate agent configurations, the only work remaining prior to agent deployment was to provision the necessary Active Directory containers. Microsoft IT took the following steps:

1. Microsoft IT gathered a list from its asset database of all of the server names, and their relevant domains, that would need to have agents installed on them.
2. From the list of server names, Microsoft IT gathered the distinct list of domain names in which agents would reside. This acted as the authoritative list of domains in which Active Directory containers needed to be provisioned.
3. Microsoft IT mapped the domain names to one or more management groups in both the pre-production and production environments. A single domain may have been mapped to multiple management groups for the following reasons:
   • Some domains contained systems that multiple management groups would manage.
   • Some domains contained more agents than could fit into one management group, so agents from that domain would need to be split between multiple management groups.
4. For each management group/domain name pair, Microsoft IT worked with the Active Directory administrators to have them run the MOMAdAdmin.exe tool (provided as part of the support tools on the Operations Manager 2007 installation CD) to provision the necessary containers. This tool creates the following:
   • The Operations Manager container under the root of the domain if it does not already exist.
   • A container under the Operations Manager container with the same name as the management group.
   • Within the management group container, two service connection point (SCP) objects and one security group. After those containers were provisioned, the monitoring engineering team within Microsoft IT had the necessary rights on those containers to do the remainder of the provisioning.
5. Through the administration section of the operations console, Microsoft IT created agent assignment rules for each relevant domain for the specified management group. After Microsoft IT created these assignment rules, it executed workflows on the root management server. These workflows created the following for each management server or gateway server in the management group, within the relevant domain:
   • An SCP for the management server.
   • Two security groups (primary and secondary) to provide permissions to the SCP objects. The primary security group contains the list of computer objects to be directly managed by the management server that the SCP object represents. The secondary security group contains the primary groups from other management servers' SCPs and is used to control agent failover behavior.

Deploying Agents

As was the case with Operations Manager 2005 SP1, Operations Manager 2007 provides the ability to perform push installations for the operations console. Push installations are those in which administrator or technicians perform an upgrade for a user, either directly or through software such as Systems Management Server.

In its Operations Manager 2005 SP1 infrastructure, Microsoft IT had invested a tremendous amount of time and resources to automate these push installations. Although this automation was extremely beneficial for Microsoft IT, it had the significant downside of being a proprietary solution that functioned only on Operations Manager 2005 SP1. It relied on Operations Manager 2005 itself to deploy software, which is not the specialty of that product.

With Operations Manager 2007, and specifically, now that agents can discover their configurations from Active Directory, Microsoft IT has moved to deploying agents through SMS advertisement. The SMS advertisement installs the agent in such a way that the agent will automatically discover its configuration from Active Directory, both initially and on an ongoing basis. With Operations Manager 2007, agent deployment occurs in two parts: deploying agents via SMS, and letting agents enter management groups via Active Directory.

Deploying Agents via SMS

The monitoring engineering team within Microsoft IT worked with the SMS deployment team to create a package that would install the Operations Manager 2007 agent. The package contained the agent installation files (copied from the installation CD). The package worked as follows:

1. It identified the platform of the system that it was running on (i386 or x64).
2. It executed the installer by using the following command.

MSIEXEC.exe /i MOMAgent.msi /qn /l*v MOMinstall.log USE_MANUALLY_SPECIFIED_SETTINGS=0

"MSIEXEC.msi /I MOMAgent.msi": Install MOMAgent.msi

"/qn /l*v MOMinstall.log": Save the MSI install log to a file

"USE_MANUALLY_SPECIFIED_SETTINGS=0": Provide no configuration to the agent, which results in the agent defaulting to discovering configuration from AD.

Microsoft IT then targeted the package at a collection of servers that needed to have agents installed.

Letting Agents Enter Management Groups via Active Directory

At this point, the agents had been deployed via SMS and were running on the servers, but those agents were doing nothing because they had not been given permissions to discover any management groups in Active Directory. All that remained was to let those agents enter their respective management groups by giving them systems rights to view the necessary containers that were previously provisioned in Active Directory.

Most customers will use the assignment rules in the management server properties to control allowing agents to enter management groups, but Microsoft IT worked with the product group to develop a management pack that performs this process programmatically.

Deploying Management Packs

Microsoft IT had roughly 80 management packs deployed in its Operations Manager 2005 infrastructure. These management packs primarily consisted of standard Microsoft management packs, such as the Active Directory management packs, and management packs written specifically for custom applications. As such, when Microsoft IT began its deployment work, it needed to evaluate new 2007 management packs as well as its existing management packs from 2005. Microsoft IT reviewed all relevant management packs and categorized them into three steps:

1. Identify and convert existing Operations Manager 2005 management packs to be deployed on Operations Manager 2007.
2. Identify and disable existing Operations Manager 2005 management packs not to be deployed on Operations Manager 2007.
3. Deploy new Operations Manager 2007 management packs.

Identify and Convert Existing Management Packs

The first step that Microsoft IT took in deployment of management packs was to identify those management packs from the Operations Manager 2005 SP1 infrastructure that would need to move forward to the Operations Manager 2007 deployment. This group of management packs consisted of primarily custom management packs that had been developed in Microsoft IT to monitor internal applications. This group did, however, include some Operations Manager 2005 retail management packs that did not have 2007 equivalents released. After Microsoft IT identified the entire list of management packs to be converted, it performed the deployment process one management pack at a time by using the following steps:

1. Export the Operations Manager 2005 SP1 management pack as an AKM file, the default file format for Operations Manager management packs.
2. Use MP2XML.exe from the Operations Manager 2005 resource kit to convert the AKM file into an XML format.
3. Run MPConvert.exe from the installation directory of an Operations Manager 2007 management server against the Operations Manager 2005 management pack XML file to convert it into an Operations Manager 2007 management pack XML file.
4. Import the management packs into the Operations Manager 2007 management groups.

After the converted version of an Operations Manager 2005 management pack was running in the Operations Manager 2007 infrastructure, Microsoft IT disabled the management pack in the Operations Manager 2005 infrastructure. Microsoft IT can fall back to the Operations Manager 2005 management pack, by re-enabling it, if necessary.

Identify and Disable Existing Management Packs Not to Be Deployed

After all of the necessary management packs were converted into Operations Manager 2007, some remained in Operations Manager 2005 that were not going to be implemented. Either the monitoring contained in these management packs was no longer relevant, or components of other management packs in the Operations Manager 2007 infrastructure effectively covered the areas that these older management packs were designed to monitor. In either case, Microsoft IT disabled the management packs, which left them available to be re-enabled if a fallback is necessary.

Deploy New Operations Manager 2007 Management Packs

Because Microsoft IT was primarily working with beta management packs from development groups, it expended a greater level of effort to tune the new Operations Manager 2007 management packs than other IT organizations would likely require. Microsoft IT runs these management packs to improve them before they are posted on the Web for external consumption.

Microsoft IT deployed the Operations Manager 2007 management packs in its environment for the following technologies:

• Operations Manager 2007(installed by default)
• SQL Server
• Microsoft Exchange Server
• Active Directory
• Windows Internet Information Services
• Windows Server operating systems
• Windows Terminal Services
• Windows Clustering

Deploying the User Interfaces

After Microsoft IT completed the process of building the infrastructure and deploying both agents and management packs, it needed to make the various UIs available to the consumers of the services. The consumers of Microsoft IT's monitoring services are located all over the world and range from data-center technicians, to service owners, to various operations teams located in offices separated by several time zones. Microsoft IT needed to make a number of interfaces available to accommodate each team's needs. The interfaces that Microsoft IT used include the following:

• Operations console. Consumers of Microsoft IT's centralized monitoring service can work with operational data, and administer rules, in the production infrastructure via the Operations Manager 2007 operations console. Through the use of user roles in Operations Manager 2007, Microsoft IT can limit the scope of data and rules that the various users can work with. The operations console also provides access to reporting based on the data warehouse.
• Notifications. With the introduction of the notification features built into Operations Manager 2007, Microsoft IT can offer its monitoring customers the option of subscribing to e-mail notifications generated in response to alerts. Some teams in turn have opted to use notifications solely instead of monitoring the operations console, because this approach better accommodates their workflows. Additionally, service managers and application owners can set up notification subscriptions so that they can be informed of outages that may affect their overall service.
• Web console. The Web console is a Web-based alternative to the operations console that provides a trimmed-down set of functionality to its users. As Microsoft IT evaluated the uses of the Web console, it identified its primary consumers as follows:
  • Customers who prefer not to install an application to access monitoring data.
  • Monitoring customers located in smaller branch offices and low-bandwidth subsidiaries.
  • Subscribers to alert notifications. E-mail notifications can be configured to contain a hyperlink that links to the relevant alert in the Web console.
• Integration with the Microsoft IT ticketing system. Many of the operations teams within Microsoft have well-established workflow processes that are driven by tickets in the internal ticketing system at Microsoft. To accommodate those existing processes, Microsoft IT invested in a ticketing system connector that converts alerts raised in the Operations Manager 2007 infrastructure into trouble tickets in the ticketing system.

Deploying Additional Integration

As was the case with its Operations Manager 2005 deployment, Microsoft IT worked with internal development and support teams, as well as some third-party vendors, to build some add-in solutions. These solutions were oriented toward either integrating the Operations Manager 2007 infrastructure with external systems, or providing additional functionality based on requirements from various monitoring customers within Microsoft. The primary product features that were used to develop these solutions are the Operations Manager 2007 connector framework and the SDK.

Partner Connectors

Numerous Microsoft partners supply connectors that allow their applications to integrate with Operations Manager. Because most enterprise operations teams' workflow begins with the generation of an alert, integrating monitoring systems with the configuration management database, ticketing systems, and dashboards streamlines the support workflow. Microsoft IT has integrated its network monitoring systems with Operations Manager to provide a single operations console and present a holistic service view.

Microsoft IT uses an EMC product called Smarts to monitor its network devices. By acquiring a connector from EMC, Microsoft IT has been able to push network alert data into Operations Manager. In doing so, Microsoft IT can put all of its fault alert data into one operations console, thus bringing more efficiency to its operations teams. Staffing an operations console can be expensive. The ability to provide all the alert information in one UI reduces the amount of staff required to watch the console.

Custom Connectors

Microsoft IT wrote a connector to integrate its Operations Manager implementation with its trouble ticketing system. This connector performs several functions that provide benefits to the operations teams that consume the alert data. In some cases, this connector has completely removed the need for staff dispatchers to route tickets to the appropriate queues. In other cases, it has reduced the amount of time needed to generate tickets and significantly improved the accuracy of the ticket information.

All server hardware alerts are automatically routed through the ticketing system connector and translated into trouble tickets. Additionally, when these tickets are translated, they are coded with the appropriate ticketing system values and placed in the correct data-center work queue. If a hard disk drive fails on a server, Operations Manager 2007 generates an alert that is routed to the ticketing system connector. The connecter then adds the appropriate ticketing system values and pushes the alert to the ticketing system Web services. The ticketing system Web services generate a ticket.

Beyond auto-ticketing functionality, the end users of the operations console can also use ticketing functionality via the use of customer resolution states that Microsoft IT set up. When a user wants to generate a ticket from an alert in the console, he or she sets the alert's resolution state to one of three options, or states, for ticketing purposes. In turn, the ticketing system connector is subscribed to all alerts with these custom resolution states that allow the connectors to dynamically detect alerts that they need to take action on. With this ticketing functionality integrated into the operations console, an operations staff member can triage an alert and perform one of the following actions:

• Determine whether the alert is invalid and set the status to Resolved, which removes the alert from view.
• Determine whether the alert was generated for a known issue and set the resolution state to the "append" custom resolution state. The ticketing connector then associates the alert with the ticket of the previously detected issue. After the ticket is closed and the problem is resolved, the ticketing connector resolves the alert in the Operations Manager 2007 infrastructure.
• Determine whether the alert is a new, valid issue and set the resolution state to the "create" custom resolution state. The ticketing connector then forwards the alert details to the ticketing system, and a new ticket is created. As with the append functionality, when the resulting ticket is closed, the connector resolves the alert in the Operations Manager 2007 infrastructure.

SDK Integration

Anything that can be done in the Operations Manager UI can be done programmatically via the SDK. This ability has enabled Microsoft IT to automate several administrative tasks. One good example of such a task is the way that Microsoft IT has integrated with its configuration management database to create computer groups dynamically.

Microsoft IT uses an internally developed configuration management database. Starting with hardware procurement, assets are entered into the configuration database system. After a valid host name is entered, SMS gathers information about the device and registers it in the database. Other information, such as the particular service offering for the device, is stored in this database as well. By using the Operations Manager SDK, Microsoft IT was able to build a management pack that uses this asset information to group these servers logically according to service offering.

Microsoft IT has written a management pack that uses the SDK to extend the base system class within Operations Manager 2007. Operations Manager 2007 operates on a class-based structure. When the monitoring infrastructure discovers a server, it assigns a set of logical classes to the server. These classes serve as descriptors for the managed device. One class assigned to every device that Operations Manager 2007 discovers is the base system class. Within this base system class, several attributes can be attached to further describe how the system should view this device.

Microsoft IT also wrote a management pack that placed a service offering attribute on every new base system class. This management pack simply queried the asset management database to get the assigned service offering that should be attached to the managed server. After the attribute was attached to the device, the management pack placed the managed device in the correct computer grouping. Consequently, rules applied to the device based on the computer grouping. Thus, Microsoft IT was able to programmatically target specific rules assets based on their service offerings. This solution resolved what could have otherwise caused a burden on administrative overhead.

Best Practices

During the development of the centralized monitoring service, Microsoft IT was involved in every stage of development of Operations Manager 2007, from reviewing and providing feedback on design specifications to deploying multiple builds of the product over the various milestones across thousands of servers within Microsoft. Based on its experience, Microsoft IT proposes the following best practices for an organization that is evaluating and deploying Operations Manager 2007:

• Before any deployment planning, spend time with the groups that will be the end users of the monitoring solution. Gather their requirements and suggestions, and develop deployment plans and objectives based on the information provided.
• Before planning the deployment of Operations Manager 2007 and designing a new monitoring infrastructure, consider the new server roles and features introduced in Operations Manager 2007. Noteworthy roles and features include:
  • The gateway and root management server roles.
  • Clustering support for the Operational database and root management server.
  • Automatic discovery of agent configuration from Active Directory.
  • Granular rights delegation via user roles.
  • Management packs based on health models.
  • Product connectors.
  • Operations Manager 2007 SDK.
• When planning and evaluating for scalability, consider both the number of systems to be monitored and the number of rules that will be run. Both of these are significant factors that contribute to the amount of data in the Operational database.
• Pay careful attention to the performance capabilities of the server (or servers) that will host the Operational database. The Operational database is the central repository and processing point for all data in a management group. Properly planning this server's capabilities and capacities is fundamental to a good infrastructure.
• Use the configuration advice provided in the Operations Manager 2007 Help files and System Center Capacity Planner to help make informed decisions in the infrastructure design phase.
• If migrating from Operations Manager 2005 to Operations Manager 2007:
  • Closely evaluate which management packs can be carried forward and converted, and which do not need to be carried forward or can be replaced with new Operations Manager 2007 management packs.
  • Try to run the two infrastructures in tandem during the transition to allow for a phased deployment; turn off functionality in Operations Manager 2005 after it has been turned on in Operations Manager 2007.
• Wherever possible, deploy management servers and gateway servers in pairs to provide redundancy.
• Consider using SMS to distribute Operations Manager 2007 agents because it can significantly reduce agent deployment costs and complexity.
• Consider using Active Directory integration to allow agents to automatically detect their monitoring configurations.
• Consider deploying management packs in batches. Allow time between deployment of batches of management packs to review and validate the data that each batch generates.

Conclusion

One of the unique aspects of IT within Microsoft is that the engineering teams that plan and implement the deployments of Microsoft technologies often work directly with the groups that are developing the software. The relationships between the IT teams and the product groups are fundamental to the development of a viable enterprise technology, and Microsoft IT views being the "first and best customer" of Microsoft as one of its most vital business objectives.

The planning and deployment of Operations Manager 2007 has been an opportunity for Microsoft IT to improve its service offerings and to provide feedback based on its experience and the needs of the organization.

Deploying Operations Manager 2007 gave Microsoft IT the ability to provide end-to-end service management in a scalable, centralized monitoring infrastructure. Operations Manager 2007 provided new features and functionality that enabled Microsoft IT to streamline how it identifies and resolves issues that affect the health of distributed IT services. These features and functionality also enabled Microsoft IT to take advantage of much of the work that it had invested in deploying previous monitoring solutions.

For More Information

For more information about System Center Operations Manager 2007, visit the product pages at: http://www.microsoft.com/systemcenter/opsmgr/default.mspx

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2008 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, SQL Server, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.