System Management

Essential Tools for MOM Administration

Dave Cristini and Matt Goedtel and Eric Kamor

At a Glance:

• MOM Resource Kit basics
• Management Pack Toolkit utilities
• Troubleshooting tools
• MOM Power Toys

Resource kits have always provided administrators with a Swiss Army knife of useful add-ons, and the one for Microsoft Operations Manager (MOM) is no exception.

At the time of its release, MOM 2005 included a resource kit, but it was refreshed in February 2006 and contains some nifty new tools, divided into five categories as shown in Figure 1. We'll explore only the first three here. Look for a follow-up in the coming months where we'll discuss the connectors and accelerators.

Figure 1 MOM Resource Kit Categories

Management Pack Tools

There may be over 170 management packs in the MOM catalog, but if you are like most MOM administrators, you have internally developed line-of-business apps you need to manage for which you can't find an off-the-shelf solution. The 11 tools that make up the management pack toolkit can help you develop, document, test, and maintain your customized rules.

Management Pack Wizard This wizard lets you quickly and easily create a basic management pack to monitor services, performance, and events for an application. It also lets you create a custom state role with subcomponent detail—the same type of component/subcomponent state health you get from other management packs. For instance, the Base Operating System Management Pack shows overall operating system health, with details for disks, processors, and services. You can manage nearly any app with MOM in just a few minutes.

To monitor services, launch the wizard and browse to any machine to specify what services MOM should watch. The wizard creates all the necessary service discovery and monitoring rules—no need to know that you are looking for a 21207 event and parameter 5 must match a specified string! You can even define which subcomponent's health state this is associated with.

Performance threshold monitoring is just as easy and lets you specify both yellow (warning) and red (critical) threshold values (see Figure 2).

Figure 2** Performance Thresholds **

And it's the same for event log monitoring. If you've used the MOM 2000 Import Dynamic Link Library feature, you'll find the Management Pack Wizard to be a much more powerful replacement. This portion of the wizard lets you point to any machine and read the list of message DLLs that are registered to write to the event viewer. Simply select the event source, and the list of event IDs and messages from that provider is presented. Select the ones you want MOM to monitor, edit the message or alert severity, even combine items (the wizard creates the necessary regular expression syntax), then save your selections. The result is a custom management pack .akm file you can import into your MOM Administration Console with full state health awareness. Be careful, though—since you can't easily uninstall a management pack, be sure to test it first!

Remove Blank Rule Names If you are importing any custom management packs from a MOM 2000 Service Pack 1 (SP1) environment, you'll want to run them through the Remove Blank Rule Names utility first or your import into the MOM 2005 infrastructure may fail. This utility simply validates that each processing rule has a name, generating a new one if necessary. Believe it or not, you could have had an unnamed or blank rule in the past. Now every rule must have a name.

Response Tester Developing and debugging scripts run by MOM outside of the infrastructure is difficult. MOM executes scripts under the agent Action Account and accesses the internal object model presented by the MOMHost process. ResponseTest.exe helps you overcome those challenges by emulating the MOM agent environment. With ResponseTest.exe you can now test scripts before introducing them into your infrastructure. For more information, check out the discussions of the Response Tester utility on Brian Wren's Scripting MOM series at the TechNet Script Center.

Event Creator Do you have rules that should respond to a particular event log entry and want to test them? Event Creator allows you to specify the event log, event source, event ID, calling user, and event type, then it creates the event log entry for you. Let's say you need to verify that MOM gets an alert for a serious problem in Active Directory® or another critical application. You can't simply force the problem to occur to create the real event, but you can use Event Creator to generate it for you. Figure 3 shows how we've created an event from source COM+ with eventID 5069, as an error. The event viewer shows the results as seen in Figure 4.

Figure 3** Building an Event **

Figure 4** Test Event in Event Viewer **

Event Creator needs to be installed on the machine from which you want to generate the alert. It installs quickly, but doesn't create an icon or an entry in a program group; you'll find it in the %Program Files%\EventCreator folder. Note that hidden in the documentation folder of your resource kit install is an XLS file which lists the internally generated MOM event IDs and their text. This can be very useful for testing MOM health-related rules.

Managed Code Response Utility One of the MOM 2005 alert responses lets you call a method on a managed code assembly. Unless you're a developer, you probably won't know what to enter into each field or where to get that info. The Managed Code Response Utility can get the information for you (see Figure 5). Run MCRUtil.exe against the assembly name to get the full assembly name, use the /type switch to get the types, and the /method switch to get the methods it provides. Now that you've got the information you need, you can use the tool to execute the managed code on your MOM management server or MOM agent.

Figure 5** Managed Code Response Utility **

Resultant Set of Rules Need to generate a list of the rules targeted to a particular agent? The Resultant Set of Rules tool is what you want. You can think of it as the MOM equivalent of the Group Policy Resultant Set of Policy (RSoP) tool. When you run RSOR.exe against a specified MOM agent, this tool creates a text file listing all the rules applied to that agent. There are two things to be aware of: the tool doesn't include Overrides and it drops the results into the folder C:\ResultantSetOfRules.

Configure Action Account If you're using a specific account (instead of the default LocalSystem) as your agent Action Account, chances are you've received alerts that MOM can't monitor its performance or something similar. The problem is that the Action Account needs certain permissions to carry out some of the monitoring, which it may not have in a low-rights scenario. The Configure Action Account utility can set those permissions for you. You may need to run it on a Windows Server® 2003 hosted agent not using LocalSystem as its Action Account. When run, it creates the Action Account membership in the Users and Performance Monitor Users local groups and grants the necessary security rights for the MOMHost process.

Convert Management Packs to XML If you need to document your management packs, the Convert Management Packs to XML tool is one way to do it. Run MP2XML.exe against an .akm file and it generates an XML version of the management pack. You can then open the XML document in Microsoft® Excel® or Word, select the fields you want, and there you have it—documentation of your rules. One caveat: you can't go the other way—convert an XML document into an .akm file.

Management Pack Differencing Tools Combined with the output of the Convert Management Packs to XML utility, the Management Pack Differencing Tools can identify what has changed between two versions of a particular management pack, with either GUI or command-line output. Once you export your management pack and convert it to XML, you can load it into the differencing tool and see what has changed. You can even automate this process to perform a periodic comparison to a user-defined lockfile (a file containing a list of rules that should not change).

Rule and Group Toggle Utility Do you need to bulk-enable or disable rules or computer groups and dread the thought of clicking so many times in the administrator console? Relax! The Rule and Group Toggle Utility makes it easy. You can run RuleUtil.exe with a /list switch to get the list of rule group or computer group names, their object IDs, and the current state (enabled or disabled) for each one. With that, you can run RuleUtil.exe with the /enable or /disable switch and pass the name of the group to perform the action against—very useful during testing when you may need to enable or disable one or more processing rule folders of a management pack.

Business Activity Monitoring Wizard If you're monitoring BizTalk® Server, the Business Activity Monitoring (BAM) Wizard can help create the necessary scripts in MOM to retrieve key performance indicators (KPIs) that you define.

Troubleshooting Tools

The MOM Resource Kit includes a selection of tools that ease setup, maintenance, and troubleshooting.

Cleanup MOM No doubt you've seen the fire alarms that say "break in the event of an emergency." The Cleanup MOM utility is just that sort of tool, to be used only as a last resort. Cleanup MOM is designed to remove MOM from a system, including the Windows® Installer registry keys relevant to the installation of MOM server or agent components. It will remove MOM 2000 (including SP1) and MOM 2005 (including beta and release candidate installations).

The normal methods of removing the agent or components should always be tried first. Do note that if you use this tool to remove an agent from an agent-managed system, its primary Management Server will not be aware of this and will alert on a heartbeat failure. To clean the management server, you will need to manually delete the entry from the MOM Administrator Console. This program should be run on the agent-managed server locally.

Management Group Utility Sometimes you may have to move an agent-managed server from your test lab into production in haste, perhaps without removing the agent or forcing it to unmanaged management mode and then deleting the cache directory. In that case, the rules are orphaned on the agent-managed server. The Management Group Utility (MGUtil.exe) can help in cleaning up the configuration the server received from the management group membership. Execute MGUtil.exe /r to remove any management groups the agent reported to.

This tool can also assist in situations where you have a multi-homed agent and you want to identify which management groups the agent reports to. Simply run MGUtil.exe without any switches to receive a list. As with other resource kit tools, this tool must be run locally.

MOM Information Utility The MOM Information Utility can clear the queue on an agent-managed system, and dump the rules, VarSets values, and responses from a MOM agent to an XML file when you need to troubleshoot or review your management pack rules. It also lets you debug custom scripts and place an agent into maintenance mode. That last feature gives you extra flexibility—you can put your server into maintenance mode by configuring a script or batch file with Task Scheduler, without having to use the Operators console. MOMInfo.exe can place an agent into maintenance mode infinitely or for a certain timeframe, by calling:

MOMInfo.exe /maintenancemode:<Timeout>

Timeout equals number of seconds; the default is -1 (infinite).

MOM Inventory If you need to open a case with Microsoft Product Support Services (PSS), you may be asked for detailed information from your agent managed system or management server. If so, MOM Inventory is just the tool to help. It is designed to collect vital information, including Windows Installer logs, trace logs, MOM registry information, configuration information stored in the database and locally, running processes on the system, and event logs.

This tool should be run locally and requires the Microsoft .NET Framework. The program has a graphical interface; to execute it, simply click the Run Collection button and specify where you want to save the CAB file that stores all the collected data. As Figure 6 shows, you can monitor the utility's progress as it runs.

Figure 6** Gathering MOM Inventory **

MOM Trace Log Viewer Ever look at the .mc8 files under the %SystemRoot%\Temp\Microsoft Operations Manager folder on an agent-managed server and wonder what was in these files and how to view them? Well wonder no more: the MOM Trace Log Viewer will shed some light on the matter. To view these log files, copy MOMLogViewer.exe to the agent or management server, run the executable, and point it to the specific log file you need to view. To see just a glimpse of what the Log Viewer displays, check out Figure 7.

Figure 7** Viewing a MOM Trace Log **

By default, MOM Trace Log Viewer will display the entire contents of the file, but you can filter within a specific time, by thread, and so on. If you're dealing with a large log file, a handy feature lets you create bookmarks for specific entries by highlighting that line and pressing Ctrl+B. You can then jump to a bookmark by pressing Ctrl+G (or by selecting a bookmark from the Edit menu). Note that the log viewer shows a snapshot of the trace log, not a real-time view of events as they occur.

MOM Power Toys

The MOM Resource Kit contains a set of useful tools called the MOM Power Toys, designed to simplify some of the administrative tasks and integration points of a MOM 2005 deployment.

Agent Helper The Agent Helper automatically allows management of agents when there is no heartbeat present by restarting the agents and reinstalling them as needed. This is an update to the MOM 2000 utility and leverages the managed code response feature of MOM 2005. This tool is extremely helpful as a last-ditch effort to maintain client health. Agent Helper will attempt three checks on the agent: verifying the MOM service exists; if so, restarting it; and if not, performing an agent reinstall.

Alert to RSS Utility The Alert to RSS Utility can provide current information about the number of unresolved alerts on the management server, categorized by resolution state, as well as a summary of the number of computers being managed. The utility is based on RSS 2.0 and can be used to publish other data from MOM via the MOM APIs. It must be installed on a MOM management server and must be compiled before first use.

Computer Group Hierarchy Utility The Computer Group Hierarchy Utility helps to export your computer group hierarchy and recreate it elsewhere, as well as create computer groups based on Active Directory containers and their hierarchies. It can be used when you need to replicate work done on one management group to another disconnected management group. The utility needs to be run on the MOM server. The output is in XML and can be used to recreate the data on the target system.

Console Scope Utility You can use the Console Scope Utility to automatically update console scope membership, as well as to mirror Active Directory security group membership to a console scope. This helps you synchronize Active Directory group members with MOM 2005 console scope members. Console Scope runs only on a MOM management server and could be run as a scheduled task to keep group members in sync with scope users.

DAS Role Update Utility The DAS Role Update Utility removes the BUILTIN\Administrators group from the MOM Administrator role. The default behavior from a security perspective is that the MOM Administrator and Operator consoles are accessible to the local administrators group. This utility will remove that access so that only the MOM Administrators group can access those consoles.

Management Group Utility If you have orphaned management group data on your agents from test management groups, the Management Group Utility can help to identify and clean up the data. This utility lets you list and remove management groups from the agent side, and it will list the current management groups an agent reports to, and diagnose and remove any corrupt management groups.

MOM Remote Prerequisite Checker The MOM Remote Prerequisite Checker can determine whether the required services and ports are accessible in order to push an agent or run discovery. It is a great tool for troubleshooting security and network issues for MOM deployments, and lets you export results to a log file. This is a standalone GUI utility and can be run under alternate credentials.

Operators Console Notifier Have you ever wanted a pop up dialog to tell you about open alerts in MOM? Then take a look at the Operators Console Notifier, which alerts operators through a pop-up from the System Tray. This tool notifies MOM users and administrators about changes in the alert state. When a change is detected in any of the counters, the changes are shown in bold until the notification is closed. The Operators Console Notifier requires both the .NET Framework and the MOM console to be installed. The utility is right-click enabled to force a manual update or auto-hide notifications (see Figure 8).

Figure 8** Console Notifications **

Password Updater Password Updater is a handy new Power Toy that aids in updating many aspects of Windows that require a security principle, including services, COM+ objects, scheduled tasks, and IIS virtual directories. This tool can help to automate the password or account update of the MOM Data Access Server (DAS) COM+ object on MOM management servers, the SystemCenterDTSPackageTask scheduled task on the MOM reporting server, and the MOMCONN NT service on any source management group management servers. This utility can reference a text file for a bulk password change among large groups of servers. We've presented Password Updater from the perspective of its usefulness to MOM, but this utility could also be used for updating passwords in bulk on any system.

SharePoint Web Part The SharePoint® Web Part enables the presentation of MOM data in a browser-based console, along with the RSS feed. This lets you show MOM data from within SharePoint, by taking advantage of the MOM 2005 SDK. The utility requires that the management server be running both MOM 2005 and either Windows SharePoint Services (WSS) or SharePoint Portal Server (SPS) 2003. It is recommended that you use a separate management server that does not manage agents to provide the data to SharePoint, and that the server not be listed as a failover server for the MOM agents. This is source code you can modify as necessary, but needs to be compiled before first use.

Task Launcher The Task Launcher utility starts MOM tasks from the command line and is very useful for leveraging tasks in scripts. The task must already exist in MOM in order to be used and the utility should only be used for runtime tasks—those that run on the MOM server or agent, not for console tasks. This utility requires the .NET Framework.

Exchange Server Management Pack One of the many management packs offered for MOM 2005 is the Exchange Server Management Pack. A Power Toy called Exchange Server Management Pack Configuration Wizard provides a graphical user interface for configuring it. Among the items you can configure are test mailboxes, message tracking, and service monitoring. You can use it for both Exchange Server 2000 and Exchange Server 2003.

Wrap-Up

You can download the MOM Resource Kit. You'll want to install it onto either your MOM management server or at least a workstation having the MOM consoles installed as some of the tools have dependencies on those pieces of code. As we noted, a few need to be compiled, so for those you'll also need Visual Studio® installed. If you've done any custom reporting for MOM, you probably already have that.

Dave Cristini is a Management Specialist with Microsoft in the New York region. He has been working with management solutions for the last six years. He can be reached at davidcri@microsoft.com.

Matt Goedtel is a Management Technology Solutions Professional working out of the Microsoft NY/NJ District and has been with Microsoft for over two years focusing on SMS and MOM specifically. Reach him at mgoedtel@microsoft.com.

Eric Kamor is a five-year veteran of Microsoft and works as a Management Specialist focused on the System Center product family. He can be reached at eric.kamor@microsoft.com.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.