Dynamic Provisioning

Dynamically Configure Manager Roles Using VMM and Operations Manager

Chris Adams

 

At a Glance:

  • Save time with Unattended Installs
  • Understanding VMM Libraries
  • Dynamic vs. Static Provisioning
  • Integrating VMM with Operations Manager

Contents

VMM: Building Your Nucleus
VMM Server Installation
VMM Agent Installations
VMM Library: The Building Block of Dynamic Provisioningy
Don't Send in the Clones
Working with Profiles and Templates
WAIK Up
Building Your Unattended File for Deployment
From Server to Configuration Manager Site Role Server in 10 Minutes or Less
Testing Your Dynamic-Provisioned Environment
From Static to Dynamic
Test and Verify
Putting It All Together

And it's no different for the Microsoft operations team, which runs one of the largest installations of System Center Configuration Manager 2007 in the world.

But our datacenters are changing before our eyes, especially with the introduction of virtualization. Configuration Manager 2007 leverages the Microsoft virtualization platform, using dynamic provisioning to scale its client-based services for increased agility. The 300,000-plus clients Microsoft manages forced us to think a bit outside the box in the world of scale, finding new ways to increase automation and efficiency. Our lessons learned may help you do the same.

VMM: Building Your Nucleus

The first step to successfully building a Configuration Manager infrastructure that scales quickly and massively is to break out of the "server-based" model and move to the "services-based" model. In the services world, your scale-unit will be hardware objects such as CPUs, memory, disk, and networks. The physical server where these resources exist is less important.

The services model starts with building your virtual machine (VM) infrastructure from the ground up. Before we dive too deep, though, let's get some basics out of the way. System Center Virtual Machine Manager 2008, VMM for short, offers a host of configurable options in your datacenter. VMM consists of four components, all of which have various scale-units:

  1. Virtual Machine Manager Server
  2. Virtual Machine Manager Library
  3. Virtual Machine Manager Agent
  4. Virtual Machine Manager Self-Service Portal

The foundation is built around the concept of dynamic provisioning. (The first three items are covered in this article. For more information on VMM Self-Service Portal, see System Requirements: VMM Self-Service Portal.)

VMM Server Installation

The brains of the outfit, VMM Server, depends on the VMM database, which should run in a SQL Server cluster. The cluster ensures that the job engine that performs all the VM host and guest tasks, such as creating and moving VMs, along with storage determination, is highly available.

VMM Server is a single server that provides administrators a view into their physical hosts and virtual assets. Utilizing hardware technologies such as direct-attached storage (DAS) and storage-area networks (SAN) that are already available in your datacenter allows you to transform your environment into a dynamic provisioning masterpiece.

Installing VMM the first time involves installing two components: VMM Server (which includes the database), and the first instance of your VMM Library. A great deal of planning should go into this step; a good resource can be found at Planning for High Availability.

VMM Agent Installations

The next step is to ensure your Windows-based virtualization servers are available and ready for your VMM infrastructure. Windows Server 2008 with the Hyper-V role and the stand-alone Hyper-V Server are the choices here. VMM offers two methods to set up virtualized hosts: Active Directory-based discovery and manual installation.

We will focus on Active Directory-based discovery, because it's the most common. This process is fairly straightforward; the only real decisions concern the default VM storage location (for example, D:\VMs), and whether to install the Hyper-V role in Windows Server 2008 (if not already enabled).

The VMM Agent is installed on each Windows-based server that runs as Local System; it communicates to the host via Windows Remote Management (WinRM). The Agent installation process enables WinRM to allow HTTP-based port communication between the managed host and the VMM Server.

Once the installation is complete, the VMM Administrator Console should list the host(s) available in your datacenter.

VMM Library: The Building Block of Dynamic Provisioning

The nucleus is in place. Now what? Time to head to the library. The VMM Library provides the foundation for dynamic provisioning. The Library is the primary scale-unit for VMM. You can have multiple Library servers in your infrastructure, while you can't have additional VMM Servers. The Library is the building block for deployment and provisioning. During installation, the first VMM Library is placed on the VMM Server; additional Libraries should be placed on other servers. At Microsoft, we keep our Library installations as close as possible to the hosts they are serving. For instance, you might have a host group consisting of eight Hyper-V servers that service your Seattle datacenter, while a secondary datacenter in Boston has a single Hyper-V host (see Figure 1). In Seattle you would provide multiple Library servers to service the host group, but in Boston you would need just a single Library server.

figure1_hi-res.gif

Figure 1 Using multiple Libraries with System Center Virtual Machine Manager.

These datacenter Libraries store the virtual hard disks, scripts and images. Having Libraries close to the hosts they serve allows provisioning time to improve.

Once you have added the appropriate Library servers to your infrastructure, the next step is to understand how to provision your desired services.

Don't Send in the Clones

There are several provisioning techniques available when utilizing virtualization. One is cloning, where the entire machine is stored for later use, including the machine-specific information such as machine name, IP address and profile-specific data.

Cloning is handy in some scenarios, but for dynamic provisioning it creates more headaches than solutions. The first problem is that cloning requires the computer to remain a part of your infrastructure (in terms of Active Directory, Operations Manager and so on), even though the system itself isn't online (it's stored in the Library). This causes a lot of false positives—systems showing up as available even when they're offline—when you're building dynamic services in your datacenter.

The second downside to cloning is the storage requirement. Every scale-unit (defined as one server = one scale-unit) is stored completely, leading to substantial storage requirements for VMs. For example, a cloned Configuration Manager 2007 Distribution Point (DP) includes three virtual hard disks totaling 130GB in size. This is for a single DP; for Microsoft it takes six potential DPs to run just one site. Thus, the cloning process in this case would require 780GB of Library storage. This adds up extremely fast.

The "base image" method of provisioning is generally more desirable. In this method, a base operating system image is created with nothing installed but the default services required to run the OS. Microsoft, for example, has three base images—Windows Server 2003 R2, Windows Server 2008 and Windows Server 2008 R2—stored as virtual hard disks (VHDs) in its Library. Each base image is roughly 6GB to 10GB, which is much more efficient than the 130GB needed for cloning.

Working with Profiles and Templates

After determining your provisioning method, you next create profiles. Profiles are VMM-wide entities that define, in a very general way, either the hardware or OS. The Hardware profile defines items such as number of CPUs, number and type of hard disks (such as IDE or SCSI) and network configuration. Guest OS profiles define Windows-based characteristics about particular roles being provisioned. This includes items such as machine name, product keys and domain-related information.

Dynamic provisioning is predicated on hardware and guest OS profiles. You build out client-based Configuration Manager roles, but the principles are the same for all profiles. For more details on creating your hardware and guest OS profiles and templates, see my blog at Virtual Machine Manager Library: This ain’t got no Card Catalog and you are the Crazy Librarian.

When the profiles are done, you create a template. Templates detail the hardware and OS characteristics for your server, rather than defining them machine by machine. The key difference between profiles and templates is the specificity: templates are very precise, while profiles are not. Profiles, for example, include the type of network adapter to build, but no connection-specific information about that adapter. Such information is stored as part of the template.

So the three-step process is to create a hardware profile, then a guest OS profile, followed by the site-specific template. To apply this to the Seattle-Boston example, you would start with Configuration Manager DPs and their separate host groups in the Seattle and Boston datacenters. First, create the hardware profiles; next, set the OS information for the entire infrastructure; finally, build templates for each site based on these profiles. Since the networks for Seattle and Boston are different, the customization gained through templates is necessary.

WAIK Up

With the profiles and templates done, it's time to move on to dynamic provisioning of the OS. For Microsoft, that means using the Windows Automated Installation Toolkit (WAIK), which allows customization of a Windows-based installation (see Figure 2). In the dynamic provisioning world, a full operating system installation isn't feasible, as it simply takes too long.

figure2_hi-res.gif

Figure 2 Step-by-Step: Breaking down dynamic provisioning at the operating system layer.

We use a process you're probably familiar with, called sysprep, that cleanses an installed system of any machine and user-specific information. On a typical sysprepped server, the first boot asks the user a set of questions to which they provide answers; the server does the rest. The answers are automatable through the creation of an unattended installation file. Depending on the OS, the unattended file is stored in the Library and referenced as part of the guest OS profile.

The fun part of breaking down a deployment is learning, step by step, what must get completed prior to deploying a new service. After determining these steps, you execute a game plan to solve them via your unattended file.

Building Your Unattended File for Deployment

Some key items in our unattended files at Microsoft include the following:

  1. Adding Local Administrators
  2. Enabling Remote Desktop
  3. Disabling IE Hardening for Administrators
  4. Setting display resolution and organizational information
  5. Turning off Server Manager and Initial Configuration Tasks

These items are necessary in typical datacenters. There are a great deal more available; you can check out the WAIK reference guide at Unattended Windows Setup Reference to learn how to configure other options.

Figure 3 presents an unattended install file for deploying our Configuration Manager site roles dynamically. This XML snippet includes only the items listed above; the entire XML file is available on my blog at SCVMM Sample Unattend.XML for Windows Server 2008 & R2 .

Figure 3 Unattend.xml Snippet

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
  <settings pass="specialize">
<component name="Microsoft-Windows-TerminalServices-LocalSessionManager"
           processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35"
           language="neutral" versionScope="nonSxS" 
           xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State"
           xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"> 
<fDenyTSConnections>false</fDenyTSConnections>
</component>
<component name="Microsoft-Windows-IE-ESC" processorArchitecture="amd64" 
           publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"
           xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State"
           xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
<IEHardenAdmin>false</IEHardenAdmin>
</component>
  </settings>
  <settings pass="oobeSystem">
      <UserAccounts>
<DomainAccounts>
<DomainAccountList wcm:action="add">
<Domain>Contoso</Domain>
<DomainAccount wcm:action="add">
<Name>Chris</Name>
<Group>Administrators</Group>
</DomainAccount>
</DomainAccountList>
</DomainAccounts>
            <AdministratorPassword>
              <Value>Pa$$w0rd!</Value>
              <PlainText>true</PlainText>
            </AdministratorPassword>
      </UserAccounts>
     </component>
  </settings>
  <settings pass="generalize">
    <component name="Microsoft-ServerManager-SvrMgrNc" processorArchitecture="amd64" 
      publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"
      xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State" 
      xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
      <DoNotOpenServerManagerAtLogon>true</DoNotOpenServerManagerAtLogon>
    </component>
    <component name="Microsoft-Windows-OutOfBoxExperience" processorArchitecture="amd64"
      publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"
      xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State" 
      xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance">
      <DoNotOpenInitialConfigurationTasksAtLogon>true</DoNotOpenInitialConfigurationTasksAtLogon>
    </component>
  </settings>
</unattend>

Save this as your unattended file to the Library share. To access the file, refresh your Library server by right-clicking on the Library server in the VMM Administrators console and selecting Refresh.

After the refresh, the unattend.xml is available to add to the guest OS profile created earlier. Click on Answer Files to highlight the selection, then click Browse in the right-hand pane; locate the XML file and click OK.

From Server to Configuration Manager Site Role Server in 10 Minutes or Less

We've seen how to effectively deploy a new Windows server from a base image using unattend.xml. Although the configuration options are vast, few are more useful than the auto logon functionality provided as part of the WAIK. This feature provides a great deal of flexibility and is what enables the entire concept of runtime provisioning—including only what you need, when you need it.

The <AutoLogon> XML element allows you to set the default logon credentials; it works with LogonCount to determine the frequency of auto logon, which enables the triggering of "GUIRunOnce events." These settings, stored as part of the guest OS profile, run synchronously on the server upon the first login. Figure 4 gives some examples of GUIRunOnce usage scenarios.

Figure 4 GUIRunOnce Scenarios
Step Goal
Install Server Features Your working server might require Windows features such as IIS, Clustering, or other functionality
Configure Features For some applications to function, it is necessary to modify or add configuration for particular features
Deploy Applications Some applications have requirements for certain applications are installed prior to installing them such as SQL server, .NET Framework
Configure Applications Some applications require specific configuration to complete the installation and bring the service online

These are only a few of the possibilities; understanding them lays the groundwork for making a vanilla Windows server a Configuration Manager 2007 Site Role Server, such as a Distribution Point, without the administrator doing a thing.

To add the automatic logon to your already created unattend.xml so that GUIRunOnce will occur without any user intervention (that is, a user logon), insert the XML shown in Figure 5.

Figure 5 Adding AutoLogon Code to an unattend.xml file

<settings>
<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" 
  publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"
  xmlns:wcm="https://schemas.microsoft.com/WMIConfig/2002/State">
      <AutoLogon>
            <Password>
              <Value>Pa$$w0rd!</Value>
              <PlainText>true</PlainText>
            </Password>
            <Username>administrator</Username>
            <LogonCount>1</LogonCount>
            <Enabled>true</Enabled>
      </AutoLogon>
</settings>

Next, set up the provisioning necessary to enable a Configuration Manager Site System. The first step is to identify what actions are necessary to completely install a specific Site System. For example, let's assume these specific tasks were required to successfully create a DP; to accomplish this using GUIRunOnce requires the following steps:

  1. Install IIS
  2. Install BITS Server feature
  3. Install WebDAV
  4. Configure WebDAV in IIS configuration
  5. Add Central\Primary Site server to Local Administrator group for server
  6. Deploy distribution point site system
  7. Add all packages to Distribution Point

These tasks should be tackled in a sequential manner, because each step is dependent on the successful completion of the previous one. Figure 6 provides simple guidance on how to accomplish each task.

Figure 6 Creating a Configuration Manager Distribution Point
Step Command
Install IIS serverMgrCmd –install IIS –allSubFeatures
Install Bits serverMgrCmd –install BITS
Install WebDAV Msiexec –qna webdav.msi
Configure WebDAV Cmd –c \\server\share\scripts\ConfigureDP.bat
Site Server to Local Admin Net localgroup –add domain\siteservername$ administrators
Deploy Site System & Role Cmd –c \\server\share\scripts\DeployDP.vbs
Deploy Packages Cmd –c \\server\share\scripts\DeployPackages.vbs

Note about Figure 6: The scripts for ConfigureDP.bat, DeployDP.vbs, and DeployPackages.vbs are in my blog article Dynamically Deploying ConfigMgr Site Roles using Scripts.

Testing Your Dynamic-Provisioned Environment

After laying the foundation, be prepared for tweaking, because it's seldom that everything will work correctly the first time through. But the beauty is that once it works, very little will ever need to change in the future.

The next step is to watch it deploy and see how much time it takes. It's important to note just how long it takes for a complete deployment until the services are online, to ensure a good user experience.

After that, test your deployment using a Windows PowerShell script or the VMM Administration Console. The easiest method is to use the Administration Console, although the Windows PowerShell script is how dynamic provisioning is put into production. To test your provisioning, do the following:

  1. Open the VMM Administration Console
  2. Click New Virtual Machine from the Action menu
  3. Select the option to create from a template and select the appropriate template based on the hardware and guest OS profile
  4. Select the appropriate host
  5. Click to automatically Start and Save state
  6. Click the check box to automatically start the virtual machine after creation
  7. Click Create

Depending on the size of your base image, your deployment should take roughly five to 15 minutes. Upon completion, access the server either through Remote Desktop or within the Administration Console. Do this by selecting the VM in the console, right-clicking and selecting "Connect to virtual machine." Then check to ensure that the expected configuration is complete.

From Static to Dynamic

What's been demonstrated so far is static provisioning; making it dynamic means that your infrastructure should have the means to monitor your applications and take prescriptive actions when needed.

Thus far, we've seen that we can effectively get a Configuration Manager site system and role provisioned using VMM. The final step is to integrate VMM and System Center Operations Manager 2007 to enable your VMs to expand and contract as necessary.

A guide to integrating Operations Manager 2007 and VMM can be found at Configuring Operations Manager Integration with VMM 2008. Integration allows monitoring alerts in Operations Manager to take actions using VMM's Windows PowerShell interfaces, such as the creation of servers that provide services to clients.

Showing how we did this for Microsoft's Configuration Manager infrastructure may provide a blueprint for your environment.

The first step was to ensure that Operations Manager correctly identified servers providing services such as Distribution Points, so that targeting could take place. To simplify this process, we inserted GUIRunOnce, which added specific Registry keys and values on the server. This ensured that when the Operations Manager agent was installed, the specific type of role was watermarked and identifiable. For example, we placed a value called ContosoDP in the HKLM\Software\Microsoft\TechNetMag with value called ContosoDP that is a DWORD type of 0.

The management pack XML snippet can be found at Building a “Sample” SCOM Management Pack for Dynamically Building Virtual Machines based on Performance of physical or virtual machines. It's called "DynamicDP," and in this case triggers the creation of a new DP. For demonstration purposes, we kept the value for connections and the sampling set low; these values would get tweaked based on performance characteristics in your environment.

Our DynamicDP System Center Operations Manager management pack has a discovery policy that reads a Registry key created for each Configuration Manager DP in production. After discovery, we created a rule that monitors specific performance counters, such as the IIS Web Service concurrent connections; when a threshold is reached, an alert is generated. That alert changes the state of the VM in question from Healthy to Critical. That leads to our recovery action. These steps are demonstrated in Figure 7.

figure7_hi-res.gif

Figure 7 The dynamic monitoring and provisioning process.(Click the image for a larger view)

After this, we created our own custom management pack using the Operations Manager Authoring Console. The pack monitored the concurrent IIS connections for the DP; if it hit a specific threshold, the service would trip an alert.

Test and Verify

To test that everything in your environment is working as expected using the Microsoft blueprint, increase the connections on your Configuration Manager DP, causing alerts to spike. You should see the status change from Healthy to Critical. You can perform this test using an HTTP tool like TinyGet, which sends requests to a Web server, generating a high number of connections.

The following TinyGet script will generate a lot of connections (60) to a large file (300KB) named test.txt:

C:\>tinyget5 -h -status:200 -SRV:localhost -uri:/test.txt -x:60 -l:5000 -o 

After validating that the management pack is working correctly, work on building the recovery action. Start your recovery action small—for example, with the creation of a text file. The management pack recovery action sets the corrective action to take place on the Configuration manager RMS server that's been integrated with VMM. At Microsoft, we created a directory on our Configuration Manager server named "recovery" that has a DP folder with two files: dp_recovery.cmd and dp_recovery.ps1. The following is a sample of the dp_recovery.cmd that integrates with the Windows PowerShell script to create the VM:

time /t >>c:\temp\dplog.txt
C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe -PSConsoleFile "C:\Program Files\Microsoft System Center 
Virtual Machine Manager 2008 R2\bin\cli.psc1" -command ".\dp_recovery.ps1 -VMMSERVER:localhost -VMNAME:mydynamicVM"

This recovery command will load the VMM Windows PowerShell console and call our recovery script.

The final step is to configure Configuration Manager to correctly take action when a failure occurs. This utilizes Configuration Manager's Channel, Subscription and Notification settings and allows us to generate the right actions when we create our alert. Instruction on how to build the correct channel, subscription and notification settings is available on my blog at Building a “Sample” SCOM Management Pack for Dynamically Building Virtual Machines based on Performance of physical or virtual machines.

Putting It All Together

The tasks to set up physical and virtual servers are similar, but include significant differences, too. The upfront planning and work invested can save you and your team a great deal of time, as deployment of servers becomes a thing of the past, allowing you to focus your attention on scaling your resources to support your services.

At Microsoft, we have found a method to quickly and efficiently scale our Configuration Manager client services such as Management Points, Distribution Points and Software Update Points utilizing the steps outlined in this article. Dynamic provisioning is a powerful method of allowing a system to expand and contract as necessary. This lets you and your team focus on fine-tuning performance and health criteria that trigger these actions, and less on the reactive techniques that are often employed in IT shops today.

Chris Adams is a senior lead program manager in the Management & Services Division at Microsoft. He focuses on System Center Configuration Manager and System Center Virtual Machine Manager.