Appendix D - The Pilot Project

  • Article

A pilot project is a series of in-house performance tests of your client and server hardware with standard production software installed. The pilot project is performed on a small group of computers in your production environment before you deploy SMS to the organization. It essentially tests and validates all of the plans you have created during the planning phase described in this book. Running the pilot project allows you to draw conclusions and refine your SMS hierarchy design and deployment plan.

Important

Continue to review the hierarchy design as you create the pilot project and refine the deployment plan. Test it in the lab. If you have a multitiered hierarchy, test the depth of it. For example, distribute software to the lowest tier to see if it gets there fast enough. Or, run hardware inventory at the lowest tier and measure the bandwidth cost of sending the inventory back up to the central site. Make changes and test them before you deploy your SMS hierarchy in your production environment.

Understanding the Pilot Project

Your pilot project does not have to be a complete test of all system functionality. However, it allows you to analyze and confirm whether your chosen design will work well in your production environment. The object of the pilot project is to create a small-scale version of your full production deployment, not to watch SMS function in a clean environment with few obstacles. The closer your pilot project installation is to your actual network design and hardware, the more valuable the results are as you plan the deployment of SMS throughout your enterprise.

What the Pilot Project Accomplishes

Running the pilot project enables you to accomplish the following tasks:

Calculating load signature

To create an accurate deployment picture, you must understand the load signature for each site server in the SMS hierarchy design you have created. Load signature is the set of performance metrics of SMS components on a specific computer hardware configuration. Load signature is unique on each SMS site server. It is affected by a variety of factors, including hierarchy design, site design, feature options enabled, feature configuration, task frequency, and numbers of clients.

Running a pilot project provides the information you need to measure and manipulate the load signature of your systems before deployment. For more information about load signature, see Appendix F: "Appendix F - Capacity Planning for SMS Component Servers."

Evaluating the effect of using various SMS features

The pilot project will help you determine how the SMS features that you plan to enable in your SMS hierarchy — and the order in which you implement them — will affect site systems, SMS clients, and the network. Ensuring that your pilot project resembles the production environment as much as possible will result in more accurate results.

Testing your SMS project plan

In the pilot project, you actually implement the plans you created and documented for training IT staff and end users, deploying SMS sites, configuring SMS, deploying SMS clients, performing a backup and recovery of an SMS site, and all of the other tasks documented in your project plan. The pilot project is a test of all the project documentation you have created. It helps determine if your deployment and configuration plan will work in your final production deployment.

How the Pilot Project Works

To meet the objectives of the pilot project, include a variety of users and computers in the pilot program. Also, focus on the functions that you believe might cause problems in the production environment or that have caused problems in your test lab. However, remember that the pilot project is still a test phase. Try to include users who are accustomed to working with new software and with diverse configurations.

The pilot project uses a small subset of the production computers, but you conduct the deployment to these computers in a careful and controlled manner. You should carry out the pilot project in stages. For example, first test inventory functionality in your pilot project, then test software distribution functionality, and so on. However, treat the pilot project as a complete but very limited deployment. It can be tempting to let the pilot project expand until it includes all production computers, but this might create risks. Ensure that all implications and risks are understood in advance. Based on the success of your pilot project, you must decide whether to expand the pilot project into a production deployment, or dismantle the pilot project before deploying SMS in production environment.

Complete your pilot project before you proceed to full-scale production deployment. When you complete each phase of your pilot project, document your results, verify that you met your project requirements, and rework your plan if necessary. Resolve any major issues before you proceed to the full-scale deployment phase.

Creating the Pilot Project

You must determine server sizing for the pilot project to create a representative pilot environment that closely resembles your production environment, but is on a smaller scale.

After reading this section, you will be prepared to run your pilot project, collect data, and determine your individual hardware system load signatures.

Sizing Site System Hardware

When you are running a pilot project, you should use hardware that is sized similarly to what you will use in your production environment. It is a good idea to use the same servers in your pilot project that you have procured for your production deployment of SMS. To determine hardware sizing for your pilot project and for your actual deployment, use the guidelines for hardware sizing in Appendix F: “Appendix F - Capacity Planning for SMS Component Servers.”

Creating a Representative Pilot Environment

For the pilot project results to be useful, it is important that your pilot environment truly reflects your intended SMS hierarchy, with representative network connections intact and available for testing.

Your pilot project hierarchy should include a representative of each type of site system role, server, and client that will exist in your full SMS hierarchy. The central site and several of its child primary sites are important choices. However, several secondary sites and clients are also important. Your pilot project data might be distorted if you create a pilot environment that contains only computers with fast processors and a large amount of memory and does not include the less powerful, smaller computers in the hierarchy. You can gain valuable information by watching how your SMS hierarchy functions in your network as it exists today, including outdated computers and obsolete applications.

Caution

Do not use the same site code name for more than one site in your enterprise. If you are sharing or replicating WINS between two Active Directory forests, and you use the same site code in your production SMS site as the site code of a pilot SMS site in a different forest, the WINS registrations for the sites might overwrite one another. This can cause the Advanced Client to use the wrong WINS database and therefore the wrong management point, causing the Advanced Client policy to be reset in the wrong SMS site. The Advanced Client is no longer a member of the pilot site and cannot participate in the pilot project. For more information, see Appendix I: "Appendix I - Installing and Configuring SMS Clients."

Running the Pilot Project

Allocate as much time as you have available for running the pilot project. This might be a week, a month, or longer, depending on the size and complexity of your SMS hierarchy, the variety of platforms running, and the number of SMS features enabled. Involve as many members of your SMS team in the pilot project as possible, including SMS administrators, programmers, database administrators, and help desk support personnel.

During the pilot project, perform as many critical work functions as possible. These include training the SMS staff members, deploying SMS sites and clients, configuring sites, practicing your software distribution process, practicing a backup and recovery scenario, and other functions as identified in the section, “The Preplanning Phase.” Prioritize which work functions to perform based on your overall SMS objectives. For example, if you plan to use SMS for hardware inventory, software inventory, and software distribution only, then test those functions to the fullest in your pilot project.

Assuming you have already determined hardware sizing, the following steps summarize the process to run your pilot project:

Synchronize date and time

Be sure to synchronize the date and time on all the pilot computers. Synchronized system clocks enable you to identify issues occurring across several computers by matching events that are recorded in log files with the same time stamps.

Install SMS 2003

Install SMS exactly as you will during the actual deployment, based on the deployment plan you have created. This includes the sequence of primary and secondary site installations, hierarchy design, and setup options, and it includes using the data that you gathered in the previous sections to determine how to install your SMS site database server. For example, you might deploy SMS sites in this order:

  • Install SMS on the designated central site server, and be careful to install all the components you want to run in your production environment.

  • Install all child primary sites, and connect them to the central site.

  • Install secondary sites.

Install tools and enable logging and reports

As you install each SMS site, configure tasks, alerts, and the SMS status system. Install SMS and Windows performance tools to test your pilot SMS hierarchy. Most of these tools have logging capabilities that you must enable. You should also enable logging for the site by using SMS Service Manager. For more information, see Appendix F: "Appendix F - Capacity Planning for SMS Component Servers."

Configure SMS sites and site systems

Configure security settings, site boundaries, and roaming boundaries. Prepare and assign SMS site systems for each site in your hierarchy. Configure site communications. Attach SMS sites to one another to create the site hierarchy.

Deploy SMS clients

Discover and install clients that are participating in the pilot project, based on the client deployment plan that you developed.

Enable features

Enable the same SMS features that you will use in your production site.

Collect data

Use logging to collect data about server load signature.

Analyze data

Watch for bursts and other problems that affect performance. After you identify and resolve these problems, rerun the appropriate tests. For more information about bursts, see the “Burst and Recovery Effects on Design” section later in this appendix.

As part of your formal change management process and tracking system, ensure that you keep a complete record of your activities during the pilot project. This information will be of considerable use after the pilot project is complete and you are ready to refine your final site design.

Diagram the pilot project

After you work through the pilot project, ensure that you have a diagram of the design that you are using. Keep records of all activities associated with your pilot project. Your final documentation should include a record of all the changes you make during the project, in addition to any cost-benefit analyses.

The hierarchy that you developed during the initial SMS hierarchy planning stages and a diagram of the pilot project hierarchy will enable you to refine your final SMS hierarchy design. The first diagram that you created is based heavily on your organization’s business and operational issues. Your final site design will probably closely resemble this diagram. The diagram that you create while you are running the pilot project enables you to identify the hardware you require to make SMS function smoothly in your SMS hierarchy.

Using Performance Tools

A primary objective of the pilot project is to analyze your SMS hierarchy design. As you conduct the pilot project, use SMS and Windows performance tools to test performance throughout your SMS hierarchy.

To complete this analysis, you must use several different tools that are either provided with SMS 2003 or are available for download from the Microsoft Web site.

There are two types of tools in your pilot project: production tools and test tools. Production tools run in your production environment. They must be part of your load signature and should not be run remotely. SMS status messages and Network Monitor are designed for use in a production environment.

Test tools are not optimized for your production environment, and must be moved to a remote computer or disabled on the pilot project after preliminary testing. Logging is a test tool that is frequently used by SMS administrators in production. System Monitor is both a test and production tool. However, it has only a slight effect on system resources and does not significantly affect your load signature calculations, unless the frequency and number of counters that are enabled is excessive.

For more information about using these tools to analyze performance, see Appendix F: "Appendix F - Capacity Planning for SMS Component Servers."

Using Remote Monitoring

To calculate a true load signature, you must ensure that the computers and environment that you are using for testing simulate your production environment as closely as possible. Ensure that the only processes running on your pilot computers are processes that will be running in a production environment. This means that you must run some of your tools remotely or from a computer for which you are attempting to obtain a load signature.

Analyzing the Design and Making Changes

As you run the pilot project and monitor performance, keep track of anticipated design changes based on the following considerations:

  • SMS hierarchy design optimization

  • Burst and recovery effects on design

  • Total processing time and design

  • Hardware tuning

  • Software tuning

SMS Hierarchy Design Optimization

While you conduct your pilot project, identify any hierarchy or hardware problems that require troubleshooting. After you identify and correct any such problems, you are ready to retest for the problems. Testing and troubleshooting are iterative processes that can be repeated for as long as an SMS component or site does not meet the performance goals set during the initial planning stages.

The load on an SMS site or client frequently varies between idle and busy. A constantly busy computer is as uncommon as a constantly idle site. Therefore, to interpret how your system is functioning, do not wait for just hardware or system failures. You must also carefully investigate how your computers are working. Understanding where problems are likely to arise requires practice. When you are familiar with the log files, all the SMS and Windows performance tools, and performance issues, you can more quickly identify and resolve many of the issues described in the following sections.

Burst and Recovery Effects on Design

Many SMS processes generate bursts of information and load up SMS inboxes with information that is processed in a timely manner. However, if your computer cannot process this information, or if it cannot process the total amount of files that are generated at one time during bursts, you might have a serious performance issue. Ensure that you watch for burst and recovery situations that might overload your hardware resources.

Bursts might occur whenever you have a scheduled item that occurs at a specific time.

  • If you schedule an advertisement to run at a specific time for all clients in the site, a burst of status messages is likely.

  • If you schedule hardware or software inventory to run at the same time on all clients in the site, a burst of Management Information Format (MIF) files and software inventory files is possible.

  • If you schedule a particular discovery method to run at a particular time, a burst of DDRs is possible.

  • If the site must perform a resynchronization operation, a burst of full MIFs from clients in the site is likely.

  • During operations that tend to occur during certain times of the day, bursts are likely to occur. For example, when employees log on to their computers in the morning, a burst of advertisement status messages, DDRs resulting from network discovery, and hardware and software inventory files is likely.

Total Processing Time and Design

The total time it takes for your computer to process files is an important performance metric. For example, the time it takes for a client to take hardware inventory, plus the time it takes for the hardware inventory file to reach the SMS site database, is the total processing time for inventory. When you are running your pilot project, you should regularly take total time measurements for every type of process.

One basic operation to monitor on the site server is the total time that it takes the server to process objects, such as DDRs and MIF files. To do so, locate the server component that processes the object that you are interested in, and review the time stamps in the corresponding log file to determine the length of time it took to process individual objects. For accurate results, use System Monitor.

It is important to consider that using this method to monitor processing time can be somewhat misleading, because it is not obvious what other processing tasks the server might be performing during that interval. The processing of many SMS objects simultaneously by the server will affect the time required to process individual objects.

After you identify the problems in your SMS hierarchy design, you should analyze the cost in time and money that will be required to resolve each problem against the benefits of improved performance. You might have to perform this analysis several times to create a system with outstanding performance.

It is often true that a computer system could have better performance, regardless of the amount of performance tuning that has already been conducted. However, there is a point at which the time gained by increased performance is outweighed by the money and effort required to create a better system.

Identifying the cost, in both additional hardware expenditure and additional hours to troubleshoot and resolve a performance issue, can be difficult. After your pilot project has been running for some time, you will have a better idea of how to conduct your performance testing and troubleshooting. You can assume at the start, however, that disk I/O problems will be the hardest to diagnose and will require the most hours to resolve, and CPU problems will be the easiest to diagnose and resolve. However, disk I/O improvements are likely to give a very large return on the time invested in diagnosing them, so they are likely worth your time and effort.

Hardware Tuning

You can solve some problems, such as disk I/O and memory issues, without hardware tuning. For example, if you receive “Out of Virtual Memory” error messages on your site server, you might have to adjust your virtual memory sizes. See your Windows documentation for more information about adjusting virtual memory.

Some hardware issues can be solved quickly and easily by adding a physical disk or more memory. These are easy, relatively inexpensive options that will tax neither your hardware budget nor your time or staffing resources. For this reason, it is a good idea to identify these problems first and resolve them as you proceed through each iteration of your system analysis.

Testing and optimizing your disk I/O is vital to a successful SMS deployment. Although you can take several different approaches after deployment to reduce disk I/O problems, it is best to address those issues when you test your site system hardware. Both hardware and software RAID solutions can be very effective in addressing disk I/O issues.

When trying to avoid problems caused by disk I/O bottlenecks, consider the following actions. These actions provide progressively better performance (at increasing levels of expense):

  1. Add more physical disks. Installing the data device, log device, and SMS directories on separate disks will result in moderate improvement.

  2. Implement hardware RAID arrays. If you have a large number of disks in multiple volumes, successfully implementing hardware RAID will result in excellent reliability and speed improvements.

Disk I/O issues can be difficult to diagnose. For information about individual disks and software RAID, see the Microsoft Windows 2000 Resource Kit and your hard disk specifications. Diagnosing the disk I/O activities of your hardware RAID is more difficult because the built-in Windows 2000 system counters for disks do not recognize disks that are masked by a hardware RAID layer. You might consider using third-party software to identify disk I/O problems and consulting hardware RAID manufacturers to optimize systems that use hardware RAID.

For more information about hardware tuning, see Appendix F: "Appendix F - Capacity Planning for SMS Component Servers."

Software Tuning

Configure both SMS and SQL Server for optimal performance.

SMS Tuning

The intervals of all the operations that run in your site greatly affect the load that your site sustains while running every day. For example, hardware and software inventory are the most processor-intensive operations on clients and servers. Reducing the frequency of inventory collection by as much as your requirements allow frees up computing resources and enables you to maximize your hardware investment. Carefully select the low network traffic times to schedule hardware and software inventory to run on clients.

The number of SQL Server connections that are required by SMS is determined primarily by the number of remote SMS Administrator consoles that are running and connecting to it at any given time. If more than 40 simultaneous remote SMS Administrator consoles access an SMS site, you should monitor the connection status in SQL Server. For more information about setting SQL Server connections, see Appendix G: "Appendix G - Deploying and Configuring SMS Sites."

SQL Server Tuning

You can use the following settings to manually tune SQL Server for the SMS site database in your SMS hierarchy:

  • TempDB data size

  • TempDB data space available

  • TempDB log size

  • TempDB log space available

  • SMS data size

  • SMS data space available

  • SMS log size

  • SMS log space available

For more information about SQL Server tuning, Appendix F: “Appendix F - Capacity Planning for SMS Component Servers.”

Dismantling the Pilot Project

For security reasons, you might choose to not to let your production environment grow from your pilot project. After you assemble a complete record of your final design considerations, and you are ready to proceed to final site design, you can remove SMS.

The best way to prevent potential security problems is to completely reformat and prepare each computer for production from a clean state. Remove and reinstall SMS, SQL Server, and Windows 2000 to ensure that your production databases, security settings, and SMS hierarchy can be created without any configurations or data left over from your pilot project.

After you develop a final SMS hierarchy design and dismantle your pilot project, you are ready to implement your actual SMS hierarchy and deploy SMS throughout your organization. When you do so, ensure that your network is ready to grow as your organization grows. Be prepared for upgrades and basic system maintenance tasks. Understanding how upgrades will affect you and your network is critical to the continued smooth functioning of SMS.