Export (0) Print
Expand All

Improving Desktop Performance with the Risk and Health Assessment Program for Windows Desktop

Technical Case Study

Published: December 2011

The following content may no longer reflect Microsoft’s current position or infrastructure. This content should be viewed as reference documentation only, to inform IT business decisions within your own company or organization.

Is your company looking for proactive ways to improve the health and performance of computers connected to your corporate network? Learn how Microsoft Information Technology utilized the Risk and Health Assessment Program for Windows Desktop to develop a proactive client health initiative that optimized the performance of the computers on Microsoft's corporate network.

Download

Download Technical Case Study, 398 KB, Microsoft Word file

Situation

Solution

Benefits

Products & Technologies

Microsoft IT was determined to optimize the health and performance of the computers on Microsoft's corporate network and implement a sustainable, proactive client health strategy.

Through a cooperative Windows Desktop Risk Assessment Program engagement with Premier Field Engineering, Microsoft IT completed a comprehensive assessment on internal Windows 7 desktops and found ways to substantially reduce system boot times and improve end user productivity.

  • Improved desktop performance enterprise-wide by reducing system boot times.
  • Improved employee experience and positively impacted productivity.
  • Developed a culture of proactive, performance improvement within Microsoft IT.
  • Risk Assessment and Diagnostic tool provided to the customer.
  • Risk and Health Assessment Program for Windows Desktop
  • Windows 7 (32-bit and 64-bit)
  • Windows Vista (32-bit)
  • Windows XP

Situation

Microsoft Information Technology (IT) is responsible for the client health of all desktops and laptops on the corporate network across the company, supporting more than 100,000 users globally. Windows client performance in the modern enterprise can vary greatly as IT and users add a broad mixture of security, management, and other types of applications onto their Windows-based computers. Poor PC performance can directly influence employee productivity, and continued disruptions can negatively impact business productivity, which over time can significantly affect the user experience and the corporation's bottom line. In particular, the routine PC startup and logon experience tends to be a bellwether for productivity and illustrates how users feel overall about their work PC.

Although Microsoft IT had call volume data, support request ticket information, and user experience feedback concerning overall PC performance degradation, they did not have data that identified the cause of specific performance issues. Microsoft IT Support typically resolves performances issues in a reactive manner, and this highlighted Microsoft IT's challenge—employees contact Microsoft IT Support with an issue and expect a resolution. This type of reactive support focuses on solving the employee's current issue, not on collecting data and information that would allow them to take a deep dive into any potential performance issues. In many cases, the resolution only benefits the individual employee who reported the issue, when, in fact, the same issue may be affecting employees enterprise-wide.

The Client Health team within Microsoft IT was determined to develop a strategic, tactical, and sustainable plan that would enable them to identify the top PC performance degradation issues, implement timely fixes, and proactively prevent future issues. Microsoft IT recognized that, by establishing a proactive client health program, they could identify and resolve issues in the production environment before impacts to PC performance and employee productivity occurred. To do this successfully, they needed to establish a PC performance baseline that would enable them to benchmark and identify their top performance issues. The Risk and Health Assessment Program for Windows Desktop offered the toolset and knowledge transfer needed to collect and analyze relevant data to achieve success with the client health program.

Solution

Microsoft IT partnered with the Microsoft Services Premier Support team, which resulted in a Risk and Health Assessment Program for Windows Desktop (WDRAP) engagement. Microsoft IT used the knowledge and tooling gained through that WDRAP engagement to advance their client health strategy.

What is WDRAP?

The Risk and Health Assessment Program for Windows Desktop is a proactive offering to assess the health and performance of Windows-based desktop and laptop computers in a specific enterprise. The assessment focuses on key areas such as startup and logon performance, power management, hardware configuration, and network to identify risks and problems that may degrade user experience and productivity.

The primary goals of WDRAP are to:

  • Assess the health of existing PC configuration, deployment, and operational practices using a suite of data collection tools and results from staff interviews.

  • Identify key areas where a PC's configuration deviates from Microsoft best practices.

  • Diagnose real-world performance issues during the startup and logon process.

  • Empower the customer through tools and knowledge transfer to diagnose future issues and optimize their desktop experience for the long term.

A WDRAP engagement occurs in three phases:

Phase 1: Environmental Assessment Phase

The purpose of the assessment phase is to understand current pain points, review operational procedures, and identify high-risk issues. The assessment performs over 500 in-depth tests to gather configuration, performance, and operational data and analyzes the information to identify settings that deviate from best practices. In addition to technical diagnostic work, the WDRAP Premier Field Engineers (PFEs) conduct interviews with staff that focus on key operational risk areas such as Planning, Change Management, Monitoring, and Security Policy. These steps provide the PFEs with information to evaluate the overall health of the client system environment.

Phase 2: Analysis and Reporting Phase

During the analysis phase, the WDRAP engineers evaluate the assessment data for compliance with Microsoft Windows Desktop configuration and deployment best practices and other methodologies. The results can identify both significant gaps and opportunities for improvement. These results are used to generate the Key Findings report, a detailed recommendations report addressing the risks identified in the assessment phase. The WDRAP engineers ensure a high-level knowledge transfer to IT staff about how to use the data collection tool, how to interpret and understand the report findings, and how to remediate the identified issue.

Phase 3: Remediation Phase

The recommendations in the Key Findings report provide details regarding the identified issues and recommendations to improve the stability and availability of the PCs within the company. Remediation results should confirm an improvement in PC performance, resulting in a reduction in Microsoft IT support requests. Improved PC stability and availability can lead to increased user satisfaction and potential productivity gains.

Implementation

Microsoft IT developed a PC test pool to use during the assessment to create their performance baseline. The test pool was composed of a group of Windows 7 machines in two different domains, with both corporate network and remote access. The test pool spanned seven different user roles to represent a variety of programs, applications, and settings.

As part of the assessment, a small number of machines underwent in-depth analysis, reviewing startup and logon performance. The Microsoft PFE worked together with the customer engineers to complete this analysis. The goals were to understand a real world startup/logon experience in that enterprise and to identify items causing delays in this experience.

Often, issues leading to major delays of many seconds or even multiple minutes are identified that allow the customer enterprise to plan remediation based on those issues. Delays identified and corresponding remediation often involves:

  • Windows configurations poorly suited to the goals of the enterprise (including performance/productivity goals).

  • Group Policy usage patterns poorly suited to the goals of the enterprise.

  • Drivers and services versions and detrimental behaviors.

  • Add-in software behaviors.

  • Security software versions and behaviors.

  • Hardware capabilities.

WDRAP Assessment Results

From the WDRAP test results, Microsoft IT found that their systems passed the majority of the health indicator scans with good marks. There were no pervasive CPU or memory saturation issues, and many common misconfigurations were not found.

The scans did find severe issues with startup and logon performance. In many cases, users were waiting several minutes to get from "power on" to "productivity." By comparison, a clean Windows 7 system without enterprise management layers would only take around 20 seconds to do the same.

The results allowed them to create a defined baseline of PC performance and identify the top performance issues to resolve.

The top PC performance degradation issues in the MSIT environment were:

  1. Excessive disk I/O consumption during startup/logon by various installed components.

  2. Synchronous "Wait for the Network" mode Group Policy usage.

  3. Slow service start times.

Collectively, these issues identified an opportunity to include boot time optimization as part of the client health initiative. A slow boot time can cause employees to experience system startup delays and loss of productivity. Productivity losses of even 30 seconds per employee in a company this size can cost tens of millions of dollars annually.

The results of the assessment helped Microsoft IT to identify their top performance issues and develop a strategy focused on tactical, corrective actions for resolution and proactive measures to sustain the client health initiative.

Figure 1. System boot times before and after optimization

Figure 1. System boot times before and after optimization

Note: The above graph represents an illustrative subset of the delays found in the Microsoft environment. Microsoft product-centric delays have been selected for this case study; real world results will be more heterogeneous. A WDRAP engagement in your enterprise will likely show a different set of delays, which can be identified and remediated through this process.

Corrective Actions

Note: The following corrective actions apply specifically to results derived from Microsoft IT's engagement with the WDRAP team. These results are unique to Microsoft's environment and processes. Issues identified, components responsible, and corrective actions will vary by company.

  1. Remediate excessive disk I/O consumption. Using the Windows Performance Analyzer tool, Microsoft IT was able to document boot time tracing results that revealed 100% disk I/O utilization trends during the first 170 seconds of system startup. Upon further investigation, Microsoft IT determined that multiple applications accessing the disk at the same time during start up were causing significant delays and that the Microsoft Lync 2010 and Microsoft Forefront Security applications were the top two disk I/O consumers.

    • Microsoft Lync 2010 (logging configuration). By working with the Lync product team, Microsoft IT engineers were able to test and determine that verbose client side logging was configured to run on all MSIT clients, which resulted in high disk I/O and delays in startup/logon. Through testing, Microsoft IT determined that, by adjusting the Lync client side logging settings and updating processes to run logs only when necessary, they were able to net a 50-second improvement in startup time.

    • Microsoft Forefront® Endpoint Protection (version-specific high disk I/O behavior). Desktop security applications, like Microsoft Forefront Endpoint Protection, run the scanning processes during the initial system boot. This allows the program to complete necessary anti-virus scans and assessments prior to use; however, these scans are time consuming and maximize disk I/O consumption. Microsoft IT worked with the Microsoft Forefront Security team to identify specific boot time activities that could take place during the post-boot phase. By moving the identified activities from startup to the post-boot, they were able to achieve a 25-second improvement in startup time. The Microsoft Forefront Endpoint Protection team was able to include these process changes in their next version release, allowing them share this benefit with all Forefront Endpoint Protection customers.

    • In addition to the examples mentioned above, Microsoft IT identified several other high disk I/O consumers. Ongoing desktop performance optimization by Microsoft IT will continue to identify startup/logon disk I/O consumers and target them for remediation.

  2. Avoid synchronous group policy usage. Microsoft IT uses Group Policy to configure many aspects of Windows. Some usage patterns of Group Policy apply asynchronously without delaying the user logon, while others require synchronous mode to apply. Synchronous mode waits for the network to initialize at startup and logon and waits for Group Policy to finish before allowing logon to complete. This can result in significant delays. Microsoft IT found that users subject to a GPO that required synchronous mode (Software Installation related to MSIT Direct Access deployment) were delayed by 40 seconds compared to users who received an alternative GPO, which accomplished the same end goals asynchronously.

  3. Slow service start times.

    • Reduce Application Virtualization start time. A very long delay was diagnosed in the start of the "sftlist" service, which is part of Application Virtualization (App-V). Through collaboration with the App-V team, Microsoft IT was able to determine that App-V used a WMI query to gather necessary data for startup that required all other startup applications to wait until the App-V query completed. The teams were able to implement a fix that started the App-V service in 10 seconds, resulting in improving service startup times by 30-90 seconds. This change in App-V will be available to Microsoft customers in a future App-V service release.

    • In addition to App-V, additional slow starting services were identified. In your own environment, you are likely to find various slow starting services that may delay startup and logon.

Table 1. Boot time optimization corrective actions results

Corrective Actions Scenario

Benefit

Remediate excessive disk I/O consumption with Lync logging

50 seconds boot time improvement

Remediate excessive disk I/O consumption with Forefront file analysis

25 seconds boot time improvement

Apply Direct Access GPOs asynchronously

40 seconds boot time improvement

Reduce Application Virtualization service start up time

30-90 seconds improvement to service startup time

Total corrective action boot time improvement

1 minute, 55 seconds improvement

Proactive Measures

Microsoft IT realized that corrective actions only benefit existing resources and that additional progress and performance maintenance require a client health program to respond proactively. Using the knowledge gained from the WDRAP assessment, remediation planning sessions, and experience from implementing the corrective actions, Microsoft IT was empowered to enact a series of proactive measures that would embed the proactive client health focus into existing processes.

Note: This section describes proactive measures that are currently being deployed. To that effect, not all proactive measures have been completed or tested to determine their impact on system boot times.

  • Stop boot time degradation in the environment. Every application deployed to production that can potentially have a boot performance impact is run through the Change Advisory Board (CAB) review process to test the application against a set of release criteria. Microsoft IT has implemented a Boot Time Performance Measurement Process as a criterion in the CAB assessment. This will ensure that future releases of programs have minimal to zero impact on system startup times, which will help prevent further degradation of boot performance.

    • Example: A test of the Boot Performance Measurement process against a beta release of the FIM Password Reset Tool identified a negative impact on boot performance. The FIM team implemented a fix that proactively corrected the issue before the release. During testing, the fix delivered a boot time improvement of 23 seconds.

    • Example: Microsoft IT proactively identified a boot performance impact introduced by the System Center GreenUp application and, through the CAB boot performance assessment, was able to provide the findings to the product group to identify the root cause and remediation steps. The fix resulted in a 16-second boot time improvement.

  • Review and manage logon scripts for efficiency gains. In the past, changes made to the logon script were implemented to production without any type of boot performance impact testing. As part of their commitment to maintaining reduced system boot times, Microsoft IT is instituting a Boot Performance Impact process that reviews the logon script for efficiency each time a new change is made. This focus will avoid duplicate commands or entries that might cause boot time delays.

  • Review all Group Policies that apply in synchronous mode. As a result of work done during the corrective action phase, the Group Policy team is in the process of conducting an inventory review and cleanup of all production GPOs and retiring those no longer in use.

  • Implement automated boot performance tracing. To further drive the Boot Performance Optimization initiative, Microsoft IT is currently developing an automated remote boot performance tracing platform. This platform will provide a live boot performance baseline and insight into top boot performance issue affecting production systems real time.

  • Minimize the swarm of boot activity. Microsoft IT is conducting an in-depth review of startup processes to reduce the excessive number of processes that initialize during system boot.

Table 2. Boot time optimization proactive measures results

Proactive Measures Scenario

Potential boot time degradation avoided/seconds

FIM Password Reset Tool startup fix

23

Implement Boot Performance Measurement Process

16

Total boot time optimization after proactive measures

39 seconds

Total boot time optimization results (corrective actions + proactive measures)

2 minute, 34 second boot time improvement

Benefits

By including the Windows Desktop and Risk Assessment program as part of their Client Health initiative, Microsoft IT derived a number of benefits:

  • Enhanced desktop performance and improved productivity. On average, the typical user will experience a 2 minute, 34 second boot time improvement because of the corrective action taken throughout this process. This has the potential to result in the company saving tens of millions of dollars in employee productivity gains annually.

  • Established partnerships between Microsoft IT and product groups. By using the data, knowledge, and tools from the WDRAP engagement, Microsoft IT established themselves as a key partner in the pre-release of new programs and applications. They are able to test and verify the impact of new applications on system boot times and help optimize the user experience.

  • Created a culture of performance improvement. Microsoft IT began their Client Health initiative with the direct expectation that they would learn how to collect and analyze data that would help them to identify their top performance issues. An unexpected benefit of the WDRAP engagement resulted in a change to the culture within Microsoft IT. Client health is no longer a reactive process; it is embedded in the processes and procedures within Microsoft IT and is considered with each new application and process introduced.

  • Potential productivity savings. After completing the remediation steps, Microsoft IT calculated the value that this work delivered. The approach and the formula used are presented (in the table below) in a way that should help you to perform a similar value calculation for your environment. This value will vary by organization; the boot time difference, number of full-time employees (FTE's), and average cost of an FTE will affect your potential savings figure.

Table 3. Calculating your potential productivity savings

Minutes to boot (initial average)

3

Minutes to boot (remediated)

2.5

Potential improvement

3-2.5=

0.5

# boots per day

1

# FTEs

50,000

# working days in year

245

Minutes gained per year

0.5*1*50,000*245=

6,125,000

Hours gained per year

6,125,000/60=

102,083

Days gained per year

102,083/8=

12,760

Fully-burdened average annual FTE cost

$120,000

Cost of FTE per day ($)

$120,000/245=

$489.80

Value uplift over cost

30%

Value of FTE per day ($)

489.80*(1+30%)=

$637

Annual savings

12,760*637=

$8,125,000

Conclusion

Microsoft IT knew that PCs on the corporate network could benefit from improved performance; they had user feedback and Microsoft IT Support request ticket data to prove it. What they did not know was specifically what performance problems existed in the production environment or the impact of those issues. Microsoft IT decided to develop a proactive client health program that could identify and resolve issues in the production environment before impacts to PC performance and employee productivity occurred, and they engaged the Microsoft Services Premier Support Risk and Health Assessment Program for Windows Desktop (WDRAP) team to help them collect and analyze relevant client health data.

The success of the engagement with the WDRAP has markedly changed the approach to PC client health management within Microsoft IT. The WDRAP offered the toolset, data, and knowledge transfer needed to achieve success with the client health program and help Microsoft IT to develop a client health strategy focused on proactively resolving and preventing issues. Microsoft IT is now able to maintain a PC performance baseline, work collaboratively with product groups to mitigate and prevent new performance issues, improve productivity, and reduce IT support requests.

In continued support of the client health program, Microsoft IT has incorporated an annual review of boot time performance in collaboration with Microsoft Premier Field Engineering. Using the WDRAP tools to analyze and review PC performance, an annual review will allow them to keep their baseline stable over time, establish comparisons, and identify trends. Microsoft IT is also working on expanding client health initiatives to monitor images installed on machines, proactively test the boot performance of the quarterly operating system release, include production systems, and develop an automated process to reduce support requests.

Note: The authors of this paper would like to thank the FEP, App-V, and Lync teams at Microsoft for their responsiveness and help in remediating issues found for the benefit of MSIT and all Microsoft customers.

For More Information

WDRAP Resources

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

© 2011 Microsoft Corporation. All rights reserved.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Windows Desktop Risk and Health Assessment Program, Windows 7, Windows XP and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback
Show:
© 2014 Microsoft