Microsoft IT Tests LOB Compatibility with Office 365 ProPlus
Published: July 2013
The following content may no longer reflect Microsoft’s current position or infrastructure. This content should be viewed as reference documentation only, to inform IT business decisions within your own company or organization.
Microsoft IT gains additional efficiencies during the new Office deployment by incorporating changes into their mature application compatibility testing process.
|Intended Audience||Products & Technologies|
Article, 118 KB, Microsoft Word file
Application compatibility testing is critical for the successful deployment of new software in any enterprise environment. Microsoft Information Technology (Microsoft IT) faces the same challenge maintaining an application portfolio that consists of approximately 1,500 applications in its internal line-of-business (LOB) application portfolio. Prior to each new client product release (Windows®, Microsoft Office, and Microsoft Internet Explorer®), including the early validation of pre-release software (also known as dogfood testing), Microsoft IT conducts compatibility testing on the most critical LOB applications and websites used within Microsoft. Testing results provide feedback to Microsoft product groups to help improve Microsoft products before their release and allow LOB application owners to update their code and inform users of any issues or workarounds to help ensure a good user experience.
With each testing program that Microsoft IT conducts, they review the processes of past programs and introduce new efficiencies wherever possible, helping to reduce their IT costs. This article describes the processes that Microsoft IT used for their application compatibility testing program during the new Office deployment.
The first step in testing for application compatibility is understanding the applications present in an environment and identifying the subsets that are business-critical. This information will enable you to build a test plan and test matrix of the type and level of testing that should be performed on each key application.
Application Portfolio Management
Microsoft IT uses an Application Portfolio Management (APM) tool to track a wealth of historical application data. This enables them to identify which applications are most likely to provide valuable feedback to the product group.
From an application portfolio perspective, Microsoft IT tracks LOB dependencies on Office, Windows, Internet Explorer, and the Microsoft .NET Framework to help target groups of applications that are dependent on specific client technologies or products. They also use two types of application designations, primary and secondary.
- Primary. Critical applications required for business continuance, such as payroll and expense reporting, procurement, hiring and resource management tools, sales, and so on. Testing of primary applications is required during all test cycles.
- Secondary. Less-critical key applications that the business depends on and identified through voluntary participation from internal groups. Testing of secondary applications is voluntary and may occur at any point during the test cycle.
Tracking this information helped Microsoft IT focus their testing on a much more manageable subset of the application portfolio.
Your testing plan needs to account for not only the LOB applications with dependencies on Office, but also any changes to the operating system and browser that may have an impact on your end users.
Number and Timing of Test Passes
Internal testing for both Windows® 8 and the new Office started within weeks of each other. Since Microsoft IT identifies which LOB applications have an Office and a Windows dependency in their APM tool, they were able to gain efficiency and reduce the impact on their internal testing resources by combining testing of both products later in the release cycle. Early in the testing process, Microsoft IT does not combine products in order to more easily identify the source of any issues found.
For the new Office release, Microsoft IT focused iterative (repeated) testing on approximately 59 primary applications during each compatibility test cycle. In addition, Microsoft IT performed voluntary testing on secondary applications on an ad-hoc basis, increasing coverage to 148 Office-dependent LOB applications tested on Windows 7. Later, integration with Windows 8 testing increased the number of unique LOB applications tested with the new Office to 266. Helpdesk calls were monitored as a means of validating that the right set of applications were being tested.
Testing Configuration Matrix
As part of the application compatibility testing for the new Office deployment, Microsoft IT needed to develop a test matrix that enabled them to test various combinations of OS, browser version, version of Office, and deployment means to ensure the best end user experience. This matrix resulted in testing LOB applications on both the Windows 7 and Windows 8 platforms with Internet Explorer 10, deploying Microsoft Office 2010 and Microsoft Office 365® ProPlus in stand-alone and side-by-side scenarios, and evaluating using MSI versus Click-to-Run as a delivery mechanism (in the past, Microsoft IT has always used the MSI version of Office).
Microsoft IT developed a plan that used testing best practices to limit the number of configuration changes introduced into the environment. Since Office 2010 and Windows 7 were both known compatible environments, their test plan started with testing Office 2010 on Windows 8 and testing Office 365 ProPlus on Windows 7. These two scenarios mirrored the most common configurations seen in the Microsoft user's environment, where the largest set of users were running Windows 7 and a pre-release version of Office 365 ProPlus. Then combining the testing of Windows 8 and Office 365 ProPlus as both deployments gained critical mass and would soon be the dominant configuration in the environment.
Microsoft IT tested the following configurations during the new Office testing program:
Windows 7 SP1
Office Professional 2013
Internet Explorer 9
Windows 7 SP1
Office 2010 + Office 365 ProPlus side-by-side
Internet Explorer 9
Office 2010 + Office 365 ProPlus side-by-side
Internet Explorer 10 Classic Browser
Office 365 ProPlus
Internet Explorer 10 Classic and Modern Browsers
There are a large number of compatible configurations available. Customers can adopt Office Professional Plus 2013 (MSI) or Office 365 ProPlus (Click-to-Run), running on Windows 7 or Windows 8. This allows customers a wide variety of options for how they want to run their IT.
Internally at Microsoft, Microsoft IT validates everything that is being released and has tested the various compatible configurations. Customers need to take an approach that matches the configuration they will deploy.
The following provides some guidance on items to consider when moving to the new Office.
- Transitioning to Windows 7 or Windows 8. If you are running Windows XP and are contemplating moving to Windows 7 with the new Office, Microsoft IT has tested both Windows 7 and Windows 8 with the new Office, helping to ensure that the new Office would work for customers transitioning to either platform.
- Classic versus Modern browser. While testing browser compatibility was not strictly a requirement for the new Office deployment, approximately 94 percent of Microsoft IT's application portfolio is web-based (the remainder is installed desktop/smart client applications developed with .NET), so Microsoft IT focused compatibility testing on browser updates to learn how this might affect any key internal web-based LOB application. As Windows 8 offers two different browser experiences, Microsoft IT recommends testing LOB applications for compatibility against both the Classic and Modern browsers. If incompatibilities occur during testing with the Modern browser, customers may be able to use their business critical applications with Classic browser while they upgrade their applications to be more WC3 standards compliant or until the application is retired.
Note: For more information about Windows 8 and Internet Explorer 10 considerations, see Managing Windows 7 and Windows 8 Side-by-Side at http://technet.microsoft.com/library/dn151702.aspxand Compatibility in Office 2013 at http://technet.microsoft.com/library/ff394407.aspx.
Testing should be automated where possible. A virtual machine (VM) testing environment provides for an efficient, safe, and non-destructive way to test configurations before a wide-scale rollout without impacting the day-to-day hardware. In addition, how you report the status of applications tested should be consistent and easily understood by all teams reviewing reports.
Open Testing Model
While application compatibility testing has been part of previous deployments, Microsoft IT took some new approaches to increase efficiency in this deployment. To minimize cost and effort associated with the testing process, Microsoft IT leveraged desktop virtualization and virtualization management features available in Windows Server® 2008 R2 and Microsoft System Center Virtual Machine Manager 2008 R2. Using dynamically provisioned virtual machines (VMs), Microsoft IT tested LOB client applications on pre-configured virtual desktops. This provided a higher level of control and efficiency by providing identical configurations for all testers, eliminating the need and time required to build out physical machines. Issues found were attributable only to the new software, because the only change to the testing environment was the version of Office. All of the other software components installed were known to be compatible from previous testing programs.
Another big change was that Microsoft IT moved to a new open testing model. In the past, testers were required to submit VM requests via email, and then had to wait for notification when their pre-configured VMs were available, delaying the testing process. With the new open testing model, based on Microsoft System Center Virtual Machine Manager technology, testers can provision one or more VMs using templates through a self-service portal. Testers can interact with the VM as soon as the creation process is complete. This change vastly reduced lost testing time, eliminating the need for testers in remote locations to wait for their requests to be completed by Puget Sound-based support personnel.
Looking ahead, Microsoft IT plans to increase their efficiency further in two ways—first, by changing the way that VMs are hosted using System Center Virtual Machine Manager 2012. And second, they plan to increase efficiency by moving from a distributed voluntary model, with testers in various Microsoft IT groups performing LOB application testing when requested to a centralized dedicated group whose sole purpose is to repeatedly test Microsoft IT’s primary applications year-round on new client configurations. Moving to a dedicated group will significantly reduce the number of testers participating while still providing the same valuable feedback, and result in continuous feedback to Microsoft product groups rather than only periodic feedback during release cycles.
A new scorecard was created leveraging Microsoft SQL Server® Reporting Services that allowed key test managers to monitor the progress of their test teams in real time. Test managers were able to refresh the report at any time to show the latest testing data entered in Microsoft IT’s LOB portfolio management tool. The new scorecard included a report divided by organization and group. Each group was monitored with a percentage score tracking completion of their testing of the applications most critical to daily business. Prior to the new reporting solution, participation and current status were tracked in separate reports that were manually updated with the results of SQL queries made against the LOB application portfolio database. The data displayed was static until the next time the report was manually updated.
Participating in pre-release testing of Microsoft client products is an integral part of Microsoft’s culture. With the inclusion of non-Microsoft IT groups, the LOB client application compatibility stakeholder community swelled from about 500 members to over 850. Of those individuals, approximately 150 were directly involved in the testing of LOB applications. An Active Directory®-based security group was created to manage membership and to provide a means to easily communicating status updates and changes to all stakeholders. It also provided access to security-restricted resources, such as the Product Group’s bug database, and Microsoft IT’s LOB Application Compatibility Microsoft SharePoint® site, where testing program reports, bug templates, and the LOB testing schedule were published.
Pilot programs are familiar to anyone who has worked on a deployment project. In a pilot program, an application or product is deployed to a small control group of users, typically skilled users in multiple affected groups. The small number prevents costly down time to a business group if issues are encountered, yet the number must be large enough to establish statistically meaningful field test data.
Deployment Milestones and Targets
Microsoft IT takes a ramped approach to deployment. Microsoft IT tests pre-release and dogfood builds of products released to internal Microsoft users prior to the final Release to Market (RTM) build.
This ramped process was no different with the new Office deployment.
- Initial release was to a controlled group of full-time employees, approximately 31,000 users, with the builds hosted on a restricted access share to control distribution.
- Beta 2 (also known as Customer Preview) release was open to the entire company with a deployment goal of 55,000.
- RTM release ramped up to full deployment with the goal of 95 percent of all employees using the new Office.
Beyond the testing that occurs in the IT labs, the user community provides a larger, shallow level testing of LOB applications as they are using the application in day-to-day business and running into interactions, scenarios, and/or other configurations that Microsoft IT cannot cover in their clean install approach. VMs represent a pristine environment, whereas the user's desktop may have additional items installed that can impact a LOB application.
Note: For more information on deploying Office 365 ProPlus, see Microsoft IT Deploys and Manages Office 365 ProPlus at http://technet.microsoft.com/library/dn283378.aspx.
Microsoft IT strives to achieve a high level of compatibility before deploying a new product, such as Office, but does not require 100 percent compatibility before internal deployment. Microsoft IT moves forward without major impact in productivity before product development work is complete. Where there are remaining issues with the new product, Microsoft IT focuses on finding workarounds to mitigate the impact to early adopters.
Microsoft IT publishes results of application compatibility testing through the Microsoft internal website, IT Web. For the new Office deployment, IT Web provided users with a single, trusted source for information to learn about the state of LOB applications, any known issues or workarounds to allow the user to make an informed decision whether to move forward with adopting the new Office. Microsoft IT created a distribution group that was used to send notification when application compatibility results changed. Users could join the group with a single click from the application’s compatibility results page on IT Web. This allowed users to move to the new Office as soon as any issues with their key applications were resolved.
Providing a centralized mechanism to make users aware that there are application issues is one of the enablers that allows for a rolling deployment.
Application compatibility testing is crucial for the successful deployment of new software in an enterprise environment. Microsoft IT has developed an efficient approach to LOB application compatibility testing and applied that approach when deploying the new Office. Microsoft IT recommends that you:
- Understand your environment. Survey of your systems and identify both LOB application dependencies with the new Office but also on any OS and browser dependencies. Store historical data to help you focus your testing and avoid having to test your entire application portfolio. Test a subset of your portfolio, testing either the most complex applications, applications that historically have been known to have issues, or the applications that are most critical for the business. By picking the right subset of applications that leverage Office products and technologies the same way, you can become efficient and reduce the number of applications that you need to test.
- Create a test plan. A test plan and test matrix should be sufficient to enable you to test the various configurations in your environment and to represent the typical LOB application to minimize the need to test every LOB application.
- Use testing best practices. Introduce a single change into a controlled test environment to make it easier to identify the source of compatibility issues.
- Gain efficiencies through automation. Using a test environment based around VM technology will enable you to perform a large number of tests in a short period without impacting your day-to-day hardware.
- Use a rolling deployment. Use a pilot approach, deploying the new Office to a small set of users, and leverage their feedback on interaction with their application portfolio to identify issues. Make sure that you provide users with guidance on what issues exist and any workarounds so they can continue to be productive and you can continue to resolve compatibility issues.
- Benefit from the side-by-side installation opportunity presented by Click-to-Run. The new Office Click-to-Run deployment model allows subscription customers to install the new Office alongside an existing Office release, which can help unblock any LOB applications that are not yet compatible with the new Office.
For More Information
For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:
To learn more about the new Office deployment at Microsoft, review the two companion articles:
- Microsoft IT Deploys and Manages Office 365 ProPlus http://technet.microsoft.com/library/dn283378.aspx
- Microsoft IT Helps Users Embrace Office 365 ProPlus http://technet.microsoft.com/library/dn283375.aspx
© 2013 Microsoft Corporation. All rights reserved.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Active Directory, Internet Explorer, Office 365, SharePoint, SQL Server, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.