Technical White Paper

Published: May 22, 2007

Executive Summary

This white paper is for Microsoft customers who plan to deploy Microsoft® Office SharePoint® Server 2007 on their networks. It shares the experience of Microsoft Information Technology (Microsoft IT) in expanding the hosted team site and portal offerings at Microsoft and upgrading them from Microsoft Office SharePoint Portal Server 2003 to Office SharePoint Server 2007. Microsoft IT requirements are among the most challenging in the world. Therefore, this paper should provide helpful guidance for any Microsoft customer who is planning to deploy Office SharePoint Server 2007 on a large scale.

Over the past several years, Microsoft IT has collaborated with the Office SharePoint Server 2007 and Windows® SharePoint Services version 3.0 product teams at Microsoft to test and share feedback on new features and functionality of the products. As the first and largest customer to deploy Office SharePoint Server 2007 and Windows SharePoint Services 3.0, Microsoft IT has had a unique opportunity to apply improved capabilities to its service offerings and to provide operational validation on the functionality, scale, and performance of the platform. Microsoft IT used pre-release versions of Office SharePoint Server 2007, Windows SharePoint Services 3.0, and Microsoft SQL Server™ 2005.

The paper describes the SharePoint infrastructure at Microsoft and discusses the pros and cons of various upgrade options. It then describes the actual upgrade process that Microsoft IT is using, and it describes the business benefits experienced thus far. Finally, the paper details lessons learned and best practices.

Introduction

Windows SharePoint Services is the technology in Microsoft Windows Server® 2003 that enables workers to collaborate in browser-based workspaces. Because it provides a manageable infrastructure and extensible application platform, Windows SharePoint Services offers a foundation for collaboration applications that includes a common framework for document management and a common repository for storing documents of all types. In addition, it provides common, flexible administration and deployment tools, and it builds on and exposes key Windows Server services like Windows Workflow Services and Windows Rights Management Services.

Office SharePoint Server 2007 delivers highly scalable collaboration solutions that connect sites, people, and business processes, and as a result facilitates knowledge sharing. It also extends the capabilities of Windows SharePoint Services by providing organizational and management tools for SharePoint sites, and by enabling teams to publish information to the entire organization. Figure 1 provides a high-level overview of the technologies that support Office SharePoint Server 2007.

Figure 1. Technologies that support Office SharePoint Server 2007

Office SharePoint Server 2007 unifies portal and content management, business insight, search, and business process capabilities to help enterprises improve their organizational effectiveness. By supporting all of the intranet, extranet, and Web applications across the enterprise with one integrated platform, it helps reduce the number of platforms that must be maintained. In addition, it greatly simplifies the end-user experience through integration with browsers, e-mail, and familiar desktop applications in the Microsoft Office system.

As one of the world's leading technology companies, Microsoft is committed to exploring the possibilities of electronic information-sharing and collaboration not only in the products that it creates but also by deploying those products internally. Office SharePoint Server 2007 and its companion technology, Windows SharePoint Services 3.0 in Windows Server 2003, are the platforms that enable intranet-based collaboration at Microsoft. As such, a large majority of the more than 100,000 Microsoft employees and vendors use Office SharePoint Server 2007 on a daily basis.

Office SharePoint Server 2007 is used for a wide variety of business processes. Portals aggregate corporate information for users based on their Active Directory® directory service profiles. Team sites enable project-specific teams at Microsoft to share, track, and manage information and documents. MySites form the foundation of the Office SharePoint Server 2007 hierarchy, enabling individuals to share information related to their expertise and role. Employees also use an increasing number of user-controlled knowledge management strategies—such as Wikis and blogs—to share and improve their work. Slowly, Microsoft is also integrating line-of-business (LOB) data into the company's Web-enabled information ecosystem.

The system contains over 24 million discrete documents and other files, a number that is growing by about a million a month. Employees rely heavily on the robust search functionality of Office SharePoint Server 2007 to help them find what they need. Document life-cycle features and processes help determine when documents can be deleted, reducing the volume of files.

An ever-increasing number of employees use the People search feature, which helps them find human resources within the company. MySites propagate changes made to individual profiles to the rest of the system, and they display knowledge about an individual's roles, responsibilities, and information assets. The many thousands of Microsoft partners can also participate in the collaboration cycle by using the hosted extranet sites, which are outside the internal network, providing a separate environment for collaboration with no access to the company's intranet. For all these reasons, the widespread use of Office SharePoint Server 2007 inside Microsoft is one of the company's fundamental competitive advantages.

This paper describes Microsoft IT's experience deploying Office SharePoint Server 2007, which involved not only upgrading from SharePoint Portal Server 2003 but also architecting a new service with 2007 features and services, along with engineering a new farm and a tiered service offering. The experience within Microsoft was in some ways unique—the company had a unique degree of access to the inner workings of the product and participated in early testing before the product was released. Every company is different and must design and implement deployments of any size in a way that makes the most sense for it.

At the same time, many of the experiences of Microsoft apply generally, especially to companies that are deploying Office SharePoint Server 2007 to thousands of users. Examples of these experiences include choosing a deployment methodology, understanding the advantages of using a shared services architecture, planning and scheduling, communicating with internal stakeholders, measuring product performance, and choosing how to implement various product features. These topics are covered in detail in this paper, and they provide insight into how an enterprise company uses Office SharePoint Server 2007. Finally, the lessons learned and best practices provided near the end of the paper will apply to almost any large deployment of Office SharePoint Server 2007.

Internal Deployment of Office SharePoint Server 2007

Previous Environment

Microsoft IT's previous strategy for the SharePoint service offering was primarily focused on storage, Web-based access to user-generated content, and centralized services such as search and profiles. The 2003 products provided minimal governance and IT management of customization and minimal workflow capabilities. The Web publishing features of SharePoint Portal Server 2003 did not enable full Internet-based information portals; these features required additional Web content management services to do so. To allow significant customization required isolating the property to dedicated hardware.

Users throughout the company had adopted the practice of creating on-demand sites for a variety of reasons. However, a lack of comprehensive life-cycle management features resulted in content persisting long past its usefulness, adding to the cost of management, storage, and backup. Protecting sensitive data was difficult because users and administrators could not classify sites, audiences, and content at a sufficiently granular level.

The absence of a recycle bin for users to recover deleted content was an operational burden. Workers avoided adopting personal sites as replacements for services like MyDocs because Microsoft IT did not allow customizations of personal sites. In addition, the SharePoint product lacked offline capability, which also prohibited retirement of MyDocs. Finally, corporate strategy did not necessarily encourage regional subsidiaries to use central SharePoint services. Thus, several subsidiaries chose other solutions or self-hosted SharePoint environments.

Service Goal

The aim of Microsoft IT was to provide a Web-based solution that supported general collaboration, Web-based publishing, and Web-based dashboard functionality to all employees and partners at Microsoft. The company wanted a service that could scale to meet the needs of the individual, team, and enterprise audiences.

The solution had to be available to all users connected to the corporate network and facilitate collaboration with third-party partners. End users work all over the world. The service had to provide an acceptable level of performance for all users directly connected to the corporate backbone of the closest regional deployment.

Office SharePoint Server 2007 needed to be able to provide security for content based on varying sensitivity levels. Users could not host medium business impact (MBI) content, high business impact (HBI) content, or personally identifiable information (PII) on sites open to a reasonably sized audience. Allowing all authenticated users access to MBI, HBI, and PII data was not acceptable.

Supported Audiences

Microsoft IT identified four major audiences that require Office SharePoint Server 2007 support.

Office Dwellers (About 25,000 Employees, 40% of Full-Time Staff)

Office dwellers—typically employed in marketing, administration, development, testing, and similar activities—may attend meetings in conference rooms but rarely leave the building in which their office or cubicle is located. They primarily use a desktop computer for collaboration, and most of this collaboration takes place within small, distinct groups.

Campus Nomads (About 25,000 Employees, 40% of Full-Time Staff)

Typically a program manager, service manager, or product manager, the campus nomad is a roaming user who tends to use a laptop computer with wireless connectivity as the primary collaboration environment. Campus nomads participate in or facilitate a large number of meetings and are rarely in their offices.

Remote Users (About 15,000 Employees, 20% of Full-Time Staff)

Consulting services, client support services, sales, and technical specialists make up the category of remote users. These individuals do not spend significant time connected directly to the corporate network. They connect by using a variety of technologies to access information, data, and documents. This user base is most concerned with remote and offline access to content; network performance is a secondary concern.

Business Partners (About 100,000 Users)

This group consists of consumers of Microsoft business services. These users have little or no direct connectivity to the Microsoft internal environment. Instead, they connect via the Internet to extranet sites.

Desired Features

Overall, users at Microsoft wanted the ability to customize how team or project content is stored and shared. Other features commonly desired included:

• Improved e-mail and discussion functionality with support for incoming e-mail storage within different list types.
• Better programmability and an improved administration user interface.
• More granular Information Rights Management.
• Two-way synchronization between Microsoft Office Outlook® 2007/Microsoft Exchange Server and Office SharePoint Server for tasks, calendars, and e-mail discussions.
• Better people functionality with a focus on social networking, People search, and personalization.
• Improved search crawling and relevance.
• Out-of-the-box integration between LOB data and services and portals. Integration with business intelligence functions such as the Microsoft Office Excel® Calculation Services engine, key performance indicators (KPIs), dashboards, and report centers.
• Improved workflow and document life-cycle management features.
• A two-step recycle bin available to end users.
• Offline content support with one-way synchronization in Office Outlook 2007 and two-way synchronization with Microsoft Office Groove® 2007.
• A clear difference between team and portal sites.

Business Benefits of the Move to Office SharePoint Server 2007

Microsoft IT will not be able to quantify the full range of benefits gained by moving to Office SharePoint Server 2007 until the migration is complete later in 2007, but many business advantages have already surfaced:

• Easier publishing. The simplified content publishing process helps publishers and analysts for internal portals present content to users more quickly.
• Improved site-level management. Administrators of upgraded sites can better manage their own site security, storage, distribution, and management. The tools that enable them to do so are easy to use and integrated into the 2007 Office system applications.
• Business data integration. Microsoft IT has enabled crawling and exposure of data contained in the company's Siebel customer relationship management (CRM) system through Business Data Catalog. Business Data Catalog provides a way to integrate business data from back-end server applications within Office SharePoint Server 2007 by building an XML schema to map to the data source. Business Data Catalog, along with Excel Services and custom coding, enables workers to add key data from various business applications to Office SharePoint Server 2007 lists, Web Parts, search, user profiles, and custom applications.
• Fewer user requests to Microsoft IT for document recovery. Due to the addition of a two-stage recycle bin available to users, Microsoft anticipates a savings of 20-25 person-hours per week and a 75 percent reduction in database maintenance related to recovery.
• Real-time access to Excel Services. Users can access real-time, interactive Microsoft Office Excel 2007 spreadsheets from a Web browser through Excel Services running on Office SharePoint Server 2007. For example, the 2007 Office system client team maintained counts and amounts of deployment numbers of different versions of the 2007 Office system clients during internal testing. The pages refreshed on viewing, always showing the most current data.
• Less coding. Office SharePoint Server 2007 provides standard workflows such as a change management process, project portfolio management, and a change advisory board, reducing the amount of server-side code that Microsoft IT needs to write. Windows Workflow Foundation integrates workflow coding into the Microsoft .NET Framework and provides an easy way to automate repeated development tasks.
• Automatic synchronization. MySites and lists automatically synchronize with Office Outlook 2007, whereas Microsoft Office system applications and Windows Internet Explorer® 7 integrate with Office SharePoint Server 2007 Really Simple Syndication. (RSS) feeds. Microsoft IT used the RSS feeds to send "tips and tricks" for using Office SharePoint Server 2007 to users via Office Outlook 2007.
• Improved content policies and regulatory compliance. Microsoft IT can better specify security settings, storage policies, and auditing policies, helping control and manage potentially sensitive business information more effectively. The 2007 Office system renders policy settings onto client applications, helping users see and comply with regulatory requirements.
• Scheduled content approval and deployment. Content authors can create and submit content for approval and scheduled deployment to intranet or Internet sites.
• Improved search. Office SharePoint Server 2007 Enterprise Search represents a significant improvement over the search engine technology of SharePoint Portal Server 2003. Features like duplicate collapsing, spelling correction, and alerts improve the relevance of the results.
• Lower costs. Microsoft IT decreased the total cost of maintaining team sites. The recycle bin reduces hours lost to document recovery. Enhanced site-level administration features make administering sites easier and spread the administrative workload across a larger group of personnel.
• Server consolidation. Microsoft anticipates that the move to Office SharePoint Server 2007 will reduce server hardware by 30 percent, because services can now be combined and the new portal architecture makes less of an impact on the servers and reduces unnecessary hardware choices.

Team Structure

The Messaging and Collaboration Services (MACS) team within Microsoft IT has primary responsibility for deployment of the Office SharePoint Server 2007 solution, but the solution both relies on and serves multiple other groups. Figure 2 illustrates the responsibilities of the MACS team.

Figure 2. Messaging and Collaboration Services responsibilities

Solution Design

Microsoft IT hosts Office SharePoint Server 2007 in three data centers, one each in Dublin, Singapore, and Redmond, WA. Approximately 9 out of 10 users in each region are local to that region. Figure 3 illustrates the geography of the Office SharePoint Server 2007 farms.

Figure 3. Geography of Office SharePoint Server 2007 farms

The Office SharePoint Server 2007 data center in Redmond has the largest number of servers and serves the largest number of employees. The Redmond farm is primarily for U.S. employees and for the Web properties supported in the regional data centers. It crawls content from all three regions and imports LOB data into user profiles by using Business Data Catalog functionality. The system replicates Business Data Catalog data to the regions as necessary.

The Dublin and Singapore Office SharePoint Server 2007 farms are called regional deployments. They crawl only local content, so they require less-powerful index servers than the Redmond farm. Not all of the services provided by the Redmond farm are mirrored in the regions; therefore, regional users consume some services from the Redmond farm.

Figure 4 shows the average distribution of Office SharePoint Server 2007 users at Microsoft broken down by region.

Figure 4. Distribution of Office SharePoint Server 2007 users at Microsoft

Architecture

As part of the deployment of Office SharePoint Server 2007, Microsoft IT took the opportunity to redesign the solution. In the SharePoint Portal Server 2003 deployment, personal sites and portals shared the same services, whereas team sites had no services architecture. With the Web services model of Office SharePoint Server 2007, the confusion between portals and team sites is eliminated and they can be hosted together. The Office SharePoint Server 2007 infrastructure designed by Microsoft IT provides a Shared Service Provider (SSP) farm in each region, which provides services to child farms in that region.

Microsoft chose the dedicated SSP model for the following reasons:

• Consolidation of servers. The shared services model is one factor that enables the aforementioned reduction in server hardware.
• Clarity of administrative roles. With the separation of server roles, Helpdesk and site-level administrators can make changes on child farms where content resides but do not have access to service-level settings. This limits SSP administration to the MACS team—preventing unauthorized users from damaging enterprise-wide settings.
• Performance. With the SSP servers separate from content, Microsoft IT can restart individual content servers without compromising the availability of shared services to sites that reside elsewhere.
• Security. The smaller number of personnel granted access to a dedicated SSP farm helps protect usage data and other confidential information hosted there.
• Database throughput requirements. The Redmond-hosted SSP has a very high write-disk input/output compared to regional SharePoint operations and benefits from a dedicated SQL Server instance separate from other content and SQL Server operations. The separate hardware makes operational management of this requirement much easier.

Overview of SSP Model

For services, there are three deployments internally at Microsoft. Redmond is the largest service provider in the system, providing local services in the Americas and enterprise-scoped search. Singapore and Dublin are regional deployments that are self-contained. However, a custom-built profile replication engine synchronizes changes made to local profile stores with the other two regions.

Figure 5 illustrates the interaction between regional SSP farms.

Figure 5. Interaction between regional SSP farms

Each regional SSP farm has three nodes: two Web/query nodes and one index node. SSP-provided services include Search, People, Audiences, Profiles, Business Data Catalog (where enabled), and Usage Analysis Services. Each regional SSP crawls that region's content every 24 hours. The exception is Redmond, which also crawls content across all three regions weekly and imports Business Data Catalog data into local profiles. Because of this, Redmond requires a dedicated SQL Server instance to support its crawling function.

Child Farms

Child farms host sites and portals. Each child farm is dedicated to a particular class of sites, exists within a single data center, and consists of two or more load-balanced Web front ends. Some child farms contain a third node that is a dedicated search target, and host files for the indexes point to the dedicated front ends. A particular child farm may also have a local SSP that provides Excel Services to the sites on that farm. The Web front ends use various hardware configurations; processors may be 32 bit or 64 bit, dual core, with one or two CPUs, and have from 4 to 16 gigabytes (GB) of random access memory (RAM). Child farms are not federated across the Microsoft wide area networks (WANs) with the regional SSPs. This means that the local farms consume from the local SSP. For example, farms in Dublin consume only services from Dublin.

Interaction Between Child Farms and SSPs

The communication between child farms and SSPs occurs through Web service calls and, less frequently, direct database connections. Properties hosted on the child farm will be rendered even if the SSP farm is offline, but SSP-provided services like Search and People will not work. Figure 6 illustrates the interaction between the child farms and SSPs.

Figure 6. Interaction between child farms and SSPs

Excel Services

For optimal performance, Excel Services needs to run on a local server. Many regional child farms have two dedicated Excel Services servers, although Microsoft IT has learned that not all sites generate enough traffic to justify this level of service.

SQL Server Infrastructure

The Office SharePoint Server 2007 solution relies on a 64-bit SQL Server 2005 infrastructure. Each cluster contains multiple active nodes, in addition to single passive and backup nodes. The active nodes are clustered through Windows Clustering, a feature of Windows Server 2003. To run SQL Server 2005, Microsoft IT uses HP D585 eight-way servers with 8 GB of RAM, which use a shared storage area network (SAN). In its next fiscal year, Microsoft IT will examine cheaper storage solutions as the SAN environment comes to its end of warranty. Microsoft IT will also re-examine the multiple active/passive solution to determine whether a simple active/passive pair is more manageable.

Service Levels

The deployment of SharePoint Portal Server 2003 at Microsoft was a one-size-fits-all model that faced many challenges from the beginning. Microsoft IT had several customers that wanted to deploy custom code or request exceptions outside the constraints of the service definition. Tracking and maintaining these exceptions became both expensive and difficult. Running a model where there was no chargeback and no differentiation between highly customized services with dedicated hardware and a standard collaboration site quickly became unmanageable at scale. Therefore, Microsoft IT decided to define multiple tiers of service for its Office SharePoint Server 2007 service and hosting solution. The tiers were broken into three categories.

The Platinum tier:

• Hosts dedicated hardware for one or many properties owned by a single business team at Microsoft.
• Offers the maximum support for server-side customization.
• Is generally represented by high-value Web properties within the company.

The Gold tier:

• Hosts several customers on the same hardware but provides isolation via application pools for each customer.
• Permits some customization on the platform.
• Distributes the costs of hardware and hosting across customers.

The Silver tier:

• Supports thousands of customers per server.
• Allows no server-side customizations.
• Hosts the utility service at no charge but limits the amount of supported storage.

Any Microsoft employee can start a Silver-level site, and the resources are self-provisioned. Moving toward such a user-provisioned model to support the majority of the organization's intranet needs enables Microsoft IT to focus its efforts on maintaining the higher-level tiers of support for critical services.

Deployment Process

Microsoft IT has implemented Office SharePoint Server 2007 in multiple phases, from beta testing among relatively small groups of users in the early stages of product development, to the new farms made available across the environment at Beta 2, and finally to the current, enterprise-wide upgrade to the first Release Candidate (RC) version, RC0. This section focuses on the general process that Microsoft IT has used in performing large-scale upgrades to Office SharePoint Server 2007. This process includes:

• Choosing an upgrade method
• Planning
• Scheduling
• Preparing the environment
• Performing the upgrade
• Communication
• Post-upgrade support

Choosing an Upgrade Method

There are three main ways to perform an upgrade to Office SharePoint Server 2007:

• In-place upgrade
• Gradual upgrade
• Database attach

Microsoft IT used each of the three strategies in different regions and scenarios. However, the team overwhelmingly favored gradual upgrade and database attach due to the quantities of sites and data involved, in addition to the large number of users affected.

In-Place Upgrade

An in-place upgrade upgrades all SharePoint sites at once on the original hardware. IT personnel completely overwrite the old version with the new version and change the databases to work with the new version. All sites are unavailable to site visitors during upgrade, and the outage window for all users is the full time necessary to upgrade the entire server or server farm. For a large deployment, this can represent a significant amount of downtime.

In addition, an in-place upgrade is reversible only by rebuilding the previous system and restoring backups of the databases; users cannot view the previous versions of the sites after upgrade. Because of these limitations, Microsoft IT used this approach for build-to-build upgrades, where the team detached all of the databases from the farm (except for the configuration and content database that had the root site) and upgraded the other databases in parallel on other servers. After that operation was successful, Microsoft IT performed and in-place upgrade of the remaining database. This process helped mitigate the potential of data loss; if the upgrade had failed, Microsoft IT could have recovered the databases from backup rather than having to rebuild the farm.

Gradual Upgrade

In a gradual upgrade, one or more site collections can be updated individually. Before upgrading, the site administrator copies data from the original database to a new database. Original data is maintained until the site administrator explicitly deletes it. Although this approach allows upgraded sites to be rolled back if necessary, it also doubles, and sometimes triples, the amount of SQLServer storage space required. A built-in redirector component of Office SharePoint Server 2007 sends user requests back to the 2003 environment if a particular site is awaiting upgrade.

Although it is more complex and resource intensive, this more granular approach confines downtime to the specific sites being upgraded, which is a significant advantage in environments that have many frequently used sites. In addition, customizations can be handled on a case-by-case basis. Microsoft IT used gradual upgrades for team site collections in Redmond. Personal sites were upgraded as part of beta testing, but it proved to take longer and be more expensive. Team sites and personal sites in other regions were upgraded through database attach.

The approach worked as planned except for one minor but instructive problem. In a gradual upgrade, the original URLs point to the upgraded versions of the sites. Newly created URLs point to non-upgraded sites. Because Microsoft IT considered non-upgraded team sites secondary, it appended "TeamV2" to non-upgraded URLs. This confused end users, leading some of them to think that "TeamV2" designated the Office SharePoint Server 2007 sites. "Team2003" or "TeamOriginal" would have been better choices.

Database Attach

The third upgrade method, and the one that Microsoft IT used in most cases other than for team sites, is database attach. It is essentially an in-place upgrade performed on a copy of the content. First, Office SharePoint Server 2007 is installed on new hardware. Then, content databases are moved to the new hardware.

Finally, the upgrade process runs, building the Office SharePoint Server 2007 sites based on the databases. The upgrade does not touch the old databases; sites can be rolled back as long as the old databases are not taken offline. Microsoft IT used this approach when moving services or content to new hardware. Only content is actually moved in a database attach, so search scopes, settings, and customizations need to be reapplied.

Although all sites on a particular server are unavailable during a database attach, the process is fast. Microsoft IT found parallel farms to be very helpful during database attach upgrades. The team set up single-node farms with their own content databases. The team ran five or six at a time to make the upgrade faster, attaching everything to the final farm when they were finished. Microsoft IT recommends this approach for large farm upgrades.

Upgrade Performance

Table 1 shows the performance of upgrade methods.

Table 1. Measured Performance of Various Upgrade Methods on Microsoft IT Hardware

Planning

In a complex IT environment such as that of Microsoft, planning and preparation were essential for a successful enterprise-wide upgrade to Office SharePoint Server 2007. Microsoft IT prepared detailed plans and schedules for the upgrade and put considerable effort into preparing the environment to make the upgrade as easy as possible. Of course, customers will generally not have access to pre-release versions of software nor reason to test and install every version. Nonetheless, many of the pre-upgrade procedures that Microsoft IT used are generally applicable—and some of the procedures that the team skipped are instructive as well.

Early in the process, Microsoft IT set up Office SharePoint Server 2007 environments on virtual machines for engineering and operational testing. Testing was useful for identifying potential problems, verifying fixes, and validating settings and configurations before deploying the upgrade process.

Microsoft IT performed "dry runs"—basically, full upgrades that never go live to users—wherever possible. Dry runs became particularly important for database attach situations. Because that upgrade methodology depends on a large number of manual settings for success, dry runs helped Microsoft IT identify orphan and problematic sites before an outage occurred.

A very important step that Microsoft IT learned during the planning phase was the language packs have to be installed before an upgrade for the upgrade to be successful. Two planning steps that Microsoft IT did not take could have saved time and effort. The team also learned that it needed to create a process to track and report exceptions, site collections that the team had to upgrade either sooner than or later than planned. Poorly managed exceptions become time consuming and can negatively affect upgrade schedules.

Scheduling

Because nearly everyone at Microsoft relies on the SharePoint solution in one way or another, Microsoft IT had to schedule upgrades in a way that minimized the impact on day-to-day business. In gradual upgrades, the team could upgrade site collections one or a few at a time, in each case affecting a relatively contained user group for about four hours. Although database attaches are generally more efficient than gradual upgrades, they affect larger numbers of users at once. Therefore, Microsoft IT performed database attaches over weekends. The team also found that it worked best to schedule personal site upgrades based on site collection speed. Because of the relatively low site count, team and portal site upgrade schedules were based on database size.

Detailed schedules were very important to the process, but so was flexibility. In one case, backup delays during a database attach set the schedule back six hours—a significant delay when Microsoft IT had only from Friday evening to Monday morning to complete the upgrades. Even with thorough preparation, the team had to be flexible to deal with product bugs, missing language packs, stalled backups, exceptions to the schedule, missed settings, and re-installation of Web Parts and customizations. When the team set a realistic schedule, it could manage end users' expectations more easily. This also required updating the calendar frequently and providing it to everyone involved.

Microsoft IT learned not to allow upgrades to run at the same time as backup services. Microsoft IT had typically always performed backups before upgrades, but the SQL Server environment is shared, and the team was not upgrading everything on a SQL Server instance at the same time. This meant that the team had to allow for backups during the upgrade schedule in some instances.

Preparing the Environment

In addition to planning and scheduling, Microsoft IT optimized the infrastructure in the months leading up to actual upgrades. This not only made upgrades easier to perform, but also was a step toward maximum hardware consolidation.

First, Microsoft IT performed a thorough audit to determine exactly what it had, including hardware, software, databases, and numbers and types of sites. As much as possible, Microsoft IT balanced content loads among databases. The team also merged and split databases to reduce the likelihood of having to deal with extremely large ones. In addition, Microsoft IT consolidated server farms where possible.

Microsoft IT used the prescan tool (Prescan.exe) that ships with Windows SharePoint Services and Office SharePoint Server 2007. Prescanning can be performed on an entire farm or just a set of URLs—and is required before upgrading. It automatically discovers and reports customized site templates, custom Web Parts, and orphaned objects that could cause the upgrade process to fail. To validate that all orphaned objects were located, Microsoft IT used the 2003 orphan tool and wrote a Structured Query Language (SQL) process that might find a few orphaned objects that both of the other tools did not find. Microsoft IT allocated several days after each pre-upgrade scan to fix potential problems that the tool discovered.

Performing the Upgrade

The systematic process of upgrading to Office SharePoint Server 2007 is available from other sources and will not be exhaustively described here. However, Microsoft IT discovered several commonly overlooked practices that may be of value to customers who are considering or undertaking such an upgrade.

From an operational standpoint, Microsoft IT made two key discoveries. First, it found that if only one person was capable of implementing a particular task, that person's absence due to illness or conflicting schedules could stall the whole process. Microsoft IT found that having somewhat redundant skill sets among team members was useful, especially for mission-critical knowledge. Second, it discovered that consistently labeled logs of the entire process—from the preparation and prescanning of the environment through user service requests after the sites, portals, and services went live—were particularly useful to the product group's investigation of bugs and technical issues.

A content-related issue that Microsoft IT faced during the upgrade was deciding whether to force the reghosting of pages. When a SharePoint page is created, the layout template resides in the file system of the Web front end, whereas the content exists on the Microsoft SQL Server. If that site is customized through Microsoft Office SharePoint Designer 2007, the page becomes "unghosted," meaning that the customized layout template is subsequently stored on the SQL Server as well. During an upgrade, if pages remain unghosted, the layout files may not be properly upgraded and will continue to point to SharePoint Portal Server 2003 site definitions.

Although Office SharePoint Server 2007 features like the recycle bin would technically exist after the upgrade for unghosted pages, they would not be available to end users. Setting the pages to reghost during the upgrade resets them to the standard site definition, enabling the use of new features, but it requires replacing the customizations. Reghosting was not forced for two important reasons: Microsoft IT had a schedule to follow to get valuable feedback back to the product groups, and the team wanted to allow end users the choice when working on customizing their sites. Microsoft IT had to consider whether replacing the customizations after reghosting would require more work than adding Office SharePoint Server 2007 features to an unghosted page.

Finally, upgrading to Office SharePoint Server 2007 can be data intensive. In addition to pre-upgrade efforts to shrink databases that were already near maximum size, Microsoft IT set databases, temporary databases, and log files to autogrow, ensuring that enough space existed to finish the upgrades. Many of the databases were sized in advance to minimize the fragmentation that growth causes.

Communication

Because the upgrade to Office SharePoint Server 2007 affected nearly everyone at Microsoft at one point or another, communication during the process was essential. In most ways, Microsoft IT was very successful at communicating with end users, although communication was a learning process. For example, Microsoft IT team members found that using images such as screenshots helped them communicate with personnel who possessed varying levels of technical skills and language boundaries. Managing user expectations was easier when Microsoft IT informed users about limitations in advance, such as the fact that sites could not be reverted during the upgrade process. Also related to user expectations, Microsoft IT found that committing to hard deadlines created more problems than it solved.

Internal to Microsoft IT, an established triage process helped the team synchronize on a daily schedule to track and assign open issues. Early in the upgrade process, Microsoft IT met once a week to address issues and exceptions. As the deployment dates grew closer, the team met once a day.

Post-upgrade Support

As with any complex IT effort, completing the upgrades did not end Microsoft IT's work. However, the detailed upgrade logs that the team had kept made addressing post-upgrade issues much easier. The team found that locking previous site versions to prevent users from modifying old sites was important. The team also dedicated substantial resources to training support staff and users throughout and after the upgrade process.

Three teams monitor the Office SharePoint Server 2007 environment by using Microsoft Operations Manager 2006 along with various add-in packs. The Data Center Operations team monitors the infrastructure—operating system, hardware, and infrastructure covered by the Web Sites and Services management pack. Data Center Operations provides continuous coverage and a 30-minute response time on Web Sites and Services outages. The MACS team monitors Office SharePoint Server 2007, SharePoint Portal Server 2003, Windows SharePoint Services version 2.0, Windows SharePoint Services 3.0, Internet Information Services (IIS), Windows Clustering, and Distributed File System (DFS). MACS team members currently use pagers for critical alerts, but they expect to outsource server monitoring and initial response to another group, enabling the team to focus on improving systems rather than handling emergencies. The team currently uses MOM 2005 and is considering a move to Microsoft System Center—the new, comprehensive system monitoring solution. Finally, the Platform Services IT group monitors SQL services, DFS, and Windows Clustering on the SQL Servers.

Microsoft IT uses a service level agreement (SLA) to prioritize support issues. Technical support is available 24 hours a day, 7 days a week. Tickets are resolved according to four levels of priority, with business-critical issues having precedence over mission-critical ones. Microsoft IT does not fix broken customizations. The team's goal is 99.9 percent availability. Automatic backups of databases occur once every 24 hours, and Microsoft IT retains backups for two weeks.

Deployment Results

Table 2 compares the original SharePoint Portal Server 2003 offering with the current Office SharePoint Server 2007 offering.

Table 2. Comparison of Previous and Current SharePoint Environments

Although the number of team sites and site collections has decreased, the amount of content stored has almost doubled. Yet, because of pre-upgrade optimization and the flexible architecture, Microsoft IT projects that the total number of servers used to host SharePoint sites will drop from 99 to 69. The Windows SharePoint Services 3.0 HostHeader sites are a deprecated service that Microsoft IT is moving to the new Office SharePoint Server 2007 platform at the time of this writing.

Performance

A Minimum Performance Level (MPL) serves as Microsoft IT's goal for performance from a user standpoint. Customers not local to a given region but still using that region's farms are typically the ones whose results fall outside the MPL. Microsoft IT's SQL Server infrastructure in Dublin has a heavier workload than those in Redmond or Singapore, contributing to somewhat lower performance for Europe, Middle East, and Africa (EMEA) users. In addition, all users rely on the Redmond farm for certain, non-mirrored services; regional users experience longer wait times for those services simply due to distance. Microsoft IT constantly analyzes the environment, attempting to bring all employees up to the MPL.

Table 3 indicates user compliance with MPL goals at Microsoft.

Table 3. Microsoft User Compliance with MPL Goals

Services

This section provides an overview of various services provided by the configuration choices that Microsoft IT made after the deployment of Office SharePoint Server 2007.

Search and Crawl

Each region has a single Office SharePoint Server 2007 index. After upgrades are complete, high-priority Office SharePoint Server 2007 content will be crawled once every 24 hours, whereas the lowest-priority content will be crawled about once a week. This frequency will vary somewhat among regions. The performance goal is 20 documents per second averaged over multiple crawls.

Deprecated services like the Windows SharePoint Services 3.0 HostHeader sites still require a full crawl, as opposed to the more efficient index-based crawling that Office SharePoint Server 2007 enables. Therefore, search performance on the Windows SharePoint Services 3.0-based properties is not equal to that on upgraded sites. Eventually, Microsoft IT will transition the entire environment to Office SharePoint Server 2007.

Dedicated query servers in the SSP farms process search requests, with an average response time of two seconds per query. Microsoft IT is in the process of adding RAM to the query servers to increase performance. In addition, Microsoft IT has enabled limited crawling of Siebel CRM data by using Business Data Catalog. The Redmond deployment receives the largest number of queries—approximately 575,000 per month.

Excel Services

Most Silver-level farms have dedicated Excel Services servers that are part of the child farm rather than the SSP farm. The Redmond deployment serves Excel Services from the SSP farm. The maximum file size is set at 100 MB. Microsoft IT enables refreshing external data through a published, low-permission account that allows users (who have already been authenticated elsewhere) to log on and retrieve the data. Although Office SharePoint Server 2007 can be set to use only Office Data Connection (.odc) files in determining what constitutes a trusted data source, the Microsoft IT solution uses embedded connections as well. Eventually, Microsoft IT may implement a process whereby users can request new trusted locations for Excel Services worksheets.

Business Data Catalog

Business Data Catalog runs only in Redmond because 99 percent of the data sources are local to that deployment. Currently, Microsoft IT has enabled Business Data Catalog in a limited way, allowing profiles to import CRM data from the Siebel CRM system. The Business Data Catalog feature is extremely powerful and has great potential at Microsoft, but planning it properly is important. First, Business Data Catalog can expose sensitive data. It can also import unwieldy quantities of data if it not properly configured. Finally, mapping LOB data to an Office SharePoint Server 2007 interface requires an XML schema to translate between them, which can be a challenging exercise. For these reasons, Microsoft IT is being conservative in implementing Business Data Catalog to balance both the impact for crawl and load on the servers with the business benefit.

Profile Import and Synchronization

All regions import Active Directory profiles. As mentioned earlier, in Redmond, the system imports specified LOB data into user profiles. A full profile import happens once a week, whereas incremental imports happen every 24 hours. The system handles more than 130,000 Active Directory records.

The Office SharePoint Server 2007 deployment within Microsoft has MySite hosts in each region, together containing more than 55,000 personal sites. These sites enhance the social networking features of Office SharePoint Server 2007. Microsoft IT added custom code to replicate user changes to personal sites worldwide in seconds by using a change log. The search index reflects the changes after the next crawl.

InfoPath

Microsoft Office InfoPath® 2007 is a child farm feature rather than an SSP feature, although because the architecture has multiple Web front-end servers, it does rely on the regional SSP to store session state. Microsoft IT has enabled Office InfoPath 2007 on all Office SharePoint Server 2007 child farms. Silver-level farms do not support forms that contain code, but Gold-level and Platinum-level farms do.

Future Efforts

Microsoft IT plans to use single sign-on (SSO) for Business Data Catalog functionality on Gold-level and Platinum-level farms within the next 12 months. The team also recently got approval to implement Microsoft Office Project Server 2007 enterprise wide. Microsoft IT uses Microsoft Internet Security and Acceleration Server 2006 to provide security-enhanced access to certain Office SharePoint Server 2007 properties via the Internet. Eagerly anticipating the release of Microsoft Office PerformancePoint™ Server 2007, Microsoft IT is planning to take advantage of expanded scorecard functionality of ProClarity at both Gold and Platinum service tiers to determine how well those solutions scale. Finally, Microsoft IT plans to move most properties that still rely on Windows SharePoint Services 3.0 to Office SharePoint Server 2007—especially important for testing the next generation of server products for the Microsoft Office system.

Lessons Learned

This section lists the lessons learned during the upgrade of Office SharePoint Server 2007 Beta 2 on the worldwide servers that host the internal collaboration services that Microsoft IT manages. The scope of Beta 2 upgrades from SharePoint Portal Server 2003 to Office SharePoint Server 2007 included Redmond team and portal sites, in addition to personal sites and shared services in Redmond, Dublin, and Singapore.

The following lessons relate to project successes:

• A process that helps the team make efficient decisions about exceptions and open issues will help keep the project on schedule. Microsoft IT met weekly in the early stages of the deployment process and had daily meetings as deployment dates grew closer.
• Upgrading is an excellent opportunity to organize and solidify the collaboration infrastructure. Some cleanup tasks, like running Prescan.exe, are mission critical because they help resolve issues like orphaned objects, which can cause the upgrade to fail. Others, like optimizing database size and deleting unused content, may not be necessary but are a good idea. Not only can they help the upgrade run more smoothly, but they can also make the new environment easier to use—and can help reduce costs through a more efficient use of hardware. Microsoft IT was meticulous about pre-upgrade optimization, and the team experienced the benefits firsthand.
• Upgrading customizations and Web Parts so that they function properly after the upgrade can be time-consuming. Microsoft IT allocated several days after each time it ran Prescan.exe to fix such issues.
• Dry runs may seem like wasted effort, but upgrading Office SharePoint Server 2007 in an enterprise environment is complex no matter how it is done—and fixing mistakes midstream not only is time-consuming but also affects end users. In Microsoft IT's case, performing dry runs helped identify and resolve many issues that could have upset the upgrade itself.

The following lessons relate to problems that Microsoft IT encountered:

• Backups cannot be performed while upgrades are running. Microsoft IT tried to minimize such conflicts but did not avoid them altogether. Backups should be performed and validated—and automatic backup processes should be disabled—before upgrades begin.
• Microsoft IT found that defining roles and responsibilities during the process was difficult. The team could have put more effort into structuring the team beforehand. In particular, the team should have considered whether sufficient redundancy of mission-critical skills existed. After an upgrade is underway—particularly in the case of database attach upgrades—the absence of a team member can cause schedule delays if nobody else on the team shares that person's skill set. This is especially true if the upgrades run though the day and night.
• Required language packs should be identified and installed beforehand. Language packs can generally be installed during an upgrade, but this is time-consuming and might cause problems. Microsoft IT found this step especially important for the database attach upgrades that it performed.
• End users at Microsoft would have benefited from a pre-defined support and escalation policy. Microsoft IT defined the policy after the upgrade was completed, leading users to have unrealistic expectations. More generally, managing user expectations early and proactively tends to increase users' satisfaction.

Best Practices

Microsoft IT recommends the following best practices for deployment of an Office SharePoint Server 2007 solution within an enterprise. In some cases, deploying the solution without following these suggestions is possible, but this may result in unnecessary expense or effort.

Best practices for initial activities are as follows:

• Perform a thorough audit of all products and platforms that the upgrade may affect, including Windows Server, SQL Server, IIS, Windows SharePoint Services, and prior versions of SharePoint Products and Technologies, in addition to portals, sites, and content.
• Optimize and clean up current environments. These activities include:
• Merging and splitting databases to optimize sizes.
• Balancing content loads among databases.
• Consolidating server farms where possible.
• Make sure that level settings are consistent across the environment to be upgraded.
• Identify and install missing language packs. Doing so during the upgrade itself is inefficient and can cause problems.
• Document initial activities and discoveries made during the process, creating a reference in case personnel need to look back during the upgrade to see what actions were performed and to help make future upgrades easier.

Best practices for the schedule part of planning are as follows:

• Schedule and analyze prescans by using the prescan tool (Prescan.exe) at least two days before each upgrade.
• Leave space on the schedule to catch up after unexpected events. Upgrading is a complex process with a big impact on the business. It should be completed efficiently but not rushed.
• Upgrades can fail for a variety of reasons. Have a contingency plan in place in case minor or major roadblocks occur.
• Schedule database upgrades based on free space.
• Determine the priority of exceptions and to what degree they will be allowed to affect the schedule. Communicate this information to affected users.
• Frequently update the calendar and provide it to all teams involved.
• Schedule, perform, and confirm backups before performing upgrades. Ensure that backups and upgrades are not scheduled to run at the same time, because this will cause upgrade failure.
• Schedule time and resources for testing and previews of the new functionality.
• In performing a database attach, the number of sites is the most important factor in how long the upgrade will take. If many sites exist on a farm, schedule upgrades based on the number of sites. If there are fewer sites (as may be the case with team and portal sites), schedule them based on the quantity of data.

Best practices for the communication part of planning are as follows:

• Before upgrading, communicate that sites cannot be accessed or reverted during the upgrade.
• Define the upgrade process thoroughly before sending communications about the schedule and process. Not doing so will confuse end users and may encourage them to think the schedule is negotiable.
• Avoid committing to specific dates where possible.
• Use images in communications to explain the upgrade process to non-technical personnel.
• Send notifications to site owners and administrators often—before, during, and after the upgrade.
• Finishing the upgrade is not the end of the upgrade process. Define, convey, and implement a post-upgrade communication plan.
• Define the support process for exceptions and escalations. If some technical issues are not supported, make sure that users know beforehand.

Best practices for the planning process are as follows:

• Training is key to getting the most benefits from Office SharePoint Server 2007, even if users are familiar with Windows SharePoint Services 2003 and SharePoint Portal Server 2003. Plan to invest time, resources, and money in training users.
• Determine whether to reset pages to site definitions (reghost) or leave customizations intact during the upgrade.

Best practices for the upgrade process are as follows:

• Perform a dry run whenever feasible. Identifying and fixing problems beforehand will increase the likelihood of maintaining the schedule after the upgrade begins.
• Do not avoid testing because of limited hardware resources. Use virtual machines for testing if excess hardware capacity is not available.
• Data requirements usually grow during an upgrade; exceeding the capacity of a database will cause the upgrade to fail. Increase the target databases to appropriate sizes before the upgrade process and set databases, temporary databases, and log files to autogrow.
• Watch for upgrade problems caused by full-text search. Moving the SQL Server database to a different server or disabling full-text search on the SQL Server computer can mitigate the impact.
• Maintain a list of which sites have been upgraded.
• Move exception sites to a new content database via the Stsadm.exe command-line tool so that they do not interfere with the general upgrade process and schedule.
• Shut down and restart (bounce) SQL Server computers between SQL Server instances.
• It is difficult to complete tasks correctly the first time no matter how carefully custom templates, definitions, and Web Parts are configured before the upgrade. If performing a gradual upgrade or database attach, maintain a copy of the previous environment to allow for rollbacks if necessary.
• Do not finalize the upgrade until you are sure the upgrade is finished. Finalizing the upgrade removes the connection to the previous version and deletes any temporary data, and it is irreversible.
• Allow only one administrator at a time to make changes to the configuration database.

Best practices for validation are as follows:

• Document the entire process in as much detail as possible. This will enable personnel to track problems back to their sources and will make future upgrades easier.
• Develop custom user guides to help users understand the specifics of your Office SharePoint Server 2007 environment.
• Lock previous site versions after the upgrade to prevent users from making updates to old sites. This not only prevents general confusion, but also helps avoid a situation in which misapplied updates are lost when old versions are taken offline.

Conclusion

Office SharePoint Server 2007 facilitates the creation and deployment of feature-rich and content-rich Web sites and helps organizations integrate business processes and applications through a full set of collaboration and personalization features for information workers. Enhanced features as compared to previous versions include an improved administration model, new compliance features and capabilities, better support for network configuration, and enhanced extensibility of the object model that makes custom applications and components easier to deploy.

These key benefits help streamline business processes and make information easy to share through integration with familiar desktop applications and tools. In addition, instead of relying on separate fragmented systems, customers can now take advantage of a single integrated platform in Office SharePoint Server 2007 that can support all of the intranets, extranets, and Web applications across an enterprise.

Microsoft uses Office SharePoint Server 2007 to facilitate information sharing and collaboration both internally and with external partners. In general, employees find it intuitive and easy to use, but the infrastructure itself has many complex parts. Part of the company's upgrade efforts included simplifying and streamlining that infrastructure to make it more robust and less expensive, especially by implementing a shared services architecture.

Microsoft found that meticulous planning was the key to rolling out Office SharePoint Server 2007 with the least inconvenience to end users. These efforts included extensive pre-deployment testing and validation in addition to dry-running the upgrades by using virtual machines. Auditing and optimizing the pre-installation environment help smooth the upgrade process and help achieve the greatest cost efficiencies.

Because of its planning and preparation, Microsoft IT completed most upgrades on schedule with few negative impacts on users. A few simple steps—like consistently pre-installing language packs and avoiding conflicts between backups and upgrades—would have helped the process go even more smoothly. In the end, however, users not only were largely satisfied with the upgrade process, but also found the enhanced features of Office SharePoint Server 2007 to be very beneficial in enabling collaboration and communication across the enterprise.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information through the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

© 2007 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Excel, Groove, InfoPath, Internet Explorer, Outlook, PerformancePoint, SharePoint, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

All other trademarks are property of their respective owners.