Best Practices for an Enterprise Deployment of Microsoft Project Server 2002

Article
02/20/2014

Archived content. No warranty is made as to technical accuracy. Content may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Published: February 1, 2003

By Lou Lucarelli, Microsoft Corporation

Applies to:
Microsoft Project Professional 2002
Microsoft Project Web Access 2002
Microsoft Project Server 2002

Summary: This article discusses best practices for a large, successful deployment of Microsoft Project 2002, with specific implementation tips, frequently asked questions, troubleshooting tips, and deployment issues and data. (19 printed pages)

Introduction

Discover how the Microsoft® Project team hosts its own Microsoft Project Server for a large number of Microsoft Project Professional 2002 and Microsoft Project Web Access 2002 users in a multi-site deployment. This enterprise wide deployment services over 900 projects and 3500 users. Starting off with the top five lessons learned, we then delve into the details of how this multi-site deployment's architecture provides for each hosted group, separate instances of Microsoft Project Server 2002, Internet Information Services (IIS) 5.0 virtual directories, databases, and SharePoint™ Team Services from Microsoft 1.0 project subwebs, all sharing a single hardware configuration. We publish our best practices, with specific implementation tips and frequently asked questions by internal customers, and end with troubleshooting tips and deployment issues and data.

Top Five Deployment Guidelines

When deploying Microsoft Project Server in the enterprise, we recommend that you follow these general guidelines. For more details, see the "Enterprise Deployment for Internal Hosting," "Operations Timeline," and "Best Practices" sections of this study.

Plan for growth during initial deployment Deploy Microsoft Project Server in an environment that can grow in parallel to the growth of your organization's project management needs. Planning for scalable growth minimizes changes to your original deployment as the number of users increases, as new server connections and network traffic can have a significant effect on overall system performance. Deploy with a data transfer rate of 100 megabytes (MB) per second (or better) throughout your entire hosting environment for all servers in your Microsoft Project Server deployment. Add additional hardware (more processors, more memory, or more servers) as they are needed.
Maximize the throughput of the database server The Microsoft Project Server database computer often becomes overloaded with network traffic. Irregularities, including SQL Server errors, publishing problems, and inconsistent open/load times may be experienced if the load on the database server is consistently high. When the load on the server hosting the Microsoft Project Server database consistently reaches 60 to 65 percent of total capacity of the % Processor Time setting in the Microsoft Windows® System Monitor, we advise either upgrading the hardware capabilities of the server or adding a second server to offload some of the processing. Use multiple network interface cards to increase the number of users who can access the database. Finally, use a static IP address for the database computer, which improves SQL Server connection times when data connections may be slow (for instance, if the client is connecting from a remote geographic connection).
Deploy SharePoint Team Services on production hardware If you are using SharePoint Team Services to manage documents and issue tracking for your projects, plan on deploying what you will eventually need for full production. Limit the IIS load on each server running SharePoint Team Services to about 750 subwebs, due to the fact that SharePoint Team Services 1.0 will not operate in an IIS-clustered environment. Additionally, it is cumbersome to migrate a SharePoint Team Services implementation across hardware. For every project published to Microsoft Project Server, a subweb (virtual directory) is created on the server running SharePoint Team Services.
Offload the Views Notification services to a dedicated server Views Notification services can place an enormous load on the Microsoft Project Server. We recommend moving the Views Notification services during the initial deployment. If Views Notification services are offloaded after your deployment of Microsoft Project Server, then you can run into other serious issues, including updating the registry in the Views Notification server, and provisioning errors.
Monitor system performance continually Be proactive about monitoring your system, seeking indications of trouble rather than reacting to trouble on a case-by-case basis. Limitations within your network and the SQL Server computer are the biggest causes of performance issues for Microsoft Project Server. Use System Monitor to keep an eye on the servers in your Microsoft Project Server deployment. Standardize your troubleshooting procedures so that you will have a ready checklist when problems do crop up. This list will not only help you find the causes of problems, but also implement solutions more quickly.

Enterprise Deployment for Internal Hosting

The Microsoft Project team needed to deploy Microsoft Project Server using an architecture that allowed for rapid growth in the number of users connecting to the server (see Figure 1 for an illustration of the hardware used in this effort). Each group of users would have both enterprise features and documents and issues tracking available to them. To minimize the workload of the IIS server, servers running SharePoint Team Services, the Microsoft Project Server database, and eventually the Views Notification services were offloaded to separate servers instead of residing on the server hosting both IIS and Microsoft Project Server. Virtual servers were used to allow each project group its own instance of Microsoft Project Server and related features.

This design choice allowed separate groups within the company to have their project management data isolated from other groups while utilizing the same set of server computers. To increase throughput, 100 MB network interface cards were used on every server. All servers were placed in the same testing environment in a secure area of a single domain.

Figure 1: The eventual configuration of the Microsoft Project Server hosting deployment

Server Hardware

Table 1 lists the specific hardware specifications for each server used during internal hosting of Microsoft Project Server. Microsoft Project Server runs on an IIS server that can be clustered. SharePoint Team Services 1.0 runs on an IIS server that cannot be clustered.

Table 1 Server hardware specifications

Microsoft Project Server IIS Server specifications

Manufacturer / Model	Compaq ProLiant DL580
Operating System	Windows 2000 Server, Service Pack 3
Processor	4 x 700 MHz Intel Pentium III
RAM	1.0 GB
Hard Drive	4.0 GB SCSI
Network Interface Card	2 x 100 MB Compaq NC3134 Fast Ethernet

Microsoft Project Server Database Server specifications

Manufacturer / Model	Compaq ProLiant DL580
Operating System	Windows 2000 Advanced Server, Service Pack 2
Processor	4 x 900 MHz Intel Pentium III
RAM	3.0 GB
Hard Drive	2 x 9.1 GB Ultra2 SCSI internal hard drives; 14 x 18.2 GB Ultra3 SCSI external hard drives
Network Interface Card	2 x 100 MB Compaq Fast Ethernet

SharePoint Team Services Server #1 specifications

Manufacturer / Model	Dell OptiPlex GX150
Operating System	Windows 2000, Service Pack 2
Processor	700 MHz Intel Pentium III
RAM	512 MB
Hard Drive	6.0 GB
Network Interface Card	100 MB Fast Ethernet

SharePoint Team Services Server #2 specifications

Manufacturer / Model	Dell Precision 420
Operating System	Windows 2000, Service Pack 2
Processor	2 x 933 MHz Intel Pentium III
RAM	512 MB ECC
Hard Drive	2 x 36.0 GB
Network Interface Card	100 MB Fast Ethernet

Views Notification Server specifications

Manufacturer / Model	Compaq W6000 Workstation
Operating System	Windows 2000 Advanced Server, Service Pack 2
Processor	2.0 GHz Intel Pentium IV
RAM	787 MB
Hard Drive	18.0 GB SCSI
Network Interface Card	100 MB Fast Ethernet

Software Installation

We set up a number of different IIS virtual directories and corresponding registry entries in order to facilitate hosting multiple groups on the same server while maintaining distinct databases for each group (this technique is referred to as multi-site deployment in this case study). The easiest method to perform this step is to use the Site Editor Utility, a utility that allows you to manage your Microsoft Project Server sites, on your IIS server. It is available as a download in the Microsoft Project Server 2002 Resource Kit. For further registry management information, see the Microsoft Project Server Registry and Global Settings: Microsoft Project 2002 article.

Caution Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

There were no other special steps performed during the installation of Microsoft Project Server. Specifying the separate databases, the services for Microsoft SQL Server™ Analysis Services, and the server running SharePoint Team Services were normal procedures performed during Microsoft Project Server setup.

Operations Timeline

This section presents a month-by-month timeline, running from May 2002 through October 2002. The initial deployment of Microsoft Project Server experienced much growth over a short timeframe; consequently, this rapid growth drove many improvements, which we discuss in the following section.

May 2002

We began hosting Microsoft Project Server for internal Microsoft users. Initial use was light and steady. We established a maintenance process that identified the areas that were causing the most trouble for our end-users and set up a process to track system performance using System Monitor. Figure 2 illustrates the initial deployment of the Microsoft Project Server and database servers with the Views Notification services on the Microsoft Project Server and a single network interface card in the database server.

Figure 2: Initial network topology between IIS server and database server

Figure 2: Initial network topology between IIS server and database server

June 2002

We started to see a jump in overall network traffic.

July 2002

We began to see a large number of users put stress on all servers in the deployment. Continuous monitoring of the server hardware indicated that the database server was not able to handle the load.

We ran into the following issues:

Significant, stressful loads on several servers in the deployment To pin down the problem-causing components, we used System Monitor to analyze all components in the system. Because we began monitoring immediately after deployment and thus had established baselines to review, we were quickly able to identify the individual component that was the slowest or most prone to failure. For more information, see the "Tracking system performance using System Monitor" section later in this article.
The database server was identified as the primary source of poor network performance We configured a new, more powerful database server and migrated the existing data to the new server. Once the new database server was up and running with Microsoft Project Server, we continued monitoring the new server using System Monitor. For more information, see the "Tracking system performance using System Monitor" section later in this article.
The network connection between the database server and the IIS server experienced heavy loads when views were being published To solve this problem, we offloaded the Views Notification services to a dedicated server. This moved processor-intensive tasks like the XML requests used to build OLAP cubes and e-mail notifications and reminders services. We also added network interface cards to ensure a dedicated network connection for each of the servers. This allowed the IIS server time for other tasks. We found that the strain on the IIS server processing views and generating cubes became the primary performance issue when our hosting effort reached about 40 separate project groups (varying in size from 10 to 200 resources). Figure 3 illustrates the IIS server, an offloaded Views Notification server and a database server, each with its own network interface card. For more information, see the "Offload the Views Notification service to a separate server" section later in this article.

Figure 3: Updated network topology among the IIS server, the database server, and the newly-added Views Notification server

August 2002

By August, the load on our servers was roughly 10 times as high as the load in late June.

We ran into the following issue:

General network congestion between the Microsoft Project Server and the database server We detected a high amount of network traffic going in and out of the database server, and the network interface card on the database server was being saturated. We noticed that as the number of hosted organizations increased, network traffic metrics did not seem to fluctuate with variations in the number of logged-on users. We used System Monitor to detect this problem by viewing the Network Interface: NIC#1 bytes total/sec and packets/sec counters from the database server's network interface card side-by-side with the Web Service: Default Web Site: Total NonAnonymous Users counter. Over the course of several days, we collected data during peak usage times (11:00 A.M. to 4:00 P.M.) which indicated that, while the number of users accessing the Microsoft Project Server was fluctuating throughout the time period, the values for the network counters were almost completely static.
This analysis was further confirmed by a check of the CPU Utilization counter on the database server, which indicated underutilization of this resource (less than 40 percent) despite a growing number of concurrent users. To enhance throughput of network traffic in and out of the database server, a second network interface card was added and assigned a static IP address. We then modified registry settings on the Views Notification server and the Microsoft Project Server to address the two separate network interface cards. Figure 4 shows the revised network topology among the IIS server, Views Notification server, and the database server, with the addition of a second network interface card in the database server.

Figure 4: Updated network topology with the addition of a second network interface card in the database server

October 2002

Growth was steady. Usage loads were still very large, but generally running smoothly.

We ran into the following issue:

Certain customers were experiencing slow network speeds We were able to determine that this problem was not specific to Microsoft Project. When we changed the database connection information in the registry of the IIS server from the Microsoft Project Server friendly name to a static IP address, the connection speed improved. For more information, see the "Configure the Microsoft Project Server IIS connection information as a static IP address" section later in this article.

Statistics over time

Table 2 shows the growth of individual users during the Microsoft Project Server hosting effort. As of October 20, 2002, there were 85 hosted groups with 982 total projects and 3,528 total resources. For more statistics, see Appendix C.

Table 2 Visitor session statistics

Timeframe	Visitors	Unique Visitors	Visitor Minutes
April 8 – May 5, 2002	244	59	1,774
May 6 – June 2, 2002	262	54	2,672
June 3 – June 30, 2002	726	130	6,847
July 1 – July 28, 2002	3,069	467	38,067
July 29 – August 25, 2002	4,832	666	48,604
August 26 – September 22, 2002	4,956	772	44,453
September 23 – October 20, 2002	5,630	772	48,579

Figure 5 shows the increase in usage of Microsoft Project Server by the number of groups. Each group has its own database server instance and IIS virtual root in Microsoft Project Server.

Figure 5: The increase of hosted groups within Microsoft

Best Practices

This section details the best practices resulting from the experience of hosting Microsoft Project Server in a multi-site environment.

Best Practice #1: Plan for Growth During Initial Deployment

Anticipate how your hosting environment will scale over time. If you anticipate moderate to high growth, develop your system with robust hardware or create a system setup that is flexible and versatile to allow for expansion.

Microsoft Project Server / IIS Server

Microsoft Project Server was deployed on a single node IIS cluster, allowing for future growth by adding additional IIS server computers to the cluster.

Microsoft Project Server Database Server

This server stores the Microsoft Project Server databases for each group of users. There is a high volume of network traffic to and from the database server, including opening and saving projects, and the processing of views. It is beneficial to have multiple network connections to the database server, more so than any other server in the deployment.

SharePoint Team Services

The servers running SharePoint Team Services store documents and issues for the users of Microsoft Project Server. Microsoft Project Server requires proper provisioning for SharePoint Team Services to work properly. About 750 subwebs can exist on a server running SharePoint Team Services before performance issues arise, so two servers were configured during installation to handle an expected increase in the number of users.

Note: It is not possible to deploy SharePoint Team Services on an IIS cluster. Multiple servers running SharePoint Team Services can only be used in a multi-site configuration.

Views Notification Server

The Views Notification server has three primary functions: publishing views in Microsoft Project Web Access, notifications and reminders e-mail services, and OLAP cube generation. For smaller deployments, the Views Notification process can be run on the Microsoft Project Server computer. For large deployments, as described in this article, we recommend that the Views Notification server be offloaded during initial deployment.

Optimize network bandwidth among servers

Server connections and network traffic can have a significant effect on overall system performance. If possible, maintain a data transfer rate of 100 MB-per-second (or better) throughout your entire hosting environment for Microsoft Project Server. The server hosting the Microsoft Project Server database tends to be limited by network traffic. If possible, use multiple network interface cards to increase the number of users that can access the database server.

Use non-expiring credentials for user accounts

For security purposes, large organizations typically implement user domain authentication credentials that expire periodically. Passwords that are used to access the Views Notification server and the server running SharePoint Team Services can cause a lot of difficulty for your end-users if they expire. Therefore, use credentials from a local Windows NT user account with a non-expiring password when running Microsoft Project Server COM+ packages—for example, PSCOMPlus.exe. This will prevent you from having to rerun the COM+ package later if the password expires and prevents unwanted failures for users accessing SharePoint Team Services or the SQL Server Analysis Services OLAP features.

We also found it beneficial to use the same Windows credentials for logging on to both the IIS and Views Notifications servers (after the Views Notification service was separated to its own server). If this password expires, it is best to log off from both computers, log back on, and then verify that all services restarted correctly.

Best Practice #2: Maximize Throughput on the Database Server

The Microsoft Project Server database server is the prime candidate as one of the initial bottlenecks as your deployment grows. When the load on your database server hosting consistently reaches 60 to 65 percent of total capacity of the % Processor Time setting in System Monitor, we advise either upgrading the hardware capabilities of the server or adding a second server to offload the views processing.

Add a second network interface card to the database server

To maximize network throughput to and from the database server, add a second network interface card.

Configure the Microsoft Project Server IIS connection information as a static IP address

Use a static IP address for the server hosting the Microsoft Project Server database. In the registry settings on that server, refer to the database computer with its IP address rather than its friendly name.

In the registry on the Views Notification server, change all instances under WebClient Server and the ODBC.INI from the name of the database server to the static IP address given to the new network interface card added to the Microsoft Project Server. This causes all connections to the database server to be made to the new network interface card, which has been given the static IP address and is not registered with the Microsoft Project Server DNS name.

Making this change allows both the Views Notification server and the Microsoft Project Server to communicate with the database directly without having to compete for network bandwidth (two servers, two network interface cards).

To change the database server connection information in the registry, in the HKEY_LOCAL_MACHINE\Software\Microsoft\Office\10.0\WebClient Server\<customer>\Datasets\Application\connectInfo section of the registry, change the Data Source property from the server name to the static IP address.

Best Practice #3: Deploy SharePoint Team Services on Production Hardware

Once you have deployed Microsoft Project Server, depending on your configuration, it can then be difficult to make changes to a server running SharePoint Team Services.

For every project published to Microsoft Project Server, a subweb (virtual directory) is created on the server running SharePoint Team Services. The sizing of the server running SharePoint Team Services depends on the estimated usage load (number of users, projects, and content stored on the server). A general guideline is to limit the IIS load on each server running SharePoint Team Services to about 750 subwebs.

If you are deploying Microsoft Project Server in a multi-site environment, scaling out and adding SharePoint Team Services servers is the easiest way to increase the number of subwebs as the number of projects in the Microsoft Project Server database increases. If you are not deploying Microsoft Project Server in a multi-site environment, anticipate the number of subwebs you will need on the server running SharePoint Team Services and make your hardware decisions based on these future projections.

Servers running SharePoint Team Services cannot be clustered

SharePoint Team Services 1.0 delivers content that resides on a server running IIS as well as the server that stores the Microsoft Project Server database. Documents reside on the front-end computer, but some information regarding these items also resides in the database. Issues content, on the other hand, resides exclusively in the Microsoft Project Server database. SharePoint Team Services relies on Windows NT security groups for the subwebs from information stored on the front-end computer. These are the biggest reasons why IIS clustering is not possible. Database clustering is possible, but considering that the database loads are minimal, this is not required.

Hardware recommendations for the server running SharePoint Team Services

Since clustering servers that run SharePoint Team Services 1.0 is not possible, it is important to estimate the expected future load on the server running SharePoint Team Services. Once you have set up a server running SharePoint Team Services, if your deployment is single-site (one Microsoft Project Server database is shared by everyone), then you cannot choose to use an additional server if you run into high usage and content loads. The only option in this scenario is to migrate the subweb to a new server, which is not a trivial process. To ensure minimal downtime, we advise that you install a server that is robust enough to handle your expected load.

Note: Extra RAM improves subweb creation time and page rendering times; however, it is subject to the same 750 subweb limitation.

Deploying additional servers running SharePoint Team Services

Every server running SharePoint Team Services works independently. Adding additional servers to a multi-site Microsoft Project Server deployment is as simple as making an entry to the list of servers running SharePoint Team Services in the Connect to servers page in Microsoft Project Web Access. This is only possible in a multi-site deployment.

How to gauge when to add a new server running SharePoint Team Services

Any of the following are indicators that you need to add a server running SharePoint Team Services to your Microsoft Project Server deployment:

The number of hosted groups is reaching a limit where a cumulative number of subwebs is expected to reach 750.
SharePoint Team Services pages fail to load in Microsoft Project Web Access.
Timeout errors occur when trying to create subwebs or synchronize users.
SharePoint Team Services pages are slow to return control to the user, after creating or uploading a list item.
You receive Object Provider errors about the SharePoint Team Services items not being entered into the Microsoft Project Server database.

When a new SharePoint Team Services server is added, all new groups will get sites created for them on the new server, while old groups (including new projects they publish) will stay on the old server.

Best Practice #4: Offload the Views Notification Services to a Separate Server

A dedicated Views Notification server is the best way to eliminate performance bottlenecks between the database server and IIS server.

Without a dedicated Views Notification server, there is a great deal of network traffic between the IIS and database server during view publication. A dedicated Views Notification server can eliminate this network bottleneck by reducing direct interaction among the IIS and database servers.

Even if you eliminate the single network interface card bottleneck on the database server, the amount of publishing, cube generation, and scheduling data being sent from the Microsoft Project Server to the database server will still be a large bottleneck. Use System Monitor to track the Bytes Sent/Sec and Bytes Rec/Sec on the database server. If you find these metrics approaching the hardware limits, this is a good indication that a separate Views Notification server with its own connection to the database server is needed.

For detailed information regarding how to offload the Views Notification server, refer to the article Distributed Deployment: Microsoft Project 2002 on the Microsoft Project Technology site on TechNet.

Best Practice #5. Monitor System Performance Continually

Continuous, proactive monitoring of your Microsoft Project Server deployment is the best way to maintain the overall health of your system.

Use System Monitor to watch specific performance counters, which provides an idea of the load, stress, and communications on each computer in the system.
Keep a close watch on the ViewDrop folder in Microsoft Project Server, which is often the best place to receive advanced warning of views publishing problems.
Periodically check the Event Viewer on all systems.

Tracking system performance using System Monitor

Use System Monitor to watch specific performance counters (see System Monitor settings table) to obtain a better idea of the load, stress, and communication on each of the servers and components that are part of the system.

When using System Monitor, it is best to capture baseline measurements when the load on the system is minimal. This will enable you to make comparisons as the load on the servers increases. Table 3 lists System Monitor settings we focused on when watching the load on the servers and components during internal hosting.

Table 3 System Monitor settings

Setting	Description
% Processor Time	If the processors have a heavy load, the CPUs will start backlogging jobs. If all processors are busy managing the workload, other sessions will be waiting, and high wait times will manifest themselves as symptoms on client computers. Add a counter for each processor in the computer.
Pages/Sec	Helps identify how much information is being loaded into and out of memory per second. This is more of a summary metric. If this is consistently high, you will experience processes waiting to be loaded into memory.
% Disk Time	Gives a summary of bytes read and written per second (to or from each logical disk). If this metric is unusually high, you can experience long wait times since the disks are one of the slowest components in the system. If this metric is consistently high, you may experience significant performance problems. Add a counter for each logical disk drive in the computer.
Bytes Sent/Sec	Indicates how much data is being sent from the server computer back to the client computers. If the data being sent from the server back to the clients is approaching available network bandwidth, less information is passing through the system and clients may lose connections or experience long response times. Add a counter for each network card in the computer.
Bytes Received/Sec	Tells how much data is being sent from the client computers to the server. Similar to Bytes Sent/Sec, if the data being sent from the clients is approaching the available network bandwidth, less information is getting through the system and clients may not log on to the server (timeout errors or query timeouts) or may lose connection to the server computer.

Two readily identifiable issues that you can detect early when using System Monitor include:

Bottlenecks caused on servers with a single network interface card—add a second network interface card.
Poor performance caused by insufficient memory on a server—add more memory.

Monitor the ViewDrop folder for back-ups

Keep an eye on the ViewDrop folder. If there are publishing problems with the Microsoft Project Server, this is one of the first places to spot the problem. Sometimes the XML files in the ViewDrop folder take significant time to process. OLAP cube builds generally take a long time, as do publishing large projects. If you see XML files that are not getting processed in a timely manner, or if you see build up in the ViewDrop folder, then investigate the cause of the problems:

Be sure the Microsoft Project Views Notification service is still running.
Check to see if the VWNotify.exe process (the process that processes the XML files) is running and using the CPU.
If one or more files seem to be blocking processing in the ViewDrop folder, try manually removing them from the folder, and adding them back after some of the backlog has been processed. Monitor this process closely.

Check the Event Viewer for system errors

Check the Event Viewer for errors on the IIS server that may be causing problems. Verify the frequency of errors and look for errors that may explain specific problems that you are experiencing.

Troubleshooting Tips

If you do run into problems, use the following checklist to help determine the cause:

Can you ping each of the servers on your system?
Can you log on to Microsoft Project Web Access using Microsoft Project Server authentication?
Can you log on to Microsoft Project Web Access using Windows authentication?
Can you open an enterprise project from the Microsoft Project Server database?
Check all of the Microsoft Project Server services running on the IIS server. Have any of them stopped?
Check for error messages in the Event Viewer on the IIS server, in both the Application Log and the System Log.
Are there any error messages on the Views Notification server? Are all the services running on the Views Notification server?
If the error is publishing-related, when you publish information to the system do you see an XML file appear in the ViewDrop folder on the Microsoft Project Server? If there are XML files in the ViewDrop folder, how old are those files? If necessary, you may need to temporarily remove the old XML files out of the ViewDrop folder. To do this, create a temporary folder and cut and paste the XML files into the temp folder. When the folder is clear, add the XML files back into the ViewDrop folder one at a time. Verify that VWNotify.exe is running in the Task Manager on the Views Notifications server.

Conclusion

Microsoft Project Server has been successfully deployed in a multi-site configuration within Microsoft. This successful deployment was aided by planning for scalable growth, maximizing the throughput of the database server, deploying SharePoint Team Services on production hardware, offloading the Views Notification services, and monitoring system performance. A set of best practices and troubleshooting tips have been developed from the lessons learned in the internal hosting deployment, and are included in this document.

Appendix A: Customer Support Processes

We found that the key to a successful multi-server hosted environment is quick and accurate customer support. The Microsoft Project customer support is a multi-layered process. Our first layer of support is an internal site where we published a Frequently Asked Questions (FAQ) page which answers the most commonly asked questions.

FAQ Page for Internal Customers

Members of your organization will have questions about how to use Microsoft Project Professional, Microsoft Project Web Access, and their features such as views, e-mail notifications, publishing projects, and how to fix errors. Keep track of the questions as you encounter them and then create a page in Microsoft Project Web Access or the Microsoft Project Guide to address frequently asked questions.

Some of the questions we encountered are:

How do I set up e-mail notifications and reminders?
I have set up notifications, but I am only receiving them embedded in e-mail messages, not as an Outlook task. Why?
How do I prevent Microsoft Project Server from sending out e-mail notifications when new or changed assignments are published?
I am getting strange dialog boxes when opening Microsoft Project Professional using Terminal Server. Any ideas?
How do I build an OLAP cube?
How can I force check-in a project?
How do I set up a digital dashboard to show projects in the Project Center?

To handle other requests, we developed a support infrastructure to provide specialized services in the areas of general troubleshooting, database troubleshooting, publishing, scheduling, enterprise features, Microsoft Project Web Access, notifications, reminders, and timesheets, plus interaction with SharePoint Team Services.

Provisioning New Sites

Whenever you decide to set up new servers in your hosting environment, make sure to have a system in place to provision the sites. Provisioning new sites is a very replicable process and one that could be easily automated.

Our process for provisioning new sites is the following:

The customer navigates to the hosting request page located on the Microsoft intranet, and enters the following information into the form:

group name, short name (the name of their virtual directory), contact/administrator, the full name (for the administrator), a brief group description, user information, the expected number of projects and users, and a space for comments or additional information.

When completed, the request is submitted to the Microsoft Project Server administrator.
When the request is accepted, the information is passed on to a database. A hosting tool checks for new entries every 20 minutes.
A virtual directory is created that points to the installed files on the server, and the configuration for the new virtual directory is set.
A new registry key for the site in Microsoft Project Server is set, and an e-mail message is sent out notifying of the new site.
The database is updated with site information; a flag in the database is set that indicates the virtual directory has been created.
SharePoint Team Services subwebs are created; they are provisioned for the new Microsoft Project Server virtual directory.
A record about the server running SharePoint Team Services is inserted into the Microsoft Project Server database, and the database is updated with SharePoint Team Services information. A flag in the database is set when this is completed.

Appendix B: List of All Issues Encountered

The following list details the types of issues we encountered during our Microsoft Project Server deployment:

Network / WAN We experienced some performance issues with customers who had slower network connections (less than 100 MB). When a connection is slower, there can be problems connecting to the database server. These problems can be reproduced by opening SQL Server Enterprise Manager and attempting to connect to the Microsoft Project Server database server from that application. One workaround for this problem is to use a terminal server that is set up properly for your Microsoft Project Server deployment.
Breaking out Views Notification services We ran into the following problems after offloading the views process on to a separate server: provisioning errors and modifications which needed to be made to registry settings. To avoid these problems, we recommend that you deploy a separate Views Notification server during your initial deployment.
Windows password expiration We found that it is beneficial to use the same Windows credentials for logging on to both the IIS and the Views Notification servers. However, when the password expires, it is best to log out, then log on to each server, and then verify that the services started up correctly.
Domain disjoins If one of your servers is disjoined from the domain, it is necessary to rejoin the domain and reboot the server. Then verify that all system components are running correctly in your Microsoft Project Server environment.
Expired user passwords When a Windows NT user password that is being used to provide credentials to a Microsoft Project Server COM+ package expires, users attempting to connect to SharePoint Team Services or the Views Notification server (through OLAP Analysis Services, specifically) are not able to connect. To restore this functionality, you will need to re-run the PSCOMPlus.exe tool with the new user credentials. This will require a short period of downtime for all users who require this password. To avoid this problem, use a local Windows NT user account with a non-expiring password.
Registry settings on the IIS and Views Notifications servers If the registry settings on either the IIS or Views Notification servers are changed unintentionally (the GUID of the Session Manager, for example), there can be serious problems.
Errors in published data Occasionally, XML files sent to the ViewDrop folder to be processed contain errors and are not processed correctly. Consequently, XML files in the ViewDrop folder begin to pile up behind the file with the error. This problem cannot be fixed until the XML file that contains the error is removed from the folder.
Slow network connections and network congestion If the network is unusually slow, it can make the system appear to be down. The ViewDrop folder can fill up with files and users experience brief outages in both enterprise Project availability and access to Microsoft Project Web Access. Note that a slow network connection or network congestion are not always the reasons why XML files pile up in the ViewDrop folder.
Heavy load on the database server When the database server reaches a consistent load across all processors at around 70 to 75 percent, we experienced problems accessing projects stored on the server. The problems first started occurring for users of Microsoft Project Professional. As the load increased, this affected users of Microsoft Project Web Access. The problems included slow connection times to the Microsoft Project Server, slow or unresponsive opening and saving of projects, publishing errors ("the database server cannot be accessed," for example), and a general unresponsiveness. We determined that when the database server reached a consistent load at around 60 to 65 percent, it was beneficial to add a second database server to the configuration and move selected organizations data to this new server. Take the following steps when adding a new database server:
1. Set up a new server and configure it for use with Microsoft Project Server.
2. Copy all data files and log files (including the master database) over to the new server.
3. Overwrite the existing master database with the master database from the old server to preserve log on information.
4. Detach the databases (except for the master) and move the files to new locations.
  
  Note: Consider moving the data files to one logical drive (RAID-10 array) and transaction log files to another logical drive (a separate RAID-10 array).
  
  Then re-attach the databases.
5. Set up Windows NT security groups and match them from the old server. This includes both users and administrators of SharePoint Team Services.
6. Remove the original computer entry from the domain and register it the same way as the original computer on the domain.
7. Update the file paths in the provisioning application and verify that the connections still work.
8. Verify that all connections to SharePoint Team Services are still provisioned correctly and that the STSAdmin and STSUser accounts are valid with the new database server.
9. After DNS is updated, run a backup of the new database server and check the Microsoft Project Server to verify that the connections are functional and that all content on the server is correct.
Note: Consider using dynamically-generated scripts to facilitate this process. For example, run the DBCC ShrinkDatabase and DBCC ShrinkFile scripts on the data files and log files to reduce their sizes. Rebuild the index using the DBCC DBReindex script.
Database connections lost after reboot Some users of Microsoft Project Server lose read/write access to the TempDB table in the database server after rebooting to the database server. If this happens, you will have two indicators. The first indicator is the presence of a large number of XML files in the ViewDrop folder (check the Event Viewer application log for text similar to "Server user '<username>' is not a valid user in the database "tempdb"). The other indicator is when you attempt to use Microsoft Project Professional to log on to Microsoft Project Server—as the user is being authenticated, you receive the error message: "An unexpected error occurred. The database could not be initialized because it is not configured correctly, it does not support required functionality, or it is not installed on your computer. Contact your System Administrator for the database." To resolve this issue, you need to verify the permissions to the TempDB table on the database server.
Our users were asking us questions about how to use the product We developed a FAQ page and linked our customers to it through the intranet.
SQL Server and SSL Security Error We ran into issues where the SQL Server 2000 server had a certificate installed on it with the same name as the SQL Server. This problem and its solution are described in the Knowledge Base article SQL Server 2000 Installation or Local Connections Fail with "SSL Security error :ConnectionOpen (SECDoClientHandshake())" Error Message.

Appendix C: October 2002 Server Usage Statistics

Table 4 shows customer load for October 2002. These statistics are real numbers, but the internal source names have been changed for privacy.

Table 4 Server usage statistics for October 2002

Organization	Total Projects	Total Resources	Average Tasks/Project	Average Assn/Project
Project Group 1	3	1	22	22
Project Group 2	3	12	209	218
Project Group 3	4	48	246	370
Project Group 4	2	100	247	250
Project Group 5	20	245	85	120
Project Group 6	0	5	0	0
Project Group 7	2	13	27	35
Project Group 8	2	19	102	103
Project Group 9	16	23	22	22
Project Group 10	0	8	0	0
Project Group 11	11	4	6	6
Project Group 12	0	21	0	0
Project Group 13	5	12	11	17
Project Group 14	3	6	100	100
Project Group 15	0	6	0	0
Project Group 16	7	9	17	19
Project Group 17	2	5	15	32
Project Group 18	4	5	3	3
Project Group 19	15	18	33	39
Project Group 20	3	25	255	258
Project Group 21	16	67	41	25
Project Group 22	2	20	0	0
Project Group 23	5	8	11	11
Project Group 24	2	15	155	157
Project Group 25	2	149	2	2
Project Group 26	21	14	39	46
Project Group 27	2	2	30	30
Project Group 28	2	1	4	4
Project Group 29	2	5	106	106
Project Group 30	5	33	18	39
Project Group 31	7	34	206	393
Project Group 32	3	7	18	18
Project Group 33	300	808	16	21
Project Group 34	6	24	127	133
Project Group 35	3	10	14	16
Project Group 36	3	54	111	125
Project Group 37	7	8	13	16
Project Group 38	178	174	25	27
Project Group 39	15	219	42	62
Project Group 40	0	43	0	0
Project Group 41	9	25	4	4
Project Group 42	20	245	85	120
Project Group 43	3	13	11	11
Project Group 44	13	129	86	95
Project Group 45	180	247	28	32
Project Group 46	2	5	34	34
Project Group 47	3	150	506	725
Project Group 48	3	105	509	544
Project Group 49	3	42	14	15
Project Group 50	0	5	0	0
Project Group 51	1	2	2	2
Project Group 52	13	122	87	92
Project Group 53	37	122	112	120
Project Group 54	3	19	45	55
Project Group 55	9	14	21	24
Project Group 56	0	3	0	0
Totals and Averages	982 total	3,528 total	39 average	46 average

Lou Lucarelli is a Software Test Engineer for Microsoft Project and has been testing software for six years. He is also the Dogfood (alpha and beta version) Administrator for Microsoft Project Server 2002, which is hosted internally at Microsoft.