Inside Microsoft.comMaking the Move to x64

Chris St. Amand

For many years, one of the biggest challenges faced by the Microsoft.com Web site engineers was the virtual memory limitations of the 32-bit operating system. With hundreds of different applications running on the site, there was often contention for the 2GB address limit that the front-end Web servers had to deal with. In the days of Windows® 2000 and IIS 5.0, tools such as Cluster Sentinel and IISRecycle were used to pull servers out of rotation and restart IIS when virtual memory ran low. Later, Windows Server™ 2003 and IIS 6.0 helped considerably by introducing app pool recycling with virtual memory thresholds. But, the root problem remained; the servers still had a 2GB memory limit because of the limitations of the 32-bit operating system. As applications became more complex in ASP.NET, application pools were recycling as frequently as every five minutes, causing huge performance problems. Something had to be done.

Enter Windows Server 2003 x64 Edition

About one year prior to the RTM release of the x64 operating system, build 1171 of the Windows Server 2003 x64 Edition operating system was released for testing. This build supported IIS 6.0 and ASP.NET running under WoW64 (the x86 emulator that allows 32-bit Windows-based applications to run on 64-bitWindows).

Working closely with some of the core Windows and IIS engineers, MSCOM System Engineers employed a white box AMD64 reference machine to test this build (see Figure 1 for white box specifications). It quickly became apparent that the extra virtual memory space was solving a lot of the problems that had plagued MSCOM in the 32-bit OS.

Figure 1 Microsoft.com White Box

Hardware Migration

After testing the Web applications on the reference hardware, this same machine was used to handle part of the MSCOM production traffic. Watching the machine handle the live traffic confirmed the need to migrate to this platform. Now that tests concluded that migration would provide solutions, the question was when and how to migrate.

One of the compelling features of the x64 platform is that you can run the x86 operating system on it without reducing performance. Knowing the x64 hardware could be quickly repurposed to run the current x86 operating system made the hardware purchasing decision very easy.

Figure 2 shows the premigration x86 hardware as well as the new 64-bit hardware to which MSCOM migrated.

Figure 2 Old and New Hardware

Here’s the path we followed in deciding to migrate our servers:

1. Validate and test the new platform in a lab setting on a single server.
2. Revalidate those same scenarios with a single machine taking live traffic.
3. Phase in the new platform when ready, since the x64 hardware can run the x86 OS without performance penalty, immediately maximizing the advantages of the new hardware investment.

Software Migration

One of the biggest concerns MSCOM initially had in moving to this new platform was the amount of work involved in porting legacy applications. MSCOM quickly realized that the WoW64 layer would eliminate the need for doing any work to port legacy applications. This was quite advantageous because there was no need to touch a single line of code or recompile any older 32-bit ISAPIs in order to move to the x64 OS. Also, because the x64 processors have x86 registers, the performance hit when running in this configuration is negligible.

MSCOM had dependencies that required us to run in 32-bit IIS worker processes under WoW64. First, MSCOM was running ASP.NET 1.1, and there isn’t a 64-bit edition of version 1.1 of the .NET Framework. That is an option today with version 2.0 of the .NET Framework, but it wasn’t available at the time.

Second, MSCOM was using ISAPI extensions and filters in production, and some of these were compiled years ago using the x86 platform.

Finally, MSCOM was running some custom legacy COM objects that were still being used by older ASP applications.

In order for the Microsoft.com site to continue to function, the 32-bit IIS worker processes needed to run so that these components could work properly. The Knowledge Base article "How to run 32-bit worker processes on 64-bit" explains how to set a metabase key that tells IIS to use 32-bit worker processes instead of the native 64-bit worker processes (see "Windows Server 2003 SP1 enables WOW64 compatibility for 32-bit Web applications in IIS 6.0").

In the end, no code changes were required to move over to the x64 platform. Legacy ISAPI and COM components migrated without issue, as did ASP.NET 1.1 applications. In addition, 32-bit processes running under WoW64 got 4GB of addressable virtual memory. Fortunately, kernel memory wasn’t sacrificed in order to squeeze out more virtual memory for the user mode processes. Even with 32-bit IIS worker processes, the kernel runs as native 64-bit.

Redirection Behaviors and Dependencies

It is important for systems engineers to understand how the WoW64 redirection behavior works. There is file system and registry redirection that’s in place to help 32-bit processes work in the 64-bit environment.

It is important for systems engineers to understand how the WoW64 redirection behavior works. There is file system and registry redirection that’s in place to help 32-bit processes work in the 64-bit environment.

File System Redirection Any time a 32-bit process attempts to access c:\windows\system32 the WoW64 layer redirects it into c:\windows\syswow64 which contains all of the 32-bit Windows binaries. This prevents a 32-bit process from trying to load a 64-bit binary. Any scripts or tools running in a 32-bit process that is referencing this directory will be automatically redirected to the syswow64 directory.

Registry Redirection A similar redirection exists for accessing the registry. Any 32-bit process trying to read or write to HKEY_LOCAL_MACHINE\Software gets redirected to HKEY_LOCAL_MACHINE\Software\Wow6432Node\. This allows separate configurations to be maintained for 32-bit and 64-bit processes. Any custom settings or keys set in this node may need to exist in both keys, as 32-bit processes will be redirected to this new branch.

Script Host Bitness Any scripts with dependencies on x86 COM objects need to be run with the 32-bit cscript or wscript. Specify either the 32-bit or 64-bit version of the script host, depending on scripting needs. Any scripts that access the redirected file system or registry are using the correct bitness script host. If a particular script runs with the 32-bit script host, it may use the WoW64 redirection and not have the expected outcome.

64-Bit Benefits

The performance benefits seen from the x64 OS have been substantial. Because of the larger virtual memory space available to the IIS worker processes, server performance has improved dramatically. In certain cases the memory usage of application pools that were previously recycling every five minutes leveled off and stopped recycling altogether. Meanwhile, poorly behaving applications that were leaking memory or abusing the ASP.NET cache could be identified. Before the migration, with a 2GB address space it was extremely difficult to differentiate between an application that was leaking and an application that was simply caching as part of its normal behavior.

There are 232 possible combinations that a 32-bit address can take (meaning 4GB worth of addressable space). On the x86 operating system, half of that is allocated for the kernel, and the other half is given to user mode processes such as IIS worker processes. Since the addressing spaces of the user mode processes are independent of each other, each process can reference up to 2GB of memory.

On the x86 platforms, one workaround we attempted was the 3GB switch, which gives 3GB of virtual address space to the user mode processes and only 1GB to the kernel. The results were disastrous. Giving the kernel only 1GB just wasn’t enough for the networking stack to keep up on our servers. Also, in attack scenarios (such as SYN attacks), there wasn’t enough pool memory for TCP/IP to fend off even the smallest attacks our site regularly experiences.

The x64 architecture changes all of that though, even while running 32-bit IIS worker processes. With a 64-bit addressing space, there is a possibility of 264 unique address combinations (16 exabytes). The x64 OS currently uses 43 bits for addressing, giving the kernel 8TB of addressable memory, and leaving the other 8TB for user mode processes. (See "Processor and memory capabilities of Windows XP Professional x64 Edition and of the x64-based versions of Windows Server 2003"). Figure 3 shows a comparison of memory limits.

Figure 3 Memory Comparisons

Given that 32-bit processes no longer need to share half the addressing space with the kernel, they’re given the full 4GB addressing space to use running under WoW64. This means our systems run with a native 64-bit kernel and 32-bit IIS worker processes that get 4GB of addressing space (instead of 2GB on the x86 OS).

Figure 4** x64 Server Processor Utilization **

One of the areas that saw a dramatic improvement in performance was processor utilization. Because the application pools were no longer recycling, or were at least recycling at a rate in the range of weeks rather than minutes, the performance implications were significant. Figure 4 and Figure 5 allow you to compare the processor utilization of our x86 servers and an x64 server.

Figure 5** x86 Server Processor Utilization **

There are two things of interest to note in the CPU utilization graphs. First, the average processor utilization of the x64 server is only about half that of the x86 server (34 percent versus 64 percent). Second, on the x86 server there are noticeable spikes where the CPU was running at 100 percent capacity for a sustained period of time. This was due to one or more app pools recycling because they had run out of virtual memory. On the x64 server these spikes don’t occur because the app pools don’t run out of memory.

The impact of this on the rest of the server is huge. Application response times improve and content is delivered to the clients much more quickly. Using the Microsoft® Server Performance Advisor, it becomes obvious that this is the case when you look at the response times of the different components of our site (see Figure 6).

Figure 6 Response Times

The Server Performance Advisor (SPA) used to generate these numbers is available free at Microsoft Windows Server 2003 Performance Advisor.

All in all, the key things to remember from this migration include the significantly improved application response times thanks to larger virtual memory space on the x64 platform, the significant drop in CPU utilization due to less frequent recycling, and the server consolidation made possible by better performance on each server.

Lessons Learned

As with any platform migration there are sure to be hiccups or unexpected scenarios while validating the new platform. Take a lesson from our experience and remember to always verify and ensure that any third-party dependencies are compatible with the x64 operating system. Also ensure that any component that requires a kernel-mode driver has an x64 compatible driver, because x86 drivers cannot be used on the x64 operating system.

In addition, MSCOM had to verify the compatibility of third-party products like antivirus software, backup software, imaging and deployment software, and common administration tools that use filter drivers (such as filemon, regmon, and so on). So when you consider migration, keep these dependencies in mind.

Be sure to fully understand any third-party dependencies and the availability of x64 compatible versions. Become familiar with the WoW64 redirection behaviors that occur in both the registry and file system, and understand script dependencies so you know which script host to run it with, as explained in the sidebar "Redirection Behaviors and Dependencies."

Future Directions

After transitioning to the x64 OS, the memory limitations that MSCOM struggled with for years suddenly were eliminated. At the time of this writing, Microsoft.com is running the 32-bit version of IIS and ASP.NET running on Windows Server 2003 x64 Edition, and enjoying all the benefits.

The ongoing transition to the x64 platform in our environment is easier because the hardware and software migration is easier. The migration steps outlined here allow for a phased approach that enables proper testing, while letting you capitalize on the x64 hardware investment immediately.

This migration is merely a stepping stone in a transition to running in a native 64-bit x64 environment. Still, this configuration has already yielded tremendous results. In addition, as we migrate to the newly released ASP.NET 2.0, we will be able to run certain applications in a full native 64-bit IIS worker process, increasing available virtual memory from 4GB to 8TB. I think we’ve only seen the beginning of what the x64 platform has to offer!

Chris St. Amand has spent five years on the Microsoft.com Operations team as the lead systems engineer on properties such as microsoft.com and msdn.microsoft.com. He is currently a senior systems engineer on the Microsoft.com Operations Debug team. You can reach Chris at chrisst@microsoft.com.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.