At CeBIT 2008, which took place this past March in Germany, Steve Ballmer discussed how the Datacenter Group at Microsoft is making efforts to develop sustainable datacenters and share the best practices it learns in the process. A major example of these efforts is the Microsoft datacenter located in Quincy, WA—the facility is
Contents
powered entirely by hydroelectricity. And the facility is sizeable. If you walked around one complete building (there are currently two buildings, with more being built over time), including the space for IT, power and cooling equipment, the receiving area, and the office space, you would travel 1.3 miles.
I recently had a chance to take a tour of the Quincy datacenter, and now I get to share some of the methods used in building and operating a sustainable datacenter. (There is a video of the Quincy datacenter available at download.microsoft.com/download/8/7/d/87d2d871-471e-44a3-bfd0-c02d3248b8cb/Energy%20Efficiency%20in%20Datacenters-022808-Med.wmv.)
In the first installment of this column, I introduced the idea of "Putting IT on a Diet" (see technet.microsoft.com/magazine/cc462802). This concept, as well as other key elements of environmental sustainability, can be found in how Microsoft designs and operates its datacenters. Here is a look at eight strategies Microsoft employs to make the most out of every bit of energy the Quincy datacenter consumes.
The lifespan of a datacenter can be 20 years. One key design goal is to minimize the resources required to build and operate the facility over that entire lifespan. Over a 20-year period, equipment will undoubtedly change. To simplify improvements that will lead to better efficiency over time, Microsoft designed the Quincy building in a modular fashion, allowing components to be upgraded without taking down the entire facility. With only about 50 people required to operate the facility, workflow efficiency is important to ensure the best use of those resources and allow work to happen quickly.
When you look at the Quincy datacenter building, it looks like one building. However, it is actually two separate buildings. The most efficient datacenter is a full datacenter. Microsoft, therefore, filled the first building to capacity before putting a single piece of equipment into the second building. This approach is being repeated throughout the site, as there is room for three more datacenter buildings, and one of these buildings can be built in as short a time as 10 months. By building the datacenter infrastructure when it's needed, Microsoft can improve the power and cooling efficiency of the datacenter and eliminate the waste that results from over provisioning capacity.
One technique a Microsoft engineer explained to me is the concept of designing around the sweet spot of performance for each piece of equipment. Every piece of equipment has a range of optimal performance, and holistically achieving the right combination improves the overall efficiency.
Microsoft evaluated multiple permutations of equipment capacities and efficiencies to find the right combination for their power and cooling needs. Unfortunately, the exact combination is unique for every environment and even Microsoft continuously evaluates and optimizes the equipment combinations as new technologies are developed. So there is no magic recommendation here, other than that you'll need to test a variety of configurations to find the best configuration for your environment. And keep testing over time to find improvements.
- 4. Organize for Collaboration
You may be thinking, "I'm a system administrator and it sounds like the Microsoft datacenter efficiencies were done by the facilities staff." Given the dynamic IT environment and how quickly technology changes, a tight partnership exists between the IT pro and the facilities staff. In fact, facilities program management and the technical program manager report to the same datacenter manager who in turn oversees the whole datacenter operation. The accountability for sustainable and efficient operations falls under one organization.
The Quincy datacenter incorporates energy monitoring of all systems, providing real-time data. The data is recorded and archived for historical trending and analysis. The staff focuses on electricity as the most important resource, not space. Power stranded in one area, that therefore can't be used is one type of the waste Microsoft looks for. The reports are shared with hundreds of Microsoft employees to help them understand the efficiencies of the operations and how much energy is consumed to provide particular services.
Common questions asked by people on staff include: What is the overall efficiency of the datacenter (measured by PUE)? What are the actual versus calculated power usages? How much capacity does my group have? What are the electricity costs for my business unit?
- 6. Efficient Commissioning
Efficiency starts the moment servers are delivered pre-populated in racks. This eliminates the bulky packaging typical used for server shipments. The only packaging required is the wooden shipping crate, which is recyclable. Just a matter of hours after a server shipment arrives, equipment is ready to be deployed and operating. The staff estimates that they can commission the space of one of these buildings in as little as three months.
- 7. Prepare for High Loads
One of the downsides of running at higher efficiencies is the equipment is run at high load conditions. Since the current industry norm is underutilization, this can cause problems.
As Microsoft was filling up the server rooms, some pieces of equipment began to display problems as they neared capacity. When contacted, vendors were actually surprised to hear that Microsoft was consistently running this equipment at high load conditions, as most of their customers do not come close to capacity ratings.
So Microsoft operations recommends that if you are going to run at high loads, make a high load test part of the acceptance criteria. Keep this in mind for projects such as server consolidation, as a higher power concentration in a rack can create high loads.
The Quincy datacenter is located in the middle of farmland near the Columbia River. Besides offering a good source of electricity, this provides an ample supply of water. However, water for agriculture does not provide a lot of waste water since the water evaporates or soaks into the ground.
Datacenters use a lot of water for the cooling system. After use, the water is sent to the local waste water treatment facility. Unfortunately, the waste water system in Quincy was not ready to handle the quantity of water resulting from a datacenter. In this case, Microsoft built a water treatment facility to reduce the waste water, drastically lowering the datacenters water use. For another facility (located in San Antonio, TX), Microsoft developed a plan to use gray water from other industries, making their water use a non-issue.
Wrapping Up
When you put IT on an energy diet and start designing more sustainable solutions, it's a good idea to see what you can learn from others. Seeing what issues other organizations have encountered and how they have addressed these issues will give you ideas on what to look for and what you can do to improve your operations.
One of the goals of the Microsoft Datacenter Group is to help the overall industry find ways to operate more efficiently. To help spread more efficient practices, the group participates in industry organizations such as The Green Grid (www.thegreengrid.org), Climate Savers (www.climatesaverscomputing.org), ASHRAE (www.ashrae.org), AFCOM (www.afcom.com), and The Uptime Institute (uptimeinstitute.org).
I'd like to thank the datacenter staff in Redmond, WA and Quincy for providing me with the opportunity to see how Microsoft runs a sustainable, energy-efficient datacenter.