Supercomputer, Super Savings – NERSC Data Center

National Lab Demonstrates Its Commitment to Energy Efficient Scientific Research in Supercomputer Data Center

Shyh Wang Hall

Project Size and Savings

IT Load: 5 MW,  PUE: < 1.1

Payback: < 3 years,  Annual Energy Savings: 2,300,000 kWh,  Annual Water Savings: 200,000 gallons
 

Project Summary

The National Energy Research Scientific Computing Center (NERSC) is a high performance computing center owned by Lawrence Berkeley National Laboratory (LBNL) and housed in Shyh Wang Hall. Its mission is:

to accelerate scientific discovery at the DOE Office of Science through high performance computing and data analysis.

To do this, the facility has been equipped with supercomputers that require a A LOT of cooling. Data center efficiency is often characterized using a ratio called power utilization effectiveness (PUE). An ideal PUE is 1.0, and the NERSC data center was designed to achieve a PUE of less than 1.1, about 90% more efficient than typical data centers.
 
We began working with LBNL after much of NERSC’s IT infrastructure was built out to help ensure that PUE goal was met. To produce results quickly, we work closely with the LBNL sustainability team and facilities staff to brainstorm ideas and start implementation as soon as we agree on the approach. We use trends from the building automation system (BAS) to both identify trends as well as optimize operation after the measures are implemented.
 

Efficient Design, but High Energy Needs

The data center was designed to be highly efficient, employing both a chiller-less cooling water system and air side economizers. Most of the IT load in the data center consists of two Cray supercomputers consuming roughly 5,000 kW, cooled by integrated cooling water coils and fans. It also houses approximately 500 kW of IT equipment in hot/cold aisle configuration. There are also floors of office space served by rooftop units which receive condenser water from the data center cooling towers.

Cray Supercomputers

Finding Savings in Good Design

Despite the highly efficient baseline, we identified over 2 million kWh in potential energy savings. In the first 6 months, we helped implement, optimize and verify energy savings over 700,000 kWh. Since then, we’ve helped implement an additional 300,000 kWh of savings and additional measures are in currently progress.
 

Optimizing the Hydronic Cooling System

One energy saving project currently being implemented is the optimization of NERSC’s hydronic cooling system, which involves several phases. The following graph shows the progression of implementing savings after the first phase. The purple line shows the baseline energy consumption and the green line shows the consumption after optimizing the condenser water reset based on outside air wet bulb temperature – nice and smooth.

This is only the first step in optimizing the system, so we didn’t dial it in perfectly. Next steps include:

• Reduce bypass flow in closed loop.
• Turn off redundant tower water pump.
• Install booster pump to serve RTUs and allow closed loop pumps to operate at reduced dP all year.
• Reset the closed loop supply water temperature to reduce energy consumption of supercomputers.

Into the supercomputers

We have begun to implement measures by working directly with Cray to optimize the operation of the fans within the computers. This presents the opportunity for us to drive the building-level cooling systems harder to reduce the supercomputer energy, optimizing the facility’s energy consumption. This is the first project in the world that this has been done with Cray supercomputers.
 

Ongoing Commissioning

Over their 2018 fiscal year, we are continuing the implementation of measures and improving our ongoing commissioning process by using SkySpark. This tool will allow us to import building data from multiple sources into one platform that we can use to automate a significant amount of data analysis. We use it as a tool for efficiently detecting problems in operation and for optimizing controls sequences as the facility is built out over time.
 
It is our privilege and pleasure to continue supporting LBNL’s sustainability efforts working together to identify energy saving measures, quantify financial metrics, support implementation and verify energy savings in many of their facilities.
 
This project was presented at the 2017 Better Buildings Summit in Washington, DC. You can view the slides from that presentation here.