日本HP主催イベント HPCフォーラム2015 トラックB(コミュニティトラック)
HP Tech Power Club 第3回スケールアウト分科会
データセンターの電力を少なくデザインするには?
常温水冷スパコン の誕生秘話
ヒューレット・パッカードカンパニー
最上級テクノロジスト
Nicolas Dobé, Ph.D
-From 1990 to 2010, Renewables grew from 1.12 % share to 2.3% share, worldwide. But yet, because the total energy demand is rapidly increasing worldwide, we are more dependent on fossil fuels, not less
-the worldwide energy situation is very real, in some geographies like Japan where Nuclear reactors were shut down, it’s even worse
-as an industry, the IT sector is not helping much, as the percentage of total electricity going into datacenters has doubled in the last five years, growing from 1.5% of total US production to > 3% today
-metrics like PUE did help a lot for driving datacenter energy efficiency
-EnergyStar programs also drove some good behavior on the server side
We worked collaboratively with NREL on their Peregrine supercomputer, which we’re now, with minor modifications, making available as the Apollo 8000 System. This system is a water-cooled supercomputer with 4x the teraflops per square foot than air-cooled systems. Along with that performance density, it uses 28% less energy, and can help save up to 3,800 tons of CO2.
With XL730f, 144 teraflops/rack
With NVIDIA GPUs XL750f, 273 teraflops/rack
With Xeon Phi XL740F, 246 teraflops/rack
Up to 144 servers in 50U
4X teraflops per sq ft compared to air cooled in rack space
2.12x teraflops/sq ft compared to water-cooled IBM Power 775
2.36x teraflops/per sq ft compared to Cray CS300 LC
Save 1217 sq. ft. compared to air cooled 1-petaflop system
Save 195 sq. ft. vs. IBM Power 775
Save 236 sq. ft. vs. Cray CS300 LC
==========================================
Efficiency
Industry’s only efficient liquid cooling without the risk
Save up to $1M in energy over 5 years for each MW of IT in the data center compared to air cooled systems
Get up to 40% more FLOPS/watt than air cooled systems
For the same performance, HP Apollo 8000 System consumes up to 28% less energy than air cooled systems
Air cooled systems consume 2.6X more energy for facility infrastructure as compared to HP Apollo 8000
Air cooled systems use up to 1.2X more MW power per 5.6 petaflop data center
Avoid up to $7.1M of new facility CapEx with water cooling to reach 5.6 petaflops
====================================================
Sustainability
Redefining data center energy recycling
Save up to 3,800 tons of CO2 per year vs. an air cooled data center 3MW water vs. air cooled
CO2 = 790 of cars
Now we want to talk about what the HP Apollo 8000 system consists of…As you can see it consists of an IT rack called the HP Apollo f8000 rack capable of hosting 144 servers, three tray versions, XL730f ( compute tray) XL740f( Intel accelerator tray), and the XL750f( GPU tray)
These server trays are based on the dry disconnect technology that ensures efficient heat transfer btwn water and server trays without the risk of competive liquid cooled systems
Dry disconnect servers enable ease of serviceability similar to air cooled servers
This solution also comes with integrated FDR IB switches called HP InfiniBand Switch for Apollo 8000 System each of which is capable of providing 36 FDR IB ports
There is also a iCDU ( Intelligent cooling distribution unit) that isolates the facility water and the secondary water maintains sub atmospheric pressure across the cooling circuit and ensures that the liquid cooled system is resilient
1. Front Server
2. Rear Server
Each server has the following:
3. Two Intel® Xeon® E5-2600 processors
4. Sixteen (16) DDR4 DIMM slots per server
5. Optional one (1) SSD
6. FLOM FDR InfiniBand Adapter
7. 1Gb Ethernet Adapter ***
Water stays isolated from server nodes
Thermal bus bar-
No “make or break” water connections during node service events
Integrated thermostatic temperature control ensures water is heated to highest possible temperature while maintaining IT component temperatures within spec
Let me walk you through some of the technologies in this system. Dry-disconnect server trays provide liquid cooling without breaking any water connections. Those trays also make this system easy to service without any worries. Facility water is isolated from the water inside of the rack, and it takes ASHRAE-spec water. Now NREL has ASHRAE-spec water coming into the building, so you’ll hear engineer Nic Dube say that they filled it with a garden hose (lol).
A starting configuration has one IT rack holding up to 144 servers, and one Intelligent Cooling Distribution Unit (iCDU) Rack that supports up to 4 IT racks. The iCDU has 320kW capacity so you can cool higher-performing technologies including higher-bin processors and accelerators.
This is a converged system, with built-in management infrastructure. The Apollo 8000 System Manager aggregates/consolidates management for quick and easy insight and management for all the system components. Meanwhile, the engineering team eliminated some of the steps in the power infrastructure for efficiency, offering up to 80KW per rack with four 30A 3-phase 380-480VAC.