2. Energy hungry datacenters
• Electricity used by data centers has doubled between 2000 and 2005
alone!
• Servers are becoming more powerful , dense and more in number as
well as storage becoming larger and larger
• Availability needs are on the rise
ALL OF THE ABOVE EQUALS
• MORE power consumed by the servers AND more consumed for
cooling
3. Challenges
• Energy demanding servers pose several challenges
– Cost
– Energy waste
– Power availability
– Cooling
– Hot spots
– Carbon footprint
4. Motivations for energy efficiency
quote from Meijer Huber, LRZ
• Energy Efficiency and SuperMUC
• Motivation
•Academic and governmental institutions in Bavaria use electrical
energy from renewable sources
•We currently pay 15.8 Cents per KWh
•We already know that we will have to pay at least 17.8 Cents per
KWh in 2013
5. Motivations for energy efficiency
quote from Steve Hammond, NREL
Motivation
• Data centers are highly energy-intensive facilities
• 10-100x more energy intensive than an office.
• Server racks well in excess of 30kW.
• Surging demand for data storage.
• ~3% of U.S. electricity consumption.
• Projected to double in next 5 years.
• Power and cooling constraints in existing facilities.
Sustainable Computing Why should we care?
• Carbon footprint.
• Water usage.
• Mega$ per MW year.
• Cost OpEx > IT CapEx!
Thus, we need a holistic approach to sustainability and TCO for the
entire computing enterprise, not just the HPC system
6. PUEs in various data centers
Source: Intel
Global bank’s best data center (of more than 100) 2.25 Air
EPA Energy Star Average 1.91 Air/Liquid
Intel average >1.80 Air
ORNL 1.25 Liquid
Google 1.16 Liquid coils,
evaporative tower,
hot aisle
containment
Leibniz Supercomputing Centre (LRZ) 1.15 Direct liquid
National Center for Atmospheric Research (NCAR) 1.10 Liquid
Yahoo Lockport *(PUE declared in project) 1.08 Free air cooling +
evaporative cooling
Facebook Prineville 1.07 Free cooling,
evaporative
National Renewable Energy Laboratory (NREL) 1.06 Direct Liquid +
evaporative tower
8. Ways to improve PUE and energy efficiency
Total vs local energy optimization
9. Ways to improve PUE and energy efficiency
Acting at different levels
IT equipment level
• Increasing processor efficiency
• Increasing memory efficiency
• Increasing storage efficiency
• Optimizing networks (i.e. 3d-Torus vs fat tree networks)
• Optimizing algorithms
• Optimizing software (i.e. locality…)
• Optimizing the jobs scheduling to maximizing processors utilization
Data center level
• 50% of the energy entering a data centre goes into the «house load»,
so it used for ancillary activities not directly related to the IT equipment
• Reducing the house load bring a considerable improvement of the
data centre energy efficiency
10. 3 main opportunity areas for energy efficiency
IT equipment
1 Maximize Flops / Watt
Maximize efficiency
Data Center
2 Reduce House Load
Reduce Cooling
Energy consumption
Optimize power
Data Center or
conversion
3 ecosystem
Reuse thermal energy
12. Energy efficient design
• Eurora has been designed using standard component but making
choices for the best energy efficiency possible
• Eurora could benefit from the Eurotech experience of making the
power conversion chain efficiency of the Eurotech Aurora system
progressively increased from 89% to 97%
The approach has been:
• Choice of the most efficient components in the market. That
is, choosing components that minimize energy consumption
giving the same functionality and performance
• Choice of the best «working points» to top the components
efficiency curves
• Water cooling to lower the working temperature of components
and maximize their efficiency and eliminate fans
13. Gain DC/DC conversion efficiency
• In the DC/DC choice a gain of over 2% in efficiency, from 95,5 % to
98%
• Choice of the optimal current (I) to work on the top of the conversion
curves
Existing DC/DC conversion New upgraded DC/DC conversion
14. Water cooling and efficiency
178 nodes – AMD Opteron 6128HE CPUs (Magny Cours) - 16GB RAM Measuremets
taken by LRZ
• With aircooling the CPU’s operate at about 5°C below maximum case
temparture
• Normal operation of an water cooled server is with water of 20°C, which is
about 40°C below the maximum case temperature
15. Water cooling = No fans, Low noise
• Ventilators consume 5-8% of peak power…per se a small contribution but the
SUM of all of the contributions described gives a considerable positive delta in
energy efficiency
16. Coldwater cooling
• Cold water often need chillers to be generate so it impacts negatively
the PUE
• Ideally cold water should be generated by natural sources like lakes,
rivers or by natural sources of cool, like cold climates, high mountain
or geothermal exchange
• Eurotech can design solutions that accommodate the use of natural
sources of cooling
20. Reducing cooling energy
Ways to reduce cooling energy consumption
• Air cooling optimization (hot and cold aisle containment…)
• Free cooling: avoid compressor based cooling (chillers) using cold air coming from
outside the data center. Possible only in cold climate or seasonal
• Free cooling with heat exchangers (dry coolers). Dry coolers consume much less
energy than chillers!
• Liquid cooling to increase the cooling efficacy and reduce the power absobed by chillers
• Liquid cooling with free cooling: the liquid is not cooled by chillers but by dry coolers
• Hot liquid cooling allows the use of dry coolers all year round and also in warm climates
• Liquid cooling using a natural source of
• Alternative approaches: spray cooling, oil submersion cooling
Eurotech Aurora approach:
• Direct Hot Water Cooling with no chillers but only dry coolers
22. Pumps consume energy but they
can control the flowrate
Increasing the flowrate is less
Chiller s energy demanding that swicthing on
Dry Coolers
a chiller
LOOP #1
LOOP #2
heater
By pass
23. Advantages of the Eurotech approach
Hot liquid cooling no chillers save energy
• Avoid/limit expensive and power hungry chillers with the only
cooling method that requires almost always dry coolers only
• Minimize PUE and hence maximize energy cost savings
• Reuse thermal energy for heating, air conditioning, electrical
energy or industrial processes
• “Clean” free cooling: no dust, no filters needed to filter
external air
Direct liquid cooling via cold plates effective cooling
• Allow very limited heat spillage
• Maximize the effectiveness of cooling allowing for hot water
to be used (up to 55 °C inlet water)
Comprehensive more efficiency
• Cools any source of heat in the server (including power
supply)
28. Minimize waste: thermal energy re-use
Three Stages Cooling + Heat Recovery
1 MW
0.13 MW
Computing Computing Computing
system system system
20° C 25° C 30° C
rack 1 rack 2 rack #n
Liquid to Liquid
Heat exchanger
Liquid to Liquid Heat
exchanger
0.87 MW
30° C 55° C
PUE < 1 !! Thermal energy re-use
29. Minimize waste: thermal energy re-use
• The ability to effectively re-use the waste heat from the outlets increases
with higher temperatures.
• Outlet temperatures starting from 45°C can be used to heat buildings,
temperatures starting from 55°C can be used to drive adsorption chillers.
• Higher temperatures may even allow for trigeneration, the combined
production of electricity, heating and cooling
• Warm water can be used also in industrial processes
30. Thermal energy recovery and swimming pools
Swimming pool 50 m, 4 lanes, 2m deep that looses 2°C per day if not heated
The heat exchange system has 90% efficiency
Volume water = 2,50m x 4 x 50m x 2m = 1000m^3 = 10^6 litri = 10^6 Kg
Water specific heat= specificheat = 4186 Joule / Kg K
Water target temperature = 28°C
How much power do I need to keep the swimming pool at 28°C?
P(W) = Q(Joule)/t(sec) = m(kg) * c_specif (Joule/Kg K) * deltaT (K)/t(sec) = 10^6
Kg * 4186 Joule/Kg K * 2K ( 24*60*60 sec ) = 96900 W = 96,9 KW
So we need a supercomputer generating roughly 110 kW.
Assuming an energy efficiency of 900 Mflops/W…
…to heat the swimming pool we would need to install a 100 Tflop
system.
That is, one Eurotech Aurora HPC 10-10 rack
33. Total cost of ownership
A comparison between datacenters: initial cost
Both datacenters with roughly 1MW of IT equipment
installed
OPTIMAL Air Cooled Hot Liquid Coloed
Values in K$ Datacenter (PUE = 1.8) datacenter (PUE=1.05)
Cost of IT (HW and SW) $8,200 $8,200
Facilities (building, raised floor,
fire system...) $960 $410
Racks and rack mngt software $220 $100
Liquid cooling $0 $620
Total for network equipment $710 $710
Cooling infrastructure/plumbing $4,280 $580
Electrical $5,710 $3,880
TOTAL INVESTMENT COST $20,080 $14,500
34. Total cost of ownership
A comparison between datacenters: annualized TCO
Both datacenters with roughly 1MW of IT equipment
installed
OPTIMAL Air Cooled Hot Liquid Coloed
Values in K$ Datacenter (PUE = 1.8) datacenter (PUE=1.05)
Cost of energy $2,690 $1,060
Retuning and additional CFD $5 $0
Total outage cost $440 $370
Preventive maintenance $150 $150
Annual facility and infrastructure
maintenance. $460 $220
Lighting $4 $2
Annualized 3 years capital costs $3,480 $3,440
Annualized 10 years capital costs $1,420 $720
Annualized 15 years capital costs $100 $40
ANNUALIZED TCO $8,749 $6,002
35. GREEN considerations
A comparison between datacenters
OPTIMAL Air Cooled Hot Liquid Coloed
Datacenter (PUE = 1.8) datacenter (PUE=1.05)
Total tons CO2 in 5 15,500 26,600
years
Tons of CO2 saved 11070 0
CO2 savings
36. CO2 equivalent
11000 tons of saved CO2 are equivalent
to
1500 cars that do not circualte for 1 yesr
11500 saved adult trees
15 Km2 of rain forest left untouched
Editor's Notes
NREL = National Renewable Energy Laboratory
This slide shows that to getPUEs < 1.8 dedicatedinfrastructureisneeded to implementeither free cooling or liquidcooling
