Lenovo HPC: Energy Efficiency and Water-Cool-Technology Innovations

Energy Efficiency and
Water-Cool-Technology
Innovations
2018 Lenovo - All rights reserved.
Karsten Kutzer | April 10th 2018 | Swiss Conference 2018
Acknowledgments: Luigi Brochard, Vinod Kamath, Martin Hiegl (Lenovo)
Julita Corbalan (BSC)

2
Why care about Power and Cooling?
Increasing
Electricy Cost
Performance-
Power relation
Application
Diversity
Waste Heat
Reuse
Data Center
limitations
Leading the Industry in Energy Aware HPC

3
0
200
400
600
800
1000
1200
1400
1600
1800
2000
60
80
100
120
140
160
180
200
220
240
2006-06-01
2006-11-01
2007-04-01
2007-09-01
2008-02-01
2008-07-01
2008-12-01
2009-05-01
2009-10-01
2010-03-01
2010-08-01
2011-01-01
2011-06-01
2011-11-01
2012-04-01
2012-09-01
2013-02-01
2013-07-01
2013-12-01
2014-05-01
2014-10-01
2015-03-01
2015-08-01
2016-01-01
2016-06-01
2016-11-01
2017-04-01
Intel Xeon Processor & Spec_fp Rate
TDP CFP2006 Rate2018 Lenovo - All rights reserved.
Performance-Power relation
500400320300 35024020585 12075
NVIDIA / AMD GPU
XEON
PHIAMD
NERVANA/CREST
NVIDIA SXM
• Maintaining Moore’s Law with increased competition
is resulting in higher component power
• Increased memory count, NVMe adoption, and I/O
requirements are driving packaging and feature
tradeoffs (superset of features doesn’t fit in 1U)
• Shared cooling fan power savings no longer exist
for dense 2S nodes architectures due to non-
spreadcore CPU layout high airflow requirements
 For highest performance systems will have to
reduce density or move to optimized cooling.
ARM SOC
Haswell
Sandy Bridge / IvyBridge

Application Diversity
• CPU bound BQCD case
• Node runs on full Power
• CPU provides full performance
while running at full power
• Memory bound BQCD case
• Node still runs on full Power
• CPU provides less performance
while still running at full power
0.00
100.00
200.00
300.00
400.00
500.00
600.00
1
9
17
25
33
41
49
57
65
73
81
89
97
105
113
121
129
137
145
153
161
169
177
185
193
201
209
217
225
DC node[W]
CPU pkg 0 [W]
RAM pkg 0 [W]
CPU pkg 1 [W]
RAM pkg 1 [W]
0.00
100.00
200.00
300.00
400.00
500.00
600.00
1
9
17
25
33
41
49
57
65
73
81
89
97
105
113
121
129
137
145
153
161
169
177
185
193
201
209
217
225
DC node[W]
CPU pkg 0 [W]
RAM pkg 0 [W]
CPU pkg 1 [W]
RAM pkg 1 [W]
Turbo ON: 157 GFlops Turbo ON: 65 Gflops
SD650 with 2 sockets 8168 and 6 x 16GB DIMMs; room temp = 21°C, inlet water = 45°C, 1.5 lpm/tray
How much energy do we waste on non-CPU bound application?

Waste Energy reuse - ERE
Energy Waste Direct Reuse Indirect Reuse
How much energy do we waste by not using the system heat?
Pictures: Leibniz Supercomputing Centre

6
Energy Aware HPC
Best CPU choice with max TDP supported
Best performance fully utilizing the system
Best TCO / Performance for maximized ROI
Best use of limited DataCenter capacities
Best Carbon Footprint for eco responsible HPC

7
The three Pillars
Leading the Industry in Energy Aware HPC
Hardware Software Infrastructure

Hardware

9
Direct Water Cooling
Water Cooling Technologies

10
• Standard Air flow with
internal fans cooled with
the room climatization
• Broadest choice of
configurable options
supported
• Relatively inefficient cooling
• Air cooled but heat
removed with RDHX
through chilled water
• Retains high flexibility
• Enables extremely tight
rack placement
• Potentially room neutral
• Most heat removed by
onboard-waterloop with
up to 50°C temperature
• Supports highest TDP CPU
at densest footprint
• Higher performance
• Free cooling
Air Cooled Air Cooled
w/ Rear Door Heat Exch.
Direct Water Cooled
Lenovo Cooling Technologies
Choose for broadest choice
of customizable options
Choose for max performance
and high energy efficiency
Choose for increased energy
efficiency with broad choice
PUE ~2.0 – 1.5
ERE ~2.0 – 1.5
PUE ~1.4 – 1.2
ERE ~1.4 – 1.2
PUE <=1.1
ERE <=1.1

Return on Investment for DWC vs RDHx
• New data centers: Water cooling has immediate payback.
• Existing air-cooled data center payback period strongly depends on electricity rate
DWC RDHx
$0.06/kWh $0.12/kWh $0.20/kWh

12
Rear Door Heat Exchanger
Up to 27°C Cold Water Cooling
Up to 100% Heat Removal Efficiency on 30kW
No moving parts or power required
Tenthousands of nodes install base
Long
ago
2009
2010

Lenovo Rear Door Heat Exchanger
Feature RDHx2
3.500 times more efficient than cold air
Air
Movement
Provided by the systems in the rack
Heat
removal
At 18oC Water temp, 27oC inlet air temp:
100% for 30kW; 90% for 40kW
Water
temperature
• Min 18° C / 64.4° F for ASHRAE Class 1
• Min 22° C / 71.6° F for ASHRAE Class 2
• Max 27°C
Water
Volume
9 Liters / 2.4 Gallons
Water Flow
Rate
• Min 22.7 liters / 6 gallons per minute
• Max 56.8 liters / 15 gallons per minute
Door
Dimensions
• Depth: 129mm/5in.
• Height: 1950mm/76.8in.
• Width: 600mm/23.6in.
Door
Assembly
Weight
• Empty: 39kg/85lbs
• Filled 48kg/105lbs
Connection • ¾ inch quick connect
(Supply: Parker SH6-63W; Return: Parker SH6-62-W; or equivalent)
© Torsten Bloth

Lenovo RDHx2 – Typical Environment

15
Direct “Hot” Watercooling
2012
2014
2018
>24.000 nodes globally
Up to 50°C Hot Water Cooling
Up to 90% Heat Removal Efficiency
World Record Energy Reuse Efficiency
30+ patents on market leading design

Lenovo ThinkSystem SD650
Feature SD650
Processors
2 Intel “Purley” Generation processors per node
• Socket-F for Intel Omnipath supported
• >120W all Skylake Shelves supported
Form factor 1U Full wide tray double-node / 6U12N Chassis
Memory Slots
Max Memory
• 12x DDR4 (R/LR) 2667MHz DIMM
• 4x Intel Apache Pass DIMM ready
Storage • 2x SATA slim SSD / 1x NVMe, 2x M.2 SATA SSD
NIC 1x 1 GBaseT, 1x 1 GbE XCC dedicated
PCIe
1x x16 PCIe for EDR Infiniband / OPA100
1x x16 ML2 for 10Gbit Ethernet (in place of Storage)
Power 1300W/1500W/2000W Platinum and 1300W Titanium
USB ports Up to 1x front via dongle cable + 1x internal (2.0)
Cooling
• No fans on chassis, PSU fans only
• Up to 50°C warm water circulated through cooling
tubes for component level cooling
System
MGMT / TPM
XCC, dedicated port or shared
TPM, Pluggable TCM
Dimensions 915mm depth, front access w/ front I/O
© Torsten Bloth

17
Top-Down View
ThinkSystem SD650
Water Inlet *)
Water Outlet
Power
Board
CPUs
6 DIMMs
per CPU
2 AEP
per CPU
x16 PCIe Slot
Disk Drive
M.2 Slot
50°C
60°C
two nodes sharing a tray and a waterloop
*) inlet water temperature 50°C with configuration limitations (45°C without configuration limitations)

SD650 Improved Node Water Cooling Architecture
• Focus on maximizing efficiency for high
(up to 50°C) inlet water temperatures
• Device cooling optimization by minimizing
water to device temperature differences
– dT CPU < ~0.1 K / W
– dT Memory < ~1 K / W
– dT Network < ~1 K / W
• Direct water cooling of processors,
memory, voltage regulation devices and
IO devices (Network and Disk)
• Water circuit traverses all critical
components to optimize cooling.
DISK
Conductive
plate
Memory
Water
chanels

HPL Temperature & Frequency on SD650 with 8168
PL2 (short term RAPL limit) is 1.2 x TDP PL1 (long term RAPL limit) is TDP
Non AVX instructions AVX instructions Non AVX instructions
SD650 with 2 sockets 8168 and 12 x 16GB DIMMs; room temp = 21°C, inlet water = 40°C, 1.5 lpm/tray

Performance Optimization
• ThinkSystem SD530 – Standard Performance
– ~ 2.15 TeraFlop/s sustained HPL
w/ SKL 6148 20C 2.4Ghz 150W
– /s sustained HPL
w/ SKL 6148 20C 2.4Ghz 150W
• ThinkSystem SD650 – High Performance Mode
– ~ 2.34 TeraFlop/s sustained HPL
w/ SKL 6148 20C 2.4Ghz 150W
HPC [GF] AC node DC node CPU Temp
Turbo OFF 2152.7 400.1 368.0 81.8
Turbo ON 2147.2 400.4 368.3 82.1
Turbo OFF
Turbo ON
Turbo OFF 2342.0 472.5 434.7 36.8
Turbo ON 2333.4 473.2 435.4 36.9
SD530 and SD650 with 2 sockets 6148 and 12 x 16GB DIMMs; room temp = 21°C, inlet water = 18°C, 1.5 lpm/tray
+9% +18%

Software

22
MANAGINGREPORTING
Becoming Energy Aware

SD650 – DC Power Sampling/Reporting Frequency
• AC power at chassis level
(through FPC)
– With xCAT
– With ipmi
• DC power and energy at
node level through XCC
– With hw_usage library
– With ipmi
– With RAPL
– With Allinea
– With LSF or LEAR
NM/ME
HSC
RAPL
CPU/memory
(energy MSRs)
XCC/BMC
1Hz
10Hz
1KHz
Meter
500Hz
Sensor
200Hz1Hz
High Level Software
HSC –node
power
XCC/BMC
FPGA
100Hz
100Hz
100Hz
New for Lenovo
ThinkSystem SD650
10KHz
Sensor

24
Bulk 12V Node 12V
SD650 – advanced Accuracy for Power and Energy
• Node DC Power readings
– Better than or equal to +/-3% power reading accuracy
– down to the node’s minimum active power (~40-50W DC).
– Power granularity <=100mW
– At least 100Hz update rate for node power readings
• Node DC Energy meter
– Accumulator for Energy in Joules (~10 weeks until meter overflow)
XCC
ME (Node
Manager)
SN1405006
(used for
capping)
FPGA
(FIFO)
ipmi raw
oem cmd
Rsense
INA226
(used for
metering)
High accuracy, fast sampling Maintains compatibility with Node Manager

Energy Aware Run time: Motivation
• Power and Energy has become a
critical constraint for HPC systems
• Performance and Power consumption
of parallel applications depends on:
– Architectural parameters
– Runtime node configuration
– Application characteristics
– Input data
• Manual “best” frequency
– Difficult to select manually and it is a time
consuming process (resources and then
power) and not reusable
– It may change along time
– It may change between nodes
Configure
application for
Architecture X
Execute with N
frequencies:
calculate time
and energy
Select optimal
frequency

27
EAR – Automatic and Dynamic CPU Frequency
• Architecture characterization
• Application characterization
– Outer loop detection (DPD)
– Application signature computation (CPI,GBS,POWER,TIME)
• Performance and power projection
• Users/System policy definition for frequency selection (with thresholds)
– MINIMIZE_ENERGY_TO_SOLUTION
- Goal: To save energy by reducing frequency (with potential performance degradation)
- We limit the performance degradation with a MAX_PERFORMANCE_DEGRADATION threshold
– MINIMIZE_TIME_TO_SOLUTION
- Goal: To reduce time by increasing frequency (with potential energy increase)
- We use a MIN_PERFORMANCE_EFFICIENCY_GAIN threshold to avoid that application that do not scale
with frequency to consume more energy for nothing

EAR – Functional Overview
Learning Phase (at EAR installation*)
Execution Phase (loaded with application)
Kernel
Execution
Coefficients
Computation
Coeffcients
Database
Dynamic Patter Detection
detects outer loop
Compute power and
performance metrics
for outer loop
Energy Policy
read
CPUFrequency
* or every time cluster configuration is modified
(more memory per node, new processors ...)
Optimal frequency
calculation

BQCD_CPU with EAR MIN_ENERGY_TO_SOLUTION
0
5000
10000
15000
20000
25000
30000
0
131843
225947
299779
390597
515154
1285534
2011522
2553471
3023401
3533108
3883160
4591497
5063327
5621436
6088931
6599507
6954154
7478425
7922780
8294067
8793099
9248717
18777011
49344177
79886200
110390411
140874404
171355272
201829244
232294824
262738421
293165791
323574238
353978730
384358646
414755367
445175899
475596024
506014315
536410396
566819213
Acuumulated me
BQCD_CPU: Outer loop size detected(mpi rank 0)
2300000
2350000
2400000
2450000
2500000
2550000
2600000
2650000
0
134249
232508
321534
408810
692318
1506881
2107093
2762515
3130317
3787863
4367471
4882332
5566174
5918062
6438408
6951142
7481230
7985974
8504949
8813306
9337736
25467570
56976636
88462174
120870973
152311968
184688272
217057025
249418854
280817893
313128553
345428274
377717332
410009928
442326512
474646835
506014315
538310924
570616511
Frequency
Accumulated me
BQCD_CPU:Frequency(mpi rank 0)
0
5000
10000
15000
20000
25000
30000
0
130783
220771
295401
383475
461384
1079442
1830691
2338181
2840801
3298260
3792725
4365887
4864175
5392171
5828340
6403551
6693295
7204960
7666811
8017997
8521646
8974694
9337736
26419798
56021729
85604660
115152228
144683945
174209742
203733324
233245157
262738421
292216336
321677811
351131866
380564787
410009928
439469002
468944263
498411778
527870308
557327668
575261394
Acuumulated me
BQCD_CPU: Outer loop size detected(mpi rank 8)
2440000
2460000
2480000
2500000
2520000
2540000
2560000
2580000
2600000
2620000
0
132238
248215
386318
502916
846670
1414410
2166140
2657547
3182790
3679084
4148797
4872170
5253045
5903995
6272346
6776910
7296748
7802636
8272221
8640726
9145024
9656532
36068860
66603483
97121470
128576097
159059861
190494052
221913387
253303684
283750918
315114433
346461071
377801000
409146018
440503053
471881334
502293490
533647990
565005225
Frequency
Accumulated me
BQCD_CPU:Frequency(mpi rank 8)
M
P
I
R
A
N
K
0
M
P
I
R
A
N
K
8
Big loop detected
Policy is applied
F: 2.6Ghz2.4Ghz
230
235
240
245
250
255
260
265
270
275
0
131843
229109
304274
398976
671428
1330758
2082498
2573903
3099153
3595445
4065140
4788521
5169396
5820376
6187342
6693295
7213097
7718993
8188571
8557082
9061369
9572891
35985216
66519836
97037820
128492446
158976207
190410395
221829728
253220023
283667255
315030767
346377404
377717332
409062348
440419381
471797662
502209816
533564316
564921550
Avg.Power(W)
Accumulated me
BQCD_CPU:Measured POWER(mpi rank 0)
245
250
255
260
265
270
0
131195
244291
380251
491701
689551
1366600
1933029
2593333
3105484
3613768
3963772
4669953
5145362
5703804
6013462
6681447
7034739
7555744
8004770
8374090
8868621
9330380
15973526
43705120
73298174
103789543
133340894
163821371
194299015
224765387
254257203
284701562
315114433
345511942
375902594
406291401
436701498
467128200
497543500
527003712
557411343
575230251
Avg.Power(W)
Accumulated me
BQCD_CPU:Measured POWER(mpi rank 8)
Power is reduced
0
200000
400000
600000
800000
1000000
1200000
0
134249
232508
321534
408810
692318
1506881
2107093
2762515
3130317
3787863
4367471
4882332
5566174
5918062
6438408
6951142
7481230
7985974
8504949
8813306
9337736
25467570
56976636
88462174
120870973
152311968
184688272
217057025
249418854
280817893
313128553
345428274
377717332
410009928
442326512
474646835
506014315
538310924
570616511
Iteraonme(usecs)
Accumulated me
BQCD_CPU:Measured Itera on me(mpi rank 0)
0
200000
400000
600000
800000
1000000
1200000
0
132238
248215
386318
502916
846670
1414410
2166140
2657547
3182790
3679084
4148797
4872170
5253045
5903995
6272346
6776910
7296748
7802636
8272221
8640726
9145024
9656532
36068860
66603483
97121470
128576097
159059861
190494052
221913387
253303684
283750918
315114433
346461071
377801000
409146018
440503053
471881334
502293490
533647990
565005225
Iteraonme(usecs)
Accumulated me
BQCD_CPU:Measured Itera on me(mpi rank 8)
Iteration time is
similar

Infrastructure

PUE, ITUE, TUE and ERE
• Power Usage Effectiveness (PUE) says how much power a datacenter uses is not used for computing.
• It is the ratio of total power to the power delivered to computing equipment.
• It does not take into account how effective a server uses the Power it gets.
• Ideal value is 1.0
• IT Usage Effectiveness (ITUE) measures how much power a system uses is not used for computing.
• It is the ratio of the power of IT equipment to the power of the computing components.
• Multiplied with the PUE it gives the Total-Power Usage Effectiveness (TUE)
• Ideal value is 1.0
• Energy Reuse Effectiveness (ERE) integrates the reuse of the power dissipated by the computer.
• It is the ratio of total power considering also reuse to the power delivered to computing equipment.
• An ideal ERE is 0.0. If no reuse, ERE = PUE
𝑃𝑈𝐸 =
𝑇𝑜𝑡𝑎𝑙 𝐹𝑎𝑐𝑖𝑙𝑖𝑡𝑦 𝑃𝑜𝑤𝑒𝑟
𝑇𝑜𝑡𝑎𝑙 𝐼𝑇 𝑃𝑜𝑤𝑒𝑟
𝐼𝑇𝑈𝐸 =
𝑇𝑜𝑡𝑎𝑙 𝐶𝑜𝑚𝑝𝑢𝑡𝑒 𝑃𝑜𝑤𝑒𝑟
𝐸𝑅𝐸 =
(𝑇𝑜𝑡𝑎𝑙 𝐹𝑎𝑐𝑖𝑙𝑖𝑡𝑦 𝑃𝑜𝑤𝑒𝑟 − 𝑇𝑜𝑡𝑎𝑙 𝑅𝑒𝑢𝑠𝑒 𝑃𝑜𝑤𝑒𝑟)

32
• Standard Air flow with
internal fans cooled with
the room climatization
• Broadest choice of
configurable options
supported
• Relatively inefficient cooling
• Air cooled but heat
removed with RDHX
through chilled water
• Retains high flexibility
• Enables extremely tight
rack placement
• Potentially room neutral
• Waste heat reused to
generate coldness to cool
non-DWC components
• Retains highest TDP,
footprint and performance
• Potentially all system heat
covered through DWC
• Most heat removed by
onboard-waterloop with
up to 50°C temperature
• Supports highest TDP CPU
at densest footprint
• Higher performance
• Free cooling
Air Cooled Air Cooled
w/ Rear Door Heat Exch.
Direct Water Cooled
w/ Adsorption Chilling
Direct Water Cooled
Lenovo Cooling Technologies
Choose for broadest choice
of customizable options
and high energy efficiency
Choose for increased energy
efficiency with broad choice
and max energy efficiency
PUE ~2.0 – 1.5
ERE ~2.0 – 1.5
PUE ~1.4 – 1.2
ERE ~1.4 – 1.2
PUE <=1.1
ERE <=1.1
PUE <=1.1
ERE <1

Value of Direct Water Cooling with Free Cooling
• Reduced noise level in the DataCenter
• Reduced server power consumption
– Lower processor power consumption (~ 5%)
– No fan per node (~ 4%)
• Reduce cooling power consumption
– At 45°C free cooling all year long ( ~ 25%)
• Energy Aware Scheduling
– Only CPU bound jobs get max frequency (~ 5%)
• CAPEX Savings
– Less conventional chillers for the Computing System
Energy Savings
35-40%
Total Saving

34
Adsorption Chilling
The method of using solid materials
for cooling via evaporation.
• Adsorption chiller consists of two identical
vacuum containers, each containing two heat
exchangers – and water.
– Adsorber (Desorber)
Coated with the adsorbent (e.g. zeolite)
– Evaporator (Condenser)
Evaporation and condensation of water
• Adsorption process has 2 phases
– in the adsorption phase the water on the
evaporator is taken in by the coated material in
the adsorber. Through that evaporation the
evaporater and the water flowing through it does
cool down while the adsorber fills with water
vapor and heats up the water flowing through it.
When the adsorber is saturated the process is
reversed.
– in the desorption phase hot water is passed
through the adsorber acting as a desorber rather
as its desorbing the water vapor and dispensing it
to the evaporator which is acting as condenser at
that point condensing the vapor back to water.
Again the process is reversed when the adsorber
is emptied.
Module 1
Desorption
Hot Water from
Compute Racks
52°/46 °C
Condensation
Cooling Water
to Hybrid
Cooling Tower
26°/32°C
Adsorption
Cooling Water
to Hybrid
Cooling Tower
26°/32°C
Evaporation
Chilled Water
to Storage
etc. Racks
23°/20°C
Desorber Condenser
Module 2
Adsorber Evaporater

Value of Direct Water Cooling with Adsorption Chiller
• Reduced noise level in the DataCenter
• Maximum TDP CPU Choice
• Reduced server power consumption
– Lower processor power consumption (~ 5%)
– No fan per node (~ 4%)
• Reduce cooling power consumption
– At 50°C free cooling all year long (~ 25%)
– Heat Reuse generate 600kW cooling capacity (> 5%)
• Energy Aware Runtime
– Frequency optimization during runtime (~ 5%)
• CAPEX Savings
– Less conventional chillers for the Computing System
Energy Savings
40 - 50%
Total Saving

Lenovo HPC: Energy Efficiency and Water-Cool-Technology Innovations

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Lenovo HPC: Energy Efficiency and Water-Cool-Technology Innovations

Ähnlich wie Lenovo HPC: Energy Efficiency and Water-Cool-Technology Innovations (20)

Mehr von inside-BigData.com

Mehr von inside-BigData.com (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Lenovo HPC: Energy Efficiency and Water-Cool-Technology Innovations