1. Heiko J Schick – IBM Deutschland R&D GmbH
January 2011
Experiences in Application Specific
Supercomputer Design
Reasons, Challenges and Lessons Learned
© 2011 IBM Corporation

2. Agenda
 The Road to Exascale
 Reasons for Application Specific Supercomputers
 Example: QPACE QCD Parallel Computing on the Cell Broadband Engine™ (Cell/B.E.)
 Challenges and Lessons Learned
2 © 2009 IBM Corporation

3. Where are we now? Blue Gene/Q !!!
 BG/Q Overview
– 16384 cores per compute rack
– Water cooled, 42U compute rack
– PowerPC compliant 64-bit microprocessor
– Double precision, quad pipe floating point acceleration on each core
– The system is scalable to 1024 compute racks, each with 1024 compute node ASICs.
 BG/Q Compute Node
– 16 PowerPC processing cores per node, 4w MT, 1,6 GHZ
– 16 GB DDR3 SDRAM memory per node with 40GB/s DRAM access
– 3 GF/W
• 200GF/Chip
• 60W/Node (All-inclusive: DRAM, Power Conversion, etc.)
– Integrated Network
3 © 2009 IBM Corporation

4. Projected Performance Development
Almost a doubling every year !!!
4 © 2009 IBM Corporation

5. The Big Leap from Petaflops to Exaflops
 We will hit 20 Petaflop in 2011/2012 …. Now beginning research for ~2018 Exascale.
 IT/CMOS industry is trying to double performance every 2 years.
HPC industry is trying to double performance every year.
 Technology disruptions in many areas.
– BAD NEWS: Scalability of current technologies?
• Silicon Power, Interconnect, Memory, Packaging.
– GOOD NEWS: Emerging technologies?
• Memory technologies (e.g. storage class memory)
 Exploiting exascale machines.
– Want to maximize science output per €.
– Need multiple partner applications to evaluate HW trade-offs.
5 © 2009 IBM Corporation

6. Extrapolating an Exaflop in 2018
Standard technology scaling will not get us there in 2018
BlueGene/L Exaflop Exaflop compromise Assumption for “compromise guess”
(2005) Directly using traditional
scaled technology
Node Peak Perf 5.6GF 20TF 20TF Same node count (64k)
hardware 2 8000 1600 Assume 3.5GHz
concurrency/node
System Power in 1 MW 3.5 GW 25 MW Expected based on technology improvement through 4 technology generations. (Only
Compute Chip compute chip power scaling, I/Os also scaled same way)
Link Bandwidth 1.4Gbps 5 Tbps 1 Tbps Not possible to maintain bandwidth ratio.
(Each unidirectional
3-D link)
Wires per 2 400 wires 80 wires Large wire count will eliminate high density and drive links onto cables where they are
unidirectional 3-D 100x more expensive. Assume 20 Gbps signaling
link
Pins in network on 24 pins 5,000 pins 1,000 pins 20 Gbps differential assumed. 20 Gbps over copper will be limited to 12 inches. Will need
node optics for in rack interconnects.
10Gbps now possible in both copper and optics.
Power in network 100 KW 20 MW 4 MW 10 mW/Gbps assumed.
Now: 25 mW/Gbps for long distance (greater than 2 feet on copper) for both ends one
direction. 45mW/Gbps optics both ends one direction. + 15mW/Gbps of electrical
Electrical power in future: separately optimized links for power.
Memory 5.6GB/s 20TB/s 1 TB/s Not possible to maintain external bandwidth/Flop
Bandwidth/node
L2 cache/node 4 MB 16 GB 500 MB About 6-7 technology generations with expected eDRAM density improvements
Data pins associated 128 data 40,000 pins 2000 pins 3.2 Gbps per pin
with memory/node pins
Power in memory I/O 12.8 KW 80 MW 4 MW 10 mW/Gbps assumed. Most current power in address bus.
(not DRAM) Future probably about 15mW/Gbps maybe get to 10mW/Gbps (2.5mW/Gbps is c*v^2*f for
random data on data pins) Address power is higher.
6 © 2009 IBM Corporation

7. Building Blocks of Matter
 QPACE = QCD Parallel Computing on the Cell Broadband Engine™ (Cell/B.E.)
 Quarks are the constituents of matter which strongly interact exchanging gluons.
 Particular phenomena
– Confinement
– Asymptotic freedom (Nobel Prize 2004)
 Theory of strong interactions = Quantum Chromodynamics (QCD)
7 © 2009 IBM Corporation

8. Balanced Hardware
 Example caxpy:
Processor FPU throughput Memory bandwidth
[FLOPS / cycle] [words / cycle] [FLOPS / word]
apeNEXT 8 2 4
QCDOC (MM) 2 0.63 3.2
QCDOC (LS) 2 2 1
Xeon 2 0.29 7
GPU 128 x 2 17.3 (*) 14.8
Cell/B.E. (MM) 8x4 1 32
Cell/B.E. (LS) 8x4 8x4 1
9 © 2009 IBM Corporation

9. Balanced Systems ?!?
10 © 2009 IBM Corporation

10. … but are they Reliable, Available and Serviceable ?!?
11 © 2009 IBM Corporation

11. Collaboration and Credits
 QPACE = QCD Parallel Computing on the Cell Broadband Engine™ (Cell/B.E.)
 Academic Partners
– University Regensburg S. Heybrock, D. Hierl, T. Maurer, N. Meyer, A. Nobile, A. Schaefer, S. Solbrig, T. Streuer, T. Wettig
– University Wuppertal Z. Fodor, A. Frommer, M. Huesken
– University Ferrara M. Pivanti, F. Schifano, R. Tripiccione
– University Milano H. Simma
– DESY Zeuthen D.Pleiter, K.-H. Sulanke, F. Winter
– Research Lab Juelich M. Drochner, N. Eicker, T. Lippert
 Industrial Partner
– IBM (DE, US, FR) H. Baier, H. Boettiger, A. Castellane, J.-F. Fauh, U. Fischer, G. Goldrian, C. Gomez, T. Huth, B. Krill,
J. Lauritsen, J. McFadden, I. Ouda, M. Ries, H.J. Schick, J.-S. Vogt
 Main Funding
– DFG (SFB TR55), IBM
 Support by Others
– Eurotech (IT) , Knuerr (DE), Xilinx (US)
13 © 2009 IBM Corporation

12. Production Chain
Major steps
– Pre-integration at University Regensburg
– Integration at IBM / Boeblingen
– Installation at FZ Juelich and University Wuppertal
14 © 2009 IBM Corporation

13. Concept
 System
– Node card with IBM® PowerXCell™ 8i processor and network processor (NWP)
• Important feature: fast double precision arithmetic's
– Commodity processor interconnected by a custom network
– Custom system design
– Liquid cooling system
 Rack parameters
– 256 node cards
• 26 TFLOPS peak (double precision)
• 1 TB Memory
– O(35) kWatt power consumption
 Applications
– Target sustained performance of 20-30%
– Optimized for calculations in theoretical particle physics:
Simulation of Quantum Chromodynamics
15 © 2009 IBM Corporation

14. Networks
 Torus network
– Nearest-neighbor communication, 3-dimensional torus topology
– Aggregate bandwidth 6 GByte/s per node and direction
– Remote DMA communication (local store to local store)
 Interrupt tree network
– Evaluation of global conditions and synchronization
– Global Exceptions
– 2 signals per direction
 Ethernet network
– 1 Gigabit Ethernet link per node card to rack-level switches (switched network)
– I/O to parallel file system (user input / output)
– Linux network boot
– Aim of O(10) GB bandwidth per rack
16 © 2009 IBM Corporation

15. Root Card
(16 per rack) Backplane
(8 per rack)
Node Card
(256 per rack)
Power Supply and Power Adapter Card
(24 per rack)
Rack
17 © 2009 IBM Corporation

16. Node Card
 Components
– IBM PowerXCell 8i processor 3.2 GHZ
– 4 Gigabyte DDR2 memory 800 MHZ with ECC
– Network processor (NWP) Xilinx FPGA LX110T FPGA
– Ethernet PHY
– 6 x 1GB/s external links using PCI Express physical layer
– Service Processor (SP) Freescale 52211
– FLASH (firmware and FPGA configuration)
– Power subsystem
– Clocking
 Network Processor
– FLEXIO interface to PowerXCell 8i processor, 2 bytes with 3 GHZ bit rate
– Gigabit Ethernet
– UART FW Linux console
– UART SP communication
– SPI Master (boot flash)
– SPI Slave for training and configuration
– GPIO
18 © 2009 IBM Corporation

17. Node Card
Network Processor Network PHYs
PowerXCell 8i (FPGA)
Memory Processor
19 © 2009 IBM Corporation

18. Node Card
DDR2 DDR2
DDR2 DDR2
800MHz
I2C
Power SPI RW
Subsystem (Debug)
PowerXCell 8i
FLEXIO FLEXIO
Clocking
6GB/s 6GB/s
RS232
SPI
I2C
SP FPGA Virtex-5
UART
Freescale
MCF52211 GigE PHY
SPI 384 IO@250MHZ
Flash
4*8*2*6 = 384 IO
680 available (LX110T)
6x 1GB/s PHY
Compute Network
20 © 2009 IBM Corporation

19. Network Processor
x+
Link Slices 92 %
PHY
Interface
PINs 86 %
x-
Link
LUT-FF pairs 73 %
PHY
Interface
Flip-Flops 55 %
 
 
Network Logic
  LUTs 53 %
z-
FlexIO Routing Link BRAM / FIFOs 35 %
PHY
Interface Interface
Arbitration
FIFOs
Ethernet
PHY
Configuration Interface
Global Flip-Flops LUTs
Signals
Processor Interface 53 % 46 %
Serial
Interfaces
Torus 36 % 39 %
SPI Flash Ethernet 4% 2%
21 © 2009 IBM Corporation

20. Torus Network Architecture
 2-sided communication
– Node A initiates send, node B initiates receive
– Send and receive commands have to match
– Multiple use of same link by virtual channels
 Send / receive from / to local store or main memory
– CPU → NWP
• CPU moves data and control info to NWP
• Back-pressure controlled
– NWP → NWP
• Independent of processor
• Each datagram has to be acknowledged
– NWP → CPU
• CPU provides credits to NWP
• NWP writes data into processor
• Completion indicated by notification
23 © 2009 IBM Corporation

21. Torus Network Reconfiguration
 Torus network PHYs provide 2 interfaces
– Used for network reconfiguration b selecting primary or secondary interface
 Example
– 1x8 or 2x4 node-cards
 Partition sizes (1,2,2N) * (1,2,4,8,16) * (1,2,4,8)
– N ... number of racks connected via cables
24 © 2009 IBM Corporation

22. Cooling
 Concept
– Node card mounted in housing = heat conductor
– Housing connected to liquid cooled cold plate
– Critical thermal interfaces
• Processor – thermal box
• Thermal box – cold plate
– Dry connection between node card and cooling circuit
 Node card housing
– Closed node card housing acts as heat conductor.
– Heat conductor is linked with liquid-cooled “cold plate”
– Cold Plate is placed between two rows of node cards.
 Simulation Results for one Cold Plate
– Ambient 12°C
– Water 10 L / min
– Load 4224 Watt
2112 Watt / side
25 © 2009 IBM Corporation

23. Project Review
 Hardware design
– Almost all critical problems solved in time
– Network Processor implementation was a challenge
– No serious problems due to wrong design decisions
 Hardware status
– Manufacturing quality good: Small bone pile, few defects during operation.
 Time schedule
– Essentially stayed within planned schedule
– Implementation of system / application software delayed
26 © 2009 IBM Corporation

24. Summary
 QPACE is a new, scalable LQCD machine based on the PowerXCell 8i processor.
 Design highlights
– FPGA directly attached to processor
– LQCD optimized, low latency torus network
– Novel, cost-efficient liquid cooling system
– High packaging density
– Very power efficient architecture
 O(20-30%) sustained performance for key LQCD kernels is reached / feasible
→ O(10-16) TFLOPS / rack (SP)
27 © 2009 IBM Corporation

25. Power Efficiency
28 © 2009 IBM Corporation

26. 29 © 2009 IBM Corporation

27. 30 © 2009 IBM Corporation

28. 31 © 2009 IBM Corporation

29. 33 © 2009 IBM Corporation

30. Challenge #1: Data Ordering
 InfiniBand test failed on cluster with 14 blade server
– Nodes were connected via InfiniBand DDR adapter.
– IMB (Pallas) stresses MPI traffic over InfiniBand network.
– System fails after couple minutes, waiting endless for a event.
– System runs stable, if global setting for strong ordering is set (default is relaxed).
– Problem was in the meanwhile recreated with same symptoms on InfiniBand SDR hardware .
– Changing from relaxed to strong ordering changes performance significantly !!!
 First indication points to DMA ordering issue
– InfiniBand adapter do consecutive writes to memory, sending out data, followed by status.
– InfiniBand software stack polls regularly on status.
– If status is updated before data arrives, we clearly had an issue.
34 © 2009 IBM Corporation

31. Challenge #1: Data Ordering (continued)
 Ordering of device initiated write transactions
– Device (InfiniBand, GbE, ...) writes data to two different memory locations in host memory
– First transaction writes data block ( multiple writes )
– Second transaction writes status ( data ready )
– It must be ensured, that status does not reach host memory before complete data
• If not, software may consume data, before it is valid !!!
 Solution 1:
– Always do strong ordering, i.e. every node in the path from device to host will send data out in order received
– Challenge: IO Bandwidth impact, which can be significant.
 Solution 2:
– Provide means to enforce ordering of the second write behind the first, but leave all other writes unordered
– Better performance
– Challenge: Might need device firmware and/or software support
35 © 2009 IBM Corporation

32. Challenge #2: Data is Everything
 BAD NEWS: There is a many ways how an application can be accelerated.
– An inline accelerator is an accelerator that runs sequentially with the main compute
engine.
– A core accelerator is a mechanism that accelerates the performance of a single core.
A core may run multiple hardware threads in an SMT implementation.
– A chip accelerator is an off-chip mechanism that boosts the performance of the
primary compute chip. Graphics accelerators are typically of this type.
– A system accelerator is a network-attached appliance that boosts the performance of a
primary multinode system. Azul is an example of a system accelerator.
36 © 2009 IBM Corporation

33. Challenge #2: Data is Everything (continued)
 GOOD NEWS: Application acceleration is possible!
– It is all about data:
• Who owns it?
• Where is it now?
• Where is it needed next?
• How much does it cost to send it from now to next?
– Scientists, computer architects, application developers and system administrators needs
to work together closely.
37 © 2009 IBM Corporation

34. Challenge #3: Traffic Pattern
 IDEA: Open QPACE for border range of HPC applications (e.g High Performance LINPACK).
 High-speed point-to-point interconnect with a 3D torus topology
 Direct SPE-to-SPE communication between neighboring nodes for good nearest neighbor
performance
38 © 2009 IBM Corporation

35. Challenge #3: Traffic Pattern (continued)
 BAD NEWS: High Performance LINPACK Requirements
– Matrix stored in main memory
• Experiments show: Performance gain with increasing memory size
– MPI communications:
• Between processes in same row/column of process grid
• Message sizes: 1 kB … 30 MB
– Efficient Level 3 BLAS routines (DGEMM, DTRSM, …)
– Space trade-offs and complexity leads to a PPE-centric programming model!
39 © 2009 IBM Corporation

36. Challenge #3: Traffic Pattern (continued)
 GOOD NEWS: We have an FPGA. ;-)
– DMA Engine was added to the NWP design on the FPGA and can fetch data from main
memory.
– PPE is responsible for MM-to-MM message transfers.
– SPE is only used for computation offload.
40 © 2009 IBM Corporation

37. Challenge #4: Algorithmic Performance
 BAD NEWS: Algorithmic performance doesn’t necessarily reflect machine performance.
– Numerical problem solving of a sparse matrix via an iterative method.
– If room of residence is adequate small the algorithm has converged and calculation is finished.
– Difference between the algorithms is the number of used auxiliary vectors (number in brackets).
Source: Andrea Nobile (University of Regensburg)
41 © 2009 IBM Corporation

38. Thank you very much for your attention.
42 © 2009 IBM Corporation

39. Disclaimer
 IBM®, DB2®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xS
eries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix und
Informix® Dynamic ServerTM, IBM, BladeCenter and POWER and others are trademarks of
the IBM Corporation in US and/or other countries.
 Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United
States, other countries, or both and is used under license there from. Linux is a trademark of
Linus Torvalds in the United States, other countries or both.
 Other company, product, or service names may be trademarks or service marks of others.
The information and materials are provided on an "as is" basis and are subject to change.
43 © 2009 IBM Corporation