HPC network stack on ARM - Linaro HPC Workshop 2018

5. © 2017 Arm Limited5 VERBs API on Arm • Besides bug fixes not much work was required • Mellanox OFED 2.4 and above supports Arm • Linux Kernel 4.5.0 and above (maybe even earlier) • Linux Distribution Support – on going process • OFED – no official ARMv8 support

6. © 2017 Arm Limited6 OpenUCX v1.3.0 • https://github.com/openucx/ucx/releases/tag/v1.3.0 WWW.OPENUCX.ORG https://github.com/openucx/ucx UC-T (Hardware Transports) - Low Level API RMA, Atomic, Tag-matching, Send/Recv, Active Message Transport for InﬁniBand VERBs driver RC UD XRC DCT Transport for intra-node host memory communication SYSV POSIX KNEM CMA XPMEM Transport for Accelerator Memory communucation GPU Transport for Gemini/Aries drivers GNI UC-S (Services) Common utilities UC-P (Protocols) - High Level API Transport selection, cross-transrport multi-rail, fragmentation, operations not supported by hardware Message Passing API Domain: tag matching, randevouze PGAS API Domain: RMAs, Atomics Task Based API Domain: Active Messages I/O API Domain: Stream Utilities Data stractures Hardware MPICH, Open-MPI, etc. OpenSHMEM, UPC, CAF, X10, Chapel, etc. Parsec, OCR, Legions, etc. Burst buffer, ADIOS, etc. Applications UCX Memory Management OFA Verbs Driver Cray Driver OS Kernel Cuda

7. © 2017 Arm Limited7 UCX Features • Support for InfiniBand and RoCE: RC, UD, DC • InfiniBand: Hardware tag-matching, extended AMOs, etc. • Support for GPU Devices/Memory – AMD ROCM, Nvidia CUDA • Support for TCP (Beta) • Java bindings (Beta) • Support for Accelerated Verbs – 40% speedup on Arm compared to vanilla Verbs • Support for UGNI API for Aries and Gemini (Thanks to ORNL and LANL!) • Support for Shared Memory: KNEM, CMA, XPMEM, Posix, SySV • Support for x86, ARMv8, Power • Efficient memory polling – 36% increase in efficiency on Arm • UCX interface is integrated with MPICH, OpenMPI, OSHMEM, ORNL-SHMEM, OSSS-SHMEM, etc.

9. © 2017 Arm Limited9 Message Passing Interface - MPI De-facto standard developed by HPC community - https://www.mpi-forum.org/ Excellent overview of MPI by Jeff Squyres - https://www.slideshare.net/jsquyres/the- message-passing-interface-mpi-in-laymans-terms Node 1 Node 2

11. © 2017 Arm Limited11 HPE Comanche (Apollo 70) with Cavium Thunder X2 SINGLE core, Mellanox ConnextX-4 100Gb/s (EDR) - Bandwidth 0 2000 4000 6000 8000 10000 12000 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072 262144 524288 1048576 2097152 4194304 MB/s Message Size MPI Bandwidth Higherisbetter

12. © 2017 Arm Limited12 0 1 2 3 4 5 6 7 8 9 0 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 Micro-Second Message Size MPI Latency HPE Comanche (Apollo 70) with Cavium Thunder X2, Mellanox ConnectX-4 100Gb/s (EDR) – Latency/Ping Pong Lowerisbetter

13. © 2017 Arm Limited13 HPE Comanche (Apollo 70) with Cavium Thunder X2, Mellanox ConnectX-4 100Gb/s (EDR) – MPI Message Rate (28 cores) 0 10 20 30 40 50 60 70 80 90 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072 262144 524288 1048576 2097152 4194304 MessagePerSecond Millions Message Size MPI Message Rate Higherisbetter

15. © 2017 Arm Limited15 MPI is HUGE! • MPI Language bindings and compiler wrappers – C, Fortran • MPI runtimes – PMI, PMI-2, ORTE, PMIX, etc. • MPI Network layer – supports every possible and impossible exotic network device Open MPI MVAPICH Over 1,200,000 code lines !

16. © 2017 Arm Limited16 Let’s talk about testing … Thanks to Howard Pritchard (LANL) and Hari Subramoni (MVAPICH) for providing this information Open MPI - 150,000 (aggregated) nightly tests ! MVAPICH – 18,000 Core hours (750 days) per- commit

17. © 2017 Arm Limited17 Open MPI testing CPU arch: CPU types: x86_64 , ARMv8, PPC64le Compilers: clang37, clang38, gcc (multiple versions, 32/64bit), ibm xlc , absoft 18.0, armclang (18.3) Distros: AWS linux 17.03, CentOS, CrayCLE, etc Thanks to Howard Pritchard (LANL) for providing this information Arch x Compiler x OS x Interconnect x MPI configurations

18. © 2017 Arm Limited18 How YOU can help ? • Providing hardware and software • Running CI for MPI projects (Jenkins, etc.) • Running distributed/scale regression (MTT, etc) • Been active member of the community (mail lists, github, etc) • https://www.open-mpi.org • https://www.mpich.org • http://mvapich.cse.ohio-state.edu • Answer user questions, fix bugs, etc • Contribute new features and optimizations ! • Participate in MPI forum - https://www.mpi-forum.org

20. 2020 © 2017 Arm Limited The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. www.arm.com/company/policies/trademarks

HPC network stack on ARM - Linaro HPC Workshop 2018

Du liest eine Vorschau.

Hier ansehen

HPC network stack on ARM - Linaro HPC Workshop 2018

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie HPC network stack on ARM - Linaro HPC Workshop 2018 (20)

Weitere von Linaro (19)

Aktuellste (20)