AI Bridging Cloud Infrastructure as World's First Large-scale Open AI Infrastructure
•
3 gefällt mir•1,839 views
AI Bridging Cloud Infrastructure (ABCI) is a large-scale open AI infrastructure in Japan operated by the University of Tokyo. It provides: 1) Over 0.55 exaflops of computing power with 1088 nodes equipped with 4352 NVIDIA GPUs and 43520 CPU cores for AI and data science research. 2) Dense rack design optimized for thermal management with ambient warm water cooling to achieve high density computing. 3) Hierarchical storage including 1.6PB of local NVMe SSDs, 22PB of parallel file storage, and object storage for burst buffers and campaign storage. 4) Open access platform to accelerate joint academic-industry R&D for AI through distributed
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie AI Bridging Cloud Infrastructure as World's First Large-scale Open AI Infrastructure
Ähnlich wie AI Bridging Cloud Infrastructure as World's First Large-scale Open AI Infrastructure (20)
Mehr von Hitoshi Sato
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
AI Bridging Cloud Infrastructure as World's First Large-scale Open AI Infrastructure
- 1. / C 2 7 1 : 1 8 0
- 2. 2 . , , AI
- 3. 3 O c x Z / 6 o Z t k N x v Z N G E N G E / v Z dfl /KIJH N )6 tk e fae O /BEKM 6 E BEB kn O 6 R O fn Oe 8 R X g OiO d N/BEKMb lp JA HE J kn O O rOo fn Oe M v Z dfl + )6 G K v O O 6G HC + FFG O c x FIJ H I / Z /DB AFKJ v Z H GA8 BH GA L N 6 D R 6 / e +BL B DFK F6 / + I II E H FE F bO Od e FF G H /BEKM 6 oP Oi Q fub P+Xx aOd dfl hvn X sOo X 6 )FHJH EN N R • IW b • P e • IW b G C U bG / • P G b G C b H A b G / • IW U P • P U • IW • P H • IW P G • P P G U hvn X bdfl W U o fub U S
- 4. AI Bridging Cloud Infrastructure as World’s First Large-scale Open AI Infrastructure • Open, Public, and Dedicated infrastructure for AI/Big Data • Platform to accelerate joint academic-industry R&D for AI in Japan • Top-level compute capability w/ 0.550EFlops(AI), 37.2 PFlops(DP) ( )2 5 0.1 ,. 10 5 ## 0 # 4 Univ. Tokyo Kashiwa II Campus Operation Scheduled in 2018
- 5. • 1088x compute nodes w/ 4352x NVIDIA Tesla V100 GPUs, 43520 CPU Cores, 476TiB of Memory, 1.6PB of NVMe SSDs, 22PB of HDD-based Storage and Infiniband EDR network • Ultra-dense IDC design from the ground-up w/ 20x thermal density of standard IDC • Extreme Green w/ ambient warm liquid cooling, high-efficiency power supplies, etc., commoditizing supercomputer cooling technologies to clouds ( 2.3MW, 70kW/rack) 5 Gateway and Firewall Computing Nodes: 0.550 EFlops(HP), 37 PFlops(DP) 476 TiB Mem, 1.6 PB NVMe SSD Storage: 22 PB GPFS High Performance Computing Nodes (w/GPU) x1088 • Intel Xeon Gold6148 (2.4GHz/20cores) x2 • NVIDIA Tesla V100 (SXM2) x 4 • 384GiB Memory, 1.6TB NVMe SSD Multi-platform Nodes (w/o GPU) x10 • Intel Xeon Gold6132 (2.6GHz/14cores) x2 • 768GiB Memory, 3.8TB NVMe SSD Interactive Nodes DDN SFA14K (w/ SS8462 Enclosure x 10) x 3 • 12TB 7.2Krpm NL-SAS HDD x 2400 • 3.84TB SAS SSD x 216 • NSD Servers x 12 Object Storage for Protocol Nodes 100GbE Service Network (10GbE) External Networks SINET5 Interconnect (Infiniband EDR)
- 6. ABCI: AI Bridging Cloud Infrastructure 6 System (32 Racks)Rack (17 Chassis) Compute Node (4GPUs, 2CPUs) Chips (GPU, CPU) Node Chassis (2 Compute Nodes) NVIDIA Tesla V100 (16GB SMX2) 3.72 TB/s MEM BW 384 GiB MEM 200 Gbps NW BW 1.6TB NVMe SSD 1.16 PFlops(DP) 17.2 PFlops (AI) 37.2 PFlops(DP) 0.550 EFlops (AI) 68.5 PFlops(DP) 1.01 PFlops (AI) 34.2 TFlops(DP) 506 TFlops (AI) GPU: 7.8 TFlops(DP) 125 TFlops (AI) CPU: 1.53 TFlops(DP) 3.07 TFlops (AI) Intel Xeon Gold 6148 (27.5M Cache, 2.40 GHz, 20 Core) 0.550 EFlops(AI), 37.2 PFlops(DP) 19.88 PFlops(Peak), Ranked #5 Top500 June 2018 131TB/s MEM BW Full Bisection BW within Rack 70kW Max 1088 Compute Nodes 4352 GPUs 4.19 PB/s MEM BW 1/3 of Oversubscription BW 2.3MW
- 7. GPU Compute Nodes • NVIDIA TESLA V100 (16GB, SXM2) x 4 • Intel Xeon Gold 6148 x 2 Sockets – 20 cores per Socket • 384GiB of DDR4 Memory • 1.6TB NVMe SSD x 1 – Intel DC P4600 u.2 • EDR Infiniband HCA x 2 – Connected to other Compute Notes and Filesystems 7 Xeon Gold 6148 Xeon Gold 6148 10.4GT/s x3DDR4-2666 32GB x 6 DDR4-2666 32GB x 6 128GB/s 128GB/s IB HCA (100Gbps)IB HCA (100Gbps) NVMe UPI x3 x48 switch x64 switch Tesla V100 SXM2 Tesla V100 SXM2 Tesla V100 SXM2 Tesla V100 SXM2 PCIe gen3 x16 PCIe gen3 x16 PCIe gen3 x16 PCIe gen3 x16 NVLink2 x2
- 8. Rack as Dense-packaged “Pod” ( AB < ) 1 6 ) 0 BC / 0 BC ,3 0G < F BA -7 BH<D G D CF BA -7 FB <IF<DA CB ) 7 7 4 I 7 F<D , D 2 D .BB A 8Pod #1 LEAF#1 (SB7890) LEAF#2 (SB7890) LEAF#3 (SB7890) LEAF#4 (SB7890) SPINE#1 (CS7500) SPINE#2 (CS7500) CX40 0#1 CX2570#1 CX2570#2 CX40 0#2 CX2570#3 CX2570#4 CX40 0#3 CX2570#5 CX2570#6 CX40 0#17 CX2570#33 CX2570#34 FBB#1 (SB7890) FBB#2 (SB7890) FBB#3 (SB7890) 1/3 Oversubscription BW IB-EDR x 24 Full bisection BW IB-EDR x 72 InfiniBand EDR x1 InfiniBand EDR x6 InfiniBand EDR x4 x 32 pods
- 9. Hierarchical Storage Tiers • Local Storage – 1.6 TB NVMe SSD (Intel DC P4600 u.2) per Node – Local Storage Aggregation w/ BeeOnd • Parallel Filesystem – 22PB of GPFS • DDN SFA14K ( w/ SS8462 Enclosure x 10) x 3 set • Bare Metal NSD servers and Flash-based Metadata Volumes for metadata operation acceleration – Home and Shared Use • Object Storage – Part of GPFS using OpenStack Swift – S3-like API Access, Global Shared Use – Additional Secure Volumes w/ Encryption (Planned) 9 Parallel Filesystem Local Storage as Burst Buffers Object Storage as Campaign Storage
- 10. Performance Reference for Distributed Deep Learning 10 Better• Environments – ABCI 64 nodes (256 GPUs) – Framework: ChainerMN v1.3.0 • Chainer 4.2.0, Cupy 4.2.3, mpi4py 3.0.0, Python 3.6.5 – Baremetal • CentOS 7.4, gcc-4.8.5, CUDA 9.2, CuDNN 7.1.4, NCCL2.2, OpenMPI 2,1.3 • Settings – Dataset: Imagenet-1K – Model: ResNet-50 – Training: • Batch size: 32 per GPU, 32 x 256 in total • Learning Rate: Starting 0.1 and x0.1 at 30, 60, 80 epoch w/ warm up scheduling • Optimization: Momentum SGD (momentum=0.9) • Weight Decay: 0.0001 • Training Epoch: 100
- 11. LGD M P C N D M A D U N , L A D I M C 11 /home (GPFS) Job Job Job Job NQS Submit Scheduling script file $ qsub <option> script_filename inter-connect SSH ( ) G (High Throughput Computing)
- 12. /: / D 0 1 72 A : /: / D / 2 D 172 2 .2: , 2 ,C : ,/17/ 2 2 1 inkbc augk t v y 12 , Hgmeu P • 172 fp k Hh rs wogk L Q • lr t augk t T
- 13. . , 13 (cont’d) CUDA8.0 8.0.44 8.0.61.2 CUDA9.0 9.0.176 CUDA9.1 9.1.85 9.1.85.1 9.1.85.3 CuDNN5.1 5.1.5 5.1.10 CuDNN6.0 6.0.21 CuDNN7.0 7.0.5 CuDNN7.1 7.1.1 7.1.2 7.1.3 CUDA9.2 9.2.88.1 NCCL1.3 1.3.4 1.3.40-1 NCCL2.1NCCL2.0 NCCL2.2 2.0.5-3 2.1.4-1 2.1.15-1 2.2.12 OpenMPI MVAPICH2-GDR2.1.3 3.0.1 3.1.0 2.3a Python 2.7 3.5 3.6 Python Modules mpi4py matplotlibCython Pillow Jupyter Caffe2 CNTK ChainerMN Tensorflow MXNetNnabla
- 14. Software Stuck for ABCI • Batch Job Scheduler – High throughput computing • Minimum Pre-installed Software – Users can deploy their environments using anaconda, pip, python venv, etc. – Reduce operational cost • Container Support – Singularity for multi-node jobs w/ user customized images – Docker for single-node jobs w/ site-certified images 14 User Applications DL Frameworks Python, Ruby, R, Java, Scala, Perl, Lua, etc. ISV AppsOSSHadoop/ Spark GCC PGI OpenMPI MVAPICH2 Intel Parallel Studio XE Cluster Edition CUDA/CUDNN/NCCL GPFS BeeOND OpenStack Swift Univa Grid Engine Singularity Docker CentOS/RHEL
- 15. ABCI : Dynamic Container Deployment with HPC Linux Containers Linux Container (Singulairty, Docker) GPFS/Object Storage Compute Node Compute Node Compute Node Compute Node Container Image Container Image Container Image Container Image Job Job Job Job Job Scheduler Container Image Register/copy container images Import/copy container images Submit jobs with container images Container image repository (Dockerhub, private registry)
- 16. P D . , H O I CDHR D HAU C M P 16 CharlieCloud HPCEnterprise
- 17. ( ( D u Mvtr B pae C . : . /: CA C m p D l Mgo mi ws cn PS :: M b kpI M :: HL hpd ws D :/ I y :/ . x D fcU S I y , . / 17
- 18. ( ) sudo singularity build –sandbox tmpdir/ Singularity sudo singularity build –writable container.img Singularity sudo singularity build container.img Singularity sudo singularity build container.img docker://ubuntu sudo singularity build container.img shub://ubuntu ) S R sudo singularity shell –writable container.img D H R (, R ( ( , , , , container.img h S g singularity run container.img singularity exec container.img … singularity shell container.img a Sc ) ( e e
- 19. : /. # $ / . $ -$ -$ . $ S 19
- 20. 4 , C C GHI 4 4 ,4 M : c s a mn_e a e e s P B . . , kt M : c sP B Md v lN g I Bc s i ro . i NI up 4. 4 . - . , , . , . - 4. 4 - M : lN g l N 4. 4 g 20
- 21. - : - D 0- 8 D H 0- 6/ .8 / 8/ / - / . 0- / / 8 $ 8 - - 6/ .8 / -:- / - / . / / 6 / 21 GPU --nv
- 22. C B 3A3 : 3 0 : M C $ : 3 an f uV - a gi pN C $ c v - fo : 3 r P H ts I y m i NN _ O 3 - B leo . - m i x 22
- 23. S y x R 40 HIA ) 29 R m in - 03 H K67 O S 0D H K 0N G 9 MDIH ) R oad S HCN K M O S NHMN ) C ( C R g l S 0 HM8 C ( C S yup R b c- 4G C H M 5 R bl- : 7 M ( R - S e W - 29 U P ) S s - I D U S t r- 6IG HMNG 21 GIG HMNG. , S = CDM 1 - S h v- 23 : Better:
- 24. L i C 2 2 , 2 e D N GLbdP G GU M iA C 2 2 , 2 e 2 - ak M i P GU G L , , o n I G L c P 24 Base Drivers, Libraries on Host CUDA Drivers Infiniband Drivers Filesystem Libraries (GPFS, Lustre) Userland Libraries on Container CUDA CuDNN NCCL2 MPI (mpi4py) Mount ibverbs Distributed Deep Learning Frameworks Caffe2 ChainerMNDistributed TensorflowMXNet
- 25. 25
- 26. / . ) / m S C O = y t = i sn f y Sun v S C , =. / St i S C . / • C I /. , ) , ) , / ) O >– iU > ir ga lu bc S S ke o a 26
- 27. A P oce ʼ A Oʼ A L ʼ t uO P V P d A P S r ʼ oceA – uO V V R I • O x O o n M A () P ao oceO P A – i dP• oce O od L i dO () o d n oce O od 27
- 28. ei S Ll B sI B BCA I g e / R / ro Ra ei A L n ut 28
- 29. 29