NSCC Training Introductory Class

1. NSCC High Performance Computing Cluster Introduction [1-July-2016]

2. • Introduction to NSCC • About HPC • More about NSCC HPC cluster • PBS Pro (Scheduler) • Compilers and Libraries • Developer Tools • Co-processor / Accelerators • Environment Modules • Applications • User registration procedures • Feedback 2 The Discussion

3. 3 Introduction to NSCC

4. • State-of-the-art national facility with computing, data and resources to enable users to solve science and technological problems, and stimulate industry to use computing for problem solving, testing designs and advancing technologies. • Facility will be linked by high bandwidth networks to connect these resources and provide high speed access to users anywhere and everyone. Introduction: The National Supercomputing Centre (NSCC) 4

5. Introduction: Vision & Objectives Vision: “Democratising Access to Supercomputing” 5 Making Petascale Supercomputing accessible to the ordinary researcher1 Bringing Petascale Computing and Storage and Gigabit speed networking to the ordinary person2 Supporting National R&D Initiatives1 Objectives of NSCC Attracting Industrial Research Collaborations2 Enhancing Singapore’s Research Capabilities3

6. 6 What is HPC?

7. 7 What is HPC? • Term HPC stands for High Performance Computing or High Performance Computer • Tightly coupled personal computers with high speed interconnect • Measured in FLOPS (FLoating point Operations Per Second) • Architectures – NUMA (Non-uniform memory access)

8. Major Domains where HPC is used Engineering Analysis • Fluid Dynamics • Materials Simulation • Crash simulations • Finite Element Analysis Scientific Analysis • Molecular modelling • Computational Chemistry • High energy physics • Quantum Chemistry Life Sciences • Genomic Sequencing and Analysis • Protein folding • Drug design • Metabolic modelling Seismic analysis • Reservoir Simulations and modelling • Seismic data processing 8

9. Major Domains where HPC is used Chip design & Semiconductor • Transistor simulation • Logic Simulation • Electromagnetic field solver Computational Mathematics • Monte-Carlo methods • Time stepping and parallel time algorithms • Iterative methods Media and Animation • VFX and visualization • Animation Weather research • Atmospheric modelling • Seasonal time- scale research • - Major Domains where HPC is used 9

10. Major Domains where HPC is used • And More – Bigdata – Information Technology – Cyber security – Banking and Finance – Data mining 10

11. 11 Introduction to NSCC HPC Cluster

12. Executive Summary • 1 Petaflop System – About 1300 nodes – Homogeneous and Heterogeneous architectures • 13 Petabytes of Storage – One of the Largest and state of the art Storage architecture • Research and Industry – A*STAR, NUS, NTU, SUTD – And many more commercial and academic organizations 12

13. HPC Stack in NSCC Mellanox 100 Gbps Network Intel Parallel studio Allinea Tools PBSPro Scheduler Lustre & GPFS HPC Application software Operating System RHEL 6.6 and CentOS 6.6 Fujitsu x86 Servers NVidia Tesla K40 GPUDDN Storage Application Modules 13

14. 14 NSCC Supercomputer Architecture Base Compute Nodes (1160 nodes) Accelerated Nodes (128 nodes) Parallel File system / Tiered storage InfiniBand network - Fully non- blocking Ethernet NW GIS FAT node NUS Peripheral Servers NTU Peripheral Servers NSCC Peripheral Servers NSCC Direct users VPN

15. Login architecture 15 Login cluster 80Gb/s Link NSCC cluster

16. 16 Customized Solution

17. 17 Genomic Institute of Singapore (GIS) National Supercomputing Center (NSCC) 2km Connection between GIS and NSCC Large memory node (1TB), Ultra high speed 500Gbps enabled 2012: 300 Gbytes/week 2015: 4300 Gbytes/week x 14

18. NGSP Sequencers at B2 (Illumina + PacBio) NSCC Gateway STEP 2: Automated pipeline analysis once sequencing completes. Processed data resides in NSCC 500Gbps Primary Link Data Manager STEP 3: Data manager index and annotates processed data. Replicate metadata to GIS. Allowing data to be search and retrieved from GIS Data ManagerCompute Tiered Storage POLARIS, Genotyping & other Platforms in L4~L8 Tiered Storage STEP 1: Sequencers stream directly to NSCC Storage (NO footprint in GIS) Compute 1 Gbps per sequencer 10 Gbps 1 Gbps per machine 100 Gbps 10 Gbps A*CRC-NSCC GIS A*CRC: A*Star Computational Resource Center GIS: Genome Institute of Singapore Direct streaming of Sequence Data from GIS to remote Supercomputer in NSCC 2km

19. The Hardware EDR Interconnect • Mellanox EDR Fat Tree within cluster • InfiniBand connection to all end-points (login nodes) at three campuses • 40/80/500 Gbps throughput network extend to three campuses (NUS/NTU/GIS) Over13PB Storage • HSM Tiered, 3 Tiers • I/O 500 GBps flash burst buffer , 10x Infinite Memory Engine (IME) ~1 PFlops System • 1,288 nodes (dual socket, 12 cores/CPU E5-2690v3) • 128 GB DDR4 / node • 10 Large memory nodes (1x6TB, 4x2TB, 5x 1TB) 19

20. Compute nodes 20 • Large Memory Nodes – 9 Nodes configured with high memory – FUJITSU Server PRIMERGY RX4770 M2 – Intel(R) Xeon(R) CPU E7-4830 v3 @ 2.10GHz – 4 x 1 TB, 4x 2 TB, and 1x 6 TB Memory configuration – EDR Infiniband • Standard Compute nodes – 1160 nodes – Fujitsu Server PRIMERGY CX2550 M1 – 27840 CPU Cores – Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz – 128 GB / Server – EDR InfiniBand – Liquid cooling system

21. Accelerate your computing Accelerators nodes • 128 nodes with NVIDIA GPUs (identical to the compute nodes) • NVIDIA K40 (2880 cores) • 368,640 total GPU cores Visualization nodes • 2 nodes Fujitsu Celsius R940 graphic workstations • Each with 2 x NVIDIA Quadro K4200 • NVIDIA Quadro Sync support 21

22. NSCC Data Centre – Green features Warm water cooling for CPUs – First free-cooling system in Singapore and South-East Asia. – Water is maintained at a temperature of 40ºC. Enters the racks at 40ºC, exits the racks at 45ºC. – Equipment placed in a technical floor(18th) cool down the water down only using fans. – The system can easily be extended for future expansion. Green features of Data Centre – PUE of 1.4 (average for Singapore is above 2.5) 22 Cool-Central® Liquid Cooling technology

23. Parallel file system • Components – Burst Buffer • 265 TB Burst Buffer • 500 GB/s throughput • Infinite Memory Engine (IME) – Scratch • 4 PB scratch storage • 210 GB/s • SFA12KX EXAScalar storage • Lustre file system – home and secure • 4 PB Persistent storage • GridScalar storage • 100 GB/s throughput • IBM Spectrum Scale (formerly GPFS) – Archive storage • 5 PB storage • Archive purpose only • WOS based archive system 23

24. IME Architecture 24

25. Tiered File system 25

26. NSCC Storage 26 Tier0 BurstBuffer Tier0 ScratchFS Tier1 HomeFS Tier1 ProjectFS Tier2 Archive 265 TB 500 GB/s 4 PB 210 GB/s 4 PB 100 GB/s WOS Active Archive Infinite Memory Engine GRIDScaler GPFS® Storage HSM 5PB 20TB/h EXAScaler Lustre® Storage

27. Software Stack Operating System CentOS 6.6 Scheduler PBS Pro Compilers GCC Intel Parallel Studio Libraries GNU, Intel MKL Allinea tools GPGPU CUDA Toolkit 7.5 Environment Modules 27

28. PBS Professional (Job Scheduler) 28

29. Why PBS Professional (Scheduler)? 29  Workload management solution that maximizes the efficiency and utilization of high-performance computing (HPC) resources and improves job turnaround Robust Workload Management  Floating licenses  Scalability, with flexible queues  Job arrays  User and administrator interface  Job suspend/resume  Application checkpoint/restart  Automatic file staging  Accounting logs  Access control lists Advanced Scheduling Algorithms  Resource-based scheduling  Preemptive scheduling  Optimized node sorting  Enhanced job placement  Advance & standing reservations  Cycle harvesting across workstations  Scheduling across multiple complexes  Network topology scheduling  Manages both batch and interactive work  BackfillingReliability, Availability and Scalability  Server failover feature  Automatic job recovery  System monitoring  Integration with MPI solutions  Tested to manage 1,000,000+ jobs per day  Tested to accept 30,000 Jobs per minute  EAL3+ security  Checkpoint support

30. Process Flow of a PBS Job 1. User submits job 2. PBS server returns a job ID 3. PBS scheduler requests a list of resources from the server * 4. PBS scheduler sorts all the resources and jobs * 5. PBS scheduler informs PBS server which host(s) that job can run on * 6. PBS server pushes job script to execution host(s) 7. PBS MoM executes job script 8. PBS MoM periodically reports resource usage back to PBS server * 9. When job is completed PBS MoM copies output and error files 10. Job execution completed/user notification sent HOST A HOST B HOST C PBS SCHEDULER PBS SERVER pbsworks ncpus mem host pbsworks on HOST A pbsworks Note: * This information is for debugging purposes only. It may change in future releases. 30 Cluster Network

31. Compute Manager GUI: Job Submission Page • Applications panel – Displays the applications available on the registered PAS server • Submission Form panel – Displays a job submission form for the application selecting the Applications panel • Directory Structure panel – Displays the directory structure of the location specified in the Address box – Files panel – Displays the contents of the directory, files, and subdirectories selected in the Directory Structure panel 31 Directory Structure Files Applications

32. Job Queues & Scheduling Policies 32 External Queue Name Internal Queue Name Walltime limit Other limits Remarks largemem 24 Hours To be decided For Jobs requiring more that 4GB per core. normal dev 1 Hours 2 standard nodes per user High priority queue for testing and development works. small 24 Hours Up to 24 cores per job For jobs that do not require more than one node. medium 24 Hours Up to limit as per prevailing policies For standard job runs requiring more than one node. long 120 Hours 1 node per user Low priority queue for jobs that cannot be checkpointed. gpu gpunormal 24 Hours Up to limit as per prevailing policies For “normal” jobs which require GPU. gpulong 240 Hours Up to limit as per prevailing policies Low priority queue for GPU jobs which cannot be checkpointed.

33. Job Queues & Scheduling Policies 33 External Queue Name Internal Queue Name Walltime limit Other limits Remarks iworkq 8 Hours 1 node per user For visualisation. ime (look for it in the near future) 24 Hours Up to limit as per prevailing policies For users who wish to experiment with DDN's IME burst buffer which offers up to 500GB/s of transfer speed * Users only need to specify the 'External Queue' for job submission. Jobs will be routed to the internal queue depending on the job resource requirements.

34. Compilers & Libraries 34

35. 35 Compilers and Libraries at a glance

36. Parallel programming OpenMP • Available compilers (gcc/gfortran/icc/ifort) – OpenMP (not openmpi, Used mainly in SMP programming) • OpenMP (Open Multi-Processing) • OpenMP is an approach and OpenMPI is an implementation of MPI • An API for shared-memory parallel programming in C/C++ and Fortran • Parallelization in OpenMP achieved through threads • Programming OpenMP is easier as it involves only pragma directive • OpenMP program cannot communicate to the processor over network • Different stages of the program uses different number of threads • A typical approach is demonstrated through the below image 36

37. Parallel Programming MPI • MPI – MPI stands for Messaging Passing Interface – MPI is a library specification – MPI implementation is typically a wrapper to standard compilers such as C/Fortran/Java/Python – Typically used in Distributed memory communication 37

38. 38 Developer Tools

39. 39 Allinea DDT • DDT – Distributed Debugging tool from Allinea • Graphical interface for debugging – Serial applications/codes – OpenMP applications/codes – MPI applications/codes – CUDA applications/codes • You control the pace of the code execution and examine execution flow and variables • Typical Scenario – Set a point in your code where you want execution to stop – Let your code run until the point is reached – Check the variables of concern

40. 40 Allinea MAP • MAP – Application Profiling tool from Allinea • Graphical interface for profilling – Serial applications/codes – OpenMP applications/codes – MPI applications/codes

41. 41 Allinea MAP • Running your code with MAP – $ module load impi/5.1.2 – $ mpiicc -g -O0 -o wave_c wave_c.c – $ module load map/a.b.c – $ map mpiexec –n 4 ./wave_c 20

42. 42 Allinea MAP

43. 43 Co-processor / Accelerators

44. GPU • GPUs – Graphic Processing Units were initially made to render better graphics performance • With the amount of research put on GPUs, it was identified that GPUs can perform better with Floating Point Operations as well • The term GPU changed to GPGPUs (General Purpose GPUs) • CUDA Toolkit includes compiler, math libraries, tools, and debuggers 44

45. GPU in NSCC • GPU Configuration – Total 128 GPU nodes – Each server with 1 Tesla K40 GPU – 128 GB host memory per server – 12GB device memory – 2880 CUDA Cores • Connect to GPU server – To compile GPU application: • Submit interactive job requesting for GPU resource • Compile job using NVCC compiler – To submit GPU job • Flexible to among qsub for login nodes • OR login to compute manager 45

46. 46 Environment Modules

47. What is Environment modules • Environment modules helps to dynamically load/unload environment variables such as PATH, LD_LIBRARY_PATH, etc., • Environment modules are based on module files which are written in TCL language • Environment modules are shell independent • Helpful to maintain different version of same software • Flexibility to create module files by the users 47

48. Applications 48

49. Molecular Dynamics Computational Chemistry Compatible Applications 49

50. Compatible Applications Engineering Applications Quasiparticle calculationQuantum Chemistry Numerical Analysis Weather research 50

51. August 27, 2015 51 https://help.nscc.sg/software-list/

52. Managed Services offered 53 • Computational resources • Storage management Infrastructure Services • Hardware break fix • Software incident resolution Incident Resolution • Data management • Job management • Software installation etc., General Service Requests • Code Optimization • Special queue configuration, etc. Specialized Service Requests • Introductory class • Code optimization techniques • Parallel Profiling etc. Training Services • Portal/e-Mail/Phone • Request for a service via portal • Interactive Job submission portal Helpdesk

53. Where is NSCC • NSCC Petascale supercomputer in Connexis building • 40Gbps links extended to NUS, NTU and GIS • Login nodes are placed in NUS, NTU and GIS datacenters • Access to NSCC is just like your local HPC system 54 1 Fusionopolis Way, Level-17 Connexis South Tower, Singapore 138632

54. Supported Login methods • How do I login – SSH From a Windows PC use Putty or any standard SSH client software hostname is nscclogin.nus.edu.sg, use NSCC Credentials From Linux machine, use ssh username@nus.nscc.sg From MAC, open terminal and ssh username@nus.nscc.sg – File Transfer SCP or any other secure shell file transfer software from Windows Use the command scp to transfer files from MAC/Linux – Compute Manager / Display Manager Open any standard web browser In the address bar, type https://nusweb.nscc.sg Use NSCC credentials to login – Outside campus Connect to Campus VPN to gain above mentioned services 55

55. NSCC HPC Support (Proposed to be available by 15th Mar) • Corporate Info – web portal http://nscc.sg • NSCC HPC web portal http://help.nscc.sg • NSCC support email help@nscc.sg • NSCC Workshop portal http://workshop.nscc.sg 56

56. 57 Help us improve. Take the online survey! Visit: http://workshop.nscc.sg >> Survey

57. Help portal 58 FAQs of NSCC Enroll to NSCC https://help.nscc.sg/

58. Registration Procedures 59

59. Registration Procedure 60

60. Web Site : http://nscc.sg Helpdesk : https://help.nscc.sg Email : help@nscc.sg Phone : +65 6645 3412 61

62. User Enrollment Instructions: • Open https://help.nscc.sg • Navigate User services -> Enrollment • Click on Login • Select your organization (NUS/NTU/A*Star) from the drop down • Input your credentials Ref: https://help.nscc.sg -> User Guides -> User Enrollment guide 63

63. Login to NSCC Login nodes • Download Putty form internet • Open Putty • Type login server name (login.nscc.sg) • Input your credentials to login 64

64. Compute manager • Open Web Browser (Firefox or IE) • Type https://nusweb.nscc.sg / https://ntuweb.nscc.sg / https://loginweb-astar.nscc.sg • Use your credentials to login • Submit a sample job 65

65. Transfer files • Use FileZilla to transfer files 66

66. Creating PBS Job submission script • Use the below sample script cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS -l place=scatter cd ${PBS_O_WORKDIR} sleep 30 qsub submit.pbs 67

67. Environment module • Open Putty • Type module avail • Type module load 68

68. Compiling simple C Program • Use putty to login • Create helloworld.c #include<stdio.h> void main() { printf("Helloworldn"); } • Use module load composerxe/2016.1.150 • Type icc heloworld.c -o helloworld.o 69

69. Submit job cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=1 cd ${PBS_O_WORKDIR} ./helloworld.o 70

70. Compiling mpi C Program • Use putty to login • Create helloworld.c #include <mpi.h> #include <stdio.h> #include <string.h> #include <mpi.h> #include <stdio.h> #include <unistd.h> int main(int argc, char **argv) { int rank; char hostname[256]; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); gethostname(hostname,255); printf("Hello world! I am process number: %d on host %sn", rank, hostname); MPI_Finalize(); return 0; } • Use module load composerxe/2016.1.150 • Type icc heloworld.c -o mpihello.o 71

71. Submit job cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS –l place=scatter cd ${PBS_O_WORKDIR} mpirun ./mpihello.o 72

72. Submit pre-compiled applicatin 73 cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS –l place=scatter cd ${PBS_O_WORKDIR} mpirun ./mpihello.o

73. Using Scratch space #!/bin/bash #PBS -N My_Job # Name of the job #PBS -l select=1:ncpus=24:mpiprocs=24 # Setting number of nodes and CPUs to use #PBS -W sandbox=private # Get PBS to enter private sandbox #PBS -W stagein=file_io@wlm01:/home/adm/sup/fsg1/<my input directory> # Directory name where all the input files are alvailable # files in the input directory will be copied to scratch space creating a directory file_io #PBS -W stageout=*@wlm01:/home/adm/sup/fsg1/<myoutput directory> # Output directory path in my home directory # Once the job is finished, the files from file_io in scratch will be copied back to <myoutput directory> #PBS -q normal cd ${PBS_O_WORKDIR} echo " PBS_WORK_DIR is : $PBS_O_WORKDIR" echo "PBS JOB DIR is: $PBS_JOBDIR" #Notice that the output of pwd will be in lustre scratch space echo "PWD is : `pwd`" sleep 30 #mpirun ./a.out < input_file > output_file 74

NSCC Training Introductory Class

NSCC Training Introductory Class

Recomendados

Más contenido relacionado

Was ist angesagt?

Was ist angesagt?(18)

Destacado

Destacado(20)

Similar a NSCC Training Introductory Class

Similar a NSCC Training Introductory Class (20)

Último

Último(20)

NSCC Training Introductory Class

Hinweis der Redaktion