Elastic Fabric Adapter (EFA) is a network interface for Amazon EC2 instances that enables customers to run HPC applications requiring high levels of inter-instance communications, like computational fluid dynamics, weather modeling, and reservoir simulation, at scale on AWS. It uses a custom-built operating system bypass technique to enhance the performance of inter-instance communications, which is critical to scaling HPC applications. With EFA, HPC applications using popular HPC technologies like Message Passing Interface (MPI) can scale to thousands of CPU cores. Get a deep dive on EFA and learn how to use EFA to enhance application performance for your HPC workloads.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling HPC applications in EC2
with Elastic Fabric Adapter
Brian Barrett
Principal Engineer
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
HPC applications in AWS
What is EFA?
Getting started with EFA
EFA tech deep-dive

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Wednesday, Nov 28
High Performance Computing on AWS
3:15 – 4:15 | Aria East, Piazza Level, Orovada 2
Wednesday, Nov 28
Running High Performance Computing Workloads in the Cloud
4:00 – 5:00 | Aria West, Level 3, Starvine 3, Table 8
Wednesday, Nov 28
Deploying a Burstable and Event-Driven HPC Cluster on AWS
1:00 – 2:00 | Aria West, Level 3, Starvine 10, Table 6

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Weather simulation

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
0
20
40
60
80
100
120
140
160
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
0 50 100 150 200 250 300 350
ScaleUp
Time(s)
Cores
c4.8xlarge Time c4.8xlarge Scaleup
Structural simulation

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Trek applications for engineering:
• Computational Fluid Dynamics
• Star-CCM+ and HEEDS software
Cloud for product design and engineering
Simulations for bicycle design:
• Execute multiple simulations
in parallel
• Fully explore the design
space to make informed
decisions about drafting
techniques related to
competitive bicycling

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fluid dynamics – Ansys Fluent
C4.8xlarge instance type
140M cell model
F1 car CFD benchmark

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC in aerospace
Boom leverages Rescale and AWS to
enable supersonic travel
• Simulated vortex lift with 200M cell
models on 512+ cores
• Increased simulation throughput:
100 jobs in parallel with 6x speedup
per job → 600x speedup
• Elastic HPC capacity and pay-as-
you-go AWS clusters allow business
agility & ability to scale
“Rescale’s ScaleX cloud platform is a
game-changer for engineering. It
gives Boom computing resources
comparable to building a large on-
premise HPC center. Rescale lets us
move fast with minimal capital
spending and resources overhead.”
Josh Krall
CTO & Co-Founder

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Children’s Hospital of Philadelphia and Edico Genome
Achieve Fastest-Ever Analysis of 1,000 Genomes
Orlando, Fla., Oct 19, 2018 – The
Children’s Hospital of Philadelphia
(CHOP) and Edico Genome today set a
new scientific world standard in rapidly
processing whole human genomes into
data files usable for researchers aiming
to bring precision medicine into
mainstream clinical practice. Utilizing
Edico Genome’s DRAGENTM Genome
Pipeline, deployed on 1,000 Amazon
EC2 F1 instances on the Amazon Web
Services (AWS) Cloud, 1,000 pediatric
genomes were processed in two hours
and 25 minutes.
Genomics processing on FPGA

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EBS Volumes
Enhanced Networking
Hardware
Quick Amazon Elastic Compute Cloud (Amazon EC2)
review
c5n.18xlarge
Software
NVMe
ENA

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC software stack in Amazon EC2

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC software stack in Amazon EC2

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC network performance
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
EC2 MPI multi-stream bandwidth
Series1 Series2 Series3 Series4
0
10
20
30
40
50
60
1 2 3 4
EC2 MPI Latency

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC software stack with EFA

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HPC software stack with EFA

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EBS Volumes
Enhanced Networking
Hardware
Introducing EFA
c5n.18xlarge
Software
NVMe
ENAEFA

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
0
2000
4000
6000
8000
10000
12000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
EC2 MPI multi-stream bandwidth
Series1 Series2 Series3 Series4 Series5
0
10
20
30
40
50
60
1 2 3 4 5
EC2 MPI Latency
HPC network performance with EFA

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EFA getting started
• Supported platforms
• C5n.18xlarge, C5n.9xlarge, P3dn.24xlarge
• EFA Kernel module
• Upstream in progress
• https://github.com/amzn/amzn-drivers
• Libfabric Network Stack
• AWS-custom version for first half 2019
• MPI Implementation or NCCL
• Open MPI 3.1.3 or later or NCCL 2.3.8 or later
• Intel MPI and MPICH in development
See https://aws.amazon.com/hpc/ for more details

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EFA constraints
• Subnet-local communication
• Must have both an “allow all traffic within security group” ingress and
egress rule
• 1 EFA ENI per instance
• EFA ENIs can only be added at instance launch or to stopped instance

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s build an EFA-enabled cluster
% aws ec2 run-instances --count=4 --region us-east-1 --image-id ami-ABCD
--instance-type c5n.18xlarge --placement GroupName=ABCD
--network-interfaces DeleteOnTermination=true,DeviceIndex=0,
SubnetId=subnet-ABCD,InterfaceType=efa --security-group-ids sg-ABCD

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What we just built…
Availability Zone #1
Subnet ABCD

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s build an EFA-enabled cluster
% aws ec2 run-instances --count=4 --region us-east-1 --image-id ami-ABCD
--instance-type c5n.18xlarge --placement GroupName=ABCD
--network-interfaces DeleteOnTermination=true,DeviceIndex=0,
SubnetId=subnet-ABCD,InterfaceType=efa --security-group-ids sg-ABCD
🕰
% ssh ec2-user@ec2-1-2-3-4.compute-1.amazonaws.com
% lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev
08)
00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061
00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
00:06.0 Ethernet controller: Amazon.com, Inc. Device efa0

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And verify drivers…
% lsmod | grep efa
efa 81920 0
ib_core 266240 2 efa,ib_uverbs
% fi_info -p efa
provider: efa
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-rdm
version: 3.0
type: FI_EP_RDM
protocol: FI_PROTO_EFA
provider: efa
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-dgrm
version: 3.0
type: FI_EP_DGRAM
protocol: FI_PROTO_EFA
provider: efa;ofi_rxr
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-rdm
version: 1.0
type: FI_EP_RDM
protocol: FI_PROTO_RXR

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And verify MPI
% ompi_info | grep 'mtl: ofi'
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v3.1.3)
% mpirun -np 2 -hostfile ~/h ./ring_c
Process 0 sending 10 to 1, tag 201 (2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do I use EFA with…
AWS CloudFormation: Launch Templates
Amazon EC2 Auto Scaling Groups: Launch Templates
Spot/Spot Fleet: Launch Templates
AWS Batch: Launch Templates
Launch Templates: add InterfaceType : efa to the Network section

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Creating a launch template with EFA
{
"NetworkInterfaces": [{
"AssociatePublicIpAddress": false,
"DeviceIndex": 0,
"SubnetId": "subnet-ABCD",
"InterfaceType" : "efa"
}],
"Placement " : {
"GroupName": "ABCD"
},
"ImageId": "ami-ABCD",
"InstanceType": "c5n.18xlarge",
"CpuOptions": {
”CoreCount": 36,
"ThreadsPerCore": 1
}
}

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How we used to write HPC applications

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Libfabric changes the picture

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Libfabric Components

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EFA and libfabric: endpoints
Two native endpoint types:
• RDM (Reliable DataGram)
• DGRM (unreliable DataGRaM)
One utility endpoint type:
• RxR (RDM over RDM)
EFA protocol custom to AWS
% fi_info -p efa
provider: efa
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-rdm
version: 3.0
type: FI_EP_RDM
protocol: FI_PROTO_EFA
provider: efa
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-dgrm
version: 3.0
type: FI_EP_DGRAM
protocol: FI_PROTO_EFA
provider: efa;ofi_rxr
fabric: EFA-fe80::883:afff:fed3:776c
domain: efa_0-rdm
version: 1.0
type: FI_EP_RDM
protocol: FI_PROTO_RXR

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EFA native modes
• RDM
• Reliable, unordered datagrams
• ~8 KiB max message size
• Send/receive interface, with no tag matching
• Native multi-pathing; no “flow limit”
• DGRAM
• Unreliable, unordered datagrams
• ~8 KiB max message size
• Send/receive interface
• Subject to same “flow limit” as TCP/IP and UDP/IP over ENA

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Utility endpoint type
• RxR: build libfabrics interface over RDM
• Completion ordered datagrams
• tagged matching support (ie, MPI)
• Max message size > system memory size
• Large iovecs
• RxR developed by AWS as part of EFA
• Contributing back to Libfabric community shortly
• Currently implemented to support MPI implementations
• Future work includes supporting RMA and atomic transfer interfaces

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scalable Reliable Datagram (SRD)
• New protocol designed for AWS’s unique datacenter network
• Network aware multipath routing
• Guaranteed delivery
• Orders of magnitude lower tail latency
• No ordering guarantees
• Implemented as part of our 3rd generation Nitro chip
• EFA exposes SRD as a reliable datagram interface

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SRD link failure handling
0
5000
10000
15000
20000
25000
0 1000 2000 3000 4000 5000 6000 7000
TCP
Series1
0
5000
10000
15000
20000
25000
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
SRD
Series1

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
F.A.Q.s
• When will it reach general availability?
• First half 2019
• How do I sign up for the preview?
• https://pages.awscloud.com/elastic-fabric-adapter-preview.html
• What regions will EFA launch in?
• Any region with C5n or P3dn support
• What is your MPI latency?
• Less than 15 𝜇s ½ RTT in placement group (osu_latency benchmark)
• We will be constantly iterating behind the scenes to lower latency, including expanding
.metal options

41. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Brian Barrett
bbarrett@amazon.com

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.