Unprecedented computational power for massively parallel applications creates unprecedented storage requirements. Learn about coherent storage clusters processing millions of IOPS at submillisecond latency, how to architect storage for HPC in the cloud, and how to do it all without breaking the bank. This session incorporates live demonstrations, including APN Competency Partner solutions.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
STORAGE FOR HPC IN THE CLOUD
I s a i a h W e i n e r
S r . M g r . S o l u t i o n s A r c h i t e c t u r e
G P S T E C 3 2 4

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
I/O
Cost
TTR
National
Labs
Research,
Energy & UtilitiesGenomics
Analytics,
AI/ML
EDA M&E
Finance
HPC IS COMPLEX

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GROWTH IN CLOUD
2015 2016 2017 2018 2019 2020
70% 65%
61% 59% 58% 55%
10%
15%
26% 26% 26% 28%
10% 12% 13% 15% 16% 17%
CLOUD MARKET FORECAST
On-Prem Public Cloud Private Cloud
Source: IDC Worldwide Quarterly Cloud IT Infrastructure Tracker

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GROWTH IN STORAGE
0
10000
20000
30000
40000
50000
60000
70000
80000
2016 2017 2018 2019 2020
Exabytes
Enterprise HPC
Source: Gartner for Enterprise and IDC for High Performance

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DIY – NFS
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Server
Volume Volume
NFS
Clients
NFS
Clients
NFS
Clients

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AMAZON EFS ARCHITECTURE
Clients Clients Clients
Mount
Target
Single Namespace
Mount
Target
Mount
Target

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HPC CLUSTER NODE ANATOMY
Data
Metadata
Tiering
Backend
Routing
Monitoring
VIPs
Clustering
Storage
Access
Protocols
Frontend
Network

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenge: Local SSD performance
with centralized management. Local
SSDs required data to be copied
around, and multiple copies took up
space.
Solution delivers: Scalable,
sharable, simplified; one copy of
the data, on-par with local SSD for
performance. 0
50
100
150
200
250
Elapsed Time (Lower is Better)
Local SSD WekaIO NFSv4
SEMICONDUCTOR CUSTOMER

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenge: Small file workload with
some large files in the mix. Pre-
solution workaround: more jobs! All
the jobs!
Solution delivers: Scalable,
sharable, simplified; one copy of
the data, on-par with local SSD for
performance. 0
20
40
60
80
100
120
140
WekaIO On-Prem AFA
Elapsed Time (Lower is Better)
1 conversion 6 conversions
GENOMICS CUSTOMER

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster Sizing

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The hardware is reliable.
Trust the kernel, it is wise.
The MTBF is millions of hours.
Hardware is up until it dies.
…Is the hardware reliable, really?
DPDK + SR-IOV, SPDK, RoCE…
200K hours is more likely.
EC2 Spot could live for 15 minutes!
N O WT H E N
SOFTWARE ASSUMPTIONS

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AVAILABILITY VS. DURABILITY
% Downtime Per Year Probability of Loss
99.999 5 minutes 15 seconds 1 in 100,000
99.9999 31 seconds 1 in 1,000,000
99.99999 3 seconds 1 in 10,000,000
99.999999999 1 in 100,000,000,000

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MTBF WITH SINGLE NODE
Failure every 22 years: 1/200,000
1 x MTBF
200K hours
8.5 hour
repair

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MTBF WITH CLUSTER
12 x MTBF
200K hours
2nd failure probability:
(11 x 8.5)/200,000
2nd failure frequency:
1.9 x 2134
3rd failure probability:
(10 x 5.4)/200,000
3rd failure frequency:
4,060 x 3,688
3rd failure every
14,977,000 years:
1 out of 3,688 2nd failures
2nd failure every
4,060 years: 1 out
of 2,134 repairs
Failure every 1.9
years:
12/200,000
5.4 hour
2nd repair

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FINDING METADATA LIMITS
srun -n $i -N 128 mdtest -i 5 -b 3 -z 3 -I 10 -w 1024 -y -d $PFS/testdir
$i = number of
compute processes
$PFS = HPC storage
mountpoint
0
5000
10000
15000
20000
25000
1 2 4 8 16 32 64 128 256 512 1024 2048
Creates/second
Number of client processes
File creates/process/second (32 nodes)
Lustre (single MDS) WekaIO v3.1

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FINDING DATA PATH LIMITS
srun -n $i -N 128 ior -a POSIX -o $PFS/iortest -z -w -F -b 1g -t 1m -i 8
$i = number of
compute processes
$PFS = HPC storage
mountpoint
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
1 2 4 8 16 32 64 128 256 512 1024 2048
Throughput(MB/sec)
Number of client processes
File-per-process throughput (32 nodes)
Lustre (single MDS) WekaIO v3.1

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FINDING DATA PATH LIMITS (DIRECT I/O)
srun -n $i -N 128 ior -a POSIX -o $PFS/iortest -z -w -F -b 1g -t 1m -i 8
vs.
srun -n $i -N 128 ior -a POSIX -o $PFS/iortest -z -w -F -B -b 1g -t 1m -i 8

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TECHNOLOGY SUMMARY
• Why RPO and RTO matter for HPC in the Cloud
• Lustre
• Supports DNE – Distributed Namespace
• Still no durability after all these years
• EBS performance limitations
• WekaIO
• Distributed Metadata
• Scalable data plane
• Durable, plus S3 persistence

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EXTERNAL RESOURCES
AWS Competency Program
• https://aws.amazon.com/partners/competencies
AWS Quick Start
• https://aws.amazon.com/quickstart
AWS Marketplace
• https://aws.amazon.com/marketplace

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!