March Alluxio Community Office Hour Host: Gene Pang Email: community@alluxio.com Slack: www.alluxio.org/slack Website: www.alluxio.org

1. Alluxio Office Hour
Running ML Workloads with
Tensorflow, Alluxio and AWS S3
Gene Pang
gene@alluxio.com

2. About
Me
• Gene Pang
• Founding Engineer @ Alluxio, Inc.
• Alluxio PMC Member
• Twitter: @unityxx

3. Agenda
Machine Learning - Challenges1
Alluxio2
Demo3

4. Challenges of Deep Learning
4
Variety of storage options for users
• More data typically leads to better
modeling performance
• Distributed vs Cloud vs Local
Data Engineers and Data Movement
• Separation of Storage and Compute
• Latency increases costs and training
times
• Limited success of Data Warehouse,
Mart, and Lakes – cost of
copying/moving data is substantial

5. Alluxio
Storage
Unification
Bring diverse
storage data into a
single interface
Interact with all data
using one API
Accelerate slow data
transparently
Common
Storage API
Intelligent
Cache
3/25/19 5

6. Alluxio: Storage Unification
• Enables effective data management across different storages
3/25/19 6
UFS Namespace
s3://bucket/users
alice/ bob/
/
Alluxio Namespace
data/
reports/ sales/
users/
alice/ bob/
UFS Namespace
hdfs://data
reports/ sales/

7. Alluxio: Intelligent Cache
• Local performance from remote data using multi-tier storage
RAM SSD HDD
Hot Warm Cold
Read & Write
Buffering
Transparent to App
Policies for pinning,
promotion/demotion, TTL
3/25/19 7

8. Alluxio: Common Data Access API
• Convert from Client-side Interface to Storage API
Bigdata Filesystem API
HDFS Connector S3A Connector Swift Connector
Google Cloud
Connector
3/25/19 8
POSIX Filesystem API

9. A Common File System Abstraction
9
• Common interface across apps
• HDFS-compatible interface:
change hdfs://foo/bar to
alluxio://foo/bar
• Other interfaces: Native Alluxio Java
FS, POSIX and S3.
• Cloud storage becomes “hidden”
to apps
• Greater Flexibility
Compute Zone
Standalone or managed with Mesos or Yarn
Storage in Different Availability Zone
Either on-prem or cloud
TensorflowPrestoMR
HDFS API POSIX API

10. Alluxio: FUSE
10
Through Alluxio-FUSE, you can mount Alluxio and expose it as
a local file system on Unix
Applications can interact with Alluxio using standard POSIX
APIs (open, write, read) without any custom client integration
Note: Since Alluxio as a write-once/read-many file system, the
mounted file system will not support all POSIX workloads

11. POSIX Filesystem API
Deep
Learning
Frameworks
Unified
Data
Storage
Systems
POSIX Filesystem API
11

12. Make Distributed Data Available Locally
• FUSE Interface makes all enterprise data available locally
SUPPORTS
• HDFS
• NFS
• OpenStack
• Ceph
• Amazon S3
• Azure
• Google Cloud
IT OPS FRIENDLY
• Storage mounted into
Alluxio by central IT
• Security in Alluxio mirrors
source data
• Authentication through
LDAP/AD
• Wireline encryption
HDFS #1
Obj Store
NFS
HDFS #2
3/25/19 12

13. Overcomes I/O bottleneck on Cloud
More details at
https://www.alluxio.com/blog/flexible-and-fast-storage-for-deep-learning-with-alluxio
13

14. Conclusion
• Alluxio: Unified data access layer for
big data and ML applications
• Serve ML apps using Fuse-based
POSIX API, presenting and locally
caching large data sets from the cloud
• Try it out: www.alluxio.org/download

15. zhuanlan.zhihu.com/alluxio
www.alluxio.com
info@alluxio.com
twitter.com/alluxio
linkedIn.com/alluxio
Thank you
gene@alluxio.com