Alluxio Day XII April 28, 2022 https://www.alluxio.io/alluxio-day/ Speakers: Tianbao Ding and Haoning Sun

1. The Power of Data Orchestration:
Storage Acceleration and Servitization at Shopee
Tianbao Ding
Haoning Sun

2. Private &
Confidential
Private & Confidential 2
1 Storage Status
2
3
4
Storage Accelerationn
Storage Servitisation
Future Plan
Storage Acceleration and Servitization at Shopee
Storage Servitisation
Storage Servitization

3. Private & Confidential 3
Storage Status—Architecture
3
Data Management Platform (DMP)
Spark Flink
Yarn
Presto
App (Search,
Recommendation
etc.)
Compute
Engine
Resource
Scheduler
Storage HDFS Ozone
Platform
Product

4. Private & Confidential 4
Storage Status—HDFS
4
Metric Value
Number of Nodes Thousands
Storage Capacity Hundreds PB
Num of Files Billions
Max QPS Hundreds of thousands

5. Private & Confidential 5
Storage Status—Presto
5
Metric Value
Number of Nodes Thousands of instances
TP90 About 2 min
Input Dozens of PB per day
Number of Queries
Hundreds of thousands per
day

6. Private & Confidential 6
HDFS
Unstable
Performance
Presto
Unstable
Query
Q：Can queries run faster?
Storage Status—Presto Accelerate Query

7. Private & Confidential 7
Presto
HDFS
Presto
Alluxio
Add cache
HDFS
Storage Status—Presto Accelerate Query

8. Private & Confidential 8 8
Storage Status—Alluxio+Presto Typical Architecture
• Mount HDFS
• Presto visit HDFS via Alluxio
• Alluxio manage cache

9. Private & Confidential 9 9
Storage Status—Shortcomings
• Need specific caching policies
• Read slowly from alluxio at the first time

10. Private &
Confidential
Private & Confidential 10
1 Storage Status
2
3
4
Storage Acceleration
Storage Servitisation
Future Plan
Storage Acceleration and Servitisation at Shopee
Storage Servitisation
Storage Servitization

11. Private & Confidential 11 11
Storage Acceleration—Solution

12. Private & Confidential 12
Storage Acceleration—Architecture
Kafka HDFS
Audit
HMS
Computing
Application
Operator Hot Table
Cache Manager
Alluxio
HDFS
Load/Unload/m
ount
Path
delete/create
event
Set/Clear
partition
property
Load data
Load Table
Update Policy
Input
Output
Get tag

13. Private & Confidential 13
Presto Query
Log
Hot Table
(Hive Table)
• Partition by date
• Calculate the number of visits of table
every day
Storage Acceleration—Hot Table

14. Private & Confidential 14
• Scheduled every day
Hot Table
Most frequently
visited weighted
tables in the last
seven days
Recent m
partitions
every table
Load from
HDFS to
Alluxio
Persist
relationship
Set tag in
HMS
Note: It is an alpha version and the
subsequent iterations will be optimized
continuously
Storage Acceleration—Update Policy

15. Private & Confidential 15
HDFS
Alluxio
HMS
Presto
On Alluxio No tag
• key:cache,
value:${DC}/Alluxio/ebj@${Alluxio_nameservice}
• If partition exists, set property in partition
property
• Else, set property in table property
Storage Acceleration—HMS Tag

16. Private & Confidential 16
Example:
Storage Acceleration—HMS Tag

17. Private & Confidential 17
HDFS
Audit Log
Flink
Format:
• cmd=xxsrc=xxdst=xx
Storage Acceleration—Kafka
Kafka topic
filter

18. Private & Confidential 18
PATH
mount
unmount
load
query
HIVE TABLE
mount
unmount
load
load all the
recorded paths
load recent n
partitions
query
ADMIN
monitor and operator
Storage Acceleration—REST API

19. Private & Confidential 19
Storage Acceleration—Perf Effect

20. Private & Confidential 20
• 6 merged, 2 WIP, 1 fixed by Alluxio.
TYPE PR STATUS
Hadoop 2.10
Fix HdfsVersion miss hadoop 2.10 config merged
Fix integration/yarn/pom.xml enforcer-plugin miss hadoop
2.10.x config
merged
Fix common.go miss hadoop 2.10 configuration merged
Command Line
Improve shell command support ebj nameservice merged
Fix for Alluxio.logs.dir
fixed by
Alluxio
Web Page
Fix isMounted should not invoke ufs, if not /metrics page very
slowly
merged
Fix FormatUtils.getSizeFromBytes method should supports EB merged
NameServices
Fix unescape the ufs url of Alluxio fsadmin report metrics
result
WIP
Metrics Fix cache radio total not include cacheMisses WIP
Storage Acceleration—Community Contribution

21. Private & Confidential
Private & Confidential 21
1 Storage Situation
2
3
4
Storage Acceleration
Storage Servitization
Future Plan
Storage Acceleration and Servitization at Shopee

22. Private & Confidential
Private & Confidential 22
Storage Servitization—Status
▪ Most of data is stored in HDFS
▪ Various development languages are used
▪ HDFS has insufficient support for non Java clients

23. Private & Confidential
Private & Confidential 23
Fuse for HDFS
S3 for HDFS
▪ Deploy alluxio fuse service on physical machine
▪ Deploy alluxio fuse service on kubernetes cluster
▪ Using S3 API to access alluxio proxy service
Storage Servitization—Solutions

24. Private & Confidential
Private & Confidential 24
▪ Kernel
▪ User-level daemon
High-Level Architecture
Storage Servitization—Fuse
WHAT IS IT
▪ FileSystem in Userspace

25. Private & Confidential
Private & Confidential 25
▪ libfuse
▪ JNR-Fuse
▪ JNI-Fuse
Requirements
Implementation
Storage Servitization—Alluxio Fuse
▪ Standalone Fuse
▪ Fuse on Workers
Deployment
▪ Not support random writes
Limitations

26. Private & Confidential
Private & Confidential 26
Store Servitization—Alluxio CSI
▪ On nodeserver pod
▪ On separate pod(new feature)
Fuse Deployment mode
WHAT IS IT
▪ Standard storage interface for
containers

27. Private & Confidential
Private & Confidential 27
▪ Fuse sidecar container in a Pod to mount the
Alluxio directory
▪ Independent configuration of pods, high flexibility
▪ Each Pod runs a Fuse container without affecting
each other
▪ Each Fuse process occupies a container, so the
solution consumes more resources
Futures
Store Servitization—k8s sidecar for Alluxio
WHAT IS IT

28. Private & Confidential
Private & Confidential 28
Store Servitization—Summarize
Fuse on
K8s-csi
K8s-sidecar
Fuse on nodeserver
pod
Fuse on separate pod
maintenance
cost
high low higher higher
resource usage low lower high high
independence high low high high

29. Private & Confidential
Private & Confidential 29
▪ Bucket: A bucket is a container for objects stored in Amazon S3
▪ Object: Objects are the fundamental entities stored in Amazon S3
▪ Key: An object key (or key name) is the unique identifier for an
object within a bucket.
▪ Region: You can choose a region to store the created buckets
Store Servitization—S3
Buckets
Objects
Keys Regions
Amazon
S3
Concepts
Conception

30. Private & Confidential
Private & Confidential 30
▪ Alluxio can mount HDFS data
▪ Alluxio provides Proxy service
▪ Proxy is compatible with the basic operations of the S3 API
▪ S3 SDK supports many development languages
Store Servitization—S3 for HDFS
Access HDFS data via Alluxio using S3 protocol

31. Private & Confidential
Private & Confidential 31
▪ 1-level directory as bucket
▪ Subdirectories and file paths as key
Store Servitization—Alluxio Proxy for S3 mapping

32. Private & Confidential
Private & Confidential 32
Store Servitization—Proxy Authentication
▪ Authentication parser
▪ Validator
▪ Secret Manager
▪ Signature Calculation

33. Private & Confidential
Private & Confidential 33
Store Servitization—Service Architecture

34. Private & Confidential
Private & Confidential 34
Store Servitization—Community contribution
TYPE PR STATUS
proxy
Fix wrong format of s3 bucket creationDate merged
Support parse authorization headers for s3 proxy WIP
fuse
Fix wrong method call to get username and wrong
parameter assignment
merged
csi Replace invalid env with args in nodeserver merged
doc
Fix bug case of S3 REST API merged
Fix wrong file name in k8s doc merged
Fix ambiguous description for impersonation in CN
doc
merged
ozone Update ozone from 1.1.0 to 1.2.1 closed
▪ 6 merged, 1 WIP, 1 closed.

35. Private & Confidential
Private & Confidential 35
1 Storage Situation
2
3
4
Storage Acceleration
Storage Servitization
Future Plan
5 Future Plan
Storage Acceleration and Servitization at Shopee

36. Private & Confidential
Private & Confidential 36
▪ Speed up Spark and Hive
▪ Implement adaptive cache policy on CacheManager
▪ Support more POSIX APIs
▪ Optimize CSI
Storage Service
Future Plan
Storage speed up

37. Private & Confidential
Private & Confidential 37
Thank You
Storage Acceleration and Servitization at Shopee