Data Orchestration Summit 2020 organized by Alluxio https://www.alluxio.io/data-orchestration-summit-2020/ How to Build a new under filesystem in Alluxio: Apache Ozone as an example Baolong Mao, Sr. System Engineer (Tencent) About Alluxio: alluxio.io Engage with the open source community on slack: alluxio.io/slack

1. DATA ORCHESTRATION SUMMI
T
How to build a new Under Filesystem in Alluxio
Apache Ozone as an example
Baolong Mao | Sr. System Engineer
Alluxio PMC
Apache Ozone Committer

2. DATA ORCHESTRATION SUMMIT
Contribution From Tencent
● Ozone & COSN UFS
● Support using IP as connect host
● Make PermissionChecker Configurable
● Prometheus + Grafana monitor dashboard template & docs
● Mount table webUI
● Generic tarball script to generate customized hadoop
● More metrics: Lock pool size and block remover metrics
● Add ETAG header to s3 api proxy
● ……………………….

3. DATA ORCHESTRATION SUMMIT
Agenda
● Alluxio global namespace
● Apache Ozone
● Alluxio UFS framework
●

4. DATA ORCHESTRATION
SUMMIT
2020
● Alluxio global namespace
Introduction
How the unified namespace works in Alluxio
with different under file systems

5. DATA ORCHESTRATION SUMMIT
Alluxio Overview

6. DATA ORCHESTRATION SUMMIT
Alluxio global namespace

7. DATA ORCHESTRATION
SUMMIT
2020
Apache Ozone Introduction
Introduce Apache Ozone shortly

8. DATA ORCHESTRATION SUMMIT
Apache Ozone

9. DATA ORCHESTRATION SUMMIT
Apache Ozone

10. DATA ORCHESTRATION
SUMMIT
2020
Alluxio UFS framework
Introduce about Alluxio UFS

11. DATA ORCHESTRATION SUMMIT
•./bin/alluxio fs mount
--option
alluxio.underfs.hdfs.configuration=<DIR>/ozone-site.xml:<DIR>/
core-site.xml
/ozone o3fs://<OZONE_BUCKET>.<OZONE_VOLUME>/
How Alluxio UFS works

12. DATA ORCHESTRATION SUMMIT
Service Discovery
● Dynamically loaded
● Java ServiceLoader . Implements the
alluxio.underfs.UnderFileSystemFactory interface.
● Pointing to the class implementation by
META_INF/services/alluxio.underfs.UnderFileSystemFactory
How Alluxio UFS works

13. DATA ORCHESTRATION SUMMIT
Dependency Management
● Fat jar
● Shaded jar
● Isolated classloading
How Alluxio UFS works

14. DATA ORCHESTRATION SUMMIT
How Alluxio UFS works

15. DATA ORCHESTRATION SUMMIT
Implementing an Under Storage Extension
•Implementing the required under storage interface

16. DATA ORCHESTRATION SUMMIT
Implementing an Under Storage Extension
•Declaring the
service implementation

17. DATA ORCHESTRATION SUMMIT
Implementing an Under Storage Extension
•Add maven plugin to pom.xml for build a package
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
</plugin>
<plugin>
<groupId>com.coderplus.maven.plugins</groupId>
<artifactId>copy-rename-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

18. DATA ORCHESTRATION
SUMMIT
2020
Introduce how to implement Ozone UFS

19. DATA ORCHESTRATION SUMMIT
alluxio fs cat /ozone/B

20. DATA ORCHESTRATION SUMMIThttps://github.com/Alluxio/alluxio/pull/11396

21. DATA ORCHESTRATION SUMMIT
Hadoop compatible File system(HCFS)

22. DATA ORCHESTRATION SUMMIT

23. DATA ORCHESTRATION SUMMIT

24. DATA ORCHESTRATION SUMMIT
•https://docs.alluxio.io/os/user/edge/en/Overview.html
•https://docs.alluxio.io/os/user/edge/en/overview/Archite
cture.html
•https://docs.alluxio.io/os/user/edge/en/core-services/Un
ified-Namespace.html
•https://docs.alluxio.io/os/user/edge/en/ufs/Ozone.html
•https://docs.alluxio.io/os/user/edge/en/ufs/Ufs-Extensio
ns.html
•https://docs.alluxio.io/os/user/edge/en/ufs/Ufs-Extensio
n-API.html

25. DATA ORCHESTRATION SUMMIT
How to Join Alluxio Community
Thank you!
alluxio.io/slack