Alluxio Day VIII December 14, 2021 https://www.alluxio.io/alluxio-day/ Speaker: Ryan Blue, Apache Iceberg

1. Building an Open
Data Platform with
Apache Iceberg
Ryan Blue
Alluxio Day 8, December 2021

2. What is Iceberg?

3. What is Iceberg?

4. Where are we going?

5. Current data architecture
● Multi-engine
○ Spark for ETL, ML
○ Trino for ad-hoc, ETL
○ Flink for streaming
○ Druid for aggregates
● In the cloud (or moving)
● Hive Metastore
○ No metastore?
● Investing in data
○ In people
○ In tools
○ In infrastructure

6. But the
pieces
don’t fit
together
quite right

7. What is Iceberg?

8. What is Iceberg?
● A table format
○ Akin to columnar file formats
○ Transactional guarantees
○ Performance enhancements
● A standard for analytic tables
○ Open source spec and library
○ Integrated into query engines

9. And how does that help?

10. Object storage
The gap
Data & metadata
Compute
Apache
Spark
Catalog
???

11. Shared storage requirements
Technical:
● Must handle concurrent writes
● Must be scalable, performant
● Must be cloud native
Practical:
● Must be open source
● Must be neutral
● Must address productivity

12. Iceberg’s
goals
● Add reliable transactions
● Unlock performance
● Fix usability

13. Object storage
Open data platform
Data & metadata
Compute
Apache
Spark
Catalog
Vertical solutions Open data stack
Data
Services

14. Thank you!

15. Iceberg exists to fix
productivity

16. Lessons learned
● Avoid unpleasant surprises
○ Principle of least surprise
● Donʼt steal attention
○ Reduce context switching

17. We try to make Iceberg
invisible

18. Usability improvements
● Schema evolution
○ Instantaneous – no rewrites
○ Safe – no undead columns 🧟
○ Saves days of headache
ALTER TABLE db.tab
RENAME COLUMN
id TO customer_id
● Layout evolution
○ Lazy – only rewrite if needed
○ Partitioning mistakes are okay
○ Changes with your data
○ Saves a month of headache
ALTER TABLE db.tab
ADD PARTITION FIELD
bucket(256, id)

19. Practical improvements
● Hidden partitioning
○ No silent correctness bugs
○ No conversion mistakes
○ Query without understanding
a tableʼs physical layout
● Reliable updates
○ Stop manual cleanup
○ Use any query engine
○ Automate maintenance

20. Performance improvements
● Indexed metadata
○ Fast job planning
○ Fast query execution
○ Faster iteration
● Table configuration
○ Tune tables, not jobs
○ Automate table tuning
○ Cluster and sort from config