HBaseCon2017 Data Product at AirBnB
•
3 gefällt mir•785 views
Building data product requires having lambda architecture to bridge the batch and streaming processing. AirStream is a framework built on top of HBase to allow users to easily build data products at Airbnb. It proved HBase is impactful and useful in the production for mission critical data products. In the talk, we will present the applications to leverage HBase to compute moving average, distinct count, window based join and etc. in the streaming computation. Also, we will talk about how to leverage HBase to bridge the gap between batch and streaming queries, including building presto-hbase connector to serve near real time ad-hoc query. by Liyin Tang of AirBnB
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie HBaseCon2017 Data Product at AirBnB
Ähnlich wie HBaseCon2017 Data Product at AirBnB (20)
Mehr von HBaseCon
Mehr von HBaseCon (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
HBaseCon2017 Data Product at AirBnB
- 1. Data Product at Airbnb LIYIN TANG & JINGWEI LU
- 2. Data Infrastructure at Airbnb
- 3. Event Logs MySQL Dumps Gold Cluster HDFS Hive Kafk Sqoo Silver Cluster Spark Cluster Spark ReAi Airflow Scheduling S3 Presto Cluster AirPal SuperSet Tableau Batch Infrastructure Yarn HDFS Hive Yarn Liyin Tang and Jingwei Lu 3
- 4. Streaming at Airbnb Liyin Tang and Jingwei Lu 4 Cluster Spark Streaming Airflow Scheduling HBase Sources Kafka S3 HDFS … Sinks Datadog Kafka Dynamo DB Elastic Search …
- 6. Batch AirStream Hive Spark SQL Lambda Architecture Liyin Tang and Jingwei Lu 6 Streaming Kafka Spark Streaming HBase
- 7. Liyin Tang and Jingwei Lu Our Foundations • Combine stream with batch • Shared global state store 7
- 8. Liyin Tang and Jingwei Lu Unified API through AirStream • Declarative job configuration • Computation operator or sink can be shared by stream and batch job. • Stream source vs static source • Single driver execute stream/batch mode job 8
- 9. AirStream Liyin Tang and Jingwei Lu 9 HBase Tables Spark StreamingSpark StreamingSpark StreamingSpark Streaming Spark BatchSpark BatchSpark BatchSpark Batch Shared Global State Store
- 10. Shared Global State Store Liyin Tang and Jingwei Lu 10 DataFrame HBase Region 1 Region 2 Region N Re-partition <Region 1, [RowKey, Value]> <Region 2, [RowKey, Value]> <Region N, [RowKey, Value]> … … Puts HFile BulkLoad
- 11. Shared Global State Store Liyin Tang and Jingwei Lu 11 HBase Tables Spark Streaming/Batch Jobs Multi-Gets Prefix Scan Time Range Scan
- 12. Why HBase ?
- 13. • Rich API for point-lookups and sequential scan (TimeRange, TTL, Prefix Scan …) • Merged view based on version • Unified API for streaming writes and bulk uploads • Unified API for reading from live table and snapshot table 13 Why HBase
- 15. Merged Storage Liyin Tang and Jingwei Lu 15 Row Key R1 V100 V100 R1 V99 V99 R1 V01 V01 … … … … Time Streaming Writes Streaming Writes Streaming Writes
- 16. Merged Storage Liyin Tang and Jingwei Lu 16 Row Key R1 V100 V100 R1 V99 V99 R1 V01 V01 Time Streaming Writes Streaming Writes Streaming Writes R1 V100 100 Batch Bulk Upload
- 17. Distinct Count Liyin Tang and Jingwei Lu 17 Row Key Prefix _ R1 V102 V102 Prefix _ R2 V101 V101 Prefix _ R4 V01 V01 Prefix _ R3 V100 100 Prefix Scan with TimeRange Prefix Scan with TimeRange Time
- 18. Moving Average Liyin Tang and Jingwei Lu 18 Row Key R1 V102 102 R1 V101 101 R1 V01 V01 R1 V100 100 Count Difference/ Time Elapsed Count Difference/ Time Elapsed Time … … … … … … Window 1 Window 2
- 19. Long Window Computation Liyin Tang and Jingwei Lu 19
- 20. Spark HBase Connector Liyin Tang and Jingwei Lu 20 Zeppelin Spark HBase Connector HBase
- 21. Presto HBase Connector Schema Mapping Split -> RS Mapping Presto - HBase Connector Liyin Tang and Jingwei Lu 21 Presto HBase
- 22. Hive - HBase Connector Liyin Tang and Jingwei Lu 22 Hive HBase Hive HBase Connector Table InputFormat Snapshot InputFormat
- 23. Use Cases
- 24. Mysql DB Snapshot Using Binlog Replay
- 25. • Large amount of data: Multiple large mysql DBs • Realtime-ness: minutes delay/ hours delay • Transaction : Need to keep transaction cross different tables • Schema change: Table schema evolves Database Snapshot 25 Move Elephant
- 26. 26 20+ hr 4+ hr Airstream Job 5 mins 15 1 hr Binlog Replay on Spark
- 27. • Streaming and Batch shares Logic: Binlog file reader, DDL processor, transaction processor, DML processor. • Idempotent: Log can be replayed multiple times. • Schema changes: Full schema change history. 27 Log Parser xvid DML DDL HBASE Architecture Binlog
- 28. Realtime Ingestion & Interactive Query
- 29. Realtime Ingestion and Interactive Query Liyin Tang and Jingwei Lu 29 HBase AirStream Spark Streaming Kafka Query Engine Data Portal Spark SQL Hive SQL Presto SQL
- 30. Interactive Query in Superset 30
- 31. Summary
- 32. Unify Batch and Streaming Computation 32
- 33. 33 We are hiring