Talk - HBase with Apache Spark POC Demo by Chetan Khatri

1. HBase with Spark POC
Demo
21st Dec, 2016
Chetan Khatri - Global Lead of Data Science,
Accionlabs Inc.

2. Current Environment
● Ubuntu 16.04
● Apache Hadoop 2.7.3
● Apache HBase 1.2.4
● Apache Spark 2.0.2
● Scala 2.12
● SBT(Scala Build Tool) 0.13.13
Note: Everything is in Pseudo Distributed mode

3. Action
Read Table
Write to Table

4. Approach
1. Created Table at HBase
2. Generated Dummy Data in Spark as a RDD and Saved to HBase
3. Reading HBase Table as a HadoopRDD in Spark
4. Traversing HadoopRDD Elements and Printing it out.
5. Entire Spark Transformation coded in Scala
6. Generated Executable Uber(“Super”) Jar with SBT
7. Submitted Spark job using spark-submit

5. Source Code - SparkHbaseJob.scala

6. Deployment
1. Add SBT Assembly Plugin
a. addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
b. At Plugins.sbt file under Scala Project
2. Build Uber(“Super”) Jar
a. Sbt clean assembly
3. Submit Spark Job
a. bin/spark-submit --class hbase.spark.chetan.com.SparkHbaseJob
/home/chetan/hbase-spark/SparkMSAPoc-assembly-1.0.jar

7. Demo

8. TODO
1. Read HBase Table and Store HBaseRDD to Apache Hive with Spark.
a. Incremental load from HBase to Hive
b. SparkSQL / SparkRDD and Apache Drill Performance Benchmark for Reading
HBase.
2. Save Summarized Spark Aggregation to PostgreSQL

9. Assumed : High Level Architecture
Event Queuing Kafka Batch
Incremental load /
Transformations
Reporting Tool
(Dashboard)
Processed DW - Ad
hoc Analysis / 1st
level aggregation
Data Ingestion
Pipeline
RestfulAPI’s
Summarized DB - KPI
Reporting

10. Reference
Github: https://github.com/chetkhatri/SparkHbasePOC