An introduction to structured data processing using Data source and Dataframe API's of spark.Presented at Bangalore Apache Spark Meetup by Madhukara Phatak on 31/05/2015.

1. Structured data
processing with Spark
SQL
https://github.com/phatak-dev/structured_data_processing_spark_sql

2. ● Madhukara Phatak
● Big data consultant and
trainer at datamantra.io
● Consult in Hadoop, Spark
and Scala
● www.madhukaraphatak.com

3. Agenda
● Variety in Big data
● Structured data
● Structured data analysis in M/R
● Datasource API
● DataFrame
● SQL in Spark
● Smart sources

4. 3 V’s of Big data
● Volume
○ TB’s and PB’s of files
○ Driving need for batch processing systems
● Velocity
○ TB’s of stream data
○ Driving need for stream processing systems
● Variety
○ Structured, semi structured and unstructured
○ Driving need for sql, graph processing systems

5. Why care about structured data?
● Isn’t big data is all about unstructured data?
● Most real world problems work with structured / semi
structured data 80% of the time
● Sources
○ JSON from API data
○ RDBMS input
○ NoSQL db inputs
● ETL process convert from unstructured to structured

6. Structured data in M/R world
● Both structured and unstructured data treated as same
text file
● Even higher level frameworks like Pig/Hive interprets
the structured data using user provided schema
● Let’s take an example of processing csv data in spark in
Map/Reduce style
● Ex: CsvInRDD

7. Challenges of structured data in M/R
● No uniform way of loading structured data, we just piggyback
on input format
● No automatic schema discovery
● Adding a new field or changing field sequencing is not that
easy
● Even Pig JSON input format just limits for record separating
● No high level representation of structured data even in Spark
as there is always RDD[T]
● No way to query RDD using sql once we have constructed
structured output

8. Spark SQL library
● Data source API
Universal API for Loading/ Saving structured data
● DataFrame API
Higher level representation for structured data
● SQL interpreter and optimizer
Express data transformation in SQL
● SQL service
Hive thrift server

9. Spark SQL Apache Hive
Library Framework
Optional metastore Mandatory metastore
Automatic schema inference Explicit schema declaration using DDL
API - DataFrame DSL and SQL HQL
Supports both Spark SQL and HQL Only HQL
Hive Thrift server Hive thrift server

10. Loading structured Data
● Why not Input format
○ Input format always needs key,value pair which is
not efficient manner to represent schema
○ Schema discovery is not built in
○ No direct support for smart sources aka server side
filtering
■ Hacks using configuration passing

11. Data source API

12. Data source API
● Universal API for loading/saving structured data
● Built In support for Hive, Avro, Json,JDBC, Parquet
● Third party integration through spark-packages
● Support for smart sources
● Third parties already supporting
○ Csv
○ MongoDB
○ Cassandra (in works)
etc

13. Data source API examples
● SQLContext for accessing data source API’s
● sqlContext.load is way to load from given source
● Examples
○ Loading CSV file - CsvInDataSource.scala
○ Loading Json file - JsonInDataSource.scala
● Can we mix and match sources having same schema?
○ Example : MixSources.scala

14. DataFrame
● Single abstraction for representing structured data in
Spark
● DataFrame = RDD + Schema (aka SchemaRDD)
● All data source API’s return DataFrame
● Introduced in 1.3
● Inspired from R and Python panda
● .rdd to convert to RDD representation resulting in RDD
[Row]
● Support for DataFrame DSL in Spark

15. Querying data frames using SQL
● Spark-SQL has a built in spark sql interpreter and
optimizer similar to Hive
● Support both Spark SQL and Hive dialect
● Support for both temporary and hive metastore
● All ideas like UDF,UDAF, Partitioning of Hive is
supported
● Example
○ QueryCsv.scala

16. DataFrame DSL
● A DSL to express SQL transformation in a map/reduce
style DSL
● Geared towards Data Scientist coming from R/Python
● Both SQL and Dataframe uses exactly same interpreter
and optimizers
● SQL or Dataframe is upto you to decide
● Ex : AggDataFrame

17. RDD transformations Data Frame transformation
Actual transformation is shipped on
cluster
Optimized generated transformation is
shipped on cluster
No schema need to be specified Mandatory Schema
No parser or optimizer SQL parser and optimizer
Lowest API on platform API built on SQL which is intern built on RDD
Don’t use smart source capabilities Make effective use of smart souces
Different performance in different language API’s Same performance across all different languages

18. Performance

19. Why so fast?
https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html

20. Smart sources
● Data source API’s integrates richly with data sources to
allow to get better performance from smart sources
● Smart data sources are the one which support server
side filtering, column pruning etc
● Ex : Parquet, HBase, Cassandra, RDBMS
● Whenever optimizer determines it only need only few
columns, it passes that info to data source
● Data source can optimize for reading only those columns
● More optimization like sharing logical planning is coming
in future

21. Apache Parquet
● Apache Parquet column storage format for Hadoop
● Supported by M/R,Spark, Hive, Pig etc
● Supports compression out of the box
● Optimized for column oriented processing aka analytics
● Supported out of the box in spark as data source API
● One of the smart sources, which supports column
pruning
● Ex : CsvToParquet.scala and AggParquet.scala

22. References
● https://databricks.com/blog/2015/01/09/spark-sql-data-
sources-api-unified-data-access-for-the-spark-platform.
html
● https://databricks.com/blog/2015/02/17/introducing-
dataframes-in-spark-for-large-scale-data-science.html
● http://parquet.apache.org/
● http://blog.madhukaraphatak.
com/categories/datasource-series/