Presented at Datapalloza Seattle, Feb 10-11, 2016

1. Scala and Spark:
Coevolving Ecosystems
for Big Data
John Nestor
47 Degrees
Datapalooza Seattle
February 10-11, 2016
www.47deg.com
147deg.com

2. 47deg.com
Outline
• Scala
• Spark
• Scala Impact on Spark
• Spark Impact on Scala
• Summary
• Questions
2

3. Scala
3

4. 47deg.com
Scala History
• Scala created by Martin Odersky, EFPL Switzerland
• wrote javac, the compiler for Java
• designed Java generics
• 2004 Scala announced
• 2006 Scala 2.0
• 2011 Typesafe started, Scala 2.9
• 2012 Scala 2.10
• 2014 Scala 2.11
• 2015 Scala 2.12-RC1
• 2016 Scala now 12 years old and quite mature
4

5. 47deg.com
Why Scala?
• Open Source
• Strong typing
• Concise elegant syntax
• Runs on JVM (Java Virtual Machine)
• Supports both object-oriented and functional
• Small simple programs (REPL) through very large
multi-server systems
• Easy to cleanly extend with new libraries and DSL’s
• Ideal for parallel and distributed systems
5

6. 47deg.com
Scala: Strong Typing and Concise Syntax
• Strong typing like Java.
• Compile time checks
• Better modularity via strongly typed interfaces
• Easier maintenance: types make code easier to
understand
• Concise syntax like Python.
• Type inference. Compiler infers most types that had to be
explicit in Java.
• Powerful syntax that avoid much of the boilerplate of Java
code (see next slide).
• Best of both worlds: safety of strong typing with conciseness
(like Python).
6

7. 47deg.com
Scala Case Class
• Java version
class User {
private String name;
private Int age;
public User(String name, Int age) {
this.name = name; this.age = age;
}
public getAge() { return age; }
public setAge(Int age) { this.age = age;}
}
User joe = new User(“Joe”, 30);
• Scala version
case class User(name:String, var age:Int)
val joe = User(“Joe”, 30)
7

8. 47deg.com
Key Scala Features
• Immutable Collections
• Seq, Set, Map
• Functional Programming
• (i:Int) => i + 1
• Functions can be parameters and results
• For collections: map, filter, flatMap, groupBy, …
• Case Classes
• case class Person(name:String, age:Int)
• Define Domain Specific Languages (DSLs)
• Akka, SBT, Spray, ScalaTest and now Spark
8

9. 47deg.com
View Sample Scala Code
• Word Count in Scala
9

10. Spark
10

11. 47deg.com
Spark History
• 2009 Mesos developed at UC Berkeley AmpLab
• 2009 Matei Zaharia starts Spark as a test case for
Mesos
• 2012 Spark 0.5
• 2014 Spark 1.0, Top Level Apache Project
• 2014 Databricks started
• 2016 Spark 1.6
11

12. 47deg.com
Why Spark?
• Support for not only batch but also (near) real-time
• Fast - keeps data in memory as much as possible
• Often 10X to 100X Hadoop speed
• A clean easy-to-use API
• A richer set of functional operations than just map and
reduce
• A foundation for a wide set of integrated data
applications
• Can recover from failures - recompute or (optional)
replication
• Scalable for very large data sets and reduced time
12

13. 47deg.com
Spark Components
• Spark Core
• Scalable multi-node cluster
• Failure detection and recovery
• RDDs, Dataframes and functional operations
• MLLib - for machine learning
• linear regression, SVMs, clustering, collaborative
filtering, dimension reduction
• more on the way!
• GraphX - for graph computation
• Streaming - for near real-time
13

14. 47deg.com
Spark RDDs
• RDD[T] - resilient distributed data set
• typed
• immutable
• ordered
• can be processed in parallel
• lazy evaluation - permits more global optimizations
• Rich set of functional operations ( a small sample)
• map, flatMap, reduce, …
• filter, groupBy, sortBy, take, drop, …
14

15. 47deg.com
Spark Data Frames
• Similar to SQL tables
• can be transformed using SQL
• Focus of much of the work on Spark performance
optimization
• Unlike RDD’s, optimized knows about fields
• Dynamically typed
• Not a natural fit for Scala (more on this later)
15

16. 47deg.com
View Sample Spark Code - RDDs
• Word Count for Spark written Scala
16

17. 47deg.com
View Sample Spark Code - Data Frames
• Tweet language count using Data Frames written in
Scala
17

18. Scala Impact on Spark
18

19. 47deg.com
Spark Impact on Scala
• Written in Scala
• Scala is primary API
• Must use Scala to extend
• Source code as documentation: Scala code
• Design from Scala
• functional programming
• immutable collections
• Spark is a Scala DSL
19

20. 47deg.com
Spark Application Language Choices - 1
• 71% Scala, 58% Python
• Code length similar
• Scala faster for RDDs (no need to move serialized
data between JVM and Python)
• Scala is generally faster
• Scala provides strong typing
• Compile time checks
• Code is easier to understand
• Types help when writing and maintaining code
20

21. 47deg.com
Spark Application Language Choices- 2
• In addition to Scala and Python also supports Java and R
• If you want to build scalable high performance production
code based on Spark
• R by itself is too specialized
• Python is too slow
• Java is tedious to write and maintain
• Scala is “just right”
21

22. 47deg.com
Transformations: Parallel versus Serial
• Scala has operations that traverse sequence elements in serial
order: foldLeft, scanLeft
• Spark RDDs can only process sequence elements in parallel
• Enables full use of multiple works with multiple cores
• Serial operations like foldLeft would be slow compared to
current parallel operations
• But there are cases where foldLeft or scanLeft is really needed
• Example: time series where early event can affect later event
• Example: Sherlock, word count by story
• So should Spark add foldLeft or similar operators?
22

23. 47deg.com
Sample Spark Internal Code
• Lets examine the internal code in Spark
• Spark Context
• RDD
23

24. Spark Impact on Scala
24

25. 47deg.com
Pair RDD’s for Scala
25
Scala Sequence Key Value
Unique Set Map
Duplicate Seq
???
Odersky
Spark Sequence Key Value
Unique
Duplicate RDD Pair RDD
.distinct

26. 47deg.com
Lazy Evaluation
• Strict evaluation (Scala)
• Each transformation is evaluated as it is seen
• Easy to understand and debug
• Lazy evaluation (Spark)
• Transformations are collected into a linage graph
• Evaluated only when a final action is applied
• Allows cross transformation optimization
26

27. 47deg.com
Lazy Evaluation in Scala
• Scala has Stream (a kind of sequence)
• Elements are evaluated in order as needed
• Allows infinite collections
• But no cross transform optimization
• Lazy Spark evaluation does all elements at once
• Enables parallelism
• Scala may add lazy versions of its collections that are
more like Spark (Odersky)
27

28. 47deg.com © Copyright 2015 47 Degrees
Spark Clusters - Serialization
28
Your Spark App
Spark Context
Cluster Manager
Worker Node
Executer
Task Task
Spark Driver
Worker Node
Executer
Task Task
...
Static Code
Jar File
Dynamic Code
Closures

29. 47deg.com
Dynamic Code Serialization
• Depends on values known at run-time
• Often a function passed to Spark transformations such
as map, filter, and flatMap
• Sent from driver to workers
• Serialized on driver to byte array
• Deserialized on each work
29

30. 47deg.com
Serialization of Closures
• Often include more than needed (or expected)
• Demo
• Scala may become smarter about including less in
closure serialization (Odersky)
30

31. 47deg.com
Datasets
• DataFrames are becoming the focus of Spark
optimization
• DataFrames are dynamically typed
• Scala API would be nicer if there was static typing
• Datasets (new experimental in Spark 1.6) attempt to
solve this
• Sample Code
• It would be nice if DataBricks and Typesafe could work
together to produce a better solution
31

32. Scala and Spark in
Seattle
32

33. 47deg.com
Scala and Spark in Seattle
• Seattle Meetups
• Scala at the Sea Meetup (over 1000 members)
http://www.meetup.com/Seattle-Scala-User-Group/
• Seattle Spark Meetup (over 1400 members)
http://www.meetup.com/Seattle-Spark-Meetup/
• 47 DegreesTraining (Seattle and Worldwide)
http://www.47deg.com/events#training
• Typesafe Scala Training: Scala, Akka
• Spark Training: Programming Spark with Scala
• UW Scala Professional Certificate Program
http://www.pce.uw.edu/certificates/scala-functional-reactive-programming.html
33

34. Summary
34

35. 47deg.com
Summary
• Scala and Spark are great technologies for big data applications
• Both are functional
• Both have immutable Data
• Both can be used to build very large systems
• Typesafe and Databricks evolving relationship
• Databricks is a consumer of Typesafe components
• Typesafe now supports Spark in addition to the Scala
compiler and other Scala components
• They are working together to evolve toward ever more
coordinated and integrated ecosystem for big data
• Martin Odersky - Spark — The Ultimate Scala Collections
35

36. Questions
36