At the StampedeCon 2015 Big Data Conference: Today if a byte of data were a gallon of water, in only 10 seconds there would be enough data to fill an average home, in 2020 it will only take 2 seconds. The Internet of Things is driving a tremendous amount of this growth, providing more data at a higher rate then we’ve ever seen. With this explosive growth comes the demand from consumers and businesses to leverage and act on what is happening right now. Without stream processing these demands will never be met, and there will be no big data and no Internet of Things. Apache Spark, and Spark Streaming in particular can be used to fulfill this stream processing need now and in the future. In this talk I will peel back the covers and we will take a deep dive into the inner workings of Spark Streaming; discussing topics such as DStreams, input and output operations, transformations, and fault tolerance.

1. Lifting the hood on Spark Streaming
Andrew Psaltis (@itmdata)
StampedeCon, July 15, 2015

2. Outline
• Introduction
• DStreams
• Thinking about time
• Recovery and Fault tolerance
• Conclusion

3. About Me
Andrew Psaltis
Data Engineer @ Shutterstock
Fun outside of Shutterstock:
• Sometimes ramble here: @itmdata
• Author of Streaming Data
• Dreaming about streaming since 2008
• Content provider for SkillSoft
• Lacrosse crazed

4. Introduction

5. Why Streaming?
“Without stream processing
there’s no big data and no
Internet of Things” – Dana
Sandu, SQLstream

6. Why Streaming?
• Operational Efficiency - 1 extra mph for a
locomotive on it’s daily route can lead to $200M
in saving (Norfolk Southern)
• Predict machine failure - GE monitors over
5500 assets from 70+ customer sites globally.
Can predict failure and determine when
something needs maintenance. Single gas
turbine produces ~ 1TB of data every hour.

7. Our shared problem
Today if a byte of data was 1
gallon of water we could fill an
average house in 10 seconds, by
2020 it will take only 2.

8. What is Spark Streaming?
• Provides efficient, fault-tolerant stateful stream
processing
• Provides a simple API for implementing complex
algorithms
• Integrates with Spark’s batch and interactive processing
• Integrates with other Spark extensions

9. High-level Architecture

10. DStreams

11. Discretized Streams (DStreams)
• The basic abstraction provided by Spark Streaming
• Continuous series of RDDs

12. DStreams
• 3 Things we want to do
• Ingest
• Transform
• Output

13. Input DStreams (Ingestion)
There are 3 ways to get data in:
• Basic sources
• Advanced sources
• Custom Sources

14. Basic Input DStreams
• Basic sources
• Built-in (file system, socket, Akka actors)
• Non-built in (Avro, CSV, …)
• Not reliable

15. Advanced Input DStreams
• Advanced sources
• Twitter, Kafka, Flume, Kinesis, MQTT, ….
• Require external library
• Maybe reliable or unreliable

16. Custom Input DStreams
• Implement two classes
• InputDStream
• Receiver

17. Custom Input DStream

18. Custom Receiver

19. Receiver Reliability
Two types of receivers
• Unreliable Receiver
• Reliable Receiver

20. Receiver Reliability
Unreliable Receiver
• Simple to implement
• No fault-tolerance
• Data loss when receiver fails

21. Receiver Reliability
Reliable Receiver
• Complexity depends on the source
• Strong fault-tolerance guarantees (zero data loss)
• Data source must support acknowledgement

22. Input DStream and Receiver

23. Creating DStreams
2 Ways to create a DStream
• Input – a streaming source
• Transforming a DStream

24. Creating a DStream viaTransformation
• Transformations modify data from one DStream to another
• Two general classifications:
• Standard RDD operations – map, countByValue,
reduceByKey, join,…
• Stateful operations – window, updateStateByKey,
transform, countByValueAndWindow, …
map
Input DStream Output DStream

25. Transforming the input - Standard Operation
val
myStream
=
createCustomStream(streamingContext)
val
events
=
myStream.map(….)
batch
@
t+1
batch
@
t
batch
@
t+2
My
InputDStream
MyReceiver
map map map
events
DStream

26. Stateful Operation - UpdateStateByKey
Provides a way for you to maintain arbitrary state
while continuously updating it.
• For example – In-Session Advertising,Tracking
twitter sentiment

27. Stateful Operation - UpdateStateByKey
Need to do two things to leverage it:
• Define the state – this can be any arbitrary data
• Define the update function – this needs to
know how to update the state using the
previous state and new values
Requires Checkpointing to be configured

28. Using updateStateByKey
Maintain per-user mood as state, and update it with
his/her tweets
moods
=
tweets.updateStateByKey(tweet
=>
updateMood(tweet))
updateMood(newTweets,
lastMood)
=>
newMood
tweets
t-­‐1
t
t+1
t+2
t+3
moods

29. Transform
Allows arbitrary RDD-to-RDD functions to be
applied on a DStream
transform
(transformFunc:
RDD[T]
=>
RDD[U]):
DStream[U]
Example: We want to eliminate “noise” words from
crawled documents:
val
noiseWordRDD
=
ssc.sparkContext.newAPIHadoopRDD(...)
val
cleanedDStream
=
crawledCorpus.transform(rdd
=>
{
rdd.join(noiseWordRDD).filter(...)})

30. Joining streams
Allows you to combine two DStreams that share a
key and produce a new DStream
join(other:
DStream(K,V)):
DStream[K,(V,W)]
Example: We want to group Fitbit and MapMyRun
streams
val
musicBits
=
fitBitStream.join(mapMyRunStream)

31. Outputting data
val
myStream
=
createCustomStream(streamingContext)
val
events
=
myStream.map(….)
events.countByValue().foreachRDD{…}
batch
@
t+1
batch
@
t
batch
@
t+2
myStream
Custom
Receiver
map map map
events
DStream
foreachRDD* foreachRDD* foreachRDD*

32. DStreams
• We have our 3 Things
• Ingest – Basic, Advanced and Custom
• Transforming – Map, Join,Transform
• Output – foreachRDD
• How do we go from that to a Spark Job?

33. From Streaming Program to Spark jobs
M
T
C
FE
DStream
Graph
M
B
C
A
RDD
Graph
Block
RDD
with
data
received
in
last
batch
interval
1
Spark
job
myStream
=
createCustomStream
events
=
myStream.map(…)
events.countByValue()
.foreachRDD{…}

34. Thinking about time

35. Thinking about time
• Windowing – Tumbling, Sliding
• Stream time vs. Event time
• Out of order data

36. Windowing
• Common Types
• Tumbling
• Sliding

37. Tumbling (Count) Windowing

38. Tumbling (temporal) Windowing

39. Sliding Window

40. Spark Streaming -- Sliding Windowing
• Two types supported:
• Non-Incremental
• Incremental

41. Non-Incremental Sliding Windowing
reduceByKeyAndWindow((a,b)=>(a + b),Seconds(5), Seconds(1))

42. Incremental Sliding Windowing
reduceByKeyAndWindow((a,b) => (a + b), (a,b) => (a-b),
Seconds(5), Seconds(1))

43. StreamTime vs. EventTime
• Stream time -- the time when the record
arrives into the streaming system.
• Event time – the time that the event was
generated, not when it entered the system.
• Spark Streaming uses stream time

44. Time Skew

45. Out of order data
• Does it matter to your application?
• How do you deal with it?
If you perform analysis of ad impressions and your
computation is running at 12:01 AM, which day gets
attribution for the impression?

46. Handling Out of Order Data
Imagine we want to track ad impressions between
time t and t +1
Continuous Analytics Over Discontinuous Streams
http://www.eecs.berkeley.edu/~franklin/Papers/sigmod10krishnamurthy.pdf

47. Recovery and FaultTolerance

48. Recovery
• Checkpointing
• Metadata checkpointing
• Data checkpointing

49. Recovery
Without
With

50. Recovery
• Too frequent: HDFS writing will slow things down
• Too infrequent: Lineage and task sizes grow
• Default setting: Multiple of batch interval at least 10
seconds
• Recommendation: checkpoint interval of 5 - 10
times of sliding interval

51. FaultTolerance
• All properties of RDDs still apply
• We are trying to protect two things
• Failure of a Worker
• Failure of the Driver Node
• Semantics
• At most once
• At least once
• Exactly once
• Where we need to think about it
• Receivers
• Transformations
• Output

52. Conclusion
• Introduction
• High-level Architecture
• DStreams
• Thinking about time
• Recovery and Fault tolerance

53. Thank y u
Andrew Psaltis
@itmdata
psaltis.andrew@gmail.com
https://www.linkedin.com/in/andrewpsaltis