Processing Live data with Spark Streaming

1. SPARK STREAMING
... IN 10 MINUTES
sven.tessmann@comsysto.com
@setema

2. WHY STREAMING?

3. FROM "DATA AT REST"
TOWARDS "DATA IN MOTION"

4. LinkedIn processes 1.1 Trillion Messages a day
(2015, using Kafka)
Microsoft processes 10TB of O ce 365 Event Data a day
(2015, using Spark)

5. "FAST DATA"

6. GENERATE
"BUSINESS REALTIME"
INSIGHTS

7. WHAT ABOUT SPARK
STREAMING?

8. "scalable, high-throughput, fault-tolerant
stream processing framework"

9. Spark Ecosystem

10. DISCRETIZED STREAM (DSTREAM)
a sequence of distributed datasets (RDDs) representing a
continuous stream of data

12. DStream Transformations

13. Execution Model

14. LET'S SEE SOME CODE

15. THANK YOU!