Batch and Streaming Data Processing and Vizualize 300Tb in 5 Seconds meetup on April 18th, 2016 (http://www.meetup.com/Big-things-are-happening-here/events/229532500)

1. Section Slide Template Option 2
Put your subtitle here. Feel free to pick from the handful of pretty Google colors available to you.
Make the subtitle something clever. People will think it’s neat.
Dataflow
A Unified Model for Batch and Streaming
Data Processing
Vadim Solovey
Google Developer Expert & Trainer
vadim@doit-intl.com
Shahar Frank
Cloud Solutions Architect
srfrnk@doit-intl.com

2. Goals:
Write interesting
computations
Run in both batch &
streaming
Use custom timestamps
Handle late data
https://commons.wikimedia.org/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg

3. Data Shapes
Google’s Data Processing Story
The Dataflow Model
Agenda
Demo: Google Cloud Dataflow
1
2
3
4

4. Data Shapes1

5. Data...

6. ...can be big...

7. ...really, really big...
Tuesday
Wednesday
Thursday

8. ...maybe even infinitely big...
9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00

9. … with unknown delays.
9:008:00 14:0013:0012:0011:0010:00
8:00
8:008:00
8:00

10. 1 + 1 = 2
Completeness Latency Cost
$$$
Data Processing Tradeoffs

11. Requirements: Billing Pipeline
Completeness Low Latency Low Cost
Important
Not Important

12. Requirements: Live Cost Estimate Pipeline
Completeness Low Latency Low Cost
Important
Not Important

13. Requirements: Abuse Detection Pipeline
Completeness Low Latency Low Cost
Important
Not Important

14. Requirements: Abuse Detection Backfill Pipeline
Completeness Low Latency Low Cost
Important
Not Important

15. Google’s Data Processing Story2

16. 20122002 2004 2006 2008 2010
MapReduce
GFS Big Table
Dremel
Pregel
FlumeJava
Colossus
Spanner
2014
MillWheel
Dataflow
2016
Google’s Data-Related Papers

17. (Produce)
MapReduce: Batch Processing
(Prepare)
(Shuffle)
Map
Reduce

18. FlumeJava: Easy and Efficient MapReduce Pipelines
● Higher-level API with simple data
processing abstractions.
○ Focus on what you want to do to
your data, not what the
underlying system supports.
● A graph of transformations is
automatically transformed into an
optimized series of MapReduces.

19. MapReduce
Batch Patterns: Creating Structured Data

20. MapReduce
Batch Patterns: Repetitive Runs
Tuesday
Wednesday
Thursday

21. MapReduce
Tuesday [11:00 - 12:00)
[12:00 - 13:00)
[13:00 - 14:00)
[14:00 - 15:00)
[15:00 - 16:00)
[16:00 - 17:00)
[18:00 - 19:00)
[19:00 - 20:00)
[21:00 - 22:00)
[22:00 - 23:00)
[23:00 - 0:00)
Batch Patterns: Time Based Windows

22. MapReduce
TuesdayWednesday
Batch Patterns: Sessions
Jose
Lisa
Ingo
Asha
Cheryl
Ari
WednesdayTuesday

23. MillWheel: Streaming Computations
● Framework for building low-latency
data-processing applications
● User provides a DAG of
computations to be performed
● System manages state and
persistent flow of elements

24. Streaming Patterns: Element-wise transformations
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time

25. Streaming Patterns: Aggregating Time Based Windows
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time

26. 11:0010:00 15:0014:0013:0012:00Event Time
11:0010:00 15:0014:0013:0012:00
Processing
Time
Input
Output
Streaming Patterns: Event Time Based Windows

27. Streaming Patterns: Session Windows
Event Time
Processing
Time
11:0010:00 15:0014:0013:0012:00
11:0010:00 15:0014:0013:0012:00
Input
Output

28. ProcessingTime
Event Time
Skew
Event-Time Skew
Watermark
Watermarks describe event time
progress.
"No timestamp earlier than the
watermark will be seen"
Often heuristic-based.
Too Slow? Results are delayed.
Too Fast? Some data is late.

29. Streaming or Batch?
1 + 1 = 2 $$$
Completeness Latency Cost
Why not both?

30. The Dataflow Model3

31. What are you computing?
Where in event time?
When in processing time?
How do refinements relate?

32. What are you computing?
• A Pipeline represents a graph
of data processing
transformations
• PCollections flow through the
pipeline
• Optimized and executed as a
unit for efficiency

33. What are you computing?
• A PCollection<T> is a collection
of data of type T
• Maybe be bounded or unbounded
in size
• Each element has an implicit
timestamp
• Initially created from backing data
stores

34. What are you computing?
PTransforms transform PCollections into other
PCollections.
What Where When How
Element-Wise Aggregating Composite

35. Example: Computing Integer Sums
// Collection of raw log lines
PCollection<String> raw = ...;
// Element-wise transformation into team/score pairs
PCollection<KV<String, Integer>> input =
raw.apply(ParDo.of(new ParseFn()))
// Composite transformation containing an aggregation
PCollection<KV<String, Integer>> output = input
.apply(Sum.integersPerKey());
What Where When How

36. Example: Computing Integer Sums
What Where When How

37. What Where When How
Example: Computing Integer Sums

38. Key
2
Key
1
Key
3
1
Fixed
2
3
4
Key
2
Key
1
Key
3
Sliding
1
2
3
5
4
Key
2
Key
1
Key
3
Sessions
2
4
3
1
Where in Event Time?
• Windowing divides
data into event-
time-based finite
chunks.
• Required when
doing aggregations
over unbounded
data.
What Where When How

39. PCollection<KV<String, Integer>> output = input
.apply(Window.into(FixedWindows.of(Minutes(2))))
.apply(Sum.integersPerKey());
What Where When How
Example: Fixed 2-minute Windows

40. What Where When How
Example: Fixed 2-minute Windows

41. What Where When How
When in Processing Time?
• Triggers control
when results are
emitted.
• Triggers are often
relative to the
watermark.
ProcessingTime
Event Time
Watermark

42. PCollection<KV<String, Integer>> output = input
.apply(Window.into(FixedWindows.of(Minutes(2)))
.trigger(AtWatermark())
.apply(Sum.integersPerKey());
What Where When How
Example: Triggering at the Watermark

43. What Where When How
Example: Triggering at the Watermark

44. What Where When How
Example: Triggering for Speculative & Late Data
PCollection<KV<String, Integer>> output = input
.apply(Window.into(FixedWindows.of(Minutes(2)))
.trigger(AtWatermark()
.withEarlyFirings(AtPeriod(Minutes(1)))
.withLateFirings(AtCount(1))))
.apply(Sum.integersPerKey());

45. What Where When How
Example: Triggering for Speculative & Late Data

46. What Where When How
How do Refinements Relate?
• How should multiple outputs per window
accumulate?
• Appropriate choice depends on consumer.
Firing Elements
Speculative 3
Watermark 5, 1
Late 2
Total Observ 11
Discarding
3
6
2
11
Accumulating
3
9
11
23
Acc. & Retracting
3
9, -3
11, -9
11

47. PCollection<KV<String, Integer>> output = input
.apply(Window.into(Sessions.withGapDuration(Minutes(1)))
.trigger(AtWatermark()
.withEarlyFirings(AtPeriod(Minutes(1)))
.withLateFirings(AtCount(1)))
.accumulatingAndRetracting())
.apply(new Sum());
What Where When How
Example: Add Newest, Remove Previous

48. What Where When How
Example: Add Newest, Remove Previous

49. 1. Classic Batch 2. Batch with Fixed
Windows
3. Streaming 5. Streaming with
Retractions
4. Streaming with
Speculative + Late Data
Customizing What Where When How
What Where When How

50. Demo: Google Cloud Dataflow4

51. Cloud DataflowA fully-managed cloud service and
programming model for batch and
streaming big data processing.
Google Cloud Dataflow

52. Open Source SDKs
● Used to construct a Dataflow pipeline.
● Java available now. Python in the open beta.
● Pipelines can run…
○ On your development machine
○ On the Dataflow Service on Google Cloud Platform
○ On third party environments like Spark or Flink.

53. Fully Managed Dataflow Service
Runs the pipeline on Google Cloud Platform. Includes:
● Graph optimization: Modular code, efficient execution
● Smart Workers: Lifecycle management, Autoscaling, and
Smart task rebalancing
● Easy Monitoring: Dataflow UI, Restful API and CLI,
Integration with Cloud Logging, etc.

54. Demo Time!

55. user11_BisqueEmu,BisqueEmu,11,1460980305000,2016-04-18 04:59:27.936
user0_BisqueQuokka,BisqueQuokka,15,1460980768000,2016-04-18 04:59:28.473
user11_ArmyGreenNumbat,ArmyGreenNumbat,5,1460980768000,2016-04-18 04:59:28.473
user4_AquaEmu,AquaEmu,19,1460980768000,2016-04-18 04:59:28.473
user5_BeigeDingo,BeigeDingo,16,1460980768000,2016-04-18 04:59:28.473
user1_ArmyGreenKookaburra,ArmyGreenKookaburra,9,1460980768000,2016-04-18 04:59:28.473
user7_AquaDingo,AquaDingo,18,1460980768000,2016-04-18 04:59:28.473
user7_AmberEmu,AmberEmu,11,1460980768000,2016-04-18 04:59:28.473
user1_BeigeDingo,BeigeDingo,6,1460980768000,2016-04-18 04:59:28.473
user2_ArmyGreenKookaburra,ArmyGreenKookaburra,8,1460980768000,2016-04-18 04:59:28.473
user10_AzureBandicoot,AzureBandicoot,12,1460980768000,2016-04-18 04:59:28.473
user1_AzureBandicoot,AzureBandicoot,12,1460980768000,2016-04-18 04:59:28.473
Robot-12,BeigeQuokka,12,1460980768000,2016-04-18 04:59:28.473
user3_MagentaAntechinus,MagentaAntechinus,5,1460980768000,2016-04-18 04:59:28.473
user6_AmberEmu,AmberEmu,15,1460980768000,2016-04-18 04:59:28.473
Data

56. Wrote Reusable
PTransforms
Ran the PTransform in
streaming mode
Used event time, not
processing time
Easily adjusted windowing
and late data settings

57. Learn More!
● The Dataflow Model @VLDB 2015
http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
● Dataflow SDK for Java
https://github.com/GoogleCloudPlatform/DataflowJavaSDK
● Google Cloud Dataflow on Google Cloud Platform
http://cloud.google.com/dataflow (Free Trial!)