Over the past few months, the Apache Flink and Apache Beam communities have been busy developing an industry leading solution to author batch and streaming pipelines with Python. This was made possible by a significant effort to revamp Beam’s portability framework, build the corresponding Flink Runner, and simplify Flink’s artifact distribution & deployment mechanisms. What is the “killer big-data app” enabled by this integration: production TensorFlow pipelines. Building production machine learning pipelines that process large distributed data sets can get complex. In this talk, we will describe a set of open source libraries developed at Google, that simplify and unify pre and post processing stages for a production TensorFlow pipeline. These libraries are authored on Beam’s python SDK, and can be run on Apache Flink at scale. Last, but not least, we will describe how Beam & Flink aim to bring the power of big-data to newer audiences, in particular, developers of the Go programming language.

1. © 2017 Google Inc. All rights reserved.
Expanding the horizons of Big-Data
Apache Flink
+
Apache Beam
Presenter:
Anand Iyer
Product Manager, Google Cloud

2. © 2017 Google Inc. All rights reserved.
Rich history of collaboration
Unified
Batch & Streaming
Comprehensive
streaming
semantics &
correctness
Streaming SQL
(w/ Apache
Calcite)
Unified
Batch & Streaming
Comprehensive
streaming
semantics &
correctness
Streaming SQL
(w/ Apache Calcite)

3. © 2017 Google Inc. All rights reserved.
Flexible Big-Data Platform for Batch & Streaming
Java Python...
Machine
Learning
Genomics ...
Time
Series
...
Horizontal Framework in multiple languages
Vertical
Solutions
via domain
specific
libraries &
tools
...

4. © 2017 Google Inc. All rights reserved.
Making the power of Flink available in multiple languages

5. © 2017 Google Inc. All rights reserved.
The Apache Beam Model

6. © 2017 Google Inc. All rights reserved.
Cross-language Portability Framework
Language agnostic abstractions are at the core of the Beam Model
Language B
SDK
Language A
SDK
Language C
SDK
Runner 1 Runner 3Runner 2
The Beam Model
Language A Language CLanguage B
The Beam Model

7. © 2017 Google Inc. All rights reserved.
Prototype Flink Runner
❏ Works with Beam’s Python SDK
❏ Collaborators: Flink, Beam, Lyft,
GetInData
❏ https://issues.apache.org/jira/browse/
BEAM-2889
❏ For updates, please subscribe to
Apache Flink and Apache Beam Blogs

8. © 2017 Google Inc. All rights reserved.
Prototype Flink Runner
❏ Currently supports batch workloads
❏ Streaming capabilities on the roadmap

9. © 2017 Google Inc. All rights reserved.
Tools & libraries for compelling use case verticals

10. © 2017 Google Inc. All rights reserved.
Machine-Learning + Big-Data
Joined at the Hip

11. © 2017 Google Inc. All rights reserved.

12. © 2017 Google Inc. All rights reserved.
ML
Code
Because, in addition to the actual ML...

13. © 2017 Google Inc. All rights reserved.
ML
Cod
...you have to worry about so much more.
Configuration
Data Collection
Data
Verification
Feature Engineering
Process Management
Tools
Analysis Tools
Machine
Resource
Management
Serving
Infrastructure
Monitoring
ML
Code

14. © 2017 Google Inc. All rights reserved.
TensorFlow Transform
Consistent In-Graph Transformations in Training and Serving

15. © 2017 Google Inc. All rights reserved.
Typical ML Pipeline
batch processing
During training
“live” processing
During serving
data request

16. © 2017 Google Inc. All rights reserved.
Typical ML Pipeline
batch processing
During training
“live” processing
During serving
data request

17. © 2017 Google Inc. All rights reserved.
TensorFlow Transform
tf.Transform batch processing
During training
transform as tf.Graph
During serving
data request

18. © 2017 Google Inc. All rights reserved.
Scale to ... Bag of Words / N-Grams
Bucketization Feature Crosses
tft.ngrams
tft.string_to_int
tf.string_split
tft.scale_to_z_score
tft.apply_buckets
tft.quantiles
tft.string_to_int
tf.string_join
...
Rich collection of pre-implemented transforms

19. © 2017 Google Inc. All rights reserved.
Apply another TensorFlow Model
tft.apply_saved_model
Scale to ... Bag of Words / N-Grams
Bucketization Feature Crosses
tft.ngrams
tft.string_to_int
tf.string_split
tft.scale_to_z_score
tft.apply_buckets
tft.quantiles
tft.string_to_int
tf.string_join
...
Rich collection of pre-implemented transforms

20. © 2017 Google Inc. All rights reserved.
github.com/tensorflow/transform

21. © 2017 Google Inc. All rights reserved.
TensorFlow Model Analysis
Scalable, sliced, and full-pass metrics

22. © 2017 Google Inc. All rights reserved.
Analyzing model mistakes by subgroup
Specificity (False Positive Rate)
Sensitivity(TruePositiveRate)
ROC Curve
All groups
Learn more at ml-fairness.com

23. © 2017 Google Inc. All rights reserved.
Analyzing model mistakes by subgroup
Learn more at ml-fairness.com
Specificity (False Positive Rate)
Sensitivity(TruePositiveRate)
ROC Curve
All groups
Group A
Group B

24. © 2017 Google Inc. All rights reserved.
TensorFlow Model Analysis

25. © 2017 Google Inc. All rights reserved.
github.com/tensorflow/model-analysis
https://medium.com/tensorflow/introducing-tensorflow-model-anal
ysis-scaleable-sliced-and-full-pass-metrics-5cde7baf0b7b

26. © 2017 Google Inc. All rights reserved.
Tensorflow Extended
https://youtu.be/vdG7uKQ2eKk

27. © 2017 Google Inc. All rights reserved.
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. KDD (2017).
https://youtu.be/fPTwLVCq00U

28. © 2017 Google Inc. All rights reserved.
Beam library for transforming and processing VCF files at scale
https://github.com/googlegenomics/gcp-variant-transforms
Big-Data in Genomics

29. © 2017 Google Inc. All rights reserved.
Come join us!
Complete the Flink
Runner
Add new language
SDKs.
Javascript anyone?
Build vertical
libraries & tools