This talk shows how to productionize Machine Learning models in mission-critical and scalable real time applications by leveraging Apache Kafka as streaming platform. The talk discusses the relation between Machine Learning frameworks such as TensorFlow, DeepLearning4J or H2O and the Apache Kafka ecosystem. A live demo shows how to build a mission-critical Machine Learning environment leveraging different Kafka components: Kafka messaging and Kafka Connect for data movement from and into different sources and sinks, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy. Updated slide deck and talk from September 2018 at ApacheCon Montreal.

1. 1Confidential
Machine Learning at Extreme Scale with the
Apache Kafka Open Source Ecosystem
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
How to build a Scalable, Mission-Critical Machine Learning Infrastructure

2. 2Apache Kafka and Machine Learning – Kai Waehner
Poll
In which of these environments, if at all,
are you using Apache Kafka today?
1. On premise
2. A public cloud
3. Our own virtual private cloud
4. We do not currently use Apache Kafka

3. 3Apache Kafka and Machine Learning – Kai Waehner
What is eXtreme Scale?
• High Volume of Events (millions, billions, trillions)
• Big Data Sets for Analytics (GB, TB, PB)
• Dynamic Scalability for Training (minutes, hours, days)
• Real Time Prediction Process for Deployment (ms)
• Hybrid Deployments (different frameworks and clouds)

4. 4Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

5. 5Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

6. 6Apache Kafka and Machine Learning – Kai Waehner
Machine Learning
... allows computers to find hidden insights without being
explicitly programmed where to look.
Machine Learning
• Decision Trees
• Naïve Bayes
• Clustering
• Neural Networks
• etc.
Deep Learning
• CNN
• RNN
• Autoencoder
• etc.

7. 7Apache Kafka and Machine Learning – Kai Waehner
Real World Examples of Machine Learning
Spam Detection
Search Results +
Product Recommendation
Picture Detection
(Friends, Locations, Products)
Your Company
The Next Disruption:
Google Beats Go Champion

8. 8Apache Kafka and Machine Learning – Kai Waehner
Leverage Machine Learning to Analyze and Act on Critical Business Moments
Seconds Minutes Hours
Price
Optimization
Predictive
Maintenance
Fraud
Detection
Cross Selling
Transportation
Rerouting
Customer
Service
Inventory
Management
Windows of Opportunity

9. 9Apache Kafka and Machine Learning – Kai Waehner
Live Demo – Building an Analytic Model
Neural Networks in Action
http://playground.tensorflow.org/

10. 10Apache Kafka and Machine Learning – Kai Waehner
Languages, Frameworks and Tools for Machine Learning
There is no Allrounder à ML-independent infrastructure needed!
Portable Format
for Analytics (PFA)

11. 11Apache Kafka and Machine Learning – Kai Waehner
Machine Learning with H2O.ai
H2O Engine
R / Python /
Scala / Flow UI
Java Code

12. 12Apache Kafka and Machine Learning – Kai Waehner
Live Demo – Building an Analytic Model
Use Case:
Airline Flight Delay Prediction
Machine Learning Algorithm:
Deep Learning
using Neural Networks
Technology:
H2O.ai, TensorFlow

13. 13Apache Kafka and Machine Learning – Kai Waehner
H2O Deep Water (TensorFlow, MXNet, …)
https://h2o-release.s3.amazonaws.com/h2o/rel-vapnik/1/
docs-website/h2o-docs/booklets/DeepWaterBooklet.pdf
Deep Water
(H2O + TensorFlow)
Pre-Defined Networks
+
User-Defined Networks

14. 14Apache Kafka and Machine Learning – Kai Waehner
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

15. 15Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

16. 16Apache Kafka and Machine Learning – Kai Waehner
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka – The Rise of a Streaming Platform

17. 17Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka at Scale
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921
https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture
(2018)
(2018)

18. 18Apache Kafka and Machine Learning – Kai Waehner
Netflix’ Meson: Automation Engine for ML Pipelines
https://www.infoq.com/presentations/netflix-ml-meson

19. 19Apache Kafka and Machine Learning – Kai Waehner
Uber’s internal ML-as-a-Service Platform
https://eng.uber.com/michelangelo
• Cover the end-to-end ML
workflow: manage data,
train, evaluate, and deploy
models, make predictions,
and monitor predictions
• Supports various AI
technologies: Traditional
ML models, time series
forecasting, and deep
learning

20. 20Apache Kafka and Machine Learning – Kai Waehner
Paypal: Real Time Fraud Detection at Scale
https://www.infoq.com/presentations/paypal-data-service-fraud
“Scalable model / framework independent infrastructure for fraud detection”

21. 21Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning

22. 22Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Source Ecosystem as Infrastructure for Machine Learning
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go / .NET / Python
Kafka Producer
KSQL
Kafka
Streams

23. 23Apache Kafka and Machine Learning – Kai Waehner
Poll
Which components of the Apache Kafka open source ecosystem do you use?
1) Kafka Connect
2) Kafka Streams
3) Confluent Open Source
(Schema Registry, REST Proxy, KSQL, non-Java Clients, …)
4) Other Open Source Add-Ons (Kafka Manager, node-rdkafka Client, …)
5) Only Kafka for Messaging / Storage + Java Producers and Consumers

24. 24Apache Kafka and Machine Learning – Kai Waehner
Replay-ability – A log never forgets!
Time
Model B Model XModel A
Producer
Distributed Commit Log
Different models with same data
Different ML Frameworks
AutoML compatible
A/B Testing

25. 25Apache Kafka and Machine Learning – Kai Waehner
AutoML à No Data Scientist available for the ML Tasks?
Hidden Technical Debt in Machine Learning Systems
http://slideplayer.com/slide/10575150/
“One-Click Data-In
Model-Out simplicity”
This talk does not focus on building models, but scalable infrastructure for ML

26. 26Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

27. 27Apache Kafka and Machine Learning – Kai Waehner
Kafka Connect for Data Integration
“Kafka benefits under the hood”
Out-of-the-Box Connectivity
Data Format Conversion
Simple Message Transformation
Connect
Syslog
(“you name the interface”)

28. 28Apache Kafka and Machine Learning – Kai Waehner
Kafka Connect Configuration for MQTT Connector
curl -s -X POST -H 'Content-Type: application/json' http://localhost:8083/connectors -d '{
"name" : "mqtt-source",
"config" : {
"connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector",
"tasks.max" : "1",
"mqtt.server.uri" : "tcp://127.0.0.1:1883",
"mqtt.topics" : "temperature",
"kafka.topics" : "mqtt.",
"transforms":"filter",
"transforms.filter.type":"com.github.kaiwaehner.kafka.connect.smt.StringFilter",
"transforms.filter.topic.format":"fraud"
}
}'

29. 29Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams / KSQL for Data Preprocessing
Streams
“Kafka benefits under the hood”
Streaming ETL
Same Pipeline for Training and Serving
Input Data
Output Data

30. 30Apache Kafka and Machine Learning – Kai Waehner
KSQL – Continuous Query for Streaming ETL
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM
clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';

31. 31Apache Kafka and Machine Learning – Kai Waehner
Confluent Schema Registry for Message Validation
Input Data
Schema
Registry
Output Data“Kafka benefits under the hood”
Schema Definition + Evolution
Forward and Backward Compatibility

32. 32Apache Kafka and Machine Learning – Kai Waehner
Kafka Connect for Data Ingestion
Connect
“Kafka benefits under the hood”
Out-of-the-Box Connectivity
Data Format Conversion
Simple Message Transformation

33. 33Apache Kafka and Machine Learning – Kai Waehner
Model Training
Benefits of Public Cloud
Extreme Scale
Dynamic Instances
Special Hardware

34. 34Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

35. 35Apache Kafka and Machine Learning – Kai Waehner
Model Serving / Inference / Deployment / Scoring
Kafka Streams
KSQL
“Kafka benefits under the hood”
Continuous Stream Processing
Reuse Preprocessing Logic from Ingestion Pipeline
Serving within the application (not via REST interface)
Predictions in real time
Streams
Input Event
Prediction

36. 36Apache Kafka and Machine Learning – Kai Waehner
Stream Processing
Data at Rest Data in Motion

37. 37Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source)

38. 38Apache Kafka and Machine Learning – Kai Waehner
Stream Processing Pipeline
APIs
Adapters /
Channels
Integration
Messaging
Stream
Ingest
Transformation
Aggregation
Enrichment
Filtering
Stream
Preprocessing
Process
Management
Analytics
(Real Time)
Applications
& APIs
Analytics /
DW Reporting
Stream
Outcomes
• Contextual Rules
• Windowing
• Patterns
• Analytics
• Machine Learning
• …
Stream
Analytics
Index / SearchNormalization
Applying an Analytic Model
is just a piece of the puzzle!

39. 39Apache Kafka and Machine Learning – Kai Waehner
A complete streaming microservices, ready for production at large-scale
Word
Count
App configuration
Define processing
(here: WordCount)
Start processing

40. 40Apache Kafka and Machine Learning – Kai Waehner
Why KSQL?
Population
CodingSophistication
Realm of Stream Processing
New, Expanded Realm
BI
Analysts
Core
Developers
Data
Engineers
Core Developers
who don’t like
Java
Kafka
Streams
KSQL

41. 41Apache Kafka and Machine Learning – Kai Waehner
Trade-Offs
• subscribe()
• poll()
• send()
• flush()
• mapValues()
• filter()
• punctuate()
• Select…from…
• Join…where…
• Group by..
Flexibility Simplicity
Kafka Streams KSQL
Kafka / Consumer
Producer

42. 42Apache Kafka and Machine Learning – Kai Waehner
When to use Kafka Streams or KSQL for Stream Processing?

43. 43Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams (shipped with Apache Kafka) / KSQL (Confluent Open Source)
Map, filter,
aggregate,
apply analytic model,
„any business logic“
Input Stream
(Kafka Topic)
Kafka Cluster
Output Stream
(Kafka Topic)
Kafka Cluster
Stream Processing
Microservice
(Kafka Streams / KSQL)
Deployed Anywhere
Java App, Docker,
Kubernetes, Mesos,
“you-name-it”

44. 44Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams and KSQL
Ok. Ok. Ok.
are viable for S / M / L / XL / XXL use cases

45. 45Apache Kafka and Machine Learning – Kai Waehner
Option 1: RPC communication to do model inference
Streams
Input Event
Prediction
Request
Response
Model Serving
TensorFlow Serving
RPC (e.g. gRPC, HTTP)

46. 46Apache Kafka and Machine Learning – Kai Waehner
Option 2: Model interference natively integrated into the App
Streams
Input Event
Prediction

47. 47Apache Kafka and Machine Learning – Kai Waehner
Stream Processing vs. Request-Response for Model Serving
Pros of a Model Server:
• Simple integration with existing technologies
and organizational processes
• Easier to understand if you come from non-
streaming world
• Later migration to real streaming is also
possible
• Model management built-in for different
models, versioning and A/B testing
Cons (== Pros of Deployment in the Streaming App):
• Worse latency as remote call instead of local
inference
• No offline inference (devices, edge processing, etc.)
• Coupling the availability, scalability, and
latency/throughput of your Kafka Streams
application with the SLAs of the RPC interface
• Side-effects (e.g., in case of failure) not covered by
Kafka processing (e.g., exactly once)

48. 48Apache Kafka and Machine Learning – Kai Waehner
Use Case:
Airline Flight Delay Prediction
Machine Learning Algorithm:
Neural Network
built with H2O and TensorFlow
Streaming Platform:
Apache Kafka and Kafka Streams
Live Demo – Deployment of a Trained Model
Streams

49. 49Apache Kafka and Machine Learning – Kai Waehner
H2O.ai Model + Kafka Streams
Filter
Map
1) Create H2O DL model
2) Configure Kafka Streams Application
3) Apply H2O DL model to Streaming Data
4) Start Kafka Streams App

50. 50Apache Kafka and Machine Learning – Kai Waehner
Github Examples: Kafka + Machine Learning
https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/
https://github.com/kaiwaehner/kafka-streams-machine-learning-examples
1) git clone à 2) mvn clean package à 3) look at implementations and unit tests

51. 51Apache Kafka and Machine Learning – Kai Waehner
KSQL and Deep Learning (Autoencoder) for Fraud Detection
“CREATE STREAM FraudDetection AS
SELECT payment_id, applyFraudModel(payment_input)
FROM payment_engine;“
User Defined Function (UDF)
https://www.confluent.io/blog/build-udf-udaf-ksql-5-0
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot

52. 52Apache Kafka and Machine Learning – Kai Waehner
Use Case:
Anomaly Detection
(Payment Fraud Detection)
Machine Learning Algorithm:
Autoencoder built with H2O
Streaming Platform:
Apache Kafka and KSQL
Live Demo – Prebuilt Model Embedded in KSQL Function

53. 53Apache Kafka and Machine Learning – Kai Waehner
Github Examples: KSQL + Deep Learning
https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
https://github.com/kaiwaehner/ksql-fork-with-deep-learning-function
+ Kafka Connect
+ Elasticsearch

54. 54Apache Kafka and Machine Learning – Kai Waehner
Agenda
1) Added Business Value via Machine Learning
2) Apache Kafka Ecosystem as Infrastructure for Machine Learning
3) Data Ingestion and Preprocessing with Apache Kafka for Model Training
4) Model Inference in Real Time with Kafka Streams and KSQL
5) DevOps and Monitoring of a Machine Learning Infrastructure

55. 55Apache Kafka and Machine Learning – Kai Waehner
Automated Model Improvement with Apache Kafka and Kafka Streams
How to deploy a model
once, then monitor and
improve it
continuously?

56. 56Apache Kafka and Machine Learning – Kai Waehner
Automated Model Improvement with Apache Kafka and Kafka Streams
How to improve models?
1. Manual Update
2. Continuous Batch Updating
3. Real Time à Online Model Training
Your choice… All possible with Kafka!

57. 57Apache Kafka and Machine Learning – Kai Waehner
Caveats for Online Model Training
• Processes and infrastructure not ready
• Validation needed before production
• Slows down the system
• Only a few ML implementations à Build your own!
• Only possible for unsupervised ML (e.g. clustering)
• Many use cases do not need it
à Do it only when feasible!

58. 58Apache Kafka and Machine Learning – Kai Waehner
Continuous Batch Updating as “Best Feasible Option”
DevOps Pipeline
1. Apply the model online to make predictions
2. Collect data and train a new model
3. Automated Re-Deployment (e.g. via a Kafka Topic)
https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/
https://www.coveros.com/services/devops/

59. 59Apache Kafka and Machine Learning – Kai Waehner
Kubernetes – The Winner of the Container and DevOps Wars!
https://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html
http://techgenix.com/year-of-kubernetes/
Docker, Inc

60. 60Apache Kafka and Machine Learning – Kai Waehner
Kubernetes for Infrastructure Deployment
https://kubernetes.io/blog/2016/10/kubernetes-and-openstack-at-yahoo-japan
Stateful Backend
• Zookeeper and Kafka Broker Pods
• REST Proxy, Schema Registry
• Persistent Volumes
• Kubernetes Kafka / Confluent Operator
Stateless and Stateful Clients
• Java / .Net / Go / Python Kafka Clients
• Kafka Streams / KSQL Apps
• Scalability and Elasticity

61. 61Apache Kafka and Machine Learning – Kai Waehner
Monitoring the Infrastructure for Machine Learning
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go / .NET / Python
Kafka Producer
KSQL
Kafka
Streams
Control Center
Build vs. Buy
Hosted vs. Managed
Basic vs. Advanced

62. 62Apache Kafka and Machine Learning – Kai Waehner
Warning:
Early Stage with focus on TensorFlow Training, TensorFlow Serving, Jupyter…
Bigger ecosystem expected soon… Including Kafka components for ingestion, serving, monitoring…
Kubernetes Deployment of ML Workflows
https://github.com/kubeflow/kubeflow

63. 63Apache Kafka and Machine Learning – Kai Waehner
Poll
Which of the following use cases are you most
likely to utilize Kafka for over the next year?
1. Data pipeline (not real time, e.g. batch in Spark)
2. Data pipeline (real time)
3. Stream processing (e.g. Kafka Streams, KSQL)
4. Stream processing with machine learning
5. Other (like microservices, event sourcing, storage)

64. 64Apache Kafka and Machine Learning – Kai Waehner
Key Takeaways
à Data Scientist and Developers have to work together continuously (org + tech!)
à Mission critical, scalable production infrastructure is key for success of Machine Learning projects
à Apache Kafka Ecosystem + Cloud = Machine Learning at Extreme Scale
(Ingestion, Processing, Training, Inference, Monitoring)

65. 65Apache Kafka and Machine Learning – Kai Waehner
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Please contact me!