IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale

1Apache Kafka and Machine Learning – Kai Waehner
Industrial Internet of Things (IIoT) at Scale
Real Time Planning Optimization and Predictive Maintenance
with an Event Streaming Platform
Kai Waehner
kai.waehner@confluent.io
@KaiWaehner
Graham Ganssle, Ph.D.
graham@experoinc.com
@GrahamGanssle

Confluents - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔

3
Connected Intelligence (Cars, Machines, Robots, …)

5
Smart Retail and Customer 360

6
Intelligent Applications (Early Part Scrapping, Predictive Maintenance, …)

7
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
The world has changed!

8Best-of-breed Platforms, Partners and Services for Multi-cloud Streams
Private Cloud
Deploy on bare-metal, VMs,
containers or Kubernetes in your
datacenter with Confluent Platform
and Confluent Operator
Public Cloud
Implement self-managed in the public
cloud or adopt a fully managed service
with Confluent Cloud
Hybrid Cloud
Build a persistent bridge between
datacenter and cloud with
Confluent Replicator
Confluent
Replicator
VM
SELF MANAGED FULLY MANAGED

Data Lake
Batch
Analytics
Event
Streaming
Platform
Batch
Integration
Real Time Pre-
processing
Machine Sensors
Streaming Platform
Other Components
Real Time
Processing
(6b) All Data
(3)
Read Data
Optimization
/ Analytics
(5)
Deploy
Optimization
Model
(2)
Preprocess
Data
Model
Standard
based
Integration
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
Model Lite
Real Time App
Model Server
RPC
PLC Proprietary
based
Integration
Standard
Interface
Proprietary
Interface

Spark
Notebooks
(Jupyter)
Kafka
Cluster
Kafka
Connect
KSQL
Machine Sensors
Kafka Ecosystem
Other Components Real Time
Kafka Streams
Application
(Java / Scala)
(6b) All Data
(3)
Read Data
TensorFlow I/O
TensorFlow
(5)
Deploy Model
(2)
Preprocess
Data
TensorFlow
MQTT
File
HTTP
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
(C / librdkafka)
TensorFlow Lite
Real Time Kafka
App
TensorFlow
Serving
HTTP /
gRPC
(4)
Train Model
PLC
Beckhoff
S7
Modbus
OPC-UA
PLC4X
Connector
Kafka Connect
Standard
Interface
Proprietary
Interface

11
Confluent Platform
The Event Streaming Platform Built by the Original Creators of Apache Kafka®
Operations and Security
Development & Stream Processing
Apache Kafka
Confluent Platform
Mission-Critical Reliability
Complete Event
Streaming Platform
Freedom of Choice
Datacenter Public Cloud Confluent Cloud
Self-Managed Software Fully Managed Service

12
Confluent Platform Licensing
Open Source features
Apache Kafka®
Apache 2.0 License
Free. Unlimited Kafka brokers
Community support
Enterprise License (paid)
● Annual subscription
● 24x7 Confluent support
● Kafka Connect
● Kafka Streams
Apache ZooKeeper™
Clients
Ansible Playbooks
Community features
Connectors
Confluent Community License
Free. Unlimited Kafka brokers
Community support
REST Proxy
KSQL
Schema Registry
Commercial features
Connectors
Developer License
● Free
● Limited to 1 Kafka broker
● Community support
Evaluation License
● Free 30-day trial
● Unlimited Kafka brokers
● Community support
Control Center
Command Line Interface
Replicator
Auto Data Balancer
MQTT Proxy
Operator
Security Plugins
Role-Based Access Control (preview) ● Best-effort Confluent Support
New in CP 5.3

1313
Confluent Operator:
Apache Kafka on
Kubernetes made
simple
Run Apache Kafka and Confluent
Platform as a cloud-native application
on Kubernetes to minimize operating
complexity and increase developer
agility
Confluent Platform
Kubernetes
AWS Azure GCP
RH OpenShift Pivotal
On-Premises Cloud
Docker Images
Confluent Operator

1414
Confluent
Operator
Deploy to Production in
Minutes
Automated deployment of
Confluent Platform resources:
Brokers, ZooKeeper, Kafka Connect,
KSQL, Schema Registry, Control
Center, and Replicator
Automate Key Lifecycle
Operations
● Failover
● Automated rolling upgrades
● Elastic scalability
Deploy on Any Platform,
On-Prem or in the Cloud
Run at Scale with
Confidence
Operationalizes years of Confluent
Cloud experience into a proven,
enterprise-grade solution that you
can deploy without deep Kafka
expertise
Deploy Apache Kafka
and Confluent Platform
as a cloud-native system
on Kubernetes
Kubernetes Engine Elastic Container
Service for Kubernetes
Kubernetes Service
https://www.slideshare.net/KaiWaehner/c
onfluent-operator-as-cloudnative-kafka-
operator-for-kubernetes

Confluent Cloud
Cloud-Native Confluent Platform Fully-Managed Service
Available on the leading public clouds with mission-critical SLAs.
Serverless Kafka characteristics:
Pay-as-you-go, elastic auto-scaling, abstracting infrastructure (topics not brokers)

Confluent Cloud, What does Fully-managed Mean?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than 60 minute response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you
need
Future-proof
Mission-critical reliability
Most Kafka as a Service offerings are partially-managed

WE BRING CHALLENGING IDEAS TO REALITY
Data Science & Machine Learning
ML ops, devops, analytics integration
Rapid Prototypes (UX, Data & ML)
User Experience & Data Visualization
Full Stack Software Architecture & Development
Product Innovation
Graph Data Modeling & Visualization
Product Assessments, Roadmaps & Selection
Training
17

© 2019 Expero, Inc. All Rights Reserved
18
Biotech
Semiconductors Financial Services
Software Supply Chain
Defense & Justice 18

Expero’s main offices are in
Houston and Austin, Texas.
Delivery teams include expert staff
from around the United States,
Canada, Spain, Argentina, and
Romania. We make software for
clients in the US, Europe, Australia
and Japan.
19

Challenges
● Overwhelming Options
● Complex Alternatives
● Endless Dependencies
● Large Data Sets
● Disconnected Systems
● “Good Enough?!?”
‘Overwhelming the Human’ with Data
20

Complex Alternatives
The Plan Is Never Perfect
● The given plan is inaccurate, how do these
two human-designed options compare?
● Can I get a decent answer now instead of a
perfect answer tomorrow?
● A human can often out-think an optimizer, if
she has help visualizing constraints her
ideas break
21

Endless Dependencies
Exceptions are the Rule
● Which orders are affected by this incident?
● If this inventory shortage persists, which
manufacturing activity is at risk?
● What is the earliest I can get this package
to the final customer location?
● Which customer’s orders do I need to cut if
production or transportation falls short?
22

Large, Siloed Data
The World Keeps Happening
● Master data is fairly small
○ Routes, recipes, trucks, parts
● Transactional data is huge
○ Orders, shipments, scans
● Important data resides in multiple systems
AND spreadsheets
● Years of historical data
● A system that handles large write load while
allowing rapid access to the data is critical
for real time decision making
23

Streaming and Batch
Analytics Use Case: Supply
Chain Planning
24

Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
streaming analytics using Confluent
batch analytics using Expero ML
physical operations
miranda
viz
miranda
viz
miranda
viz
miranda
viz

Demo

Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
miranda
viz
miranda
viz
miranda
viz
miranda
viz
PLC4X
Connector
Kafka
ConnectMQTT
File
HTTP
Machine
Sensors
Kafka
Cluster
KSQL
Tensor
Flow
Kafka
Connect
Notebooks
(Jupyter)
Spark
Real Time
Kafka
App
streaming analytics using Confluent
batch analytics using Expero ML
physical operations
TensorFlow
Serving

SOLUTIONS
FOR SUPPLY CHAIN
Expero Miranda:
Solution Includes: UI, Data Model,
Backend Code, Platform Elements:
Graph, Time-series, Streaming,
NoSQL, Search & Analytics
SOLUTION : Supply Chain
● Inventory
● Planning
● Analytics
● Security
Extendable ML Architecture:
Risk, Planning, Analytics

Combination Dashboards:
• What If - Analysis
• Optimization - Outcome Analysis
29

Fulfilment Chain
Optimization
30
● Graph-based information propagation
● Temporal forecasting
● Optimal DC locations picker
● Inventory distribution
● Bin packing
● Delivery scheduling
● Flow simulation
● Dynamic routing
● MLOps

Inventory Balance
31

Business - Technical Match
● Craft a technology solution which solves a business
problem
Partner with Leading Software
● Confluent is a preferred partner in the streaming space
Delivered Functionality
● Iterate with stakeholders to ensure solution fit
6 - 8 Week PoC
● Rapid delivery of business value
32
Engagement Model

33
Foundation Session 2-3 Days
USABILITYUSEFULNESS
CONTENT
INTERACTION DESIGN
INFO ARCHITECTURE
FUNCTIONALITY
USER AUDIENCE
WHO are present and future users?
What are their goals?
WHAT functionality to keep?
WHAT to add?
HOW can UI frameworks and patterns
be employed to scale UX?
Homogenizing visual design & brand
Does terminology align with domain?
VISUAL DESIGN
Foundational
in creating
good UX
Beauty is skin deep | UX extends to Foundation

34
PROCESS FOR SUCCESS SUCCESSEDUCATE&ADVISECOLLABORATE
DELIVERMAP
MAP SOLUTION TO YOUR ENVIRONMENT
ALIGNED TO YOUR TECHNICAL NEEDS AND
PRIORITIZE USE CASES THAT ENABLE YOUR
DESIRED BUSINESS OUTCOMES
LIVE DEMO / DELIVER A SOLUTION
PROPOSAL THAT PROVIDES BEST PRACTICE
RECOMMENDATION, BUSINESS VALUE, AND
DEPLOYMENT SUCCESS PLAN
COLLABORATE THROUGH STRUCTURED
BUSINESS AND TECHNICAL DISCOVERY TO
ACCELERATE ALIGNMENT ACROSS
REQUIREMENTS, SUCCESS CRITERIA &
ROADMAP
EDUCATE & ADVISE HOW EXPERO HAS
PARTNERED WITH OTHER CUSTOMERS
ACROSS USE CASES TO DELIVER BUSINESS
AND TECHNICAL VALUE

35
Working Prototype : Engage Business Users Early
35
Prototype &
Test
Data DiscoveryState the Problem
& Zero in on User
Goals
Determine the Most
Effective
Presentation
PLAY: Rapid Pilot

36
PoC : Weekly Timeline ~6 - 8 Weeks
(Ex: Discovery & Development Stage)
Foundation FinalizeDevelopment Refinement
Detailed Design
PoC Sprint 1
First Review S1 Review S2 Demo Final Review
PoC Sprint 2 Final Sprint
* Full Team Standups
- Jira Board Features List
- Weekly Review per Sprint
PoC Sprint 3

37Apache Kafka and Machine Learning – Kai Waehner
Questions? Feedback?Let’s connect!
Kai Waehner
kai.waehner@confluent.io
@KaiWaehner
Graham Ganssle, Ph.D.
graham@experoinc.com
@GrahamGanssle
https://experoinc.zoom.us/j/6549904076

Code: Exper20KS19
20% DISCOUNT*
*Standard Priced Conference pass

IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale

Ähnlich wie IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale (20)

Mehr von Kai Wähner

Mehr von Kai Wähner (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale