Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale

629 Aufrufe

Veröffentlicht am

I did a webinar with Confluent's partner Expero about "Apache Kafka and Machine Learning for Real Time Supply Chain Optimization". This is a great example for anybody in automation industry / Industrial IoT (IIoT) like automotive, manufacturing, logistics, etc.

We explain how a real time event streaming platform can integrate in real time with the legacy world and proprietary IIoT protocols (like Siemens S7, Modbus, Beckhoff ADS, OPC-UA, et al). You can process the data at scale and then ingest it into a modern database (like AWS S3, Snowflake or MongoDB) or analytic / machine learning framework (like TensorFlow, PyTorch or Azure Machine Learning Service).

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale

  1. 1. 1Apache Kafka and Machine Learning – Kai Waehner Industrial Internet of Things (IIoT) at Scale Real Time Planning Optimization and Predictive Maintenance with an Event Streaming Platform Kai Waehner kai.waehner@confluent.io @KaiWaehner Graham Ganssle, Ph.D. graham@experoinc.com @GrahamGanssle
  2. 2. Confluents - Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔
  3. 3. 3 Connected Intelligence (Cars, Machines, Robots, …)
  4. 4. 4 Smart Cities
  5. 5. 5 Smart Retail and Customer 360
  6. 6. 6 Intelligent Applications (Early Part Scrapping, Predictive Maintenance, …)
  7. 7. 7 Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency The world has changed!
  8. 8. 8Best-of-breed Platforms, Partners and Services for Multi-cloud Streams Private Cloud Deploy on bare-metal, VMs, containers or Kubernetes in your datacenter with Confluent Platform and Confluent Operator Public Cloud Implement self-managed in the public cloud or adopt a fully managed service with Confluent Cloud Hybrid Cloud Build a persistent bridge between datacenter and cloud with Confluent Replicator Confluent Replicator VM SELF MANAGED FULLY MANAGED
  9. 9. Data Lake Batch Analytics Event Streaming Platform Batch Integration Real Time Pre- processing Machine Sensors Streaming Platform Other Components Real Time Processing (6b) All Data (3) Read Data Optimization / Analytics (5) Deploy Optimization Model (2) Preprocess Data Model Standard based Integration (8a) Stop Machine (1) Ingest Data Real Time Edge Computing Model Lite Real Time App Model Server RPC PLC Proprietary based Integration Standard Interface Proprietary Interface
  10. 10. Spark Notebooks (Jupyter) Kafka Cluster Kafka Connect KSQL Machine Sensors Kafka Ecosystem Other Components Real Time Kafka Streams Application (Java / Scala) (6b) All Data (3) Read Data TensorFlow I/O TensorFlow (5) Deploy Model (2) Preprocess Data TensorFlow MQTT File HTTP (8a) Stop Machine (1) Ingest Data Real Time Edge Computing (C / librdkafka) TensorFlow Lite Real Time Kafka App TensorFlow Serving HTTP / gRPC (4) Train Model PLC Beckhoff S7 Modbus OPC-UA PLC4X Connector Kafka Connect Standard Interface Proprietary Interface
  11. 11. 11 Confluent Platform The Event Streaming Platform Built by the Original Creators of Apache Kafka® Operations and Security Development & Stream Processing Apache Kafka Confluent Platform Mission-Critical Reliability Complete Event Streaming Platform Freedom of Choice Datacenter Public Cloud Confluent Cloud Self-Managed Software Fully Managed Service
  12. 12. 12 Confluent Platform Licensing Open Source features Apache Kafka® Apache 2.0 License Free. Unlimited Kafka brokers Community support Enterprise License (paid) ● Annual subscription ● 24x7 Confluent support ● Kafka Connect ● Kafka Streams Apache ZooKeeper™ Clients Ansible Playbooks Community features Connectors Confluent Community License Free. Unlimited Kafka brokers Community support REST Proxy KSQL Schema Registry Commercial features Connectors Developer License ● Free ● Limited to 1 Kafka broker ● Community support Evaluation License ● Free 30-day trial ● Unlimited Kafka brokers ● Community support Control Center Command Line Interface Replicator Auto Data Balancer MQTT Proxy Operator Security Plugins Role-Based Access Control (preview) ● Best-effort Confluent Support New in CP 5.3
  13. 13. 1313 Confluent Operator: Apache Kafka on Kubernetes made simple Run Apache Kafka and Confluent Platform as a cloud-native application on Kubernetes to minimize operating complexity and increase developer agility Confluent Platform Kubernetes AWS Azure GCP RH OpenShift Pivotal On-Premises Cloud Docker Images Confluent Operator
  14. 14. 1414 Confluent Operator Deploy to Production in Minutes Automated deployment of Confluent Platform resources: Brokers, ZooKeeper, Kafka Connect, KSQL, Schema Registry, Control Center, and Replicator Automate Key Lifecycle Operations ● Failover ● Automated rolling upgrades ● Elastic scalability Deploy on Any Platform, On-Prem or in the Cloud Run at Scale with Confidence Operationalizes years of Confluent Cloud experience into a proven, enterprise-grade solution that you can deploy without deep Kafka expertise Deploy Apache Kafka and Confluent Platform as a cloud-native system on Kubernetes Kubernetes Engine Elastic Container Service for Kubernetes Kubernetes Service https://www.slideshare.net/KaiWaehner/c onfluent-operator-as-cloudnative-kafka- operator-for-kubernetes
  15. 15. Confluent Cloud Cloud-Native Confluent Platform Fully-Managed Service Available on the leading public clouds with mission-critical SLAs. Serverless Kafka characteristics: Pay-as-you-go, elastic auto-scaling, abstracting infrastructure (topics not brokers)
  16. 16. Confluent Cloud, What does Fully-managed Mean? Infrastructure management (commodity) Scaling ● Upgrades (latest stable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than 60 minute response time Infra-as-a-Service Harness full power of Kafka Kafka-specific management Platform-as-a-Service Evolve as you need Future-proof Mission-critical reliability Most Kafka as a Service offerings are partially-managed
  17. 17. WE BRING CHALLENGING IDEAS TO REALITY Data Science & Machine Learning ML ops, devops, analytics integration Rapid Prototypes (UX, Data & ML) User Experience & Data Visualization Full Stack Software Architecture & Development Product Innovation Graph Data Modeling & Visualization Product Assessments, Roadmaps & Selection Training 17
  18. 18. © 2019 Expero, Inc. All Rights Reserved 18 Biotech Semiconductors Financial Services Software Supply Chain Defense & Justice 18
  19. 19. Expero’s main offices are in Houston and Austin, Texas. Delivery teams include expert staff from around the United States, Canada, Spain, Argentina, and Romania. We make software for clients in the US, Europe, Australia and Japan. 19
  20. 20. © 2019 Expero, Inc. All Rights Reserved Challenges ● Overwhelming Options ● Complex Alternatives ● Endless Dependencies ● Large Data Sets ● Disconnected Systems ● “Good Enough?!?” ‘Overwhelming the Human’ with Data 20
  21. 21. © 2019 Expero, Inc. All Rights Reserved Complex Alternatives The Plan Is Never Perfect ● The given plan is inaccurate, how do these two human-designed options compare? ● Can I get a decent answer now instead of a perfect answer tomorrow? ● A human can often out-think an optimizer, if she has help visualizing constraints her ideas break 21
  22. 22. © 2019 Expero, Inc. All Rights Reserved Endless Dependencies Exceptions are the Rule ● Which orders are affected by this incident? ● If this inventory shortage persists, which manufacturing activity is at risk? ● What is the earliest I can get this package to the final customer location? ● Which customer’s orders do I need to cut if production or transportation falls short? 22
  23. 23. © 2019 Expero, Inc. All Rights Reserved Large, Siloed Data The World Keeps Happening ● Master data is fairly small ○ Routes, recipes, trucks, parts ● Transactional data is huge ○ Orders, shipments, scans ● Important data resides in multiple systems AND spreadsheets ● Years of historical data ● A system that handles large write load while allowing rapid access to the data is critical for real time decision making 23
  24. 24. © 2019 Expero, Inc. All Rights Reserved Streaming and Batch Analytics Use Case: Supply Chain Planning 24
  25. 25. Planners forecast long term schedule Production begins IOT data from production: inventories, manufacturing machines, yield metrics Production forecast Forecasted production - plan diffs Re optimize plan based on actuals Change orders to supply chain: inventory, manufacturing schedules Change operational characteristics : plant 223 needs new Al extruder Customer delivery SLAs: actuals vs. plan streaming analytics using Confluent batch analytics using Expero ML physical operations miranda viz miranda viz miranda viz miranda viz
  26. 26. © 2019 Expero, Inc. All Rights Reserved Demo
  27. 27. Planners forecast long term schedule Production begins IOT data from production: inventories, manufacturing machines, yield metrics Production forecast Forecasted production - plan diffs Re optimize plan based on actuals Change orders to supply chain: inventory, manufacturing schedules Change operational characteristics : plant 223 needs new Al extruder Customer delivery SLAs: actuals vs. plan miranda viz miranda viz miranda viz miranda viz PLC4X Connector Kafka ConnectMQTT File HTTP Machine Sensors Kafka Cluster KSQL Tensor Flow Kafka Connect Notebooks (Jupyter) Spark Real Time Kafka App streaming analytics using Confluent batch analytics using Expero ML physical operations TensorFlow Serving
  28. 28. SOLUTIONS FOR SUPPLY CHAIN Expero Miranda: Solution Includes: UI, Data Model, Backend Code, Platform Elements: Graph, Time-series, Streaming, NoSQL, Search & Analytics SOLUTION : Supply Chain ● Inventory ● Planning ● Analytics ● Security Extendable ML Architecture: Risk, Planning, Analytics
  29. 29. © 2018 Expero, Inc. All Rights Reserved Combination Dashboards: • What If - Analysis • Optimization - Outcome Analysis 29
  30. 30. © 2019 Expero, Inc. All Rights Reserved Fulfilment Chain Optimization 30 ● Graph-based information propagation ● Temporal forecasting ● Optimal DC locations picker ● Inventory distribution ● Bin packing ● Delivery scheduling ● Flow simulation ● Dynamic routing ● MLOps
  31. 31. © 2019 Expero, Inc. All Rights Reserved Inventory Balance 31
  32. 32. © 2019 Expero, Inc. All Rights Reserved Business - Technical Match ● Craft a technology solution which solves a business problem Partner with Leading Software ● Confluent is a preferred partner in the streaming space Delivered Functionality ● Iterate with stakeholders to ensure solution fit 6 - 8 Week PoC ● Rapid delivery of business value 32 Engagement Model
  33. 33. © 2018 Expero, Inc. All Rights Reserved 33 Foundation Session 2-3 Days USABILITYUSEFULNESS CONTENT INTERACTION DESIGN INFO ARCHITECTURE FUNCTIONALITY USER AUDIENCE WHO are present and future users? What are their goals? WHAT functionality to keep? WHAT to add? HOW can UI frameworks and patterns be employed to scale UX? Homogenizing visual design & brand Does terminology align with domain? VISUAL DESIGN Foundational in creating good UX Beauty is skin deep | UX extends to Foundation
  34. 34. © 2019 Expero, Inc. All Rights Reserved 34 PROCESS FOR SUCCESS SUCCESSEDUCATE&ADVISECOLLABORATE DELIVERMAP MAP SOLUTION TO YOUR ENVIRONMENT ALIGNED TO YOUR TECHNICAL NEEDS AND PRIORITIZE USE CASES THAT ENABLE YOUR DESIRED BUSINESS OUTCOMES LIVE DEMO / DELIVER A SOLUTION PROPOSAL THAT PROVIDES BEST PRACTICE RECOMMENDATION, BUSINESS VALUE, AND DEPLOYMENT SUCCESS PLAN COLLABORATE THROUGH STRUCTURED BUSINESS AND TECHNICAL DISCOVERY TO ACCELERATE ALIGNMENT ACROSS REQUIREMENTS, SUCCESS CRITERIA & ROADMAP EDUCATE & ADVISE HOW EXPERO HAS PARTNERED WITH OTHER CUSTOMERS ACROSS USE CASES TO DELIVER BUSINESS AND TECHNICAL VALUE
  35. 35. © 2019 Expero, Inc. All Rights Reserved 35 Working Prototype : Engage Business Users Early 35 Prototype & Test Data DiscoveryState the Problem & Zero in on User Goals Determine the Most Effective Presentation PLAY: Rapid Pilot
  36. 36. © 2019 Expero, Inc. All Rights Reserved 36 PoC : Weekly Timeline ~6 - 8 Weeks (Ex: Discovery & Development Stage) Foundation FinalizeDevelopment Refinement Detailed Design PoC Sprint 1 First Review S1 Review S2 Demo Final Review PoC Sprint 2 Final Sprint * Full Team Standups - Jira Board Features List - Weekly Review per Sprint PoC Sprint 3
  37. 37. 37Apache Kafka and Machine Learning – Kai Waehner Questions? Feedback?Let’s connect! Kai Waehner kai.waehner@confluent.io @KaiWaehner Graham Ganssle, Ph.D. graham@experoinc.com @GrahamGanssle https://experoinc.zoom.us/j/6549904076
  38. 38. Code: Exper20KS19 20% DISCOUNT* *Standard Priced Conference pass

×