Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Building Event-Driven (Micro)Services with Apache Kafka

1.936 Aufrufe

Veröffentlicht am

Should we use traditional REST APIs to bind services together? Or is it better to use a more loosely-coupled protocol? This talk will dive into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which.

Veröffentlicht in: Technologie
  • ACCESS that WEBSITE Over for All Ebooks (Unlimited) ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... DOWNLOAD FULL EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH }
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • wonderul concepts , Do you have any further detailing of the Architecture or what are the challenges and how you overcome - In the same line of thoughts?
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Building Event-Driven (Micro)Services with Apache Kafka

  1. 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Building event-driven Microservices with Kafka Ecosystem Guido Schmutz London, 30.5.2018 @gschmutz guidoschmutz.wordpress.com
  2. 2. Guido Schmutz Working at Trivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Building event-driven Microservices with Kafka Ecosystem
  3. 3. Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. What about streaming sources? 5. What about integrating legacy applications? 6. What about (historical) data analytics? 7. Why Kafka for Event-Driven Microservices? 8. Summary Building event-driven Microservices with Kafka Ecosystem
  4. 4. Building event-driven Microservices with Kafka Ecosystem Where do we come from?
  5. 5. Shop Rich UI Shop Backend Application Traditional Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database Building event-driven Microservices with Kafka Ecosystem sync request/response
  6. 6. Shop UI App Business Activity Service SOA Approach Building event-driven Microservices with Kafka Ecosystem Contract-first Web Services Technical layers offer their own interfaces Reuse on each level Lower layer often wraps legacy code Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic GUI Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP
  7. 7. Shop UI App Business Activity Service Virtualized SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Building event-driven Microservices with Kafka Ecosystem
  8. 8. Building event-driven Microservices with Kafka Ecosystem What are Microservices?
  9. 9. What are Microservices? Building event-driven Microservices with Kafka Ecosystem Tightly Scoped behind clear interfaces Responsible for managing their own data (not necessarily the infrastructure) Should be highly decoupled Independently deployable, self-contained and autonomous SOA done right ?!
  10. 10. Microservice Approach Building event-driven Microservices with Kafka Ecosystem Tightly Scoped Responsible for managing their own data Highly decoupled Independently deployable, self-contained and autonomous { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST
  11. 11. Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Building event-driven Microservices with Kafka Ecosystem
  12. 12. Synchronous World of Request-Response leads to tight, point-to-point couplings Building event-driven Microservices with Kafka Ecosystem problem in lower end of chain have a ripple effect on other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State
  13. 13. Building event-driven Microservices with Kafka Ecosystem Why not Event-Driven?
  14. 14. 3 mechanisms through which services interact Request-Driven (Imperative) Event Driven (Functional) Service Logic State Event Consume OrderEvent{iPad} Command ”Order IPad” boolean order(IPad) Event Publish OrderValidatedEvent{iPad} Query ”Retrieve my Orders List<Orders> getAllOrders(for) Building event-driven Microservices with Kafka Ecosystem
  15. 15. Event-Driven (Async) Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Building event-driven Microservices with Kafka Ecosystem Event Store sync request/response async request/response async, event pub/sub
  16. 16. What about streaming sources? Building event-driven Microservices with Kafka Ecosystem
  17. 17. How to work with streaming data sources Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Event Store Location Social Click stream Sensor Data Mobile Apps Weather Data Event Stream Building event-driven Microservices with Kafka Ecosystem
  18. 18. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Streaming Analytics Architecture BI Tools SQL Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Analytics Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Store Service Building event-driven Microservices with Kafka Ecosystem
  19. 19. What about integrating legacy applications? Building event-driven Microservices with Kafka Ecosystem
  20. 20. Data Store Integrate existing systems through CDC Customer Event Store Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Stream Processing Results Stream Processor Reference / Models Dashboard Capture changes directly on database Change Data Capture (CDC) => think like a global database trigger Transform existing systems to event producer Building event-driven Microservices with Kafka Ecosystem
  21. 21. Hadoop Clusterd Hadoop Cluster Stream Processing Cluster Integrate existing systems through CDC BI Tools SQL Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Processor Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Store Service Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture Building event-driven Microservices with Kafka Ecosystem
  22. 22. What about (historical) data analytics? Building event-driven Microservices with Kafka Ecosystem
  23. 23. Streaming & (Big) Data Analytics Architecture Event Stream Hadoop Clusterd Hadoop Cluster Big Data Cluster Data Flow Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Processing Cluster Stream Processor State { } API Event Stream Event Stream SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps SQL Export SearchEvent Store Service Location Social Click stream Sensor Data Mobile Apps Weather Data Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import Building event-driven Microservices with Kafka Ecosystem
  24. 24. Why Kafka for Event-Driven Microservices? Building event-driven Microservices with Kafka Ecosystem
  25. 25. Apache Kafka – A Streaming Platform Building event-driven Microservices with Kafka Ecosystem High-Level Architecture Distributed Log at the Core Scale-Out Architecture Logs do not (necessarily) forget
  26. 26. Apache Kafka – Schema Registry Building event-driven Microservices with Kafka Ecosystem Producer Kafka Broker Consumer Schema Registry Kafka Kafka avro schema avroavro avro schema
  27. 27. Apache Kafka – scalable message processing and more! Building event-driven Microservices with Kafka Ecosystem Source Connector trucking_ driver Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka RESTful RESTful
  28. 28. Legacy Microservice Change Data Capture (CDC) Building event-driven Microservices with Kafka Ecosystem RDBMS cdc-source trucking_ driver Customer Topic elasticsearc h-sink NoSQL
  29. 29. Demo Building event-driven Microservices with Kafka Ecosystem Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic REST REST Kafka Customer
  30. 30. Hold Data for Long-Term – Data Retention Producer 1 Broker 1 Broker 2 Broker 3 1. Never 2. Time based (TTL) log.retention.{ms | minutes | hours} 3. Size based log.retention.bytes 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 --create --topic customers --replication-factor 1 --partitions 1 --config cleanup.policy=compact Building event-driven Microservices with Kafka Ecosystem
  31. 31. Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) 2015-10-02 11,Peter,Muster,3010,Berne 2016-10-04 12,Paul,Steffen,8001,Zurich 2016-12-02 21,Lisa,Meier,3043,Ittigen 2017-05-03 11,Peter,Muster,3015,Berne 2017-05-03 21,Lisa,Steffen,8001,Zurich 2017-07-03 11,Peter,Muster,3052,Zollikofen Building event-driven Microservices with Kafka Ecosystem 2015-10-02 11,Peter,Muster,3010,Berne 2016-10-04 12,Paul,Steffen,8001,Zurich 2016-12-02 21,Lisa,Meier,3043,Ittigen 2017-05-03 11,Peter,Muster,3015,Berne 2017-05-03 21,Lisa,Steffen,8001,Zurich 2017-07-03 11,Peter,Muster,3052,Zollikofen
  32. 32. Keep Topics in Compacted Form 0 1 2 3 4 5 6 7 8 9 10 11 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value 3 4 6 8 9 10 K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction Building event-driven Microservices with Kafka Ecosystem V1 V2 V3 V4 V5 V6 V7 V8 V9V10 V11 K1 K3 K4 K5K2 K6
  33. 33. Keep Topics both in Original and Compacted Form Building event-driven Microservices with Kafka Ecosystem Kafka Broker A B C B A E K A E A C K B E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K Producer Consume and Produce OR TX
  34. 34. Enrich Stream with Static Data with Kafka Streams Building event-driven Microservices with Kafka Ecosystem Order Topic Consume and Produce Customer Table Customer Topic Custome r Handler Join Customer Local State Order & Customer Topic
  35. 35. Summary Building event-driven Microservices with Kafka Ecosystem
  36. 36. Hadoop Clusterd Hadoop Cluster Big Data Summary Billing & Ordering CRM / Profile Marketing Campaigns SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Event Hub Data Flow Data Flow Change Data Capture Parallel Processing Storage Storage RawRefined Results SQL Export Microservice State { } API Stream Processor State { } API Event Stream Event Stream Search Service Location Social Click stream Sensor Data Mobile Apps Weather Data Stream Processing Microservices
  37. 37. Summary Building event-driven Microservices with Kafka Ecosystem • service autonomy is key in a Microservices Architecture! • not all communication need to be synchronous => separate into • commands • events • queries • Kafka is well suited as an event broker / event store • brings many more interesting features beyond just “message passing”
  38. 38. References Building event-driven Microservices with Kafka Ecosystem Microservices Blog Series, Ben Stopford, Confluent: • https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: • https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con • https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: • http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: • https://www.youtube.com/watch?v=B1-gS0oEtYc
  39. 39. Technology on its own won't help you. You need to know how to use it properly. Building event-driven Microservices with Kafka Ecosystem

×