Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Developer Week - Reacting to an event-driven world
Ähnlich wie Developer Week - Reacting to an event-driven world (20)
Mehr von Grace Jansen
Mehr von Grace Jansen (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Developer Week - Reacting to an event-driven world
- 1. © 2021 IBM Corporation Reacting to an event- driven world Grace Jansen Developer Advocate @gracejansen27
- 2. Let’s get some coffee… © 2021 IBM Corporation
- 3. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Coffee Lovers HTTP HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo
- 4. Event Driven Architecture © 2021 IBM Corporation S1 S3 S2 S4 Event-driven messaging backbone Microservice publishing events Microservice consuming events Event-driven messaging backbone http://ibm.biz/AdvantagesOfEDA
- 5. Barista Example: © 2021 IBM Corporation Coffee Shop Coffee Shop Barista Barista Board Coffee Lovers Coffee Lovers HTTP Orders, Queue HTTP HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 6. © 2021 IBM Corporation Apache Kafka is an open source, distributed streaming platform Stream history Immutable data Highly available Scalable Scalable consumption
- 7. © 2021 IBM Corporation Q: Is your coffee shop non-blocking and highly responsive?
- 8. © 2021 IBM Corporation Q: Is your microservice system non- blocking and highly responsive?
- 9. © 2021 IBM Corporation Q: Is your microservice system non- blocking and highly responsive? A:Yes I’m using Kafka!
- 10. © 2021 IBM Corporation Q: Is your microservice system non- blocking and highly responsive? A:Yes I’m using Kafka!
- 11. Reactive systems © 2021 IBM Corporation
- 12. Reactive Manifesto © 2021 IBM Corporation
- 13. Message-Driven Reactive Manifesto © 2021 IBM Corporation
- 14. Events “A signal emitted by a component upon reaching a given state.” © 2021 IBM Corporation A message can contain an encoded event in its payload. Messages “An item of data sent to a specific location.”
- 15. © 2021 IBM Corporation Apache Kafka is an open source, distributed streaming platform Publish and subscribe to streams of records Store records in durable way Process streams of records as they occur
- 16. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 17. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 18. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 19. Elastic Message-Driven Reactive Manifesto © 2021 IBM Corporation
- 20. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 21. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 22. Barista Example: © 2021 IBM Corporation Coffee Shop Barista Board Coffee Lovers Orders, Queue HTTP https://github.com/cescoffier/quarkus-coffeeshop-demo Event Backbone
- 23. Resilient Elastic Message-Driven Reactive Manifesto © 2021 IBM Corporation
- 24. Resilient Elastic Responsive Message-Driven Reactive Manifesto © 2021 IBM Corporation
- 25. Resilient Elastic Responsive Message-Driven Reactive Manifesto © 2021 IBM Corporation
- 26. Utilizing reactive frameworks & toolkits with Kafka © 2021 IBM Corporation
- 27. Reactive Frameworks for Kafka MicroProfile Reactive Messaging Vert.x Kafka Client © 2021 IBM Corporation
- 28. Eclipse MicroProfile © 2021 IBM Corporation An open-source community specification for Enterprise Java microservices A community of: individuals organizations vendors
- 29. MicroProfile 3.3 Stack © 2021 IBM Corporation Reactive Streams Operators 1.1 GraphQL 1.0 Standalone Projects JSON-B 1.0 JSON-P 1.1 CDI 2.0 Config 1.4 Fault Tolerance 2.1 JWT Propagation 1.1 Health Check 2.2 Metrics 2.3 OpenTracing 1.3 Open API 1.1 JAX-RS 2.1 Rest Client 1.4 Reactive Messaging 1.0 Context Propagation 1.0 Reactive Streams Not a MicroProfile spec
- 30. MicroProfile Reactive Messaging Application’s beans contain methods annotated with @Incoming and @Outgoing annotations The annotated methods are connected by named channels A channel is a name indicating which source or destination of messages is used. Channels are opaque Strings. @Incoming and @Outgoing annotations are matched up by channel names © 2021 IBM Corporation
- 31. What is Eclipse Vert.x? Polyglot Tool-kit Based on Reactor pattern Runs on the JVM Non-blocking Event-driven Includes distributed event-bus Code is single-threaded © 2021 IBM Corporation
- 32. Vert.x Demo App © 2021 IBM Corporation
- 33. Demo app © 2021 IBM Corporation https://github.com/ibm-messaging/kafka-java-vertx-starter Vert.x app produce consume
- 34. Demo app © 2021 IBM Corporation Vert.x app websocket produce consume https://github.com/ibm-messaging/kafka-java-vertx-starter
- 35. © 2021 IBM Corporation https://github.com/ibm-messaging/kafka-java-vertx-starter
- 36. © 2021 IBM Corporation
- 37. Apache Kafka Java Client © 2021 IBM Corporation
- 38. Vert.x Client © 2021 IBM Corporation
- 39. Vert.x Client © 2021 IBM Corporation
- 40. © 2021 IBM Corporation ibm.biz/ExperiencesWritingAReactiveKafkaApp
- 41. MicroProfile Interactive Demo © 2021 IBM Corporation
- 42. Skills Network Labs © 2021 IBM Corporation
- 43. Interactive Demo Utilising MicroProfile Reactive Messaging to create reactive Java microservices If you would like to follow along at the same time please go to the following link and select the “Creating reactive Java microservices” Module https://openliberty.skillsnetwork.site/cloud-native-java- made-easy-microprofile-jakarta-ee © 2021 IBM Corporation
- 44. Summary Non-reactive + Kafka != reactive Reactive characteristics and Kafka go hand in hand The open-source reactive community is on hand to help! © 2021 IBM Corporation
- 45. MicroProfile Reactive Labs https://openliberty.io/guides/?search=reactive © 2021 IBM Corporation
- 46. Reactive Systems Explained © 2021 IBM Corporation ibm.biz/ReactiveReport
- 47. Thank you Grace Jansen | @gracejansen27 © 2021 IBM Corporation Reactive resources: https://ibm.biz/IntroToReactive https://ibm.biz/GettingStartedWithReactive Getting started with Kafka: https://kafka.apache.org/quickstart https://strimzi.io Reactive Kafka libraries https://vertx.io/docs/vertx-kafka-client/java/ https://github.com/eclipse/microprofile-reactive- messaging https://projectreactor.io/docs/kafka https://github.com/akka/alpakka-kafka
- 48. Additional Slides © 2021 IBM Corporation
- 49. Resilient Elastic Responsive Message-Driven Resiliency in Kafka © 2021 IBM Corporation
- 50. Resilient Topic A Partition 1 Broker 1 Topic A Partition 1 Offline Leader © 2021 IBM Corporation
- 51. Resilient producers Delivery guarantees: At most once At least once Configuration: Acks Retries © 2021 IBM Corporation
- 52. Resilient consumers © 2021 IBM Corporation 0 1 2 3 4 5 Offset
- 53. Resilient Elastic Responsive Message-Driven Scalability in Kafka © 2021 IBM Corporation
- 54. Scalability in Kafka © 2021 IBM Corporation
- 55. Elasticity in Consumers © 2021 IBM Corporation
- 56. Consumer Groups © 2021 IBM Corporation Consumer groupA p0, offset 7 p1, offset 3 p2, offset 5 Topic 0 1 2 3 0 1 2 3 4 5 Partition 0 Partition 1 Partition 2 0 1 2 3 4 5 6 7 Consumer Consumer Consumer Consumer Consumer group B p0, offset 7 p1, offset 3 p2, offset 5 Consumer Consumer
Hinweis der Redaktion
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better Clement’s session C3 – 4pm It is possible to do http requests without blocking the thread, but even with that switch you are still approaching from a request/response perspective
- Event Driven Architecture (EDA) is a popular architectural approach that enables events to be placed at the heart of our systems Consists of Events Events are records of something that has happened, a change in state - immutable and are ordered in sequence of their creation. Interested parties can be notified of these state changes by subscribing to published events and then acting on information using their chosen business logic. An event-driven architecture, refers to a system of loosely coupled microservices that exchange information between each other through the production and consumption of events.
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application? Short answer: NO While Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
- So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
- Talking about message driven vs event driven
- Talking about message driven vs event driven
- Open sourced distributed streaming platform, often being adopted as the “de-facto” event streaming technology Arrived at the right time, captured mindshare among developers and so exploded in popularity Kafka has deliberately moved away from the word “events”… instead uses records now
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Talking about message driven vs event driven
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
- Talking about message driven vs event driven
- Talking about message driven vs event driven
- Talking about message driven vs event driven
- How do we actually achieve this when using event-driven architecture and tools like Kafka? Kafka is naturally resilient : Multiple brokers with replication of events within (Leader and at least two followers) Leader election if leader broker goes down, don’t loose connection between app and Kafka Kafka configuration can also help RESILIENT PRODUCER Delivery Guarantees (Using “at least once”, so that you guarantee delivery of event or message to Kafka) and Retries (in case sending fails) RESILIENT CONSUMER Use of manual offset commit rather than automatic, line of code in app that determines when offset should be committed so messages aren’t lost during processing by consumers. Kafka is naturally scalable: Ability to scale out work for each topic across brokers scaling load on system Use of Consumer Groups to scale out number of consumers while keeping the order guarantee and preventing duplication of processing of same message or event. BUT NOT DESIGNED FOR REACTIVE – THAT’S WHERE WE CAN UTILISE REACTIVE FRAMEWORKS AND TOOLKITS TO REALLY ENABLE ALL OF THESE BAHVIOURS, BUILDING UPON WHAT COMES AS STANDARD WITH KAFKA.
- All of these frameworks take the role of handling polling, then provide different ways for us to handle the processing and flow. MicroProfile and Alpakka provide a more constrained way of doing backpressure and flow control, whereas vertx provides the tools for doing that separation but leaves it up to the developer to decide exactly how to implement it. We will describe flow control for MP and alpakka, “how they improve flow control and allow application to follow reactive streams spec” Others: Alpakka Kafka Connector Spring WebFlux Project Reactor
- Eclipse MicroProfile is an open-source community specification for Enterprise Java microservices A community of individuals, organizations, and vendors collaborating within an open source (Eclipse) project to bring microservices to the Enterprise Java community
- The MicroProfile Reactive Messaging specifiMicroProfile Reactive Messaging makes use of and interoperates with two other specifications: Reactive Streams is a specification for doing asynchronous stream processing with back pressure. It defines a minimal set of interfaces to allow components which do this sort of stream processing to be connected together. MicroProfile Reactive Streams Operators is a MicroProfile specification which builds on Reactive Streams to provide a set of basic operators to link different reactive components together and to perform processing on the data which passes between them. When you use the MicroProfile Reactive Messaging @Incoming and @Outgoing annotations, Open Liberty creates a Reactive Streams component for each method and joins them up by matching the channel names. cation aims to deliver applications embracing the characteristics of reactive systems
- A method with an @Incoming annotation consumes messages from a channel. A method with an @Outgoing annotation publishes messages to a channel. A method with both an @Incoming and an @Outgoing annotation is a message processor, it consumes messages from a channel, does some transformation to them, and publishes messages to another channel. When you use the MicroProfile Reactive Messaging @Incoming and @Outgoing annotations, Open Liberty creates a Reactive Streams component for each method and joins them up by matching the channel names.
- Polyglot Java, Javascript, Groovy, Ceylon, Scala and Kotlin The reactor pattern is one implementation technique of event-driven architecture. In simple terms, it uses a single threaded event loop blocking on resource-emitting events and dispatches them to corresponding handlers and callbacks. It receives messages, requests, and connections coming from multiple concurrent clients and processes these posts sequentially using event handlers. The purpose of the Reactor design pattern is to avoid the common problem of creating a thread for each message, request, and connection. Then it receives events from a set of handlers and distributes them sequentially to the corresponding event handlers. It’s single-threaded – so you must not block the thread! The Kafka client is becoming more popular and e.g. it is used by SmallRye Reactive messaging
- So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
- Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
- Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
- In the original version of the application, we were having to use a for loop inside a while loop that resulted in the following flow when consuming records: Poll for records. Poll function returns with a batch of records. Iterate through each record in the batch. For each record, send the record along the WebSocket to the frontend. After iteration is complete, return to Step 1. So, while we were processing the current batch of records, no new records could be fetched.
- In the new application using the Vert.x client instead, you simply write a handler function that will be called every time a new record arrives. This allows asynchronous consuming and processing of records which fits in much better with the reactive paradigm. This change makes the flow a lot simpler. Vert.x now handles the step of polling for new records. The application code does the following: Receive a new record. Send the record along the WebSocket to the frontend. This not only allows the application to process on a per-record basis but also leaves the developer free to focus on the processing of records, rather than the work of consuming them from Kafka.
- We wanted the ability to pause and resume both producing and consuming records from the frontend. In the original application, where the default Java Kafka client was used, the while loop that calls the poll for the consumer needed to be exited and resumed based on commands from the frontend. Because this client is single threaded, the loop had to be executed in a separate thread to the main flow of the application. As we wanted to produce records on a timer, a similar while loop in a separate thread was required for the producer. Therefore, we had to deal with the headache of writing logic to control the thread lifecycle and synchronize data across threads just to get control over the flow of records. In the Vert.x application, we were able to use a combination of Vert.x features and Vert.x Kafka client features to make this flow control much easier. For the producer, we used a feature in Vert.x that allows a handler to be called on a timer basis. The handler could be called once, or multiple times, and it was easy for us to pause and resume the timer when start and stop requests arrived from the WebSocket. For the consumer in the Vert.x application, we could use the built-in back-pressure support that allows consumers to be paused and resumed. When a stop command comes from the WebSocket, the consumer is paused and the Vert.x client will stop polling for new records from Kafka. At a later time, the consumer can be resumed and the Vert.x client will resume polling and the existing handler will start receiving new records.
- So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
- Creating reactive Java Microservices Testing reactive Java microservices Consuming RESTful services asynchronously with template interfaces Integrating RESTful services with a reactive system Acknowledging messages using MicroProfile Reactive Messaging
- How do we actually achieve this when using event-driven architecture and tools like Kafka? Kafka is naturally resilient – (brokers go through
- Talking about message driven vs event driven
- Imagine a broker goes down, this means the leader of Topic A, partition 1 is offline
- Can’t do fire and forget if you want full resiliency cause if the broker goes down your messages get lost Two different guarantees, way you get them is through confirguartion At most once, you may lose some messages (not completely relisient) At least once, guaranteed delivery but may get duplicates Retries is if acks times out/fails – how many times do you retry producing the event (how will the retry affect ordering)
- Talking about message driven vs event driven
- Elasticity in Kafka itself Scale out brokers, can’t scale down (where do events go if you did?) Can scale out partitions but can’t scale them down again Can add topics, and delete topics if you don’t care about them
- To allow scalability of consumers, consumers are grouped into consumer groups. Consumer declare what group they are in using a group id For consumers we use Consumer groups to enable elasticity
- Elasticity in Kafka itself Scale out brokers, can’t scale down (where do events go if you did?) Can scale out partitions but can’t scale them down again Can add topics, and delete topics if you don’t care about them