Evolutionary Systems - Kafka Microservices
•Als PPTX, PDF herunterladen•
3 gefällt mir•464 views
In this talk we'll see how to leverage Kafka, and in particular the Kafka Processor API along with patterns such as Event Collaboration, Command Event Transformations and Stateful processing for developing the next and future proof generation of Microservice oriented systems.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (16)
Ähnlich wie Evolutionary Systems - Kafka Microservices
Ähnlich wie Evolutionary Systems - Kafka Microservices (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Evolutionary Systems - Kafka Microservices
- 1. Evolutionary Systems Kafka Microservices Stefano Rocco Milan, 4 July 2018
- 2. Evolution of the Operational Domain Monolith SOA Microservices
- 3. Microservice Based System A Microservice based system is a Network whereas nodes collaborate together via links in order to achieve an overall consistent state of the system. Nodes are responsible for carrying out logic and holding some data, links are responsible for transporting data between nodes. Different Microservices (Nodes) act in concert as one larger system. They communicate with each other, typically asynchronously, through well defined Contracts, whether via HTTP REST calls, RPC or Message Broker passing.
- 4. Microservice Based System - Each microservice carries out some business logic in order to solve a unique business capability. - Each microservice own and control its own database and no services should share a database. Avoid conflicts like competing read/write patterns, data-model conflicts. Guarantee evolutionary independence. - Adapters are responsible for inbound communication, “integrating” the outside world with the service. - Ports are responsible for outbound communication, connecting the service with the outside world, while Process/Logic Data Adapter Port
- 5. A Classic Example: E-Commerce Checkout Checkout Service Order Service Payment Service Checkout Service: Sends to the Order Service events about a new shopping cart checkout. Order Service: Create a new Customer Order and calls the Payment Service for payment processing and authorization. Payment Service: Process Customer Payments by either authorizing or rejecting them.
- 6. E-Commerce Checkout: Event Collaboration Checkout Checkout Checkout Order OrderPayment Checkout Created Checkout Created Order Created Order Created Order Created Payment Received Payment Received Order Acknowledged Order Acknowledged Order Created Payment Received Order Acknowledged Checkout Acknowledged Checkout Processed Checkout Created Kafka
- 7. Kafka Ecosystem Kafka APIs Kafka Streams Schema Registry Kafka REST Proxy Kafka Connect Kafka Message Broker
- 8. Kafka High Level Architecture Producer 1 Producer 2 Producer N Consumer 1 Consumer 2 Consumer N Broker 1 Broker 2 Broker N Zookeeper Consumer Group Kafka Cluster Produce Consume Commit OffsetWritten Offset
- 9. Kafka High Level Architecture Consumer 1 Consumer 2 Consumer N Producer 1 Producer 2 Producer 3 Topic Partition 1 Partition 2 Partition 3 Consumer 1 Consumer 2 Consumer 3 Consumer N - Different Producers can write to any topic. - A Topic is a distributed log of events. - Events are stored and distributed consistently, according an hashing algorithm, into one of the Topic Partitions. - Partitions defines the Topic’s level of parallelism for consumers reading operations. - Consumers are organized in Consumer Groups. - Each Consumer within a Consumer Group is exclusively assigned to a specific Topic Partition - Different Consumer Groups, identified by a unique clientId, can read from the same topic at the same time. - Only one Consumer per Consumer Group can read from the assigned Topic Partition - Consumers are responsible for checkpointing their position by committing the reading offset.**
- 10. How Kafka Replication Works Partition 2 Partition 3 Partition 4 Partition 2 Partition 2 Partition 3 Partition 4 Partition 4 Partition 3 Partition 1 Partition 1 Partition 1 In-Sync Replica In-Sync Replica Available for consumers
- 11. One event topic per Bounded Context CheckoutCreated CheckoutProcessed OrderCreated OrderAcknowledgedCheckoutAcknowledged PaymentReceived checkout_events order_events payment_events Checkout Service Order Service Payment Service CheckoutConfirmed OrderConfirmed PaymentConfirmed
- 12. Wrap events/messages around an Envelope { "envelope" : { "id" : "...", "Ip_address" : "...", "timestamp" : "...", "event" : { "eventType" : "CheckoutCreated", "id" : "...", ... } } } ChekoutRejected CheckoutAcknowledged CheckoutProcessed CheckoutConfirmed
- 13. Choose a safe and evolutionary protocol - Support backward, forward and full compatibility - Test against a schema. Schema Registry? - Leverage type safety. Avro, Protobuf, Thrift.. - If you want to be safe with Json you have to embrace an additive change only policy… most of the times. - Json does not support schema definition - Json is not purely type safe - If you go Json soon enough you’ll break your protocol. - Version your protocol! Version your Microservices!
- 14. Get inspired by Event Sourcing Event State T1 T2 T3 OrderCreated OrderAcknwledged OrderConfirmed Status (Created) Status (Acknowledged) Status (Confirmed)
- 15. Kafka Processor Architecture Checkout events Checkout Created Order Processor Order Store Commit Offset Put Put OK Pull Receive TOPIC EVENT PROCESSOR STATE STORE SEMANTIC DELIVERY ANTI CORRUPTION
- 16. Kafka Processor APi import com.google.gson.JsonObject; import org.apache.kafka.streams.processor.AbstractProcessor; import org.apache.kafka.streams.state.KeyValueStore; public class MyProcessor extends AbstractProcessor<String, JsonObject> { @Override public void process(final String key, final JsonObject value) { //Implement your business logic here // Eventually use the State Store final KeyValueStore<String, JsonObject> store = (KeyValueStore<String, JsonObject>) context().getStateStore("store_name"); //store.put(?, ?); //store.get(?, ?); //Eventually forward to the downstream processor context().forward(key, value); // Commit context().commit(); } }
- 17. Kafka Processor Workflow Definition (PDSL) // consumer from checkout_events topic from(“order_source”, “checkout_events”) // Process using OrderProcessor .via(“order_processor”, OrderProcessor, “order_source”) // Attach OrderStore state store .with(OrderStore, “order_processor”) // Sink to order_events topic .to(“order_sink”, “order_events”, “order_processor”)
- 18. Kafka Processor Workflow Definition Order Processor Checkout Events Payment Events Order Events Order State FROM VIA TO WITH
- 19. Semantic Delivery Reliability - At Most Once: The middleware decides for you. Not what we want! - Exactly Once: The middleware decides for you. Not what we want! - At Least Once: You controntol the middleware! However it’s not smart enough... - Effectively Once = Idempotent At Least Once!
- 20. Anti Corruption: Event Command Transformation Checkout Created Create Order Order Created Status (Created) Payment Processed Acknowledge Order Order Acknowledged Payment Confirmed Status (Acknowledged) Confirm Order Order Confirmed Order (Confirmed) T1 T2 T3 Event Command Event State ADAPTER
- 22. What about the read side and Projections? Events State Store Projector Service Your Fancy Store Processor Punctuate Web Service INTERACTIVE QUERIES