Building Event Driven Services with Kafka Streams
•
12 gefällt mir•13,034 views
The Kafka Summit version of this talk is more practical and includes code examples which walk though how to build a streaming application with Kafka Streams.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Building Event Driven Services with Kafka Streams
Ähnlich wie Building Event Driven Services with Kafka Streams (20)
Mehr von Ben Stopford
Mehr von Ben Stopford (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Building Event Driven Services with Kafka Streams
- 1. 1 Building Event Driven Services with Kafka Streams Ben Stopford Technologist, Office of the CTO @benstopford
- 2. 2 There is a book! http://bit.ly/designing-event-driven-systems
- 5. 5 Evolution!
- 6. 6 EcosystemsApp Increasingly we build ecosystems
- 7. 7 But these aren’t static, they grow 7
- 8. 8 Span data centers US East 1 US East 2
- 9. 9 Span regions or clouds
- 11. 11 Span teams and departments
- 12. 12 How do you do this?
- 14. 14 Returns Service Email Service T2’’ Tax Service Order Service Or we can use event streams
- 15. 15 One Problem is DATA
- 16. 16 Most services share the same core facts. Catalog Most services live in here
- 17. 17 A Global API A Global Source of Truth Events
- 18. 18 Events have two hats Notification Data replication
- 19. 19 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
- 20. 20 Buying an iPad (with REST/RPC) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
- 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
- 23. 23 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
- 24. 24 Stateless / Stateful Stream Processing Relates to these hats
- 25. 25 Message Broker (Kafka) Submit Order Order Created getCustomer() REST Orders Service Customer Service Webserver KAFKA KStreams Shipping Service Stateless Stream Processing Notification
- 26. 26 Message Broker (Kafka) Submit Order Order Created Orders Service Customer Service Webserver KAFKA KStreams Shipping Service KTable Customer Updated Stateful Stream Processing Data replication
- 27. 27 Events have two hats Notification Data Replication Stateless Stream Processing Stateful Stream Processing
- 28. 28 Kafka Streams is an API library 2
- 29. 29 Features • Buffering • Filters • Join (Stream-Stream, Stream-Table, Table- Table) • Aggregations (Sum, Count...) • Any arbitrary code • HA (dynamic scaling, standby replicas, checkpoints, etc.)
- 30. 30 Building a simple KStreams Email Service
- 31. 31 KAFKA Buffer 5 mins Send Email Stateless Streaming (stream-stream join)
- 32. 32 KAFKA Whole customer dataset cached in Email Service Stateful Streaming (Stream-Stream-Table join)
- 33. 33 KAFKA Whole customer dataset cached in Email Service Customer information in 3 places
- 35. 35 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Order
- 36. 36 Validate Order kStreams.stream(“Orders”) .mapValues((key, order) -> isValid(order) ? pass() : fail()) .to(“OrderValidations”);
- 37. 37 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Validate Warehouse Inventory
- 38. 38 Ensure there is enough Stock Order Event Get # IPads in Stock - Validate Quantity - Reserve Item Atomic Compare and Set
- 39. 39 Table Stream State Store “Reserved Items” Topic Orders Topic Warehouse Stock Topic Orders Topic 1. Join
- 40. 40 Table Stream State Store “Reserved Items” Topic Orders Topic Order Validations Topic 2. Reserve Stock State store behaves like a little database Warehouse Stock Topic
- 41. 41 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 42. 42 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 43. 43 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 44. 44 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 45. 45 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 46. 46 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 47. 47 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 48. 48 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 49. 49 Table Stream State Store “Reserved Items” Topic Orders Topic Inventory topic Order Validations Topic Transactional Wrapper State store behaves like a little database
- 50. 50 We need one more thing!
- 51. 51 Rekey by Product builder.stream(“Orders”) .selectKey((id, order) -> order.getProduct()) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 52. 52 Rekey does two things in this case - Collocate data that must join - Serially execute critical section against single data.
- 53. 53 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” Rekey: Collocates data that must join JOIN JOIN
- 54. 54 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” - Rekey: Serially execute critical section against single data. CaS CaS
- 55. 55 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Fraud
- 56. 56 Fraud Service “Fail” an order if the total value of orders for that user eclipses a threshold.
- 57. 57 Fraud Service CREATE STREAM OrderValidations AS SELECT userId, sum(amount) FROM Orders WINDOW TUMBLING(SIZE 60 MINUTE) GROUP BY userId HAVING sum(amount) > 10000;
- 58. 58 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Mapping Synchronous and Asynchronous
- 59. 59 Create a materialized view kStreams.stream(“Orders”) .groupByKey() .reduce((agg, v) -> v, STORE_NAME));
- 61. 61 Gotchas/Guidelines - Use Schemas (Schema registry) - Global KTables & KTables are subtly different on replay. - Stream-table joins only join one way - The defaults aren’t enough. See resiliency settings in the documentation. - Releasing - Change application id trick: - loses committed offsets - Doesn’t delete topics, leaves them hanging - Name state store changelog topics - Start simple (Stateless)
- 62. 62 So…
- 63. 63 Good architectures have little to do with this:
- 64. 64 It’s about how systems evolves over time
- 65. 65 Events provide the key to evolutionary architectures Notification Data replication
- 66. 66 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
- 67. 67 Streaming is the toolset for dealing with events as they move
- 68. 68 Thank You @benstopford Book: http://bit.ly/designing-event-driven-systems Twitter: @benstopford Slides: http://www.benstopford.com/