Amitay Horwitz - Building event sourced systems with Kafka Streams - Codemotion Milan 2018
•
4 likes•781 views
Apache Kafka is a battle-tested stream-processing platform. Combined with Kafka Streams API, it’s possible to build scalable, fault-tolerant and distributed event-driven systems. In this talk, I will describe the key components of an event sourced system and how simple building blocks can be combined using the powerful and declarative streams API to deliver business value. I will demonstrate possible solutions to the challenges of eventual-consistency and the object-relational impedance mismatch.
Recommended
Recommended
More Related Content
Similar to Amitay Horwitz - Building event sourced systems with Kafka Streams - Codemotion Milan 2018
Similar to Amitay Horwitz - Building event sourced systems with Kafka Streams - Codemotion Milan 2018 (20)
More from Codemotion
More from Codemotion (20)
Recently uploaded
Recently uploaded (20)
Amitay Horwitz - Building event sourced systems with Kafka Streams - Codemotion Milan 2018
- 1. Event Sourcing w/ Kafka Streams Amitay Horwitz | @amitayh Milan | November 29 - 30, 2018
- 4. AGENDA ✅
- 5. AGENDA ✅ • Event sourcing 101
- 6. AGENDA ✅ • Event sourcing 101 • Eventim
- 7. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams
- 8. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams • Putting it all together
- 9. BACKGROUND 🖼
- 10. A SERVICE IS BORN ✒📃💰 • Wix Invoices was incepted in mid 2015 • Rich domain model • Auditing is important for monetary products
- 22. HOW DOYOU HYDRATE YOUR DOMAIN OBJECTS? SELECT e.employee_id AS "Employee #" , e.first_name || ' ' || e.last_name AS "Name" , e.email AS "Email" , e.phone_number AS "Phone" , TO_CHAR(e.hire_date, 'MM/DD/YYYY') AS "Hire Date" , TO_CHAR(e.salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Salary" , e.commission_pct AS "Comission %" , 'works as ' || j.job_title || ' in ' || d.department_name || ' department (manager: ' || dm.first_name || ' ' || dm.last_name || ') and immediate supervisor: ' || m.first_name || ' ' || m.last_name AS "Current Job" , TO_CHAR(j.min_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') || ' - ' || TO_CHAR(j.max_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Current Salary" , l.street_address || ', ' || l.postal_code || ', ' || l.city || ', ' || l.state_province || ', ' || c.country_name || ' (' || r.region_name || ')' AS "Location" , jh.job_id AS "History Job ID" , 'worked from ' || TO_CHAR(jh.start_date, 'MM/DD/YYYY') || ' to ' || TO_CHAR(jh.end_date, 'MM/DD/YYYY') || ' as ' || jj.job_title || ' in ' || dd.department_name || ' department' AS "History Job Title" FROM employees e -- to get title of current job_id JOIN jobs j ON e.job_id = j.job_id -- to get name of current manager_id LEFT JOIN employees m ON e.manager_id = m.employee_id -- to get name of current department_id LEFT JOIN departments d ON d.department_id = e.department_id -- to get name of manager of current department -- (not equal to current manager and can be equal to the employee itself) LEFT JOIN employees dm ON d.manager_id = dm.employee_id -- to get name of location
- 23. HOW DOYOU HYDRATE YOUR DOMAIN OBJECTS? SELECT e.employee_id AS "Employee #" , e.first_name || ' ' || e.last_name AS "Name" , e.email AS "Email" , e.phone_number AS "Phone" , TO_CHAR(e.hire_date, 'MM/DD/YYYY') AS "Hire Date" , TO_CHAR(e.salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Salary" , e.commission_pct AS "Comission %" , 'works as ' || j.job_title || ' in ' || d.department_name || ' department (manager: ' || dm.first_name || ' ' || dm.last_name || ') and immediate supervisor: ' || m.first_name || ' ' || m.last_name AS "Current Job" , TO_CHAR(j.min_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') || ' - ' || TO_CHAR(j.max_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Current Salary" , l.street_address || ', ' || l.postal_code || ', ' || l.city || ', ' || l.state_province || ', ' || c.country_name || ' (' || r.region_name || ')' AS "Location" , jh.job_id AS "History Job ID" , 'worked from ' || TO_CHAR(jh.start_date, 'MM/DD/YYYY') || ' to ' || TO_CHAR(jh.end_date, 'MM/DD/YYYY') || ' as ' || jj.job_title || ' in ' || dd.department_name || ' department' AS "History Job Title" FROM employees e -- to get title of current job_id JOIN jobs j ON e.job_id = j.job_id -- to get name of current manager_id LEFT JOIN employees m ON e.manager_id = m.employee_id -- to get name of current department_id LEFT JOIN departments d ON d.department_id = e.department_id -- to get name of manager of current department -- (not equal to current manager and can be equal to the employee itself) LEFT JOIN employees dm ON d.manager_id = dm.employee_id -- to get name of location 😱
- 26. MUTABLE STATE 🚫
- 27. MUTABLE STATE 🚫 • Instead of saving the current state, we save the succession of events that brought us to this state
- 28. MUTABLE STATE 🚫 • Instead of saving the current state, we save the succession of events that brought us to this state • currentState = fold(events, emptyState)
- 33. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "DRAFT" } Line item added Line item added Line item removed time
- 34. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "SENT" } Line item added Line item added Line item removed Invoice sent to customer time
- 35. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "PAID" } Line item added Line item added Line item removed Invoice sent to customer Payment received time
- 36. EVENTIM
- 37. DESIGN GOALS 👷 • Small and simple library • Non-intrusive • Maintain data integrity • Easily add custom views
- 38. WRITE PATH ✏ Event Sourced Command Handler UserInterface
- 39. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface
- 40. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface invoice ID: 12345 expected version: 5 amount: $12.34 Add Payment
- 41. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface invoice ID: 12345 expected version: 5 amount: $12.34 Add Payment Aggregare Repository Load
- 42. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface invoice ID: 12345 expected version: 5 amount: $12.34 Add Payment Aggregare Repository Load Event Store Get events
- 43. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface invoice ID: 12345 expected version: 5 amount: $12.34 Add Payment Aggregare Repository Load Event Store Get events version: 5 customer: {...} line items: [...] balance: $12.34 Invoice #12345
- 44. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get eventsinvoice ID: 12345 expected version: 5 amount: $12.34 Add Payment version: 5 customer: {...} line items: [...] balance: $12.34 Invoice #12345
- 45. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get eventsinvoice ID: 12345 expected version: 5 amount: $12.34 Add Payment version: 5 customer: {...} line items: [...] balance: $12.34 Invoice #12345
- 46. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events
- 47. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events Payment Added: $12.34
- 48. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events Payment Added: $12.34 Status Changed: Paid
- 49. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events Payment Added: $12.34 Status Changed: Paid version: 7 customer: {...} line items: [...] balance: $0.00 Invoice #12345
- 50. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events Payment Added: $12.34 Status Changed: Paid Publish events (OCC) version: 7 customer: {...} line items: [...] balance: $0.00 Invoice #12345
- 51. WRITE PATH ✏ Event Sourced Command Handler Command UserInterface Aggregare Repository Load Event Store Get events Payment Added: $12.34 Status Changed: Paid Publish events (OCC) Event Bus
- 52. READ PATH 👓 Event Bus
- 53. READ PATH 👓 Event Bus Events
- 54. READ PATH 👓 View Projector Event Handler Event Bus Events
- 55. READ PATH 👓 View Projector Event Handler DB Event Bus Events
- 56. READ PATH 👓 View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $12.34 New 67890 Jane Doe $34.56 Sent
- 57. READ PATH 👓 View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $12.34 New 67890 Jane Doe $34.56 Sent Payment Added: $12.34
- 58. READ PATH 👓 View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 New 67890 Jane Doe $34.56 Sent
- 59. READ PATH 👓 View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 New 67890 Jane Doe $34.56 Sent Status Changed: Paid
- 60. READ PATH 👓 View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 Paid 67890 Jane Doe $34.56 Sent
- 61. READ PATH 👓 View Projector Event Handler DB Event Bus Events
- 62. READ PATH 👓 View Projector Event Handler DB Event Bus Events
- 63. READ PATH 👓 View Projector Event Handler DB Event Bus Events
- 64. READ PATH 👓 View Projector Event Handler DB Event Bus Events
- 65. READ PATH 👓 View Projector Event Handler DB Event Bus EventsUserInterface
- 66. READ PATH 👓 View Projector Event Handler DB Queries Event Bus EventsUserInterface
- 67. READ PATH 👓 Mailer ✉ Event Handler View Projector Event Handler DB Queries Event Bus EventsUserInterface
- 68. READ PATH 👓 Mailer ✉ Event Handler View Projector Event Handler Reporting 📈 Event Handler DB Queries Event Bus EventsUserInterface
- 69. PAIN POINTS 😓
- 70. PAIN POINTS 😓 • Persisting events and publishing them is not transactional
- 71. PAIN POINTS 😓 • Persisting events and publishing them is not transactional • Inherent eventual consistency is not integrated in the product (read after write)
- 72. PAIN POINTS 😓 • Persisting events and publishing them is not transactional • Inherent eventual consistency is not integrated in the product (read after write) • Rebuilding views is a complex operation
- 74. REBUILDINGVIEWS 🌄 version event_payload 1 InvoiceCreated(...)
- 75. REBUILDINGVIEWS 🌄 version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...)
- 76. REBUILDINGVIEWS 🌄 version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...) 3 LineItemAdded(...)
- 77. REBUILDINGVIEWS 🌄 version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...) 3 LineItemAdded(...) 4 InvoiceSent(...)
- 80. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload
- 81. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload 12345 1 InvoiceCreated
- 82. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded
- 83. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 67890 1 InvoiceCreated
- 84. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 67890 1 InvoiceCreated 67890 2 InvoiceDeleted
- 85. REBUILDINGVIEWS 🌄 invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 12345 3 InvoiceSent 67890 1 InvoiceCreated 67890 2 InvoiceDeleted
- 86. REBUILDINGVIEWS 🌄 invoice_id version payload timestamp 12345 1 ... 14:05 12345 2 ... 14:06 12345 3 ... 15:50 67890 1 ... 15:30 67890 2 ... 15:33
- 87. REBUILDINGVIEWS 🌄 invoice_id version payload timestamp 12345 1 ... 14:05 12345 2 ... 14:06 67890 1 ... 15:30 67890 2 ... 15:33 12345 3 ... 15:50
- 88. REBUILDINGVIEWS 🌄 invoice_id version payload order 12345 1 ... 1 12345 2 ... 2 67890 1 ... 3 67890 2 ... 4 12345 3 ... 5
- 89. KAFKA & KAFKA STREAMS 101
- 90. APACHE KAFKA • Distributed append-only log • Replicated, fault-tolerant • Often used as pub-sub or queue • Used heavily at LinkedIn, Netflix, Wix and many others
- 98. KAFKA STREAMS
- 101. STREAMS ✈ • "Data in flight" • Unbounded, continuously updating data set • Ordered, replayable, sequence of immutable data key-value pairs
- 102. TABLES 😴 • "Data at rest" • A collection of evolving facts • A point-in-time view of aggregated data
- 103. STREAM-TABLE DUALITY
- 104. STREAM-TABLE DUALITY
- 105. STREAM-TABLE DUALITY Stream of all changes to a table
- 106. STREAM-TABLE DUALITY Stream of all changes to a table Materialize a stream into a table
- 108. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1
- 109. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
- 110. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1)
- 111. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1)
- 112. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2)
- 113. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1
- 114. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1
- 115. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
- 121. STREAMTRANSFORMATIONS • map • filter • flatMap
- 122. STREAMTRANSFORMATIONS • map • filter • flatMap • …
- 123. STREAMTRANSFORMATIONS • map • filter • flatMap • …
- 133. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data
- 134. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing
- 135. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations
- 136. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app
- 137. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app • Elastic, scaleable, fault-tolerant
- 139. STREAM PROCESSING APP Streams API Your app Streams API Your app Streams API Your app
- 141. PROCESSOR API • The most low-level • Interact with state-stores, schedulers, etc. • All standard operations are implemented like this (map / filter / …) • Create your own custom processing logic!
- 142. STREAMS DSL • Programmatically describe your topology
- 143. STREAMS DSL val builder = new StreamsBuilder • Programmatically describe your topology
- 144. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") • Programmatically describe your topology
- 145. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines • Programmatically describe your topology
- 146. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("W+")) • Programmatically describe your topology
- 151. KSQL • SQL dialect for streaming data CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
- 159. HIGH LEVEL DESIGN UserInterface Command Handler Stream processing app
- 163. HIGH LEVEL DESIGN UserInterface ProjectorProjector Command Handler Command Handler Command Handler
- 165. COMMAND HANDLER
- 181. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store Short retention (~1 hour)
- 186. PROJECTOR
- 191. API DB Business Logic Persistance Service #2 Service #3 Service #4
- 193. Invoices
- 198. WINS 🏆
- 199. WINS 🏆 • Simple and declarative system
- 200. WINS 🏆 • Simple and declarative system • Atomicity - Kafka used as event-store + notification
- 201. WINS 🏆 • Simple and declarative system • Atomicity - Kafka used as event-store + notification • Eventual consistency is handled gracefully
- 202. WINS 🏆 • Simple and declarative system • Atomicity - Kafka used as event-store + notification • Eventual consistency is handled gracefully • Easy to add or change views
- 203. DEMO 🥁
- 204. TAKEAWAYS 🍔
- 205. TAKEAWAYS 🍔 • Event driven systems and event sourcing can help create very flexible and scalable systems
- 206. TAKEAWAYS 🍔 • Event driven systems and event sourcing can help create very flexible and scalable systems • Know your tradeoffs (consistency, schema evolution, debugging, error handling, …)
- 207. TAKEAWAYS 🍔 • Event driven systems and event sourcing can help create very flexible and scalable systems • Know your tradeoffs (consistency, schema evolution, debugging, error handling, …) • Kafka & Kafka Streams are powerful tools that can be employed in many use cases
- 208. Stream on!
- 209. RESOURCES 📚 • Demo code: https://github.com/amitayh/event-sourcing-kafka-streams • Event sourcing by GregYoung - https://youtu.be/8JKjvY4etTY • Martin Kleppmann - Is Kafka a Database? https://youtu.be/v2RJQELoM6Y • Kafka Streams docs - http://wix.to/00C2ADs • Blog post from Confluent - http://wix.to/Z0C2ADs
- 210. Q&A 🤓