With the rising adoption of stream- and event-driven processing microservice architectures are becoming more and more complex. One challenge that many businesses face during the initial and ongoign development of these solutions is how to properly model and maintain dependencies between microservices. One specific example for this that will be used throughout this talk cleansing and enrichment of data that has been ingested into a streaming platform. For most use cases there are a lot of minor tasks that need to be performed on every piece of data before it is fully usable for processing. Some common examples are: normalize phone numbers, normalize street addresses, geocode addresses, lookup customer data and enrich record, ... Most of these tasks are completely independent of each other, but some have dependencies to be run before or after other tasks - geocoding, for example, should be done only after address normalization has finished. Defining and orchestrating a complex graph of these operations is no small feat. This talk will focus on outlining the requirements and challenges that one needs to solve when trying to implement a flexible framework for solving this use case. It will then build on these requirements and present the blueprint of a generic solution and show how Kafka and Kafka Streams are a perfect fit to address and overcome most challenges. This talk, while offering some technical details is mostly targeted at people at the architecture, rather than the code, level. Listeners will gain a thorough understanding of the challenges that stream processing offers but also be provided with generic patterns that can be used to solve these challenges in their specific infrastructure.
How to Troubleshoot Apps for the Modern Connected Worker
Building and Evolving a Dependency-Graph Based Microservice Architecture (Lars Francke, OpenCore GmbH & Co.KG) Kafka Summit SF 2019
1. Building and Evolving a Dependency-Graph
Based Microservice Architecture
Lars Francke – Partner and Co-Founder @ OpenCore
Kafka Summit 2019 – 30 September 2019