Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell Boomi

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 11 Anzeige

Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell Boomi

Herunterladen, um offline zu lesen

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit, and when it is not. The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage, and ksqlDB as an event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit, and when it is not. The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage, and ksqlDB as an event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell Boomi (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell Boomi

  1. 1. Building stateful applications using EDA - A use case walk-through using Kafka 1 Premjit Mishra Technology Evangelist E: premjitmishra@boomi.com
  2. 2. 2 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. App Built on Low Code and Kafka Disclaimer: The views mentioned in this presentation may not reflect my organization’s views or reflect new features in Kafka. View Controller Model Request / Response OSS SDK
  3. 3. 3 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. What was the motivation to use Kafka? We had a MVC solution to build which needed: (1) A real time event platform. (2) A DB to persist some of the work-flows states. We had a Kafka installation already, so why not try it instead of provisioning for a DB.
  4. 4. 4 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. Extractor Service GCP NL API Insights Service KSQL GCP NL automaticTopic outboundTopic manualTopic reportingTopic ➢ Leads ➢ Opportunity ➢ Account Feed ➢ Case Application stack with Boomi & Kafka Connect Service Intelligence Service Analytics/Data Service Synchronize Service Tickets
  5. 5. 5 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. Dashboard Worklist Workflow Data Lineage Intelligent Router App
  6. 6. 6 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. What worked well? ➢Agility The ability to add a new feature quickly was the most important advantage we found. ➢Realtime Integration and Storage in one platform Take advantage of Events/Streaming + Storage for realtime integration with source and sink systems in one platform. ➢Schema Support We could give a structure to our events using JSON, publish it and use it in our apps for publishing and consuming from Kafka.
  7. 7. 7 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. So, what were the challenges (DB vs Kafka)? Traditional Database Kafka • Has Updates • How do I update in an append only log? ➢Used Offset Commit ➢Topic chaining.
  8. 8. 8 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. manualTopic automaticTopic How we managed Update for workitems in UI? Completed Worklist Pending Worklist 0 1 2 3 0 Commit Offset
  9. 9. 9 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. Continue.. Traditional Database Kafka Has Updates. How do I update in an append only log ? ➢ Used Offset Commit ➢ Topic chaining. Has SQL features like WHERE, JOIN and aggregate functions like SUM, AVG etc. Used ksql TABLES for grouping functions like SUM, GROUP BY. Ability to use ORDER clause Ordering not possible if # of partitions >1 Indefinite time for storage Keep the operational data for X days and move to data warehouse for longer duration,reporting and audit requirements. Theoretically, we can also have indefinite retention period in Kafka. Well defined ACID properties and Transaction Management. As it’s an append only log, transactional commit is not possible.
  10. 10. 10 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved. Key Take Aways ➢ Prototyping helps to unlock new product features or try new ways of doing things, quickly. Low code helps. ➢ Kafka with DB.
  11. 11. 11 Copyright © 2020, Boomi A Dell Technologies Business and/or its affiliates. All rights reserved.

×