Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 28 Anzeige

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter

Herunterladen, um offline zu lesen

Describing the convenience of building an event driven application using stream processing and leveraging the power of KSQL.
Events model our lives and actions be it machine or human generated. Apache Kafka used with KSQL help us transform and process events, using KSQL to express what we want to do with the respective data.
In this talk, we will be doing a live demo with a custom made application relaying any sort of data to an internal system/application/rest proxy, this will publish events which in turn will be processed and transformed via KSQL queries and analysed using a visual tool. All the while describing how just simple queries (continuous queries in terms of KSQL) can work the magic of complete applications.
We will be discussing broadly what streams and stream processing is and how does KSQL come into the picture to help us make our lives easier. This can be leveraged in any kind or nature of industry.

Describing the convenience of building an event driven application using stream processing and leveraging the power of KSQL.
Events model our lives and actions be it machine or human generated. Apache Kafka used with KSQL help us transform and process events, using KSQL to express what we want to do with the respective data.
In this talk, we will be doing a live demo with a custom made application relaying any sort of data to an internal system/application/rest proxy, this will publish events which in turn will be processed and transformed via KSQL queries and analysed using a visual tool. All the while describing how just simple queries (continuous queries in terms of KSQL) can work the magic of complete applications.
We will be discussing broadly what streams and stream processing is and how does KSQL come into the picture to help us make our lives easier. This can be leveraged in any kind or nature of industry.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter

  1. 1. Presentation Header here Stream Processing via Kafka and KSQL @namit_mahuvakar #kafkaSummit2021
  2. 2. 02. What is Stream Processing? table of contents 01. About Jupiter 03. Stream Processing with KSQLDB 04. KSQLDB in Jupiter 06. KSQL in the current Data Ecosystem
  3. 3. About Us Jupiter is a digital banking experience that provides: ● Digital banking experience for the millennials in India ● Re-imagining and re-building different banking functions, with technology-first approach ● A financial platform that constantly calculates and improves the financial wellness of users ● Smarter savings, Transparency, Personalised rewards
  4. 4. Banking Experience
  5. 5. What is Stream Processing? Logs DB events Customer Experience IoT Transactions and Purchases
  6. 6. What is Stream Processing?
  7. 7. Stream Processing with KSQL ● Immutable events being ingested ● KSQL built on top of Kafka Streams ● Continuous Query Language in SQL ● Ease of Transformation, filtering ● Creating derived streams
  8. 8. Stream Processing with KSQL create stream events_a as select * from events where type = "A"
  9. 9. Transaction Reconciliation Jupiter as a banking platform supports multiple types of transaction interfaces - ● UPI: (Unified Payments Interface) ● Bank Transfers: (NEFT, IMPS, RTGS, IFT) ● Card Transfers: (Debit Card Transactions) Transactions on certain interfaces can be initiated on the application.
  10. 10. Transaction Reconciliation Problem Statement : Need to aggregate all events to enrich the individual transaction record. Sources - ● Interface Callbacks ● Bank Accounting Systems Challenges - Time delay between events/Synchronisation ?
  11. 11. Demo
  12. 12. Rewards Eligibility Problem Statement: Utilise incoming streams of data for, ● Check rewards eligibility ● Give 10% Jewels for all UPI transactions ● Provide specific rewards to those eligible ● Utilise already consolidated derived stream
  13. 13. Demo
  14. 14. What more can you do ?
  15. 15. Stream Processing with KSQL
  16. 16. Stream Processing with KSQL
  17. 17. KSQLDB Operations and Challenges ● Deployment - Deploy via Docker/K8s, images available at docker hub ● Tear down stack ● Monitoring - Enable explicitly for JMX metrics to be available ● Richer SQL grammar required ● Point in time select queries ● Checkpointing every 100ms ● Granular insights on consumer and stream performance
  18. 18. KSQL the one for the future ? Data consumption in real time allows us to act faster on opportunities and avoiding issues as and when they occur, ● KSQL allows anyone with SQL knowledge to process any data ● No need to write any code ● Need for real time data in an abstract environment is solved easily i.e. L0 metrics for platform alerting ● Need for integration with Big Data Platforms, spill to disk or directly to Data Lake
  19. 19. Thank you, Any Questions ?
  20. 20. Use Case : Transaction Reconciliation
  21. 21. Use Case : Rewards Eligibility
  22. 22. Stream Processing with KSQL

×