Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Real Time Investment Alerts @ ING
using Apache Kafka & Spring Kafka
Marcos Maia & Tim van Baarsen
SPRING.IO BARCELONA – MA...
Introduction
Tim van Baarsen
Software Engineer at ING
@TimvanBaarsen
Marcos Maia
Software Engineer at ING
@thegroo
ING is active in more than 40 counties
3
https://www.ing.jobs
Agenda
4
• Use case @ ING
• Mifid II
• Mobile Investments App
• Overall Architecture
• Demo
• Metrics / Monitoring
• Kafka...
Disclaimer
5
Both the code and high level architecture we show here today is
not the actual product we developed within IN...
Domain Lingo
6
Tick Price update on stock=
Mifid II: “Inform customers when a stock drops -10% in price
at the end of the day!”
7
Mifid II
Regulations
!
At the end of...
Screens of our mobile Investments app
8
price going up!
specific price
-10% drop
in price
Recurring alerts
2.000 – 12.000 stock updates a second!
9
! "
The perfect “match”
10
❤
Overall architecture
11
Alert service
A
P
I
Matching
Service
Stocks
update
receiver
Internet
“SMS
Platform”
A
P
I
Receive ...
Quick Demo
12
Monitoring - Metrics
13
• Micrometer
• Think SLF4J, but for metrics (Jon Schneider)
• Application metrics
• Out of the box...
Monitoring – Real-time?
14
Kafka Streams Join
15
Matching
Service
raw-stock-quote stock-quote-tick stock-quote-meta
log-compaction
Raw-tick
stock-quo...
Local development
16
• Docker
• Kafka
• Zookeeper
• Kafka manager
• All our Spring Boot applications
• Producers for devel...
Spring Kafka
17
• Producing:
• ProducerFactory
• KafkaTemplate
• “The producer is thread safe and sharing a single produce...
Spring Kafka
18
• Integration tests
• Junit rule: EmbeddedKafkaRule
• @EmbeddedKafka
• Metrics
• Kafka consumer metrics ou...
Lessons learned
19
• Use Spring Kafka after you understand the Kafka APIs
• Error handling
• (De)serialization -> Apache A...
Lessons learned
20
• Kafka clients are backward compatible with Kafka brokers
• Kafka Streams
• KTables
• GlobalKTables
• ...
Questions
21
!
Contact
Tim van Baarsen
Software Engineer @ ING
@TimvanBaarsen
Marcos Maia
Software Engineer @ ING
@thegroo
Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka
Nächste SlideShare
Wird geladen in …5
×

Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka

932 Aufrufe

Veröffentlicht am

Slides of our talk at:

Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka.
https://2019.springio.net/sessions/real-time-investment-alerts-using-apache-kafka-spring-kafka-at-ing-bank

By Tim van Baarsen & Marcos Maia

Nowadays our customers are expecting real time interactions with investments products and services we deliver at ING Bank especially because financial market fluctuations can have a direct impact on the investments performance of our customers.

In this talk, we present how we deal with the massive stream of price updates on stocks and how we leverage many features of Apache Kafka and Spring Kafka to improve and simplify our solutions. From simple producer / consumer up to advanced usage of streams we strongly rely on Kafka to solve our functional and non functional requirements delivering high quality, real time stream processing software that is reliable, precise and extremely valuable to our customers.

Main challenges of our project One of the main challenges of this project was to deal with a massive continuous stream of stock price updates and matching them against thousands of alerts created by our customers and still keep up with sending real time alerts to them.

Veröffentlicht in: Technologie

Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka

  1. 1. Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka Marcos Maia & Tim van Baarsen SPRING.IO BARCELONA – MAY 2019
  2. 2. Introduction Tim van Baarsen Software Engineer at ING @TimvanBaarsen Marcos Maia Software Engineer at ING @thegroo
  3. 3. ING is active in more than 40 counties 3 https://www.ing.jobs
  4. 4. Agenda 4 • Use case @ ING • Mifid II • Mobile Investments App • Overall Architecture • Demo • Metrics / Monitoring • Kafka streams • Local development / Code walk through • Lessons learned • Questions
  5. 5. Disclaimer 5 Both the code and high level architecture we show here today is not the actual product we developed within ING! This example is inspired by our work
  6. 6. Domain Lingo 6 Tick Price update on stock=
  7. 7. Mifid II: “Inform customers when a stock drops -10% in price at the end of the day!” 7 Mifid II Regulations ! At the end of the day? "No, in real time!Markets in Financial Instruments Directive II = Regulatory reform financial markets European Union came into effect on January 3, 2018
  8. 8. Screens of our mobile Investments app 8 price going up! specific price -10% drop in price Recurring alerts
  9. 9. 2.000 – 12.000 stock updates a second! 9 ! "
  10. 10. The perfect “match” 10 ❤
  11. 11. Overall architecture 11 Alert service A P I Matching Service Stocks update receiver Internet “SMS Platform” A P I Receive stock price updates ~ 12.000/sec Publish stock price updates to Kafka Create alerts Delete alerts List alerts Customer receives investment alert Consume updates & match against alerts created by customers In case of a match send a sms to customer Customer alert(s) stored in database Notification Service Schema Registry Zookeeper “Push Platform” A P I !
  12. 12. Quick Demo 12
  13. 13. Monitoring - Metrics 13 • Micrometer • Think SLF4J, but for metrics (Jon Schneider) • Application metrics • Out of the box metrics • Business metrics • Vendor Neutral
  14. 14. Monitoring – Real-time? 14
  15. 15. Kafka Streams Join 15 Matching Service raw-stock-quote stock-quote-tick stock-quote-meta log-compaction Raw-tick stock-quote-ticker RocksDB Key value ING Raw-tick GOOGL Raw-tick AAPL Raw-tick Key value ING Meta-data GOOGL Meta-data AAPL Meta-data GlobalKTableKStream
  16. 16. Local development 16 • Docker • Kafka • Zookeeper • Kafka manager • All our Spring Boot applications • Producers for development • Custom Kafka command line producer • Lab project
  17. 17. Spring Kafka 17 • Producing: • ProducerFactory • KafkaTemplate • “The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.” • Consuming • Kafka consumers are not thread safe • @KafkaListener • MessageListener (many different interfaces) • Access to ConsumerRecord • Access to manual control to commit the offset • Scale up multiple consumer (threads) within the application
  18. 18. Spring Kafka 18 • Integration tests • Junit rule: EmbeddedKafkaRule • @EmbeddedKafka • Metrics • Kafka consumer metrics out of the box (Micrometer)
  19. 19. Lessons learned 19 • Use Spring Kafka after you understand the Kafka APIs • Error handling • (De)serialization -> Apache Avro • Think about # of partitions per topic upfront if possible (routing) • Can your consumers keep up? Can you scale up? • Monitor your topics / streams in production • Based on metrics -> Micrometer
  20. 20. Lessons learned 20 • Kafka clients are backward compatible with Kafka brokers • Kafka Streams • KTables • GlobalKTables • Interactive Queries • RocksDB build for ssd • Describe • processing.guarantee=exactly_once • Kafka topology • Plan ahead • small & multiple topologies
  21. 21. Questions 21 !
  22. 22. Contact Tim van Baarsen Software Engineer @ ING @TimvanBaarsen Marcos Maia Software Engineer @ ING @thegroo

×