Slides of our talk at: Spring.io Barcelona May 2019 - Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka. https://2019.springio.net/sessions/real-time-investment-alerts-using-apache-kafka-spring-kafka-at-ing-bank By Tim van Baarsen & Marcos Maia Nowadays our customers are expecting real time interactions with investments products and services we deliver at ING Bank especially because financial market fluctuations can have a direct impact on the investments performance of our customers. In this talk, we present how we deal with the massive stream of price updates on stocks and how we leverage many features of Apache Kafka and Spring Kafka to improve and simplify our solutions. From simple producer / consumer up to advanced usage of streams we strongly rely on Kafka to solve our functional and non functional requirements delivering high quality, real time stream processing software that is reliable, precise and extremely valuable to our customers. Main challenges of our project One of the main challenges of this project was to deal with a massive continuous stream of stock price updates and matching them against thousands of alerts created by our customers and still keep up with sending real time alerts to them.

1. Real Time Investment Alerts @ ING
using Apache Kafka & Spring Kafka
Marcos Maia & Tim van Baarsen
SPRING.IO BARCELONA – MAY 2019

2. Introduction
Tim van Baarsen
Software Engineer at ING
@TimvanBaarsen
Marcos Maia
Software Engineer at ING
@thegroo

3. ING is active in more than 40 counties
3
https://www.ing.jobs

4. Agenda
4
• Use case @ ING
• Mifid II
• Mobile Investments App
• Overall Architecture
• Demo
• Metrics / Monitoring
• Kafka streams
• Local development / Code walk through
• Lessons learned
• Questions

5. Disclaimer
5
Both the code and high level architecture we show here today is
not the actual product we developed within ING!
This example is inspired by our work

6. Domain Lingo
6
Tick Price update on stock=

7. Mifid II: “Inform customers when a stock drops -10% in price
at the end of the day!”
7
Mifid II
Regulations
!
At the end of the day?
"No, in real time!Markets in Financial Instruments Directive II =
Regulatory reform financial markets European Union
came into effect on January 3, 2018

8. Screens of our mobile Investments app
8
price going up!
specific price
-10% drop
in price
Recurring alerts

9. 2.000 – 12.000 stock updates a second!
9
! "

10. The perfect “match”
10
❤

11. Overall architecture
11
Alert service
A
P
I
Matching
Service
Stocks
update
receiver
Internet
“SMS
Platform”
A
P
I
Receive stock price
updates ~
12.000/sec
Publish stock price
updates to Kafka
Create alerts
Delete alerts
List alerts Customer
receives
investment
alert
Consume updates
& match against
alerts created by
customers
In case of a match
send a sms to
customer
Customer alert(s)
stored in database
Notification
Service
Schema
Registry
Zookeeper
“Push
Platform”
A
P
I
!

12. Quick Demo
12

13. Monitoring - Metrics
13
• Micrometer
• Think SLF4J, but for metrics (Jon Schneider)
• Application metrics
• Out of the box metrics
• Business metrics
• Vendor Neutral

14. Monitoring – Real-time?
14

15. Kafka Streams Join
15
Matching
Service
raw-stock-quote stock-quote-tick stock-quote-meta
log-compaction
Raw-tick
stock-quote-ticker RocksDB
Key value
ING Raw-tick
GOOGL Raw-tick
AAPL Raw-tick
Key value
ING Meta-data
GOOGL Meta-data
AAPL Meta-data
GlobalKTableKStream

16. Local development
16
• Docker
• Kafka
• Zookeeper
• Kafka manager
• All our Spring Boot applications
• Producers for development
• Custom Kafka command line producer
• Lab project

17. Spring Kafka
17
• Producing:
• ProducerFactory
• KafkaTemplate
• “The producer is thread safe and sharing a single producer instance
across threads will generally be faster
than having multiple instances.”
• Consuming
• Kafka consumers are not thread safe
• @KafkaListener
• MessageListener (many different interfaces)
• Access to ConsumerRecord
• Access to manual control to commit the offset
• Scale up multiple consumer (threads) within the application

18. Spring Kafka
18
• Integration tests
• Junit rule: EmbeddedKafkaRule
• @EmbeddedKafka
• Metrics
• Kafka consumer metrics out of the box (Micrometer)

19. Lessons learned
19
• Use Spring Kafka after you understand the Kafka APIs
• Error handling
• (De)serialization -> Apache Avro
• Think about # of partitions per topic upfront if possible (routing)
• Can your consumers keep up? Can you scale up?
• Monitor your topics / streams in production
• Based on metrics -> Micrometer

20. Lessons learned
20
• Kafka clients are backward compatible with Kafka brokers
• Kafka Streams
• KTables
• GlobalKTables
• Interactive Queries
• RocksDB build for ssd
• Describe
• processing.guarantee=exactly_once
• Kafka topology
• Plan ahead
• small & multiple topologies

21. Questions
21
!

22. Contact
Tim van Baarsen
Software Engineer @ ING
@TimvanBaarsen
Marcos Maia
Software Engineer @ ING
@thegroo