1.
Real Time Investment Alerts @ ING
using Apache Kafka & Spring Kafka
Marcos Maia & Tim van Baarsen
SPRING.IO BARCELONA – MAY 2019

2.
Introduction
Tim van Baarsen
Software Engineer at ING
@TimvanBaarsen
Marcos Maia
Software Engineer at ING
@thegroo

3.
ING is active in more than 40 counties
3
https://www.ing.jobs

4.
Agenda
4
• Use case @ ING
• Mifid II
• Mobile Investments App
• Overall Architecture
• Demo
• Metrics / Monitoring
• Kafka streams
• Local development / Code walk through
• Lessons learned
• Questions

5.
Disclaimer
5
Both the code and high level architecture we show here today is
not the actual product we developed within ING!
This example is inspired by our work

6.
Domain Lingo
6
Tick Price update on stock=

7.
Mifid II: “Inform customers when a stock drops -10% in price
at the end of the day!”
7
Mifid II
Regulations
!
At the end of the day?
"No, in real time!Markets in Financial Instruments Directive II =
Regulatory reform financial markets European Union
came into effect on January 3, 2018

8.
Screens of our mobile Investments app
8
price going up!
specific price
-10% drop
in price
Recurring alerts

9.
2.000 – 12.000 stock updates a second!
9
! "

10.
The perfect “match”
10
❤

11.
Overall architecture
11
Alert service
A
P
I
Matching
Service
Stocks
update
receiver
Internet
“SMS
Platform”
A
P
I
Receive stock price
updates ~
12.000/sec
Publish stock price
updates to Kafka
Create alerts
Delete alerts
List alerts Customer
receives
investment
alert
Consume updates
& match against
alerts created by
customers
In case of a match
send a sms to
customer
Customer alert(s)
stored in database
Notification
Service
Schema
Registry
Zookeeper
“Push
Platform”
A
P
I
!

12.
Quick Demo
12

13.
Monitoring - Metrics
13
• Micrometer
• Think SLF4J, but for metrics (Jon Schneider)
• Application metrics
• Out of the box metrics
• Business metrics
• Vendor Neutral

14.
Monitoring – Real-time?
14

15.
Kafka Streams Join
15
Matching
Service
raw-stock-quote stock-quote-tick stock-quote-meta
log-compaction
Raw-tick
stock-quote-ticker RocksDB
Key value
ING Raw-tick
GOOGL Raw-tick
AAPL Raw-tick
Key value
ING Meta-data
GOOGL Meta-data
AAPL Meta-data
GlobalKTableKStream

16.
Local development
16
• Docker
• Kafka
• Zookeeper
• Kafka manager
• All our Spring Boot applications
• Producers for development
• Custom Kafka command line producer
• Lab project

17.
Spring Kafka
17
• Producing:
• ProducerFactory
• KafkaTemplate
• “The producer is thread safe and sharing a single producer instance
across threads will generally be faster
than having multiple instances.”
• Consuming
• Kafka consumers are not thread safe
• @KafkaListener
• MessageListener (many different interfaces)
• Access to ConsumerRecord
• Access to manual control to commit the offset
• Scale up multiple consumer (threads) within the application

18.
Spring Kafka
18
• Integration tests
• Junit rule: EmbeddedKafkaRule
• @EmbeddedKafka
• Metrics
• Kafka consumer metrics out of the box (Micrometer)

19.
Lessons learned
19
• Use Spring Kafka after you understand the Kafka APIs
• Error handling
• (De)serialization -> Apache Avro
• Think about # of partitions per topic upfront if possible (routing)
• Can your consumers keep up? Can you scale up?
• Monitor your topics / streams in production
• Based on metrics -> Micrometer

20.
Lessons learned
20
• Kafka clients are backward compatible with Kafka brokers
• Kafka Streams
• KTables
• GlobalKTables
• Interactive Queries
• RocksDB build for ssd
• Describe
• processing.guarantee=exactly_once
• Kafka topology
• Plan ahead
• small & multiple topologies

21.
Questions
21
!

22.
Contact
Tim van Baarsen
Software Engineer @ ING
@TimvanBaarsen
Marcos Maia
Software Engineer @ ING
@thegroo