Suraj Pillai, Confluent, Senior Solutions Engineer To understand the importance of being event driven, we’ll examine why events have become so pivotal in our thinking today. We will then evaluate the qualities and how events have become a first-class concern for the modern organisation, as awareness of events underpins event-first thinking and design. https://www.meetup.com/Chennai-Kafka/events/276313770/

1. Meetup
Journey to Event-Driven Architecture
Suraj Pillai
Senior Solutions Engineer
Confluent
/suraj-pillai-confluent

2. Normal
Applications
(i.e., monoliths)

3. Monoliths
are hard to
think about

4. Monoliths
are hard to
change

5. 5
Re-
integration

6. There are no good
ways to integrate
microservices

7. Filesystem

9. Database

10. Integrating
Microservices
through the
database
● “I have a database and I know how to use it.”
● Eventually causes services to co-mingle.
● Violation the «bounded context»
● Great to use inside a service boundary!
● Terrible for sharing data or negotiating change.

11. RPC

12. Integrating
microservices via
RPC
● Avoids problems of database integration
● Feels natural
● Aligns with the request/response paradigm
● Problem: cascading failures
● Question: how do you debug this system?
● Answer: you build a log.
🤔

13. Events?

14. What’s an
event?

15. A shared narrative describing the
evolution of the business
over time

16. A combination of:
Notification
State transfer
What’s an event?

17. Also,
events
are immutable.
What’s an event?

18. There has been a revolution; organizations must become real time;
to become real time, they must become event driven

19. Transitioning to event-first thinking
Event-first thinking changes how you think about what you are building

20. Event-first analog: I walk into a room, generate an “entered room” event and the light turns on.
This is a reaction to an event.
Event-command analog: I walk into a room, flip the light switch and the light turns on. This is a
command.

21. Event-command pattern

22. Event-first pattern

23. Event-first pattern

24. Benefits of the event-first approach
● Decoupling: Processors don’t know anything about upstream or downstream processors
● Encapsulation: There are clean boundaries between processors.
● Evolutionary change: The system and events can change over time.
● Event sourcing: When using a log and log-aware stream processors, we gain the ability to
potentially rebuild and restore application state.

25. Evolution of
Financial Services

26. 60’s -90’s
Mainframe
Observations:
1. Primary used for reconciliation
and bookkeeping functions.
2. Long EOD processes
3. Clearing functions largely
manual.
4. Reduced fraud between
collection and reconciliation.
5. Allowed for easier and more
accurate bookkeeping and
profit reporting.
6. Gave the birth to a generation
of super-tellers
7. Most if not all functionalities
within mainframe.
3270
26

27. The early 90’s
Mainframe
Observations:
1. GUI interface at branch
resulting in higher productivity.
2. Newer services available at
branches
3. Lowered cost of operations.
4. Clearance & settlement
activities become faster.
5. EOD becomes faster and
streamlined.
6. Newer challenges - cost of
operations of branches, cash
handling, IT teams at branches
and HQ.
7. Risk management - Basel
Accords come into existence.
Normal rule based haircuts
imposed.
3270
Branch
Teller
Systems
Branch
Server
MQ
27

28. The mid 90’s
Mainframe
Observations:
1. GUI interface at branch
resulting in higher productivity.
2. Newer services available at
branches
3. Lowered cost of operations.
4. Clearance & settlement
activities become faster.
5. EOD becomes faster and
streamlined.
6. Newer challenges - cost of
operations of branches, cash
handling, IT teams at branches
and HQ.
7. A new thing called the internet
makes an appearance.
8. ERP systems optimise back-
office functions.
3270
Branch
Teller
Systems
Branch
Server
MQ
ATM Website
Tandem - Base24
ISO 8583
HR Finance Risk
Custom
28

29. The first decade of the 2000’s
Core
Banking
Observations:
1. Emergence of Core Banking
players and tremendous
optimisation of banking
operations.
2. Introduction of SOA & ESB
patterns resulting in scale and
unbundling of capabilities.
3. Emergence of new channels -
Internet and in the later half
mobile.
4. Fraud and Anti-money
laundering systems make an
entry.
5. Large IT teams and outsourcing
contracts.
6. Systems evolve into monoliths
and become difficult to change.
7. Banks become super powerful
as controllers of data and
customer information.
8. Emergence of Silicon Valley
9. Real-time payments emerge.
10. 24 x 7 services arrive.
Contact
Center
Core
enabled
branches
ATM
Internet
Banking
Tandem - Base24 HR
Finance Risk
Mobile
Banking
Account Origination Enterprise Data Warehousing
Liquidity Management Customer 360 CRM
Doc Mgmt. Campaign Mgmt. Business Analytics
ESB/ Integration Layer
Payments Credit Card AML/Fraud
SOA
Layer
29

30. The second decade of the 2000’s
Core
Banking
Observations:
1. The platform ecosystem
becomes the primary
acquisition and servicing
engine.
2. APIs reduce the cost of
transactions dramatically.
3. Microservices and event
driven architectures makes
features and functions de-
coupled which enables
faster change to systems &
processes and also faster
time to market for new
apps.
4. Security, fraud, risk & AML
become more real-time
5. Newer sources of revenue
emerge.
6. Payments becomes a
different business for banks.
7. Silicon Valley leads and IS
the competitor.
8. Insourcing happens at scale.
9. COVID -19 hastens digital
transformation as electronic
transactions dominate.
10. Cloud and Hybrid Cloud
Contact
Center
Core enabled
branches
ATM
Internet
Banking
Tandem - Base24 HR
Finance Risk
Mobile
Banking
ESB Layer
Payments Credit Card AML/Fraud
Ecosystem
External API layer
Customer
microservice
Product
Notification
Cross-Sell
Payment
Streaming Caching Internal API GW
Data Lake
Personalisation
Cust
360
Gov
Services
Microservices
30

31. Kafka- Event
Streaming
Platform

32. Streaming Platform
Pub / Sub
Storage
Processing

33. Summary:
1. Kafka used for event-
driven multi-legged
transactions spanning
Validation, Fraud
Detection, etc etc
2. Microservices developed
in Kstreams
3. Kafka Connect used to
connect other systems
such as Mainframes via
IBM MQ
Payment - Post Payment
API
Management
Layer
Validation
Fraud
Compliance
Mainframe
Payment
Commit
Connect
POST
/payment

34. Event-driven payment processing

35. Summary:
1. Get Payment status service
2. This KStreams service
joins data from multiple
Kafka topics
3. A materialized cache is
created in Kafka (indicated
by the “green” topic) which
has details on the payment
status. This data can then
be stored in a DB or
consumed by downstream
microservices
Payment- Get Payment Status
API
Management
Layer
Fraud Compliance Enrichment
Payment
Tracking
GET
/payment/{id}
Consume Events

36. Four pillars of event streaming
1. Business function: The payment processing pipeline
2. Instrumentation plane: How much money is transferred? How quickly are payments processed?
3. Control plane: Patterns to start, stop, pause, coordinate and autoscale
4. Operational plane: Dead letter queues (DLQs), error tracking & handling, deployment, config
management

37. Thank You
/suraj-pillai-confluent