“Data is the new oil”, and we all need to get our hands dirty: we build software and infrastructures to manage data properly, storing and getting it where it creates value. However, systems and teams tend to inevitably be slowed down as data grows in size and complexity. Architectural and organizational decisions can make the difference: in this talk, we’ll discuss how microservices, stream processing and event-driven architectures can help fight “data gravity”, thus enabling the development of more evolvable and reliable data-intensive applications.

1. CONFIDENTIAL INFORMATION
Fight Data Gravity With
Event-Driven Architectures
Matteo Ferroni, CTO

2. CONFIDENTIAL INFORMATION
Fully Managed
Digital Products
Bottega52© is your end-to-end partner during the
whole lifecycle of your digital asset, so you can focus
on your core business.
BOTTEGA52©

3. CONFIDENTIAL INFORMATION
Digital Products
● New opportunities and new
business models
● Efficiency through process
automation
● Improve communication,
visibility and alignment
through data
3
How they create value

4. CONFIDENTIAL INFORMATION
It All Revolves
Around Data
● Get and validate data from
users
● Process and generate new
information
● Show data in a well organized
manner
● Request and expose data for
other system
4
Moving data back and forth between
the database and the user’s screen

5. CONFIDENTIAL INFORMATION
Database,
My Old Friend
● Store the most valuable
information, reliably
● Control accesses and
modifications
● Allows complex queries,
transactions, etc.
● Grows over time, in size and
complexity (e.g., triggers,
stored procedures, etc.)
...Until it’s too big to fail.
5
At the core of every digital product

6. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION
Data Velocity
Is a Challenge
We need to store, process and access
the increasing amount of data in a
timely manner
“Data if large enough can be
virtually impossible to move.” —
Dave McCrory on Data Gravity
6
6
Source: https://datagravitas.com/2010/12/07/data-gravity-in-the-clouds/

7. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION
Business Velocity
Is a Challenge
Software and data need to evolve
with the business
● diversity of data use cases and
users
● proliferation of sources of data
7
7
“Data is the new oil” —
Reinterpretation: “To manage it
properly, your software pipelines
will get dirty (and fragile)”
Source: https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data

8. CONFIDENTIAL INFORMATION
The Rise Of
Microservices
Operational point of view:
● bounded context & team
ownership
● single process / container, for
easier evolution & delivery
● synchronous communication
(REST APIs, gRPC, etc.)
● asynchronous communication
(message passing, ESB, etc.)
8
Split the problem into smaller pieces
“Microservices are an architecture
that optimise for autonomy” —
James Lewis

9. CONFIDENTIAL INFORMATION
Information
Hiding
● Segregate design decisions
that are subject to change
● Define stable interfaces and
contracts
● Avoid access to
implementation details
Data is the ultimate
implementation detail
that can be shared.
9
A new idea, from the ‘70s

10. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION
Dichotomy
10
● Services encapsulate the data
they hold, to reduce coupling
and aid reuse
● Databases amplify the data they
hold, to provide greater value to
their user
10
From the book: “Designing Event-Driven Systems”, by Ben Stopford

11. CONFIDENTIAL INFORMATION
Tight coupling lets components extract more value
from one another
How to balance loose coupling
and information sharing?

12. CONFIDENTIAL INFORMATION
Event-Driven
Architectures
● Events as “first-class” entities
● Services emit and consume
events asynchronously —
No temporal coupling
● Better decoupling with respect
to Queries and Commands —
No expectations
12
Centralize an immutable and
append-only stream of “business
facts” — Events
Decentralize the freedom to act,
adapt, and change — Decoupling

13. CONFIDENTIAL INFORMATION
MIXING COMMANDS,
QUERIES AND EVENTS
From the book: “Designing Event-Driven Systems”, by Ben Stopford

14. CONFIDENTIAL INFORMATION
NOTIFICATION &
DATA REPLICATION
Improve autonomy, faster and (potentially) offline
data access
Event-carried state transfer
From the book: “Designing Event-Driven Systems”, by Ben Stopford

15. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION 15
15
Events, centralized:
are we sharing a
database?
How to deal with:
Events are a conscious decision to
share data, part of the external
interface
Not sharing the whole data: smaller
scope, easier to maintain, guarantee
security and privacy
● Infinite event log retention
● Scalability
● Availability
● Performance
● Contracts on data models

16. CONFIDENTIAL INFORMATION
Apache Kafka
Ecosystem
16
An open-source distributed event
streaming platform
Provides the immutable append-only
log abstraction
● Infinite event log retention
● Scalability
● Availability
● Performance
● Contracts on data models
Icons from: https://kafka.apache.org/
Image from the book: “Designing Event-Driven Systems”, by Ben Stopford

17. CONFIDENTIAL INFORMATION
DATA MODELS
EVOLUTION
Source: https://docs.confluent.io/platform/current/schema-registry/index.html
Confluent’s Schema Registry allows backward and/or
forward compatibility checks, as well as schema
versioning and sharing

18. CONFIDENTIAL INFORMATION
RICH ECOSYSTEM
FOR DATA INTEGRATION
Source: https://debezium.io/documentation/reference/architecture.html
You can easily turn a database “inside out” with:
● Debezium, a set of connectors to capture
changes in your databases
● Kafka Connect, a framework and a runtime to
stream data from and to Kafka

19. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION 19
19
Reporting and
Analytics
Operational Data Plane — Running
the business, serving the users, etc.
Analytical Data Plane — Optimizing
the business, augmenting the user
experience with intelligence
Source: https://martinfowler.com/bliki/ReportingDatabase.html
Source: https://martinfowler.com/articles/data-mesh-principles.html

20. CONFIDENTIAL INFORMATION
THE ROAD TO
DATA MESH
Inspired by microservices: domain-oriented decentralized
data ownership
Event-driven architectures to help “self-serve data”
Source: https://martinfowler.com/articles/data-monolith-to-mesh.html

21. CONFIDENTIAL INFORMATION
Conclusions
● Event-driven architecture
enable Data Velocity and
Business Velocity, from an
Operational perspective
● Data Mesh as a new paradigm,
from an Analytical perspective
● Events are “the new oil” — If
you have the right tools to
build and maintain your new
data infrastructure as a
platform
21
“Life is a series of natural and spontaneous
changes events. Don’t resist them — that only
creates sorrow. Let reality be reality.
Let things flow naturally forward.“
— Lao-Tzu, 6th–5th century BCE

22. CONFIDENTIAL INFORMATION
CONFIDENTIAL INFORMATION
© Bottega52 Srl 2021. Tutti i diritti riservati
www.bottega52.it - info@bottega52.it
22
22
22
22
22