1.
Accelerating Innovation
with Apache Kafka
Heikki Nousiainen, CTO, Aiven
Thom Duran, Director of SRE, Moogsoft

2.
Heikki Nousiainen
CTO, Aiven
htn@aiven.io
@hnousiainen
Thom Duran
Director of SRE, Moogsoft
thom.duran@moogsoft.com
@livingeek

3.
Data
MONITORING
Data
VISUALIZATION
Data
ANALYSIS
Data
STORAGE
Data
INGESTION
NO VENDOR LOCK-IN
Apache Kafka,
Kafka Connect
(25+ connectors)
Stream Processing Framework
NEXT
YEAR
Relational DBs
Flink
PostgreSQL MySQL
Non-relational / noSQL DBs
Redis
Cassandra
Elasticsearch
InfluxDB
Time Series DB
Search Engine
M3
Clickhouse Grafana
Visualization
Making developers’ lives easier
Prometheus
Jolokia Aiven metrics
Syslog
Log Integration
Metrics systems
integration
Datadog

4.
The way we use data is changing
Future proof &
Easy to scale
High
performance
Manage large
amounts of data
in real time and
at a global scale
Access & High
security
Dynamically
grow and evolve
as the data
needs change
Keep data always
accessible and
secure in the
right hands

5.
Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Decoupled Architecture
Kafka centric architecture, breaking down the monoliths.
Independent processing steps bring about faster
development cycles, scalability and resilience.

6.
Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Message Content as Contract
Versioned Message Schema as the primary
communication contract.
Rapid evolution, versioning with ability to process messages
from the past and the future.
Version transformations and parallel flows.

7.
Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Access to & Re-Use of
Existing Data
Single source of truth. Always accurate, always complete.
Tap on existing data readily available for experimentation.
Parallel flows, development & verification.
Data Projections.

8.
Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Business Agility
Common API. Common architecture stack.
Active community & expertise available.
Rich ecosystem and links to other systems and services.

9.
Decoupling
Contracts
Access
Re-Use
Agility
Continuity
Business Continuity
Open Source.
Guaranteed access to your data.
Run everywhere; revisit when necessary.

10.
Moogsoft

11.
As the world grows ever more complex, we stick to our belief that
simplicity is the key to greater achievement. And we never forget that AI
is built on human designs, dreams and desires. That’s why everything
Moogsoft makes helps people and machines work more harmoniously,
to create clarity from chaos and expedite innovation.
Who Is Moogsoft

12.
High Level Architecture
User
CDN/Ingress
Moogsoft
Services (K8)
Kafka
Data
Storage

13.
As we moved from a service that was capable of handling the requests
of a single customer to a pipeline that could manage many, we knew
Rabbit wasn’t the solution.
We went with kafka due to its ability to replay data, quick write
capabilities, and ability to act as a producer consumer event driven
workflow for our services.
Hopping off Rabbit

14.
After selecting Kafka we realized that we couldn’t do it on our own,
and wanted to lean on a service provider to manage Kafka for us.
Our requirements at a high level were:
● Dynamically scalable
● Upgradable
● IAC (preferably Terraform)
● Exportable Metrics
● Backed by strong support
The Vendor Hunt

15.
● Implement Terraform (Infra build/peering)
● Provide cert based access details
● Roll Kafka per Environment (dev/test/stage/prod)
● Engage with dev to flip Kafka library to use new auth method
and talk to Aiven
● Configure Prom to scrape metrics from Aiven
Self Serve -> Managed Kafka

16.
● Upgrades
● Scale Up
● Scale Down
● Spin up
● Tear Down
● Modify env variables
Putting it through its Paces

17.
● Right Sizing
● Troubleshooting
● Proactive Fix Actions
● General Knowledge on Kafka
Final Thoughts

18.
Thank you
Heikki Nousiainen
CTO, Aiven
htn@aiven.io
@hnousiainen
Thom Duran
Director of SRE, Moogsoft
thom.duran@moogsoft.com
@livingeek