Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 18 Anzeige

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

Herunterladen, um offline zu lesen

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

  1. 1. Accelerating Innovation with Apache Kafka Heikki Nousiainen, CTO, Aiven Thom Duran, Director of SRE, Moogsoft
  2. 2. Heikki Nousiainen CTO, Aiven htn@aiven.io @hnousiainen Thom Duran Director of SRE, Moogsoft thom.duran@moogsoft.com @livingeek
  3. 3. Data MONITORING Data VISUALIZATION Data ANALYSIS Data STORAGE Data INGESTION NO VENDOR LOCK-IN Apache Kafka, Kafka Connect (25+ connectors) Stream Processing Framework NEXT YEAR Relational DBs Flink PostgreSQL MySQL Non-relational / noSQL DBs Redis Cassandra Elasticsearch InfluxDB Time Series DB Search Engine M3 Clickhouse Grafana Visualization Making developers’ lives easier Prometheus Jolokia Aiven metrics Syslog Log Integration Metrics systems integration Datadog
  4. 4. The way we use data is changing Future proof & Easy to scale High performance Manage large amounts of data in real time and at a global scale Access & High security Dynamically grow and evolve as the data needs change Keep data always accessible and secure in the right hands
  5. 5. Decoupling Contracts Access Re-Use Agility Continuity Decoupled Architecture Kafka centric architecture, breaking down the monoliths. Independent processing steps bring about faster development cycles, scalability and resilience.
  6. 6. Decoupling Contracts Access Re-Use Agility Continuity Message Content as Contract Versioned Message Schema as the primary communication contract. Rapid evolution, versioning with ability to process messages from the past and the future. Version transformations and parallel flows.
  7. 7. Decoupling Contracts Access Re-Use Agility Continuity Access to & Re-Use of Existing Data Single source of truth. Always accurate, always complete. Tap on existing data readily available for experimentation. Parallel flows, development & verification. Data Projections.
  8. 8. Decoupling Contracts Access Re-Use Agility Continuity Business Agility Common API. Common architecture stack. Active community & expertise available. Rich ecosystem and links to other systems and services.
  9. 9. Decoupling Contracts Access Re-Use Agility Continuity Business Continuity Open Source. Guaranteed access to your data. Run everywhere; revisit when necessary.
  10. 10. Moogsoft
  11. 11. As the world grows ever more complex, we stick to our belief that simplicity is the key to greater achievement. And we never forget that AI is built on human designs, dreams and desires. That’s why everything Moogsoft makes helps people and machines work more harmoniously, to create clarity from chaos and expedite innovation. Who Is Moogsoft
  12. 12. High Level Architecture User CDN/Ingress Moogsoft Services (K8) Kafka Data Storage
  13. 13. As we moved from a service that was capable of handling the requests of a single customer to a pipeline that could manage many, we knew Rabbit wasn’t the solution. We went with kafka due to its ability to replay data, quick write capabilities, and ability to act as a producer consumer event driven workflow for our services. Hopping off Rabbit
  14. 14. After selecting Kafka we realized that we couldn’t do it on our own, and wanted to lean on a service provider to manage Kafka for us. Our requirements at a high level were: ● Dynamically scalable ● Upgradable ● IAC (preferably Terraform) ● Exportable Metrics ● Backed by strong support The Vendor Hunt
  15. 15. ● Implement Terraform (Infra build/peering) ● Provide cert based access details ● Roll Kafka per Environment (dev/test/stage/prod) ● Engage with dev to flip Kafka library to use new auth method and talk to Aiven ● Configure Prom to scrape metrics from Aiven Self Serve -> Managed Kafka
  16. 16. ● Upgrades ● Scale Up ● Scale Down ● Spin up ● Tear Down ● Modify env variables Putting it through its Paces
  17. 17. ● Right Sizing ● Troubleshooting ● Proactive Fix Actions ● General Knowledge on Kafka Final Thoughts
  18. 18. Thank you Heikki Nousiainen CTO, Aiven htn@aiven.io @hnousiainen Thom Duran Director of SRE, Moogsoft thom.duran@moogsoft.com @livingeek

×