Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 2021 Keynote

Product-thinking is making a big impact in the data world with the rise of Data Products, Data Product Managers, data mesh, and treating “Data as a Product.” But Honest, No-BS: What is a Data Product? And what key questions should we ask ourselves while developing them? Tim Gasper (VP of Product, data.world), will walk through the Data Product ABCs as a way to make treating data as a product way simpler: Accountability, Boundaries, Contracts and Expectations, Downstream Consumers, and Explicit Knowledge.

Technologie

Intelligent, real-time
backend operations

Software Architecture Historically
How do you build an
application in software?

Software Architecture Today
How do you build a
company in software?

Two movements:
Microservices & Data Mesh

General trend towards decentralization
Microservices Data Mesh
App
App App
App App
App DB
DB
DB
DB
DB
DB

Microservices break apart monolithic apps
Payment
Provider
Restocking
orders
Stock
Service
UI
Service
Order
Service

Classic Data Integration is Also A Monolith
Limited
Sources
Efﬁcient
Centralized
Data Eng
Bounded
users and
reports
Data
Warehouse

Data can also ﬂow as event streams
Order
DWH
Search
Index

The same event stream can do both
Order
DWH
Search
Index
Payment
Provider
Restocking
orders
Stock
Service
Order
Service
Event as Notiﬁcation,
Events as Data
Application plane,
Data plane
UI
Service

Time from start of adoption
Value
&
Difﬁculty
1
2
3
Application-
Level Kafka
Org-Level
Kafka
There are both organizational and technical barriers

4 Principles of Data Mesh
Domain
Ownership
Data as a
Product
Self-service
Data
Federated
Governance
Organize how data is produced Improve how data is consumed

Decentralize ownership,
organize around product
experts, create Data
Products
1. Organize around domain
experts that know the data
2. Create “Data Products”
anyone can access
3. Freedom to choose any
data tech inside a product
Data Product
Domain Experts
Data Product
Domain Experts
Data Product
Domain Experts
Data Product
Domain Experts

Why Kafka?
Real-time Scalable Multi-subscriber Universal

Works for pure events “what happened…”
EVENTS

Why is this better?
No centralized
analytic data
monolith
Data modeling
evolves with the
business
No long queue of
frustrated data
consumers
Field 1 Field 2 New Field
a 1 foo
b 2 bar

Conﬂuent Cloud: Fully Self Service
Easy cluster
creation
Easy transformations
with ksqlDB
Easy connection to
downstream systems

Conﬂuent Cloud: Fully Managed Makes It Easy
Inﬁnite Storage Elastic Expansion Automatic
Rebalancing

Companies are complex and
geographically distributed

Managing Replication
Making data accessible
across geographies and
environments is difﬁcult.
● Separate systems increase
cost and complexity
● Operational burden is high
● Offsets are inconsistent
between clusters

The challenge of decentralization: who’s in charge?
● Where do i ﬁnd
customer data?
● Which is the right ID
for a payment?
● This shipping data is
a day old, who
should I talk to?
Kafka

Introducing Stream Governance:
Data Governance for data in motion
Stream Quality
How do I avoid bad data?
Schema management:
No broken json goes in!
Stream Catalog
How do I ﬁnd the data
that I need?
Stream Lineage
Where did this data
come from?

Scalability is required for companies
to be represented in software
Kafka

Weitere ähnliche Inhalte

Was ist angesagt?

Data Lakehouse, Data Mesh, and Data Fabric (r1)

James Serra

The ABCs of Treating Data as Product

DATAVERSITY

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 4

Data Mesh for Dinner

Kent Graziano

Databricks Fundamentals

Dalibor Wijas

DW Migration Webinar-March 2022.pptx

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.

Modernizing to a Cloud Data Architecture

Introducing Databricks Delta

The traditional data warehouse has served us well for many years, but new trends are causing it to break in four different ways: data growth, fast query expectations from users, non-relational/unstructured data, and cloud-born data. How can you prevent this from happening? Enter the modern data warehouse, which is able to handle and excel with these new trends. It handles all types of data (Hadoop), provides a way to easily interface with all these types of data (PolyBase), and can handle “big data” and provide fast queries. Is there one appliance that can support this modern data warehouse? Yes! It is the Analytics Platform System (APS) from Microsoft (formally called Parallel Data Warehouse or PDW) , which is a Massively Parallel Processing (MPP) appliance that has been recently updated (v2 AU1). In this session I will dig into the details of the modern data warehouse and APS. I will give an overview of the APS hardware and software architecture, identify what makes APS different, and demonstrate the increased performance. In addition I will discuss how Hadoop, HDInsight, and PolyBase fit into this new modern data warehouse.

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...

DataScienceConferenc1

Modern Data Warehousing with the Microsoft Analytics Platform System

James Serra

Every business today wants to leverage data to drive strategic initiatives with machine learning, data science and analytics — but runs into challenges from siloed teams, proprietary technologies and unreliable data. That’s why enterprises are turning to the lakehouse because it offers a single platform to unify all your data, analytics and AI workloads. Join our How to Build a Lakehouse technical training, where we’ll explore how to use Apache SparkTM, Delta Lake, and other open source technologies to build a better lakehouse. This virtual session will include concepts, architectures and demos. Here’s what you’ll learn in this 2-hour session: How Delta Lake combines the best of data warehouses and data lakes for improved data reliability, performance and security How to use Apache Spark and Delta Lake to perform ETL processing, manage late-arriving data, and repair corrupted data directly on your lakehouse

Free Training: How to Build a Lakehouse

Data Mesh at CMC Markets: Past, Present and Future

Lorenzo Nicora

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...

Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling. In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse. She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product. This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.

Modern Data architecture Design

Kujambu Murugesan

Databricks Platform.pptx

Alex Ivy

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...

Data Con LA 2020 Description In this session, I introduce the Amazon Redshift lake house architecture which enables you to query data across your data warehouse, data lake, and operational databases to gain faster and deeper insights. With a lake house architecture, you can store data in open file formats in your Amazon S3 data lake. Speaker Antje Barth, Amazon Web Services, Sr. Developer Advocate, AI and Machine Learning

Owning Your Own (Data) Lake House

Data Con LA

Data Lakehouse Symposium | Day 1 | Part 2

A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape. Join this webinar to demystify the various architecture styles and understand how they can add value to your organization.

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...

DATAVERSITY

Was ist angesagt? (20)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

The ABCs of Treating Data as Product

Data Lakehouse Symposium | Day 4

Data Mesh for Dinner

Databricks Fundamentals

DW Migration Webinar-March 2022.pptx

Democratizing Data Quality Through a Centralized Platform

Modernizing to a Cloud Data Architecture

Introducing Databricks Delta

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...

Modern Data Warehousing with the Microsoft Analytics Platform System

Free Training: How to Build a Lakehouse

Data Mesh at CMC Markets: Past, Present and Future

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...

Modern Data architecture Design

Databricks Platform.pptx

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...

Owning Your Own (Data) Lake House

Data Lakehouse Symposium | Day 1 | Part 2

Data Architecture, Solution Architecture, Platform Architecture — What’s the ...

Ähnlich wie Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 2021 Keynote

Your mainframe and IBM i platforms do hard work for your business, supporting essential computing transactions every day. However, mainframe data does not easily integrate with the cloud platforms driving data-driven, real-time, analytics-focused business processes. Integrating data from this critical technology often results in high costs, missed deadlines, and unhappy customers. So, what can you do? Join us to hear how Precisely Connect can help use the power of Apache Kafka to eliminate data silos and make cloud-based, event-driven data architectures a reality. Start your cloud transformation journey today, knowing you don’t need to leave essential transaction data behind! Learn more about: • Where to begin your cloud transformation journey using mainframe and IBM i data and Apache Kafka • What you need to move mainframe and IBM i data to the cloud while reducing costs, modernizing architectures, and using the staff you have today • How Precisely Connect customers are using change data capture and Apache Kafka to deliver real-time insights to the cloud

Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...

Watch full webinar here: https://goo.gl/rSfYKV Gartner states in its Predicts 2018: Data Management Strategies Continue to Shift Toward Distributed, “As data management activities are becoming more widespread in both distributed processing use cases, like IoT, and demands for new types of data, emerging roles such as data scientists or data engineers are expected to be driving the new data management requirements in the coming two years. These trends indicate that both the collection of data as well as the need to connect to data are rapidly becoming the new normal, and that the days of a single data store with all the data of interest — the enterprise data warehouse — are long gone.” Data management solutions are becoming distributed, heterogeneous and extremely diverse. Attend this session to learn: • How to evolve architecture patterns in the cloud using data virtualization. • How data virtualization accelerates cloud migration and modernization. • Successful cloud implementation case studies.

Technology Overview

Liran Zelkha

Real time data integration best practices and architecture

Bui Kiet

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

2021 was a year full of unexpected data integration challenges, but one thing that didn’t change was the continued growth of the importance and value of data. By watching our customers adapt and cope through the consistent application of technology, we’ve learned that the future can be quickly adjusted to if we have up-to-date and readily available data to make decisions. As we consider the data integration landscape and look forward into 2022, we see a set of trends (some new, some old) that data leaders will need to consider as they work to provide competitive business value to their organizations: - The Continued Importance of Spatial - Data Ops as a Practice - Rising Data Volumes Demand Data Quality - Ubiquitous Hardware Supporting Augmented Reality - Agile Enterprise Integration Effortlessly Connects Systems - Real-Time Data Stream Processing - Flexible, Hybrid Deployment Options - Cost effective ARM based processing In this webinar, join co-founders Don Murray and Dale Lutz as they offer insight and predictions on what’s to come in these areas. To follow, they’ll host a Q&A session where you can get feedback and advice on solutions to your data challenges.

Oracle Coherence: in-memory datagrid

Emiliano Pecis

AWS Webcast - Informatica - Big Data Solutions Showcase

Amazon Web Services

Data Mesh in Azure using Cloud Scale Analytics (WAF)

Nathan Bijnens

7 Emerging Data & Enterprise Integration Trends in 2022

Safe Software

Watch full webinar here: https://buff.ly/2mHGaLA What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics. Attend this session to learn: • What data virtualization really is • How it differs from other enterprise data integration technologies • Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations

Why Data Virtualization? An Introduction

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Virgílio Vargas Presentations / CloudViews.Org - Cloud Computing Conference 2...

EuroCloud

Watch here: https://bit.ly/2Npt82U If you have data, you are engaged in data management—be sure to do it effectively. As organizations are assessing how COVID-19 has impacted their operations, new possibilities and uncharted routes are becoming the norm for many businesses. While exploring and implementing different deployment and operational models, the question of data management naturally surfaces while considering how these changes impact your data. Is this the right time to focus on data management? The reality is that if you have data, you are engaged in data management and so the real question is, are you doing it well? Join Brice Giesbrecht from Caserta and Mitesh Shah from Denodo to explore data management challenges and solutions facing data driven organizations.

Best Practices in the Cloud for Data Management (US)

Watch full webinar here: https://bit.ly/34iCruM Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

While many enterprises consider cloud computing the savior of their data strategy, there is a process they should be following when looking to leveraging database-as-a-service. This includes understanding their own data requirements, selecting the right cloud computing candidate, and then planning for the migration and operations. A huge number of issues and obstacles will inevitably arise, but fortunately best practices are emerging. This presentation will take you through the process of moving data to cloud computing providers.

Accelerating a Path to Digital with a Cloud Data Strategy

MongoDB

Cloud Data Integration Best Practices

Darren Cunningham

2015 APHL Annual Meeting - Racing to the Clouds: How Cloud Computing is Advan...

Eduardo Gonzalez Loumiet, MBA, PMP, CPHIMS

Your current middleware platform is costing you more than you think. It wasn't designed to support high-velocity software releases and frequent iteration of applications—prerequisites for success in today’s world. A new, modern approach to middleware is needed that enables both developer productivity and operational efficiency. Join Pivotal’s Rohit Kelapure and Perficient’s Joel Thimsen as they discuss: - The limitations of traditional middleware - The benefits of middleware modernization - Your options for modernization, including a cloud-native platform - Tips for overcoming some common challenges Presenters: Rohit Kelapure, Pivotal, Joel Thimsen, Perficient & Jeff Kelly, Pivotal (Host)

Why Your Digital Transformation Strategy Demands Middleware Modernization

VMware Tanzu

Data fabric and VMware

VMware vFabric

Learn how to accelerate and de-risk your cloud migration project Despite the surge in enterprises migrating applications to the public cloud, more than half of all projects are delayed or over budget and an even greater number are more difficult than expected.1 Cloud Migrations don’t begin when you start moving applications into the cloud. They begin with your application landscape discovery and assessment. The second phase comprises the actual migration where applications are moved to the public cloud. Working with purpose-built enterprise-grade cloud migration platforms, especially those that partner to integrate both phases greatly simplifies and accelerates projects. RISC Networks and Velostrata have teamed up to deliver this webinar where we’ll share real-world examples, tips, and tricks on crafting a seamless cloud migration from start to completion.

Risc and velostrata 2 28 2018 lessons_in_cloud_migration

RISC Networks

By managing Data in Motion, Data at Rest, and Data in Use differently, modern Information Management Solutions are enabling a whole range of architecture and design patterns that allow enterprises to fully harness the value in data flowing through their systems. In this session we explored some of the patterns (e.g. operational data lakes, CQRS, microservices and containerisation) that enable CIOs, CDOs and senior architects to tame the data challenge, and start to use data as a cross-enterprise asset.

Creating a Modern Data Architecture for Digital Transformation

MongoDB

Ähnlich wie Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 2021 Keynote (20)

Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...

Technology Overview

Real time data integration best practices and architecture

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Oracle Coherence: in-memory datagrid

AWS Webcast - Informatica - Big Data Solutions Showcase

Data Mesh in Azure using Cloud Scale Analytics (WAF)

7 Emerging Data & Enterprise Integration Trends in 2022

Why Data Virtualization? An Introduction

Confluent & GSI Webinars series - Session 3

Virgílio Vargas Presentations / CloudViews.Org - Cloud Computing Conference 2...

Best Practices in the Cloud for Data Management (US)

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Accelerating a Path to Digital with a Cloud Data Strategy

Cloud Data Integration Best Practices

2015 APHL Annual Meeting - Racing to the Clouds: How Cloud Computing is Advan...

Why Your Digital Transformation Strategy Demands Middleware Modernization

Data fabric and VMware

Risc and velostrata 2 28 2018 lessons_in_cloud_migration

Creating a Modern Data Architecture for Digital Transformation

Mehr von HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks