The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation

•

0 gefällt mir•1,488 views

This document discusses KIP-633, which proposes changing the default grace period for windowed operations in Kafka Streams from 24 hours to 0. The motivation is that the 24-hour default was causing confusion for users and suppressing results. The KIP was worked on by a team of contributors and introduces new APIs in Kafka Streams 3.0 that give developers better control over grace periods. It provides resources for migrating code to the new APIs and understanding the compatibility changes.

Technologie

KIP-633: The New Way of Configuring Grace Periods for
Windowed Operations in Kafka Streams
Israel Ekpo
Kafka Summit Americas
September 2021

About the Speaker
• Dad, Husband, Son, Brother, Friend
• Cloud Solutions Architect with Microsoft GPS
• Apache Kafka Contributor
• Open Source Proponent – Storage, Processing, Analysis
• Passionate about Event-Driven Architectures
• IzzyAcademy.com
• @IzzyAcademy – Twitter, YouTube, Github

Outline
• Motivation for KIP-633
• The Team
• The Changes
• Follow Up Resources, Next Steps

Motivation for KIP-633
• Stream-to-Stream Joins
• Stream Aggregations
• Grace period controls how long to wait
for windowed operations & process
events that arrive after a window ends
• Records coming in after the grace
period has elapsed will be dropped
from those windows.
• Results for window are
processed/finalized after grace period
• Default grace period is 24-hours

Motivation for KIP-633
• Problems and confusion for users
• Results are getting suppressed
• Results won’t show up for 24 hours
• KIP-633: Drop 24-hour default of
grace period in Streams

Motivation for KIP-633
• Set Expectations for New Users
• Eliminate Confusion for New Users
• Clarity in Usage
• Better Control in Streams Apps
• Improved User Experience

Team of Contributors
• Sophie Blee-Goldman
• Israel Ekpo
• Matthias J. Sax
• Bruno Cadonna
• Guozhang Wang
• Luke Chen

Compatibility, Deprecation, and Migration Plan
• Migrate to one of the two new APIs
• Make a conscious decision to skip the
grace period and close a window
immediately
• Still apply the old default of 24 hours
• Choose a new grace period entirely

Follow Up Resources – Check it Out!
https://github.com/izzyacademy/kip-633-demo-grace-periods
https://izzyacademy.com/tutorials
@IzzyAcademy – YouTube, GitHub, Twitter

Weitere ähnliche Inhalte

Was ist angesagt?

View recording here: https://www.confluent.io/online-talks/how-to-fail-at-apache-kafka-on-demand Apache Kafka® is used by thousands of companies across the world but, how difficult is it to operate? Which parameters do you need to set? What can go wrong? This online talk is based on real-world experience of Kafka deployments and explores a collection of common mistakes that are made when running Kafka in production and some best practices to avoid them. Watch now to learn: -How to ensure your Kafka data is never lost -How to write code to cope when things go wrong -How to ensure data governance between producers and consumers -How to monitor your cluster Join Apache Kafka expert, Pete Godfrey, for this engaging talk and delve into best practice ideas and insights.

How to Fail at Kafka

confluent

More and more Enterprises are relying on Apache Kafka to run their businesses. Cluster administrators need the ability to mirror data between clusters to provide high availability and disaster recovery. MirrorMaker 2, released recently as part of Kafka 2.4.0, allows you to mirror multiple clusters and create many replication topologies. Learn all about this awesome new tool and how to reliably and easily mirror clusters. We will first describe how MirrorMaker 2 works, including how it addresses all the shortcomings of MirrorMaker 1. We will also cover how to decide between its many deployment modes. Finally, we will share our experience running it in production as well as our tips and tricks to get a smooth ride.

Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...

HostedbyConfluent

rgpv 7th sem for it & cs Cloud computing lab record

naaaaz

Sinking data from a distributed message queue to files seems easy, right? Well, it is not trivial, but we managed to make it very complicated. At bol.com, the largest webshop of the Netherlands and Belgium, we started measuring customer behavior in real-time last year producing around 500,000 events per second. We ingest that data in Apache Kafka, a popular distributed message queue, and wanted to have that data stored in Parquet files for easier batch processing by different applications in the landscape. To do that we tried several solutions with Apache Flink, a stream processing framework, but transitioning from an event stream to windowed batch files under exactly-once guarantees proved to be very challenging. We struggled a lot with fault tolerance and, in the end, we decided to go back to basics and build a much simpler daily batch job instead of a streaming job. In this talk I’m going to share our struggle and learnings with you, especially about how (not) to deal with fault tolerance when you need exactly-once delivery guarantees.

Journeys from Kafka to Parquet

DataWorks Summit

Nick Fisk - low latency Ceph

ShapeBlue

In this training webinar, we will walk you through the basics of InfluxDB – the purpose-built time series database. InfluxDB has everything you need from a time series platform in a single binary – a multi-tenanted time series database, UI and dashboarding tools, background processing and monitoring agent. This one-hour session will include the training and time for live Q&A. What you will learn Core concepts of time series databases An overview of the InfluxDB platform How to ingesting and query data in InfluxDB

Intro to InfluxDB

InfluxData

Jenkins tutorial

HarikaReddy115

Kubernetes Scheduler deep dive

DONGJIN KIM

Ceph Tech Talk -- Ceph Benchmarking Tool

Ceph Community

A modern business operates 24/7 and generates data continuously. Shouldn’t we process it continuously too? A rich ecosystem of real-time data-processing frameworks, tools and systems has been forming around Apache Kafka that allows data to be processed continuously as it occurs. Jay Kreps will introduce Kafka and explain why it has become the de facto standard for streaming data. He will draw on practical experience building stream-processing applications to discuss the difference between architectures and the challenges each presents. Jay will then outline the Kafka Streams API, which offers new stream processing functionality in Kafka, and explain how it helps tame some of the complexity in real-time architectures. Visit www.confluent.io for more information

Distributed stream processing with Apache Kafka

confluent

Data Loss and Duplication in Kafka

Jayesh Thakrar

Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster. Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.

Grafana Loki: like Prometheus, but for Logs

Marco Pracucci

APACHE KAFKA / Kafka Connect / Kafka Streams

Ketan Gote

http://flink-forward.org/kb_sessions/joining-infinity-windowless-stream-processing-with-flink/ The extensive set of high-level Flink primitives makes it easy to join windowed streams. However, use cases that don’t have windows can prove to be more complicated, making it necessary to leverage operator state and low-level primitives to manually implement a continuous join. This talk will focus on the anomalies that present themselves when performing streaming joins with infinite windows, and the problems encountered operating topologies that back user-facing data. We will describe the approach taken at ResearchGate to implement and maintain a consistent join result of change data capture streams.

Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

Flink Forward

Timeseries - data visualization in Grafana

OCoderFest

Producer Performance Tuning for Apache Kafka

Jiangjie Qin

Nginx Reverse Proxy with Kafka.pptx

wonyong hwang

Saga pattern and event sourcing with kafka

Roan Brasil Monteiro

An Introduction to Apache Kafka

Amir Sedighi

Over the last year, we have been moving from a batch processing jobs setup with Airflow using EC2s to a powerful & scalable setup using Airflow & Spark in K8s. The increasing need of moving forward with all the technology changes, the new community advances, and multidisciplinary teams, forced us to design a solution where we were able to run multiple Spark versions at the same time by avoiding duplicating infrastructure and simplifying its deployment, maintenance, and development.

Apache Spark Streaming in K8s with ArgoCD & Spark Operator

Databricks

Was ist angesagt? (20)

How to Fail at Kafka

Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...

rgpv 7th sem for it & cs Cloud computing lab record

Journeys from Kafka to Parquet

Nick Fisk - low latency Ceph

Intro to InfluxDB

Jenkins tutorial

Kubernetes Scheduler deep dive

Ceph Tech Talk -- Ceph Benchmarking Tool

Distributed stream processing with Apache Kafka

Data Loss and Duplication in Kafka

Grafana Loki: like Prometheus, but for Logs

APACHE KAFKA / Kafka Connect / Kafka Streams

Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

Timeseries - data visualization in Grafana

Producer Performance Tuning for Apache Kafka

Nginx Reverse Proxy with Kafka.pptx

Saga pattern and event sourcing with kafka

An Introduction to Apache Kafka

Apache Spark Streaming in K8s with ArgoCD & Spark Operator

Ähnlich wie The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation

This talk will cover two years of Zabbix deployment at Globo.com - the web branch of Rede Globo, a major media player in Brazil providing access to media products to more than 1,8 million visitors hourly, 45 millions each day. The case study will include migration from legacy systems, integration, templates and deployment, suming up the challenges, solutions and the gained knowledge. Zabbix Conference 2015

Filipe paternot - Case Study: Zabbix Deployment at Globo.com

Zabbix

Meetup #3: Migrating an Oracle Application from on-premise to AWS

AWS Vietnam Community

Nous vivons une époque où tout est connecté, de nos ampoules à notre éditeur de texte, les objets et services qui nous entourent devienne de plus en plus intelligents. Pour ce faire ils génèrent des données, elles sont nécessaires au bon fonctionnement du service ou de l'objet, mais elles sont également utiles pour faire évoluer les produits. Ces données peuvent rapidement représenter de gros volumes, plusieurs dizaines voir centaines de gigaoctets. Une question se pose alors, comment traiter de tels volumes ? Comment en tirer du sens et de la valeur à cette échelle ? Avec OVHcloud Data Processing, une solution basée sur le framework Apache Spark, nous répondons à ce besoin. Venez découvrir comment vous aussi, en quelques cliques, pouvez exécuter votre code sur une infrastructure taillée pour vos besoins. Au travers de différents exemples, comme une analyse du traffic des taxis New-Yorkais, nous verrons comment Data Processing a été pensé, comment il fonctionne et comment il peut être utilisé pour valoriser vos données.

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

OVHcloud

The Day 1 Keynote at "Into the Box" conference sets the stage for an exciting journey into the world of modern development. Join us as renowned experts and industry leaders take the spotlight to explore cutting-edge tools, techniques, and frameworks that are revolutionizing the way we build software. With a focus on harnessing the power of innovation, this keynote will inspire and empower attendees to embrace the latest trends and technologies in the realm of web and mobile app development. Prepare to be captivated by insightful talks, engaging demonstrations, and a vision of a future where developers thrive in the box and beyond. Don't miss this opportunity to kickstart your conference experience with an invigorating and inspiring session.

Into The Box 2023 Keynote Day 1

Ortus Solutions, Corp

What You Missed: OpenStack Summit Austin

NetApp

Time-oriented event search. A new level of scale

DataWorks Summit/Hadoop Summit

DataEngConf SF16 - Scalable and Reliable Logging at Pinterest

Hakka Labs

At Pinterest, hundreds of services and third-party tools that are implemented in various programming languages generate billions of events every day. To achieve scalable and reliable low latency logging, there are several challenges: (1) uploading logs that are generated in various formats from tens of thousands of hosts to Kafka in a timely manner; (2) running Kafka reliably on Amazon Web Services where the virtual instances are less reliable than on-premises hardware; (3) moving tens of terabytes data per day from Kafka to cloud storage reliably and efficiently, and guaranteeing exact one time persistence per message. In this talk, we will present Pinterest’s logging pipeline, and share our experience addressing these challenges. We will dive deep into the three components we developed: data uploading from service hosts to Kafka, data transportation from Kafka to S3, and data sanitization. We will also share our experience in operating Kafka at scale in the cloud.

Scalable and Reliable Logging at Pinterest

Krishna Gade

H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...

Lucidworks

Microservices architecture has many benefits. But it comes at a cost. Running microservices and monitoring what’s going on is tedious. That’s why MicroProfile adopts monitoring as a first-class concept. In this session, learn how MicroProfile runtimes collect metrics and how to seamlessly collect them with tools like Prometheus and Grafana. Learn how MicroProfile makes it easy to connect information about interrelated service calls, how to gather the information and analyze system bottlenecks, how to deploy and scale MicroProfile applications with Kubernetes and how to react to their health status to detect and automatically recover from failures.

Control and monitor_microservices_with_microprofile

Rudy De Busscher

C.Hibbard_Platform_LSF.ppt

Chris Hibbard

OpenStack Glance Project Update

Brian Rosmaita

@KileNiklawski with @IpponUSA presents on how we built a job board in one week using JHipster. About JHipster: Our goal is to generate for you a complete and modern Web app, unifying: - A high-performance and robust Java stack on the server side with Spring Boot - A sleek, modern, mobile-first front-end with AngularJS and Bootstrap - A powerful workflow to build your application with Yeoman, Bower, Grunt and Maven

How we built a job board in one week with JHipster

Kile Niklawski

JHipster - a modern, opinionated, full stack web app generator. JHipster ties together Spring, Angular, Grunt, Bower, and much more to help you build production ready responsive web apps in a fraction of the time. In the discussion, we will look at how we built a job board in one week using JHipster and give a brief demo. From the JHipster site - Our goal is to generate for you a complete and modern Web app, unifying: - A high-performance and robust Java stack on the server side with Spring Boot - A sleek, modern, mobile-first front-end with AngularJS and Bootstrap - A powerful workflow to build your application with Yeoman, Bower, Grunt and Maven Skill level: All levels Speaker: Kile Niklawski Bio: Kile Niklawski, Architect @Ippon USA. Ippon delivers Digital, Big Data and Cloud applications on top of proven Java expertise #RVA #DC #NYC

How we built a job board in one week with JHipster - @KileNiklawski @IpponUSA

Kile Niklawski

AD1545 - Extending the XPages Extension Library

paidi_ed

Netflix oss season 2 episode 1 - meetup Lightning talks

Ruslan Meshenberg

PM, Scrum and TFS - Ivan Marković

Software StartUp Academy Osijek

Dibi Conference 2012

Scott Rutherford

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

Big Data Spain

The database team at GitHub is tasked with keeping the data available and with maintaining its integrity. Our infrastructure automates away much of our operation, but automation requires trust, and trust is gained by testing. This session highlights three examples of infrastructure testing automation that helps us sleep better at night: - Backups: scheduling backups; making backup data accessible to our engineers; auto-restores and backup validation. What metrics and alerts we have in place. - Failovers: how we continuously test our failover mechanism, orchestrator. How we setup a failover scenario, what defines a successful failover, how we automate away the cleanup. What we do in production. - Schema migrations: how we ensure that gh-ost, our schema migration tool, which keeps rewriting our (and your!) data, does the right thing. How we test new branches in production without putting production data at risk.

MySQL Infrastructure Testing Automation at GitHub

Ike Walker

Ähnlich wie The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation (20)

Filipe paternot - Case Study: Zabbix Deployment at Globo.com

Meetup #3: Migrating an Oracle Application from on-premise to AWS

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

Into The Box 2023 Keynote Day 1

What You Missed: OpenStack Summit Austin

Time-oriented event search. A new level of scale

DataEngConf SF16 - Scalable and Reliable Logging at Pinterest

Scalable and Reliable Logging at Pinterest

H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...

Control and monitor_microservices_with_microprofile

C.Hibbard_Platform_LSF.ppt

OpenStack Glance Project Update

How we built a job board in one week with JHipster

How we built a job board in one week with JHipster - @KileNiklawski @IpponUSA

AD1545 - Extending the XPages Extension Library

Netflix oss season 2 episode 1 - meetup Lightning talks

PM, Scrum and TFS - Ivan Marković

Dibi Conference 2012

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

MySQL Infrastructure Testing Automation at GitHub

Mehr von HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

Mehr von HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Kürzlich hochgeladen

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Manulife - Insurer Innovation Award 2024

The Digital Insurer

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Kürzlich hochgeladen (20)

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

🐬 The future of MySQL is Postgres 🐘

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Manulife - Insurer Innovation Award 2024

GenAI Risks & Security Meetup 01052024.pdf

Artificial Intelligence: Facts and Myths

Apidays New York 2024 - The value of a flexible API Management solution for O...

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

AWS Community Day CPH - Three problems of Terraform

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Powerful Google developer tools for immediate impact! (2023-24 C)

presentation ICT roal in 21st century education

Exploring the Future Potential of AI-Enabled Smartphone Processors

The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation

1. KIP-633: The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams Israel Ekpo Kafka Summit Americas September 2021

2. About the Speaker • Dad, Husband, Son, Brother, Friend • Cloud Solutions Architect with Microsoft GPS • Apache Kafka Contributor • Open Source Proponent – Storage, Processing, Analysis • Passionate about Event-Driven Architectures • IzzyAcademy.com • @IzzyAcademy – Twitter, YouTube, Github

3. Outline • Motivation for KIP-633 • The Team • The Changes • Follow Up Resources, Next Steps

4. Motivation for KIP-633 • Stream-to-Stream Joins • Stream Aggregations • Grace period controls how long to wait for windowed operations & process events that arrive after a window ends • Records coming in after the grace period has elapsed will be dropped from those windows. • Results for window are processed/finalized after grace period • Default grace period is 24-hours

5. Motivation for KIP-633 • Problems and confusion for users • Results are getting suppressed • Results won’t show up for 24 hours • KIP-633: Drop 24-hour default of grace period in Streams

6. Motivation for KIP-633 • Set Expectations for New Users • Eliminate Confusion for New Users • Clarity in Usage • Better Control in Streams Apps • Improved User Experience

7. Team of Contributors • Sophie Blee-Goldman • Israel Ekpo • Matthias J. Sax • Bruno Cadonna • Guozhang Wang • Luke Chen

8. API Changes in 3.0

9. API Changes in 3.0

10. Compatibility, Deprecation, and Migration Plan • Migrate to one of the two new APIs • Make a conscious decision to skip the grace period and close a window immediately • Still apply the old default of 24 hours • Choose a new grace period entirely

11. Follow Up Resources – Check it Out! https://github.com/izzyacademy/kip-633-demo-grace-periods https://izzyacademy.com/tutorials @IzzyAcademy – YouTube, GitHub, Twitter

The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation

Ähnlich wie The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation (20)

Mehr von HostedbyConfluent

Mehr von HostedbyConfluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The New Way of Configuring Grace Periods for Windowed Operations in Kafka Streams | Israel Ekpo, Microsoft Corperation