Apache Kafka 0.11 の Exactly Once Semantics

•

7 gefällt mir•6,199 views

Yoshiyasu SAEKI

Apache Kafka Meetup Japan #3 https://kafka-apache-jp.connpass.com/event/58619/ LT発表（突発）資料

Technologie

Apache Kafka 0.11
Exactly Once Semantics
/ @laclefyoshi / ysaeki@r.recruit.co.jp

• 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
3

Kafka
At Least Once At Most Once Exactly Once
9

Kafka 0.11.0.0
• 2017/06/28
• KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams
• https://issues.apache.org/jira/browse/KAFKA-4923
• Conﬂuent Blog: Exactly-once Semantics are Possible: Here’s How
Kafka Does it
• https://www.conﬂuent.io/blog/exactly-once-semantics-are-possible-
heres-how-apache-kafka-does-it/
• “In the following posts in this series, we will go into more details …”
• Exactly Once Delivery and Transactional Messaging in Kafka
• https://docs.google.com/document/d/
11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/
• KIP-98 → KAFKA-5059: Implement Transactional Coordinator
11

Producer Exactly Once
1.
2.
3. x N
4. →
16

Producer Exactly Once
1.
2.
3. x N
4.
→
→
→ 1
17

Producer Exactly Once
5.Broker Commit Marker  
COMMITTED ABORTED
↓ ↓
18

Consumer Exactly Once
6.  
↓
READ_COMMITTED
↓
READ_UNCOMMITTED
20

Consumer Exactly Once
7.
8.  
- COMMITTED  
- ABORTED Transaction List  
21

• Kafka API RecordBatch
• Kafka < 0.11 Message MessageSet
API
• Kafka 0.11 MessageSet/Message RecordBatch/Record
•
• Producer Broker Consumer
ID PID Producer ID
22

Exactly Once
• : Exactly Once
• Producer
• Broker
• Commit Marker
• Consumer
•
→ OSS

• Exactly Once
• Producer Broker
• Consumer
• 3%
•
• Conﬂuent
• OSS
Kafka
• Exactly Once
24

Weitere ähnliche Inhalte

Was ist angesagt?

Dockerからcontainerdへの移行

Kohei Tokunaga

Apache Sparkに手を出してヤケドしないための基本～「Apache Spark入門より」～（デブサミ 2016 講演資料）

NTT DATA OSS Professional Services

https://ntt-developers.github.io/ntt-tech-conference/2022/ 従来は Docker がコンテナエンジンとして幅広く利用されてきましたが、最近は Kubernetes が Docker 対応を打ち切るなど、風向きが変わってきました。本セッションでは、Docker に代わって普及しつつあるコンテナエンジンである containerd について、Docker との違いや移行方法を紹介します。 P2P でのイメージ配布など、最新の機能についても紹介します。

Dockerからcontainerdへの移行

Akihiro Suda

本当は恐ろしい分散システムの話

Kumazaki Hiroki

大量のデータ処理や分析に使えるOSS Apache Spark入門（Open Source Conference 2021 Online/Kyoto 発表資料）

NTT DATA Technology & Innovation

JVMのGCアルゴリズムとチューニング

佑哉廣岡

分散システムについて語らせてくれ

Kumazaki Hiroki

え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理（NTTデータテクノロジーカンファレンス 2019 講演資料、2019/09/05） NTTデータ技術革新統括本部システム技術本部生産技術部インテグレーション技術センタデータ活用チーム佐々木徹

え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理

NTT DATA Technology & Innovation

Hadoop/Spark で Amazon S3 を徹底的に使いこなすワザ / Hadoop / Spark Conference Japan 2019 講演者：関山宜孝（Amazon Web Services Japan）昨今 Hadoop/Spark エコシステムで広く使われているクラウドストレージ。本講演では Amazon S3 を例に、Hadoop/Spark から見た S3 の動作や HDFS と S3 の使い分けをご説明します。また、AWS サポートに寄せられた多くのお問い合わせから得られた知見をもとに、Hadoop/Spark で S3 を最大限活用するベストプラクティス、パフォーマンスチューニング、よくあるハマりどころ、トラブルシューティング方法などをご紹介します。併せて、Hadoop/Spark に関係する S3 のサービスアップデート、S3 関連の Hadoop/Spark コミュニティの直近の開発状況についても解説します。 http://hadoop.apache.jp/hcj2019-program/

Hadoop/Spark で Amazon S3 を徹底的に使いこなすワザ (Hadoop / Spark Conference Japan 2019)

Noritaka Sekiyama

Grafana LokiではじめるKubernetesロギングハンズオン（NTT Tech Conference #4 ハンズオン資料）

NTT DATA Technology & Innovation

PostgreSQLをKubernetes上で活用するためのOperator紹介！（Cloud Native Database Meetup #3 発表資料）

NTT DATA Technology & Innovation

コンテナにおけるパフォーマンス調査でハマった話

Yuta Shimada

アプリケーションの分割のアプローチ ●４つのアプローチ - ビジネスファンクション - 動詞／ユースケース - 名詞／リソース - 境界づけられたコンテキスト ● トランザクションの分割 - パイプライン化 (VETRO) - コーディネート (Saga) - 状態更新の非同期化 ( Event History - State Materialize - Domain Specific Query )

マイクロサービス　４つの分割アプローチ

増田亨

がっつりMongoDB事例紹介

Tetsutaro Watanabe

アーキテクチャから理解するPostgreSQLのレプリケーション

Masahiko Sawada

初心者向けMongoDBのキホン！

Tetsutaro Watanabe

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

マルチテナント化で知っておきたいデータベースのこと

Amazon Web Services Japan

はじめてのElasticsearchクラスタ

Satoyuki Tsukano

ストリーム処理プラットフォームにおけるKafka導入事例 #kafkajp

Yahoo!デベロッパーネットワーク

Was ist angesagt? (20)

Dockerからcontainerdへの移行

Apache Sparkに手を出してヤケドしないための基本～「Apache Spark入門より」～（デブサミ 2016 講演資料）

Dockerからcontainerdへの移行

本当は恐ろしい分散システムの話

大量のデータ処理や分析に使えるOSS Apache Spark入門（Open Source Conference 2021 Online/Kyoto 発表資料）

JVMのGCアルゴリズムとチューニング

分散システムについて語らせてくれ

え、まって。その並列分散処理、Kafkaのしくみでもできるの？ Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理

Hadoop/Spark で Amazon S3 を徹底的に使いこなすワザ (Hadoop / Spark Conference Japan 2019)

Grafana LokiではじめるKubernetesロギングハンズオン（NTT Tech Conference #4 ハンズオン資料）

PostgreSQLをKubernetes上で活用するためのOperator紹介！（Cloud Native Database Meetup #3 発表資料）

コンテナにおけるパフォーマンス調査でハマった話

マイクロサービス　４つの分割アプローチ

がっつりMongoDB事例紹介

アーキテクチャから理解するPostgreSQLのレプリケーション

初心者向けMongoDBのキホン！

KafkaとAWS Kinesisの比較

マルチテナント化で知っておきたいデータベースのこと

はじめてのElasticsearchクラスタ

ストリーム処理プラットフォームにおけるKafka導入事例 #kafkajp

Ähnlich wie Apache Kafka 0.11 の Exactly Once Semantics

データの民主化のために StackStorm を活用した事例

Yoshiyasu SAEKI

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

Chris Fregly

StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有

Yoshiyasu SAEKI

Queryable State for Kafka Streamsを使ってみた

Yoshiyasu SAEKI

Spark Streamingによるリアルタイムユーザ属性推定

Yoshiyasu SAEKI

In a recent survey, 90% of over 2400 developers reported having at least some real-time functionality in their systems. Enterprises are realizing that the ability to extract value from streaming data in near real-time is the new competitive advantage. Two technologies–Akka Streams and Kafka Streams–have emerged as popular tools to use with Apache Kafka for addressing the shared requirements of availability, scalability, and resilience for both streaming microservices and Fast Data. So which one should you use for specific use cases?

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Lightbend

My talk at Google DevFest Switzerland, Fribourg, Oct 2017. https://devfest.ch/schedule/day1?sessionId=118 Abstract: Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of information in real-time? The answer is stream processing, and the technology that has since become the core platform for streaming data is Apache Kafka. Among the thousands of companies that use Kafka to transform and reshape their industries are the likes of Netflix, Uber, PayPal, and AirBnB, but also established players such as Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: there are many technologies that need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we, as engineers, would like to work vs. how we actually end up working in practice. In this session we talk about how Apache Kafka helps you to radically simplify your data architectures. We cover how you can now build normal applications to serve your real-time processing needs — rather than building clusters or similar special-purpose infrastructure — and still benefit from properties such as high scalability, distributed computing, and fault-tolerance, which are typically associated exclusively with cluster technologies. We discuss common use cases to realize that stream processing in practice often requires database-like functionality, and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (inventory management for large retailers, patient monitoring in healthcare, fleet tracking in logistics, etc), for example in the form of event-driven, containerized microservices. We will also give a brief shout-out to the recently launched KSQL, a streaming SQL engine for Apache Kafka.

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Helena Edelson

DataMass Summit 2019 Edition --> http://summit.datamass.io There is quite a bit to learn about any stream processing engine. But at a reasonably high level they actually are very similar and have lots in common. Not only do all have to offer a high-level stream processing API to describe distributed streaming dataflows, but also a low-level API for more sophisticated streaming topologies. The engines translate the dataflow description into their internal runtime representation. That’s where the differences are and where we’ll be looking at. This talk compares two modern stream processing engines — Kafka Streams and Spark Structured Streaming. We’ll be talking about their internals and how the engines manage stateless and stateful streams. You will learn about their similarities and differences that should shed more light on the question when to use which engine.

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

Jacek Laskowski

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Trivadis

Since mid-2016, Spark-as-a-Service has been available to researchers in Sweden from the Rise SICS ICE Data Center at www.hops.site. In this session, Dowling will discuss the challenges in building multi-tenant Spark structured streaming applications on YARN that are metered and easy-to-debug. The platform, called Hopsworks, is in an entirely UI-driven environment built with only open-source software. Learn how they use the ELK stack (Elasticsearch, Logstash and Kibana) for logging and debugging running Spark streaming applications; how they use Grafana and InfluxDB for monitoring Spark streaming applications; and, finally, how Apache Zeppelin can provide interactive visualizations and charts to end-users. This session will also show how Spark applications are run within a ‘project’ on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In addition, hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.afka topics are protected from access by users that are not members of the project. We will also discuss the experiences of our users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications.

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Databricks

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Kafka Streams for Java enthusiasts

Slim Baltagi

グラフデータベース Neptune 使ってみた

Yoshiyasu SAEKI

What is Apache Kafka®?

Eventador

What is apache Kafka?

Kenny Gorman

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

Chris Fregly

"Implementing real-time data pipelines is still a challenge. Even more so for data scientists who have often been brought up with batch processing and files and typically have only heard of Kafka, but never really used it. Now if your team consists only of data scientists and you want them to implement a real-time data pipeline, your fate seems to be sealed. You need to hire real-time and streaming experts first - and you are guaranteed to lose months before you can even start implementing. But is there really no other way? At Forecasty.AI, developer of a SaaS platform for commodity price forecasting called Commodity Desk, we thought again. And came up with a way how to bridge the gap between the batch and file-based world of most data scientists and the world of real-time and streaming. Our solution is a new Open Source library allowing any Python programmer, including the aforementioned data scientists, to access the Kafka API in an easier way than ever before. kash.py (""Kafka Shell for Python"") offers a large number of easy-to-use abstractions on top of the Kafka API, including bash-inspired commands like ""ls"" or ""l"" for listing Kafka topics and ""cat"", ""head"" or ""tail"" for displaying the content of topics. kash.py bridges the gap between the file and the streaming worlds with commands like ""upload"" to upload a file to a topic, or ""download"" to download a topic to a file, and offers commands inspired by functional programming to do Kafka-Kafka/File-Kafka or Kafka-File stream processing (""map"", ""flatMap"", ""filter"" etc.) in one-liners - of course with full support for JSON Schema, Avro and Protobuf. In this session, we show you how kash.py can be used to bring the two disparate worlds of files and streaming together - and thus not only save a lot of time and money hiring real-time and streaming experts, but also make your data scientists start loving real-time."

kash.py - How to Make Your Data Scientists Love Real-time with Ralph M. Debus...

HostedbyConfluent

DataConf.TW2018: Develop Kafka Streams Application on Your Laptop

Yu-Jhe Li

Kafka Streams: The Stream Processing Engine of Apache Kafka

Eno Thereska

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

What is Apache Kafka and What is an Event Streaming Platform?

confluent

Ähnlich wie Apache Kafka 0.11 の Exactly Once Semantics (20)

データの民主化のために StackStorm を活用した事例

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有

Queryable State for Kafka Streamsを使ってみた

Spark Streamingによるリアルタイムユーザ属性推定

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Kafka Streams for Java enthusiasts

グラフデータベース Neptune 使ってみた

What is Apache Kafka®?

What is apache Kafka?

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

kash.py - How to Make Your Data Scientists Love Real-time with Ralph M. Debus...

DataConf.TW2018: Develop Kafka Streams Application on Your Laptop

Kafka Streams: The Stream Processing Engine of Apache Kafka

What is Apache Kafka and What is an Event Streaming Platform?

Kürzlich hochgeladen

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

ICT role in 21st century education and its challenges

rafiqahmad00786416

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Kürzlich hochgeladen (20)

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Exploring the Future Potential of AI-Enabled Smartphone Processors

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Why Teams call analytics are critical to your entire business

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

ICT role in 21st century education and its challenges

Ransomware_Q4_2023. The report. [EN].pdf

[BuildWithAI] Introduction to Gemini.pdf

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Axa Assurance Maroc - Insurer Innovation Award 2024

Artificial Intelligence Chap.5 : Uncertainty

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Manulife - Insurer Transformation Award 2024

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Strategies for Landing an Oracle DBA Job as a Fresher

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

AWS Community Day CPH - Three problems of Terraform

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Corporate and higher education May webinar.pptx

Apache Kafka 0.11 の Exactly Once Semantics

1. Apache Kafka 0.11 Exactly Once Semantics / @laclefyoshi / ysaeki@r.recruit.co.jp

2. • • Kafka • Exactly Once : • 2

3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) 3

4. Kafka

5. Kafka 5

6. Kafka ACK → ACK → 6

7. Kafka Broker → 7

8. Kafka → → 8

9. Kafka At Least Once At Most Once 9

10. Kafka At Least Once At Most Once Exactly Once 9

11. Exactly Once :

12. Kafka 0.11.0.0 • 2017/06/28 • KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams • https://issues.apache.org/jira/browse/KAFKA-4923 • Conﬂuent Blog: Exactly-once Semantics are Possible: Here’s How Kafka Does it • https://www.conﬂuent.io/blog/exactly-once-semantics-are-possible- heres-how-apache-kafka-does-it/ • “In the following posts in this series, we will go into more details …” • Exactly Once Delivery and Transactional Messaging in Kafka • https://docs.google.com/document/d/ 11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/ • KIP-98 → KAFKA-5059: Implement Transactional Coordinator 11

13.

14. Kafka 13

15. Kafka : Exactly Once 14

16. Producer Exactly Once 15

17. Producer Exactly Once 1. 2. 3. x N 4. → 16

18. Producer Exactly Once 1. 2. 3. x N 4. → → → 1 17

19. Producer Exactly Once 5.Broker Commit Marker   COMMITTED ABORTED ↓ ↓ 18

20. Consumer Exactly Once 19

21. Consumer Exactly Once 6.   ↓ READ_COMMITTED ↓ READ_UNCOMMITTED 20

22. Consumer Exactly Once 7. 8.   - COMMITTED   - ABORTED Transaction List   21

23. • Kafka API RecordBatch • Kafka < 0.11 Message MessageSet API • Kafka 0.11 MessageSet/Message RecordBatch/Record • • Producer Broker Consumer ID PID Producer ID 22

24. Exactly Once • : Exactly Once • Producer • Broker • Commit Marker • Consumer • → OSS

25. • Exactly Once • Producer Broker • Consumer • 3% • • Conﬂuent • OSS Kafka • Exactly Once 24

Apache Kafka 0.11 の Exactly Once Semantics

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Apache Kafka 0.11 の Exactly Once Semantics

Ähnlich wie Apache Kafka 0.11 の Exactly Once Semantics (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache Kafka 0.11 の Exactly Once Semantics