Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ewen Cheslack-Postava

•

14 gefällt mir•6,497 views

Spark Summit

Spark Summit East Talk

Daten & Analysen

BUILDING REALTIME DATA
PIPELINES WITH KAFKA CONNECT
AND SPARK STREAMING
Ewen Cheslack-Postava
Confluent

About Me: Ewen Cheslack-Postava
• Engineer @ Confluent
• Kafka Committer
• Kafka Connect Lead

Large-scale streaming data import/export for Kafka
Kafka Connect

Delivery Guarantees
• Automatic offset checkpointing and recovery
– Supports at least once
– Exactly once for connectors that support it
(e.g. HDFS)
– At most once simply swaps write & commit
– On restart: task checks offsets & rewinds

Spark Streaming
• Use Direct Kafka streams (1.3+)
– Better integration, more efficient, better
semantics
• Spark Kafka Writer
– At least once
– Kafka community is working on improved
producer semantics

Spark Streaming & Kafka Connect
• Increase # of systems Spark Streaming
works with, indirectly
• Reduce friction to adopt Spark Streaming
• Reduce need for Spark-specific connectors
• By leveraging Kafka as de facto streaming
data storage

Kafka Connect Summary
23
• Designed for large scale stream or batch data
integration
• Community supported and certified way of using
Kafka
• Soon, large repository of open source connectors
• Easy data pipelines when combined with Spark &
Spark Streaming

THANK YOU.
Follow me on Twitter: @ewencp
Try it out: http://confluent.io/download
More like this, but in blog form: http://confluent.io/blog

Add Pages as Necessary
• Supporting points go here.

Empfohlen

Spark and Bloomberg by Sudarshan Kadambi and Partha NageswaranSpark Summit

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...Spark Summit

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson

Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...Helena Edelson

Revitalizing Enterprise Integration with Reactive StreamsLightbend

Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman

2016 Spark Summit East Keynote: Matei ZahariaDatabricks

Introduction to Spark Streamingdatamantra

Empfohlen

Spark and Bloomberg by Sudarshan Kadambi and Partha NageswaranSpark Summit

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming: Spar...Spark Summit

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson

Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...Helena Edelson

Revitalizing Enterprise Integration with Reactive StreamsLightbend

Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman

2016 Spark Summit East Keynote: Matei ZahariaDatabricks

Introduction to Spark Streamingdatamantra

Operationalizing Machine Learning: Serving ML ModelsLightbend

How to deploy Apache Spark  to Mesos/DCOSLegacy Typesafe (now Lightbend)

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...DataStax Academy

SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion DubaiCodemotion Dubai

Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bour...Spark Summit

Apache Spark and Online Analytics Databricks

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache KafkaLightbend

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Exampleconfluent

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena EdelsonSpark Summit

How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Str...Lightbend

Spark Summit EU talk by Yiannis GkoufasSpark Summit

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)Spark Summit

Big Data visualization with Apache Spark and Zeppelinprajods

Data IntegrationDatio Big Data

Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time PersonalizationPatrick Di Loreto

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021StreamNative

Sa introduction to big data pipelining with cassandra & spark west mins...Simon Ambridge

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...Michael Noll

Real-time personal trainer on the SMACK stackAnirvan Chakraborty

Rethinking Streaming Analytics For ScaleHelena Edelson

Using the SDACK Architecture to Build a Big Data ProductEvans Ye

Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent

Weitere ähnliche Inhalte

Was ist angesagt?

Operationalizing Machine Learning: Serving ML ModelsLightbend

How to deploy Apache Spark  to Mesos/DCOSLegacy Typesafe (now Lightbend)

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...DataStax Academy

SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion DubaiCodemotion Dubai

Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bour...Spark Summit

Apache Spark and Online Analytics Databricks

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache KafkaLightbend

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Exampleconfluent

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena EdelsonSpark Summit

How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Str...Lightbend

Spark Summit EU talk by Yiannis GkoufasSpark Summit

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)Spark Summit

Big Data visualization with Apache Spark and Zeppelinprajods

Data IntegrationDatio Big Data

Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time PersonalizationPatrick Di Loreto

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021StreamNative

Sa introduction to big data pipelining with cassandra & spark west mins...Simon Ambridge

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...Michael Noll

Real-time personal trainer on the SMACK stackAnirvan Chakraborty

Rethinking Streaming Analytics For ScaleHelena Edelson

Was ist angesagt? (20)

Operationalizing Machine Learning: Serving ML Models

How to deploy Apache Spark  to Mesos/DCOS

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai

Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bour...

Apache Spark and Online Analytics

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Str...

Spark Summit EU talk by Yiannis Gkoufas

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

Big Data visualization with Apache Spark and Zeppelin

Data Integration

Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021

Sa introduction to big data pipelining with cassandra & spark west mins...

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Real-time personal trainer on the SMACK stack

Rethinking Streaming Analytics For Scale

Ähnlich wie Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ewen Cheslack-Postava

Using the SDACK Architecture to Build a Big Data ProductEvans Ye

Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent

Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQLconfluent

Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoVMware Tanzu

Building Realtime Data Pipelines with Kafka Connect and Spark StreamingJen Aman

Kafka connect 101Whiteklay

8th Athens Big Data Meetup - 1st Talk - Riding The Streaming Wave DIY StyleAthens Big Data

Riding the Streaming Wave DIY styleKonstantine Karantasis

Building Realtim Data Pipelines with Kafka Connect and Spark StreamingGuozhang Wang

Alpakka - Connecting Kafka and ElasticSearch to Akka StreamsKnoldus Inc.

Apache kafkaKumar Shivam

Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lightbend

Kafka Connect and Streams (Concepts, Architecture, Features)Kai Wähner

Confluent Kafka and KSQL: Streaming Data Pipelines Made EasyKairo Tavares

SSR: Structured Streaming for R and Machine Learningfelixcss

SSR: Structured Streaming on R for Machine Learning with Felix CheungDatabricks

Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...Reactivesummit

Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & KafkaAkara Sucharitakul

Streaming ETL with Apache Kafka and KSQLNick Dearden

Jug - ecosystemFlorent Ramiere

Ähnlich wie Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ewen Cheslack-Postava (20)

Using the SDACK Architecture to Build a Big Data Product

Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...

Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL

Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming

Kafka connect 101

8th Athens Big Data Meetup - 1st Talk - Riding The Streaming Wave DIY Style

Riding the Streaming Wave DIY style

Building Realtim Data Pipelines with Kafka Connect and Spark Streaming

Alpakka - Connecting Kafka and ElasticSearch to Akka Streams

Apache kafka

Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...

Kafka Connect and Streams (Concepts, Architecture, Features)

Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy

SSR: Structured Streaming for R and Machine Learning

SSR: Structured Streaming on R for Machine Learning with Felix Cheung

Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...

Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Kafka

Streaming ETL with Apache Kafka and KSQL

Jug - ecosystem

Mehr von Spark Summit

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang Spark Summit

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...Spark Summit

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang WuSpark Summit

Improving Traffic Prediction Using Weather Data with Ramya RaghavendraSpark Summit

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...Spark Summit

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...Spark Summit

Apache Spark and Tensorflow as a Service with Jim DowlingSpark Summit

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...Spark Summit

Next CERN Accelerator Logging Service with Jakub WozniakSpark Summit

Powering a Startup with Apache Spark with Kevin KimSpark Summit

Improving Traffic Prediction Using Weather Datawith Ramya RaghavendraSpark Summit

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...Spark Summit

How Nielsen Utilized Databricks for Large-Scale Research and Development with...Spark Summit

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...Spark Summit

Goal Based Data Production with Sim SimeonovSpark Summit

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...Spark Summit

Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Spark Summit

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...Spark Summit

Mehr von Spark Summit (20)

FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...

Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu

Improving Traffic Prediction Using Weather Data with Ramya Raghavendra

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...

No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...

Apache Spark and Tensorflow as a Service with Jim Dowling

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...

Next CERN Accelerator Logging Service with Jakub Wozniak

Powering a Startup with Apache Spark with Kevin Kim

Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra

Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...

How Nielsen Utilized Databricks for Large-Scale Research and Development with...

Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...

Goal Based Data Production with Sim Simeonov

Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...

Getting Ready to Use Redis with Apache Spark with Dvir Volk

Deduplication and Author-Disambiguation of Streaming Records via Supervised M...

MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...

Kürzlich hochgeladen

wepik-insightful-infographics-a-data-visualization-overview-20240401133220kwr...KarteekMane1

Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics

Real-Time AI Streaming - AI Max PrincetonTimothy Spann

Data Analysis Project: Stroke PredictionBoston Institute of Analytics

Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter

Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy

Cyber awareness ppt on the recorded dataTecnoIncentive

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024Susanna-Assunta Sansone

Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen

modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx

Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics

Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics

Unveiling the Role of Social Media Suspect Investigators in Preventing Online...Milind Agarwal

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)

Easter Eggs From Star Wars and in cars 1 and 217djon017

English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml

Principles and Practices of Data VisualizationKianJazayeri1

Semantic Shed - Squashing and Squeezing.pptxMike Bennett

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics

Advanced Machine Learning for Business ProfessionalsVICTOR MAESTRE RAMIREZ

Kürzlich hochgeladen (20)

wepik-insightful-infographics-a-data-visualization-overview-20240401133220kwr...

Bank Loan Approval Analysis: A Comprehensive Data Analysis Project

Real-Time AI Streaming - AI Max Princeton

Data Analysis Project: Stroke Prediction

Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...

Student profile product demonstration on grades, ability, well-being and mind...

Cyber awareness ppt on the recorded data

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024

Data Factory in Microsoft Fabric (MsBIP #82)

modul pembelajaran robotic Workshop _ by Slidesgo.pptx

Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...

Decoding Patterns: Customer Churn Prediction Data Analysis Project

Unveiling the Role of Social Media Suspect Investigators in Preventing Online...

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...

Easter Eggs From Star Wars and in cars 1 and 2

English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf

Principles and Practices of Data Visualization

Semantic Shed - Squashing and Squeezing.pptx

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...

Advanced Machine Learning for Business Professionals

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ewen Cheslack-Postava

1. BUILDING REALTIME DATA PIPELINES WITH KAFKA CONNECT AND SPARK STREAMING Ewen Cheslack-Postava Confluent

2. About Me: Ewen Cheslack-Postava • Engineer @ Confluent • Kafka Committer • Kafka Connect Lead

3. Traditional ETL

4. More Data Systems

5. Stream Processing

8. Separation of Concerns

9. Large-scale streaming data import/export for Kafka Kafka Connect

10.

11. Separation of Concerns

12.

13.

14. Tasks - Parallelism

15. Execution Model - Standalone

16. Execution Model - Distributed

17. Execution Model - Distributed

18. Execution Model - Distributed

19. Data Integration as a Service

20. Delivery Guarantees • Automatic offset checkpointing and recovery – Supports at least once – Exactly once for connectors that support it (e.g. HDFS) – At most once simply swaps write & commit – On restart: task checks offsets & rewinds

21. Spark Streaming • Use Direct Kafka streams (1.3+) – Better integration, more efficient, better semantics • Spark Kafka Writer – At least once – Kafka community is working on improved producer semantics

22. Spark Streaming & Kafka Connect • Increase # of systems Spark Streaming works with, indirectly • Reduce friction to adopt Spark Streaming • Reduce need for Spark-specific connectors • By leveraging Kafka as de facto streaming data storage

23. Kafka Connect Summary 23 • Designed for large scale stream or batch data integration • Community supported and certified way of using Kafka • Soon, large repository of open source connectors • Easy data pipelines when combined with Spark & Spark Streaming

24. THANK YOU. Follow me on Twitter: @ewencp Try it out: http://confluent.io/download More like this, but in blog form: http://confluent.io/blog

25. Add Pages as Necessary • Supporting points go here.