Rust with-kafka-07-02-2019

•

0 likes•988 views

Gerard Klijs

Short presentation about using rust with kafka.

Software

www.openweb.nl
Kafka + Schema Registry
in Rust
By Gerard Klijs
Developer at Openweb
gerard@openweb.nl

www.openweb.nl
Summary
▰ About me
▰ Short intro to Kafka
▰ Short intro to the Confluent Schema Registry
▰ Rust support for Kafka
▰ Using rdkafka
▰ Turning the Schema Registry code into a Crate
222

www.openweb.nl
About me
▰ About 7 years working experience as developer,
mainly java, mainly web.
▰ Some pet projects in Clojure(script)
▰ Became especially interested in rust after
reading Rust for Clojurists
▰ Less than a year since I worked through the
book, no prior C or C++ experience.
33

www.openweb.nl
Short introduction to Apache Kafka
▰ A distributed streaming platform.
▰ Developed by LinkedIn, Open sourced in 2011.
▰ Main parts of the platform run on the JVM.
▰ Librdkafka is a C/C++ library implementing both
the producer and the consumer.
▰ Very scalable.
▰ Large ecosystem.
44

www.openweb.nl
Short introduction to Schema Registry
▰ Part of the Confluent Platform, from the
company created in 2014 by the creators.
▰ Uses Apache Avro for data serialization.
▰ Schema’s are stored in Kafka, each message
contains a key.
▰ The Schema Registry itself is open source.
▰ Versions the data and at the same time makes
the messages smaller.
55

www.openweb.nl
Rust support for Kafka
▰ rdkafka: Rust wrapper for librdkafka.
▰ kafka: Rust client for Apache Kafka.
▰ No support confluent platform, but several
crates working with Apache Avro.
66

www.openweb.nl
Using rdkafka
▰ Started with the provided examples from
rdkafka and get it to work.
▰ Tried several avro and serde libraries till I found
a match that worked.
77

www.openweb.nl
Turning the Schema Registry code into
a Crate
▰ Frame part which is generic and missing in
current crates.
▰ Create new project and pasted in code.
▰ Add unit tests and documentation.
▰ Some CI, code coverage and tags.
▰ Make it more generic.
▰ Improve error handling.
▰ Use as lib in originating project.
▰ Publish as schema_registry_converter to
crates.io.
88

What's hot

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...confluent

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia

Developing Secure Scala Applications With Fortify For ScalaLightbend

End to end testing a web application with ClojureGerard Klijs

Event machinealmeidaricardo

RESTEasy Reactive: Why should you care? | DevNation Tech TalkRed Hat Developers

Let the alpakka pull your streamEnno Runne

Integrating microservices with apache camel on kubernetesClaus Ibsen

Ractor's speed is not light-speedSATOSHI TAGOMORI

How to manage large amounts of data with akka streamsIgor Mielientiev

Stateful stream processing with kafka and samzaGeorge Li

Introduction Apache KafkaJoe Stein

Reactive database access with Slick3takezoe

Lessons from managing a Pulsar cluster (Nutanix)StreamNative

Samza at LinkedIn: Taking Stream Processing to the Next LevelMartin Kleppmann

Kafka At Scale in the Cloudconfluent

Apache pulsarKotireddy Sareddy

Apache samzaHumberto Streb

pgWALSyncRumman Iftekhar

Docker and FluentdN Masahiro

What's hot (20)

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...

Developing Secure Scala Applications With Fortify For Scala

End to end testing a web application with Clojure

Event machine

RESTEasy Reactive: Why should you care? | DevNation Tech Talk

Let the alpakka pull your stream

Integrating microservices with apache camel on kubernetes

Ractor's speed is not light-speed

How to manage large amounts of data with akka streams

Stateful stream processing with kafka and samza

Introduction Apache Kafka

Reactive database access with Slick3

Lessons from managing a Pulsar cluster (Nutanix)

Samza at LinkedIn: Taking Stream Processing to the Next Level

Kafka At Scale in the Cloud

Apache pulsar

Apache samza

pgWALSync

Docker and Fluentd

Similar to Rust with-kafka-07-02-2019

Data Engineer’s Lunch #45: Apache LivyAnant Corporation

Kafka ExplainatonNguyenChiHoangMinh

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)My own sweet home!

Introduction to ScalaMohammad Hossein Rimaz

.NET RDF APIsAndrei Iacob

Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent

Apache Kafka 0.8 basic training - VerisignMichael Noll

Polyglot Plugin ProgrammingAtlassian

What's New in AWS Serverless and ContainersAmazon Web Services

Lecture 2_ Intro to laravel.pptxSaziaRahman

7 Apache Process Cloudstack Developer DayKimihiko Kitase

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre ZembStreamNative

Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll

Graph ql subscriptions through the looking glassGerard Klijs

Kafka for DBAsGwen (Chen) Shapira

Apache kafkaKumar Shivam

From java to scala at crowd mixStefano Galarraga

Why scala for data scienceGuglielmo Iozzia

A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende

Similar to Rust with-kafka-07-02-2019 (20)

Data Engineer’s Lunch #45: Apache Livy

Kafka Explainaton

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)

Introduction to Scala

.NET RDF APIs

Kafka Summit SF 2017 - Kafka and the Polyglot Programmer

Apache Kafka 0.8 basic training - Verisign

Polyglot Plugin Programming

What's New in AWS Serverless and Containers

Lecture 2_ Intro to laravel.pptx

7 Apache Process Cloudstack Developer Day

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Graph ql subscriptions through the looking glass

Kafka for DBAs

Apache kafka

From java to scala at crowd mix

Why scala for data science

A Jupyter kernel for Scala and Apache Spark.pdf

Recently uploaded

Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS LiveCall Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️Delhi Call girls

5 Signs You Need a Fashion PLM Software.pdfWave PLM

TECUNIQUE: Success Stories: IT Service providermohitmore19

Software Quality Assurance Interview QuestionsArshad QA

Diamond Application Development Crafting Solutions with PrecisionSolGuruz

Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...harshavardhanraghave

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...Health

Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...MyIntelliSource, Inc.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Steffen Staab

Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsAlberto González Trastoy

Optimizing AI for immediate response in Smart CCTVshikhaohhpro

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...MyIntelliSource, Inc.

Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfkalichargn70th171

CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️anilsa9823

Unlocking the Future of AI Agents with Large Language Modelsaagamshah0812

SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AIABDERRAOUF MEHENNI

How To Troubleshoot Collaboration Apps for the Modern Connected WorkerThousandEyes

Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...kellynguyen01

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...panagenda

Recently uploaded (20)

Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

5 Signs You Need a Fashion PLM Software.pdf

TECUNIQUE: Success Stories: IT Service provider

Software Quality Assurance Interview Questions

Diamond Application Development Crafting Solutions with Precision

Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

Optimizing AI for immediate response in Smart CCTV

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...

Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf

CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️

Unlocking the Future of AI Agents with Large Language Models

SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

Rust with-kafka-07-02-2019

1. www.openweb.nl Kafka + Schema Registry in Rust By Gerard Klijs Developer at Openweb gerard@openweb.nl

2. www.openweb.nl Summary ▰ About me ▰ Short intro to Kafka ▰ Short intro to the Confluent Schema Registry ▰ Rust support for Kafka ▰ Using rdkafka ▰ Turning the Schema Registry code into a Crate 222

3. www.openweb.nl About me ▰ About 7 years working experience as developer, mainly java, mainly web. ▰ Some pet projects in Clojure(script) ▰ Became especially interested in rust after reading Rust for Clojurists ▰ Less than a year since I worked through the book, no prior C or C++ experience. 33

4. www.openweb.nl Short introduction to Apache Kafka ▰ A distributed streaming platform. ▰ Developed by LinkedIn, Open sourced in 2011. ▰ Main parts of the platform run on the JVM. ▰ Librdkafka is a C/C++ library implementing both the producer and the consumer. ▰ Very scalable. ▰ Large ecosystem. 44

5. www.openweb.nl Short introduction to Schema Registry ▰ Part of the Confluent Platform, from the company created in 2014 by the creators. ▰ Uses Apache Avro for data serialization. ▰ Schema’s are stored in Kafka, each message contains a key. ▰ The Schema Registry itself is open source. ▰ Versions the data and at the same time makes the messages smaller. 55

6. www.openweb.nl Rust support for Kafka ▰ rdkafka: Rust wrapper for librdkafka. ▰ kafka: Rust client for Apache Kafka. ▰ No support confluent platform, but several crates working with Apache Avro. 66

7. www.openweb.nl Using rdkafka ▰ Started with the provided examples from rdkafka and get it to work. ▰ Tried several avro and serde libraries till I found a match that worked. 77

8. www.openweb.nl Turning the Schema Registry code into a Crate ▰ Frame part which is generic and missing in current crates. ▰ Create new project and pasted in code. ▰ Add unit tests and documentation. ▰ Some CI, code coverage and tags. ▰ Make it more generic. ▰ Improve error handling. ▰ Use as lib in originating project. ▰ Publish as schema_registry_converter to crates.io. 88

9. www.openweb.nl Questions?

Rust with-kafka-07-02-2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Rust with-kafka-07-02-2019

Similar to Rust with-kafka-07-02-2019 (20)

Recently uploaded

Recently uploaded (20)

Rust with-kafka-07-02-2019