Jay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyft + Priya Shivakumar, Confluent Cloud

•Download as PPTX, PDF•

0 likes•1,436 views

The document discusses how companies are increasingly becoming software and using more software. It contrasts traditional databases with newer approaches using event streaming and stream processing. Key points include: - Traditional databases assume data is passive, but with event streaming data can be active - Event streaming platforms like Kafka support storing and transmitting data as event streams - Stream processing tools like KSQL allow querying and transforming infinite event streams - This represents a shift from databases as passive data stores to active stream processing systems - KSQL aims to provide a streaming SQL engine that treats event streams as tables, providing an event streaming database

Technology

Marc Andreessen:
Software is Eating the World

Strong Form
Companies are
BECOMING SOFTWARE
Weak Form
Companies are
USING MORE SOFTWARE

Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
2

Loan Application Using Software
BORROWER
1
CREDIT
OFFICER
3
LOAN
OFFICER
5
RISK
OFFICER
4
APPROVE
DENY
6
APPLICATION
FORM
2
HUMAN CENTRIC
1-2 WEEKS

Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2

Loan Application in Software
BORROWER
1
APPROVE
DENY
3
LOAN APP UI
CREDIT
SERVICE
RISK
SERVICE
CRM
SERVICE
$ !
2
SOFTWARE CENTRIC
SECONDS

Using Software:
Classic Three Tier Architecture
USER UI SERVICE DATABASE

Becoming Software:
Services Talking To Each Other With APIs
SERVICESERVICESERVICE SERVICE

IS MORE
SOFTWARE
THE USER OF
THE SOFTWARE

We have hundreds
of databases...
FUNDAMENTAL ASSUMPTION:
DATA IS PASSIVE

Fundamental Assumption: Data is Passive
QUERY
UPDATE

Good for Building:
CRUD
Applications
Synchronous
Services

Unless there is a user and UI waiting,
why should it be synchronous?

EVENT
STREAMING
PLATFORM
APP
APP APP
APP
APP APP
APP
APP
DB
DB
DB
APP
APP
APP
APP
DB
DB
DB
APP
APP

Kafka is a Foundation for Event Streams
0 1 2 3 4 5 6 7 8LOG
READS
WRITES
DESTINATION
SYSTEM A
DESTINATION
SYSTEM B

Solution: Stream Processing
INPUT STREAMS OUTPUT STREAMS
STREAM
PROCESSOR

KSQL
Continuous queries
on infinite streams

TRADITIONAL
DATABASE
EVENT STREAM
PROCESSING
SELECT * FROM
DB_TABLE
CREATE TABLE T
AS SELECT * FROM
EVENT_STREAM
Active Query: Passive Data:
DB Table
Active Data: Passive Query:
Event Stream

TABLES STREAMS
USER
JAY
SUE
FRED
CREDIT_SCORE
695
430
710V1
V3
V2
PAYMENTS
42
18
65
...
USER
JAY
SUE
FRED
...

STREAMS
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = ’all_payments’,
key = ’user’,
value_format= ’avro’);

TABLES
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user;

But wait, some things are still
synchronous...

PUSH PULL
APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit score is 695
What is Jay’s credit score now?
695
APP

PUSH PULL
SELECT user, credit_score
FROM credit_history
WHERE ROWKEY = ‘jay’;

Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
STREAM
PROCESSING
CONNECTOR APPDB

STREAM
PROCESSING
Today: Integrating all these systems is up to you
CONNECTOR
CONNECTOR
APP
DB
APP
DB
CONNECTOR APPDB
EXTRACT STORE
LOAD STORE
TRANSFORM

How can we make this simpler?
KLIP 7: cnfl.io/ksql-klip-7
KLIP 8: cnfl.io/ksql-klip-8
github.com/confluentinc/ksql

Solution: Fewer moving parts
33
APP
DB
APP
DB
APP
KSQL

PUSH PULL
SELECT user, credit_score
FROM credit_history
WHERE ROWKEY = ‘jay’
EMIT CHANGES;
SELECT user, credit_score
FROM credit_history
WHERE ROWKEY = ‘jay’;

This is a natural generalization
of databases.

Architecture of a Distributed Database
LOG
DB
DB
DB
SERVING
NODES

Architecture of a Distributed Stream Processing System
LOG
DB
DB
DB
STREAM
PROCESSORS

KSQL: Streaming SQL Engine => Event Streaming Database
LOG
STREAM
PROCESSORS
DB
DB
DB
DB
DB
DB
SERVING
NODES
KSQL

Data Warehousing
JURY RIGGED ETL
FACTS DIMS
REPORTING

A data warehouse is NOT the central
nervous system of a digital business.

EVENT
STREAMING
PLATFORM
APP
APP APP
APP
APP APP
APP
APP
KSQL
DB
DB
APP
APP
APP
APP
KSQL
DB
DB
APP
APP

In fact this makes it easier to use them...
CREATE SINK CONNECTOR elasticConnector WITH (
‘connector.class’ =
'...ElasticsearchSinkConnector',
‘topics’ = 'CREDIT_SCORES',
‘connection.url’ = 'http://localhost:9200',
‘type.name’ = 'kafka-connect',
...
);
APP
KSQL
CONNECTOR
APP

TARGET FIRST RELEASE:
NOVEMBER
KLIP 7: cnfl.io/ksql-klip-7
KLIP 8: cnfl.io/ksql-klip-8
github.com/confluentinc/ksql

What's hot

Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture. Things we’ll discuss decoupling your system at scale via event-driven architectures include: – Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows. – Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern – How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea) – Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way. – Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems – Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem. – The simplicity and power of streaming SQL with microservices

Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...

confluent

This talk includes an introduction to the Kafka ecosystem as well as event-driven microserivces, culminating with 10 rules that help with the design of such systems: 1. Don’t use Kafka for shopping carts! 2. Pick Topics with Business Significance 3. Decouple publishers from subscribers 4. Use the log to regenerate state 5. Apply the Single Writer Principal 6. Leverage keeping datasets inside the broker 7. Prefer stream processing over maintaining historic views 8. Sometimes you need historic views. => Replicate Read Only 9. Use Schemas 10. Consider “Stream Management” Services

10 Principals for Effective Event Driven Microservices

Ben Stopford

APAC ksqlDB Workshop

confluent

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

confluent

Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story

confluent

How Priceline uses Kafka Streams technology to effectively save TBs on daily licenses of our monitoring systems. Kafka Streams powers a big part of our analytics and monitoring pipelines and delivers operational metrics transformations in real time. All logs and operational metrics from all of the APIs of Priceline’s products flow into Kafka and is ingested into our Monitoring System Splunk for Alerting and Monitoring. We have now implemented data transformations, aggregations and summarizations using Kafka Streams technologies to effectively eliminate PCI/PII violations on the log data; do aggregations on metrics to avoid ingesting sub-second metrics and ingest metrics only at the granularity that we need to. We will cover the need for custom Serdes, custom partitioners, and why we don’t use the confluent registry. You will also learn how Priceline uses a self service model to configure its streams, topics and consumers using Data Collection Console, which is our UI for managing the Kafka streaming pipelines.

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...

confluent

10 Principals for Effective Event-Driven Microservices with Apache Kafka

Ben Stopford

In this talk we'll look at the relationship between three of the most disruptive software engineering paradigms: event sourcing, stream processing and serverless. We'll debunk some of the myths around event sourcing. We'll look at the inevitability of event-driven programming in the serverless space and we'll see how stream processing links these two concepts together with a single 'database for events'. As the story unfolds we'll dive into some use cases, examine the practicalities of each approach-particularly the stateful elements-and finally extrapolate how their future relationship is likely to unfold. Key takeaways include: The different flavors of event sourcing and where their value lies. The difference between stream processing at application- and infrastructure-levels. The relationship between stream processors and serverless functions. The practical limits of storing data in Kafka and stream processors like KSQL.

Event Sourcing, Stream Processing and Serverless (Ben Stopford, Confluent) K...

confluent

Have you ever tried to debug a production outage, when your system comprises apps your team has written, third-party apps your team runs, with logs going into some system, application performance metrics going into another system, and cloud platform metrics going somewhere else? Did you find yourself switching tabs, trying to correlate metrics with logs and alerts and finding yourself in a huge tangle? It is a nightmare. In the data world, we talk about aggregating all our data so we can derive new insights quickly, but what about our operational data? Observability is your ability to be able to ask questions of your system without having to write new code, or grab new data. When you've got an observable system, it feels like you have debugging superpowers, but can be challenging to even know where to start. If you can even convince your colleagues to start, finding the right tools can be challenging. In this talk Inny and Andrew will talk about what monitoring and logging are not sufficient anymore (if they ever were), observability basics, and demo an observability platform that you can use to start your observability journey today.

Observability for developer ( Inny So & Andrew Jones, ThoughtWorks) Kafka Su...

confluent

Building Event-Driven Services with Apache Kafka

confluent

Slides from Neha Narkhede's keynote at the dotScale conference in Paris on April 24th, 2017. There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it. This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.

dotScale 2017 Keynote: The Rise of Real Time by Neha Narkhede

confluent

Today’s products - devices, software and services - are well instrumented to permit users, vendors and service providers to gather maximum insight into how they are used, when they need repair and many other operational insights. Ensuring that products can rapidly adapt to a constantly changing environment and changing customer needs requires that the events they generate are analyzed continuously and in context. Insights can be synthesized from many sources in context - geospatial and proximity, trajectory and even predicted future states.Customers, vendors and service providers need to analyze, learn, and predict directly from streaming events because data volumes are huge and automated responses must often be delivered in milliseconds. To achieve insights quickly, we need to build models on-the-fly whose predictions are accurate and in sync with the real world, often to support automation. Many insights depend on analyzing the joint evolution of data sources whose behavior is correlated in time or space.In this talk we present Swim, an Apache 2.0 licensed platform for continuous intelligence applications. Swim builds a fluid model of data sources and their changing relationships in real-time - Swim applications analyze, learn and predict directly from event data. Swim applications integrate with Apache Kafka for event streaming. Developers need nothing more than Java skills. Swim deploys native or in containers on k8s, with the same code in each instance. Instances link to build an application layer mesh that facilitates distribution and massive scale without sacrificing consistency. We will present several continuous intelligence applications in use today that depend on real-time analysis, learning and prediction to power automation and deliver responses that are in sync with the real-world. We will show how easy it is to build, deploy and run distributed, highly available event streaming applications that analyze data from hundreds of millions of sources - petabytes per day. The architecture is intuitively appealing and blazingly fast.

Continuous Intelligence for Customer Service Using Kafka Event Streams | Simo...

HostedbyConfluent

James Watters, Pivotal | Kafka Summit NYC 2019 Keynote (Spring Boot+Kafka: Th...

confluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model. Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance. Speakers: Justin Cunningham Technical Lead, Software Engineer, Yelp Gehrig Kunz Technical Product Marketing Manager, Confluent

How Yelp Leapt to Microservices with More than a Message Queue

confluent

Events, Streams, Devops and Speed - The Next Generation of Application Archit...

confluent

Telling the LivePerson Technology Story at Couchbase [SF] 2013

LivePerson

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

KafkaZone

In this presentation, I will talk about my firsthand experience dealing with the unique challenges of running Kafka at a massive scale. If you ever thought that running Kafka is difficult, this talk may change your mind and provide you with valuable insights into how to configure a Kafka cluster efficiently, how to manage Kafka for enterprise customers and how to measure, monitor and maintain the Quality of Kafka Service. Our production Kafka cluster runs over 1500+ VMs, and serves over 10 GBPS data spread across hundreds of topics for multiple teams across Microsoft. We built a self-serve Kafka management service to make the process manageable and scalable across many teams. In this talk, I will also share insights about running Kafka in Private vs multi-tenant mode, supporting failover and disaster recovery requirements, and how to make Kafka Compliant with regulatory certifications such as ISO, SOC, FEDRAMP, etc. Presented by Nitin Kumar, Microsoft Systems Track

Kafka Summit SF 2017 - Providing Reliability Guarantees in Kafka at One Trill...

confluent

Do you know who is knocking on your network’s door? Have new regulations left you scratching your head on how to handle what is happening in your network? Network flow data helps answer many questions across a multitude of use cases including network security, performance, capacity planning, routing, operational troubleshooting and more. Today’s modern day streaming data pipelines need to include tools that can scale to meet the demands of these service providers while continuing to provide responsive answers to difficult questions. In addition to stream processing, data needs to be stored in a redundant, operationally focused database to provide fast, reliable answers to critical questions. Together, Kafka and Druid work together to create such a pipeline. In this talk Eric Graham and Rachel Pedreschi will discuss these pipelines and cover the following topics: -Network flow use cases and why this data is important. -Reference architectures from production systems at a major international Bank. -Why Kafka and Druid and other OSS tools for Network Flows. -A demo of one such system.

How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi, Imply ...

confluent

What's hot (20)