Real-time Analytics with Apache Flink and Druid

•

5 gefällt mir•2,289 views

J

Slides of the talk I gave @ Berlin Buzzwords 2016.

REAL-TIME ANALYTICS WITH
APACHE FLINK
AND DRUID
Berlin Buzzwords 2016
Jan Graßegger - @gesundkrank

DATA ENGINEER @

OUR DATA
70,000EVENTS
PER
SECOND 50DIMENSIONS
20METRICS

DRUID

DRUID
‣ Online Analytical Processing (OLAP) System
‣ Column-oriented
‣ Distributed
‣ Built-in data sharding based on time windows
‣ JSON query language

DATA STRUCTURES
Column
TOP PRIVATE DOMAIN
battle.net
battle.net
noxxic.com
noxxic.com
Strings to Integers
battle.net 5
noxxic.com 6
Encoded column data
[5, 5, 6, 6]

DATA STRUCTURES
Column Bitmap Indices
battle.net [1, 1, 0, 0]
noxxic.com [0, 0, 1, 1]
TOP PRIVATE DOMAIN
battle.net
battle.net
noxxic.com
noxxic.com

FIREHOSES
FIREHOSES

APACHE FLINK

PROCESSING
?Kafka Flink Druid

TRANQUILITY

TRANQUILITY
‣ Helps ingesting real-time data into Druid
‣ Provides adapters for Samza, Spark, Storm and
Flink
‣ Standalone HTTP and Kafka applications

Kafka Flink Druid
Tranquility
PROCESSING
Replays?

LAMBDA

KAPPA

Kafka Flink Druid
Tranquility
HDFS
for replays
PROCESSING

RESULTS
▸Kappa-like architecture that’s able to do replays from
HDFS & Kafka
▸Added Flink sink to Tranquility
▸“Hacked“ replays into Tranquility
▸Real-Time Reporting

QUESTIONS?

Empfohlen

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks

Introduction to Stream Processing

Introduction to Stream Processing

Introduction to Stream ProcessingGuido Schmutz

Large Scale Lakehouse Implementation Using Structured Streaming

Large Scale Lakehouse Implementation Using Structured Streaming

Large Scale Lakehouse Implementation Using Structured StreamingDatabricks

Apache Flink Deep Dive

Apache Flink Deep Dive

Apache Flink Deep DiveDataWorks Summit

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1Databricks

The delta architecture

The delta architecture

The delta architecturePrakash Chockalingam

Azure Data Factory ETL Patterns in the Cloud

Azure Data Factory ETL Patterns in the Cloud

Azure Data Factory ETL Patterns in the CloudMark Kromer

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !

Empfohlen

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks

Introduction to Stream Processing

Introduction to Stream Processing

Introduction to Stream ProcessingGuido Schmutz

Large Scale Lakehouse Implementation Using Structured Streaming

Large Scale Lakehouse Implementation Using Structured Streaming

Large Scale Lakehouse Implementation Using Structured StreamingDatabricks

Apache Flink Deep Dive

Apache Flink Deep Dive

Apache Flink Deep DiveDataWorks Summit

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 1Databricks

The delta architecture

The delta architecture

The delta architecturePrakash Chockalingam

Azure Data Factory ETL Patterns in the Cloud

Azure Data Factory ETL Patterns in the Cloud

Azure Data Factory ETL Patterns in the CloudMark Kromer

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !

Disaster Recovery Synapse

Disaster Recovery Synapse

Disaster Recovery SynapseRicardoLinhares22

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!Guido Schmutz

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache KafkaKai Wähner

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta LakeDatabricks

Productizing Structured Streaming Jobs

Productizing Structured Streaming Jobs

Productizing Structured Streaming JobsDatabricks

Building an open data platform with apache iceberg

Building an open data platform with apache iceberg

Building an open data platform with apache icebergAlluxio, Inc.

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta LakeDatabricks

Apache Spark Overview

Apache Spark Overview

Apache Spark OverviewVadim Y. Bichutskiy

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Cathrine Wilhelmsen

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptxKshitija(KJ) Gupte

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Timothy Spann

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics PrimerDatabricks

HDInsight for Architects

HDInsight for Architects

HDInsight for ArchitectsAshish Thapliyal

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of dataRostislav Pashuto

Scalable Real-time analytics using Druid

Scalable Real-time analytics using Druid

Scalable Real-time analytics using DruidDataWorks Summit/Hadoop Summit

Weitere ähnliche Inhalte

Was ist angesagt?

Disaster Recovery Synapse

Disaster Recovery Synapse

Disaster Recovery SynapseRicardoLinhares22

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!Guido Schmutz

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache KafkaKai Wähner

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta LakeDatabricks

Productizing Structured Streaming Jobs

Productizing Structured Streaming Jobs

Productizing Structured Streaming JobsDatabricks

Building an open data platform with apache iceberg

Building an open data platform with apache iceberg

Building an open data platform with apache icebergAlluxio, Inc.

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta LakeDatabricks

Apache Spark Overview

Apache Spark Overview

Apache Spark OverviewVadim Y. Bichutskiy

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Cathrine Wilhelmsen

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptxKshitija(KJ) Gupte

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Timothy Spann

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics PrimerDatabricks

HDInsight for Architects

HDInsight for Architects

HDInsight for ArchitectsAshish Thapliyal

Was ist angesagt? (20)

Disaster Recovery Synapse

Disaster Recovery Synapse

Disaster Recovery Synapse

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!

ksqlDB - Stream Processing simplified!

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta Lake

Building Reliable Data Lakes at Scale with Delta Lake

Productizing Structured Streaming Jobs

Productizing Structured Streaming Jobs

Productizing Structured Streaming Jobs

Building an open data platform with apache iceberg

Building an open data platform with apache iceberg

Building an open data platform with apache iceberg

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta Lake

Making Apache Spark Better with Delta Lake

Apache Spark Overview

Apache Spark Overview

Apache Spark Overview

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptx

Migration to Databricks - On-prem HDFS.pptx

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processing

Introduction to Apache Flink - Fast and reliable big data processing

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

The Rise Of Event Streaming – Why Apache Kafka Changes Everything

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics Primer

Building Lakehouses on Delta Lake with SQL Analytics Primer

HDInsight for Architects

HDInsight for Architects

HDInsight for Architects

Andere mochten auch

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of dataRostislav Pashuto

Scalable Real-time analytics using Druid

Scalable Real-time analytics using Druid

Scalable Real-time analytics using DruidDataWorks Summit/Hadoop Summit

Real-time analytics with Druid at Appsflyer

Real-time analytics with Druid at Appsflyer

Real-time analytics with Druid at AppsflyerMichael Spector

Case Study: Realtime Analytics with Druid

Case Study: Realtime Analytics with Druid

Case Study: Realtime Analytics with DruidSalil Kalia

Interactive analytics at scale with druid

Interactive analytics at scale with druid

Interactive analytics at scale with druidJulien Lavigne du Cadet

Data Analytics with Druid

Data Analytics with Druid

Data Analytics with DruidYousun Jeong

Druid realtime indexing

Druid realtime indexing

Druid realtime indexingSeoeun Park

Druid at Hadoop Ecosystem

Druid at Hadoop Ecosystem

Druid at Hadoop EcosystemSlim Bouguerra

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and DruidTony Ng

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...SoftServe

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Click-Through Example for Flink’s KafkaConsumer CheckpointingRobert Metzger

Apache Kylin - OLAP Cubes for SQL on Hadoop

Apache Kylin - OLAP Cubes for SQL on Hadoop

Apache Kylin - OLAP Cubes for SQL on HadoopTed Dunning

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Apache Kylin: OLAP Engine on Hadoop - Tech Deep DiveXu Jiang

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIOJozo Kovac

Apache Provisionr (incubating) - Bucharest JUG 10

Apache Provisionr (incubating) - Bucharest JUG 10

Apache Provisionr (incubating) - Bucharest JUG 10Andrei Savu

Helio, a Continues Real-Time Fraud Detection and Monitoring Solution

Helio, a Continues Real-Time Fraud Detection and Monitoring Solution

Helio, a Continues Real-Time Fraud Detection and Monitoring SolutionAmir Sedighi

Strata lightening-talk

Strata lightening-talk

Strata lightening-talkDanny Yuan

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...Data Con LA

Druid @ branch Biswajit Das

Apache Kylin Streaming

Apache Kylin Streaming

Apache Kylin Streaming hongbin ma

Andere mochten auch (20)

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of data

Aggregated queries with Druid on terrabytes and petabytes of data

Scalable Real-time analytics using Druid

Scalable Real-time analytics using Druid

Scalable Real-time analytics using Druid

Real-time analytics with Druid at Appsflyer

Real-time analytics with Druid at Appsflyer

Real-time analytics with Druid at Appsflyer

Case Study: Realtime Analytics with Druid

Case Study: Realtime Analytics with Druid

Case Study: Realtime Analytics with Druid

Interactive analytics at scale with druid

Interactive analytics at scale with druid

Interactive analytics at scale with druid

Data Analytics with Druid

Data Analytics with Druid

Data Analytics with Druid

Druid realtime indexing

Druid realtime indexing

Druid realtime indexing

Druid at Hadoop Ecosystem

Druid at Hadoop Ecosystem

Druid at Hadoop Ecosystem

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Click-Through Example for Flink’s KafkaConsumer Checkpointing

Apache Kylin - OLAP Cubes for SQL on Hadoop

Apache Kylin - OLAP Cubes for SQL on Hadoop

Apache Kylin - OLAP Cubes for SQL on Hadoop

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Realtime streaming architecture in INFINARIO

Apache Provisionr (incubating) - Bucharest JUG 10

Apache Provisionr (incubating) - Bucharest JUG 10

Apache Provisionr (incubating) - Bucharest JUG 10

Helio, a Continues Real-Time Fraud Detection and Monitoring Solution

Helio, a Continues Real-Time Fraud Detection and Monitoring Solution

Helio, a Continues Real-Time Fraud Detection and Monitoring Solution

Strata lightening-talk

Strata lightening-talk

Strata lightening-talk

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...

Druid @ branch

Apache Kylin Streaming

Apache Kylin Streaming

Apache Kylin Streaming

Ähnlich wie Real-time Analytics with Apache Flink and Druid

Cassandra + Spark (You’ve got the lighter, let’s start a fire)

Cassandra + Spark (You’ve got the lighter, let’s start a fire)

Cassandra + Spark (You’ve got the lighter, let’s start a fire)Robert Stupp

Open Security Operations Center - OpenSOC

Open Security Operations Center - OpenSOC

Open Security Operations Center - OpenSOCSheetal Dolas

Webinar - Big Data: Let's SMACK - Jorg Schad

Webinar - Big Data: Let's SMACK - Jorg Schad

Webinar - Big Data: Let's SMACK - Jorg SchadCodemotion

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...Robert Metzger

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...javier ramirez

Introduction to apache kafka, confluent and why they matter

Introduction to apache kafka, confluent and why they matter

Introduction to apache kafka, confluent and why they matterPaolo Castagna

ThoughtWorks Technology Radar Roadshow - Melbourne

ThoughtWorks Technology Radar Roadshow - Melbourne

ThoughtWorks Technology Radar Roadshow - MelbourneThoughtworks

Real-Time Analytics with Confluent and MemSQL

Real-Time Analytics with Confluent and MemSQL

Real-Time Analytics with Confluent and MemSQLSingleStore

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg SchadSpark Summit

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...ScyllaDB

AWS Welcome to re:Invent recap - 20161214

AWS Welcome to re:Invent recap - 20161214

AWS Welcome to re:Invent recap - 20161214Amazon Web Services

Transforming the Database: Critical Innovations for Performance at Scale

Transforming the Database: Critical Innovations for Performance at Scale

Transforming the Database: Critical Innovations for Performance at ScaleScyllaDB

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Surviveconfluent

28March2024-Codeless-Generative-AI-Pipelines

28March2024-Codeless-Generative-AI-Pipelines

28March2024-Codeless-Generative-AI-PipelinesTimothy Spann

Introduction to Streaming with Apache Flink

Introduction to Streaming with Apache Flink

Introduction to Streaming with Apache FlinkTugdual Grall

Visualizing a global DNS network with open source tools

Visualizing a global DNS network with open source tools

Visualizing a global DNS network with open source toolsAPNIC

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Servicesconfluent

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...DataStax Academy

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?confluent

Managing data analytics in a hybrid cloud

Managing data analytics in a hybrid cloud

Managing data analytics in a hybrid cloudKaran Singh

Ähnlich wie Real-time Analytics with Apache Flink and Druid (20)

Cassandra + Spark (You’ve got the lighter, let’s start a fire)

Cassandra + Spark (You’ve got the lighter, let’s start a fire)

Cassandra + Spark (You’ve got the lighter, let’s start a fire)

Open Security Operations Center - OpenSOC

Open Security Operations Center - OpenSOC

Open Security Operations Center - OpenSOC

Webinar - Big Data: Let's SMACK - Jorg Schad

Webinar - Big Data: Let's SMACK - Jorg Schad

Webinar - Big Data: Let's SMACK - Jorg Schad

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...

Apache Flink Meetup Munich (November 2015): Flink Overview, Architecture, Int...

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...

Big Data analytics with Nginx, Logstash, Redis, Google Bigquery and Neo4j, ja...

Introduction to apache kafka, confluent and why they matter

Introduction to apache kafka, confluent and why they matter

Introduction to apache kafka, confluent and why they matter

ThoughtWorks Technology Radar Roadshow - Melbourne

ThoughtWorks Technology Radar Roadshow - Melbourne

ThoughtWorks Technology Radar Roadshow - Melbourne

Real-Time Analytics with Confluent and MemSQL

Real-Time Analytics with Confluent and MemSQL

Real-Time Analytics with Confluent and MemSQL

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad

Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...

Numberly on Joining Billions of Rows in Seconds: Replacing MongoDB and Hive w...

AWS Welcome to re:Invent recap - 20161214

AWS Welcome to re:Invent recap - 20161214

AWS Welcome to re:Invent recap - 20161214

Transforming the Database: Critical Innovations for Performance at Scale

Transforming the Database: Critical Innovations for Performance at Scale

Transforming the Database: Critical Innovations for Performance at Scale

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

28March2024-Codeless-Generative-AI-Pipelines

28March2024-Codeless-Generative-AI-Pipelines

28March2024-Codeless-Generative-AI-Pipelines

Introduction to Streaming with Apache Flink

Introduction to Streaming with Apache Flink

Introduction to Streaming with Apache Flink

Visualizing a global DNS network with open source tools

Visualizing a global DNS network with open source tools

Visualizing a global DNS network with open source tools

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

Managing data analytics in a hybrid cloud

Managing data analytics in a hybrid cloud

Managing data analytics in a hybrid cloud

Kürzlich hochgeladen

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...ranjana rawat

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

College Call Girls Nashik Nehal 7001305949 Independent Escort Service NashikCall Girls in Nagpur High Profile

Introduction to IEEE STANDARDS and its different types.pptx

Introduction to IEEE STANDARDS and its different types.pptx

Introduction to IEEE STANDARDS and its different types.pptxupamatechverse

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Bookingdharasingh5698

Java Programming :Event Handling(Types of Events)

Java Programming :Event Handling(Types of Events)

Java Programming :Event Handling(Types of Events)simmis5

Water Industry Process Automation & Control Monthly - April 2024

Water Industry Process Automation & Control Monthly - April 2024

Water Industry Process Automation & Control Monthly - April 2024Water Industry Process Automation & Control

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...Dr.Costas Sachpazis

Roadmap to Membership of RICS - Pathways and Routes

Roadmap to Membership of RICS - Pathways and Routes

Roadmap to Membership of RICS - Pathways and RoutesM Maged Hegazy, LLM, MBA, CCP, P3O

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...ranjana rawat

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

Porous Ceramics seminar and technical writing

Porous Ceramics seminar and technical writing

Porous Ceramics seminar and technical writingrakeshbaidya232001

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete RecordAsst.prof M.Gokilavani

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...ranjana rawat

UNIT-II FMM-Flow Through Circular Conduits

UNIT-II FMM-Flow Through Circular Conduits

UNIT-II FMM-Flow Through Circular Conduitsrknatarajan

result management system report for college project

result management system report for college project

result management system report for college projectTonystark477637

UNIT - IV - Air Compressors and its Performance

UNIT - IV - Air Compressors and its Performance

UNIT - IV - Air Compressors and its Performancesivaprakash250

Extrusion Processes and Their Limitations

Extrusion Processes and Their Limitations

Extrusion Processes and Their Limitations120cr0395

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

UNIT-V FMM.HYDRAULIC TURBINE - Construction and workingrknatarajan

Coefficient of Thermal Expansion and their Importance.pptx

Coefficient of Thermal Expansion and their Importance.pptx

Coefficient of Thermal Expansion and their Importance.pptxAsutosh Ranjan

Kürzlich hochgeladen (20)

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

Introduction to IEEE STANDARDS and its different types.pptx

Introduction to IEEE STANDARDS and its different types.pptx

Introduction to IEEE STANDARDS and its different types.pptx

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking

VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking

Java Programming :Event Handling(Types of Events)

Java Programming :Event Handling(Types of Events)

Java Programming :Event Handling(Types of Events)

Water Industry Process Automation & Control Monthly - April 2024

Water Industry Process Automation & Control Monthly - April 2024

Water Industry Process Automation & Control Monthly - April 2024

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Roadmap to Membership of RICS - Pathways and Routes

Roadmap to Membership of RICS - Pathways and Routes

Roadmap to Membership of RICS - Pathways and Routes

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

Porous Ceramics seminar and technical writing

Porous Ceramics seminar and technical writing

Porous Ceramics seminar and technical writing

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

UNIT-II FMM-Flow Through Circular Conduits

UNIT-II FMM-Flow Through Circular Conduits

UNIT-II FMM-Flow Through Circular Conduits

result management system report for college project

result management system report for college project

result management system report for college project

UNIT - IV - Air Compressors and its Performance

UNIT - IV - Air Compressors and its Performance

UNIT - IV - Air Compressors and its Performance

Extrusion Processes and Their Limitations

Extrusion Processes and Their Limitations

Extrusion Processes and Their Limitations

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

Coefficient of Thermal Expansion and their Importance.pptx

Coefficient of Thermal Expansion and their Importance.pptx

Coefficient of Thermal Expansion and their Importance.pptx

Real-time Analytics with Apache Flink and Druid

1. REAL-TIME ANALYTICS WITH APACHE FLINK AND DRUID Berlin Buzzwords 2016 Jan Graßegger - @gesundkrank

2. DATA ENGINEER @

3. OUR DATA 70,000EVENTS PER SECOND 50DIMENSIONS 20METRICS

5. DRUID ‣ Online Analytical Processing (OLAP) System ‣ Column-oriented ‣ Distributed ‣ Built-in data sharding based on time windows ‣ JSON query language

6. DATA STRUCTURES Column TOP PRIVATE DOMAIN battle.net battle.net noxxic.com noxxic.com Strings to Integers battle.net 5 noxxic.com 6 Encoded column data [5, 5, 6, 6]

7. DATA STRUCTURES Column Bitmap Indices battle.net [1, 1, 0, 0] noxxic.com [0, 0, 1, 1] TOP PRIVATE DOMAIN battle.net battle.net noxxic.com noxxic.com

8. FIREHOSES FIREHOSES

9. APACHE FLINK

10. PROCESSING ?Kafka Flink Druid

11. TRANQUILITY

12. TRANQUILITY ‣ Helps ingesting real-time data into Druid ‣ Provides adapters for Samza, Spark, Storm and Flink ‣ Standalone HTTP and Kafka applications

13. Kafka Flink Druid Tranquility PROCESSING Replays?

16. Kafka Flink Druid Tranquility HDFS for replays PROCESSING

17. RESULTS ▸Kappa-like architecture that’s able to do replays from HDFS & Kafka ▸Added Flink sink to Tranquility ▸“Hacked“ replays into Tranquility ▸Real-Time Reporting