Kafka Lambda architecture with mirroring

•Als PPTX, PDF herunterladen•

1 gefällt mir•1,074 views

Anant Rustagi

Hadoop Cluster
Hadoop Cluster
Speed Layer
Storm/Spark (Real
time processing)
Batch Layer Analytics
ELT
Kafka Cluster (Mirror)
Mirroring
Kafka Cluster
Hadoop Client for
Camus
Hive/Pig
DWH
HBase
Master Plan (Lambda Architecture)
Variety Sources
REST/IS

Kafka Cluster A
REST/IS
REST/IS
Kafka Cluster B
Mirroring
Kafka Cluster C
Kafka Cluster D
Topic : “Data”
Topic : “Data”
Kafka (Mirroring) Configuration
Integration of all the sources
coming to cluster A and B with
topic Name “Data”
Integration of all the sources
coming to cluster A and B with
topic Name “Data”
Host Configuration
-Quad-Core AMD Opteron(TM)
Processor
-8GB RAM
-320Gb
Batch Layer (Writing the data
into hadoop cluster)
Speed Layer (Feeding data from
Kafka to Storm )

Empfohlen

Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...Anton Kirillov

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

SMACK Stack 1.1Joe Stein

Kafka spark cassandra webinar feb 16 2016 Hiromitsu Komatsu

Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...Helena Edelson

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson

NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch AnalysisHelena Edelson

Lambda architectureSzilveszter Molnár

Empfohlen

Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...Anton Kirillov

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

SMACK Stack 1.1Joe Stein

Kafka spark cassandra webinar feb 16 2016 Hiromitsu Komatsu

Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...Helena Edelson

Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, ScalaHelena Edelson

NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch AnalysisHelena Edelson

Lambda architectureSzilveszter Molnár

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...DataStax Academy

Akka in Production - ScalaDays 2015Evan Chan

Streaming Big Data & Analytics For ScaleHelena Edelson

Real-time personal trainer on the SMACK stackAnirvan Chakraborty

An Introduction to Distributed Search with Datastax Enterprise SearchPatricia Gorla

Sa introduction to big data pipelining with cassandra & spark west mins...Simon Ambridge

Real-Time Anomaly Detection with Spark MLlib, Akka and CassandraNatalino Busa

Reactive dashboard’s using apache sparkRahul Kumar

Using the SDACK Architecture to Build a Big Data ProductEvans Ye

Analyzing Time Series Data with Apache Spark and CassandraPatrick McFadin

Reactive app using actor model & apache sparkRahul Kumar

The How and Why of Fast Data Analytics with Apache SparkLegacy Typesafe (now Lightbend)

Real Time Data Processing Using Spark StreamingHari Shreedharan

Getting Started Running Apache Spark on Apache MesosPaco Nathan

Apache cassandra & apache spark for time series dataPatrick McFadin

Re-envisioning the Lambda Architecture : Web Services & Real-time Analytics ...Brian O'Neill

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...DataStax Academy

Lambda Architecture Using SQLSATOSHI TAGOMORI

Alpine academy apache spark series #1 introduction to cluster computing wit...Holden Karau

Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...Lucidworks

Big Data and Fast Data - Lambda Architecture in ActionGuido Schmutz

Demystifying salesforce for developersHeitor Souza

Weitere ähnliche Inhalte

Was ist angesagt?

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...DataStax Academy

Akka in Production - ScalaDays 2015Evan Chan

Streaming Big Data & Analytics For ScaleHelena Edelson

Real-time personal trainer on the SMACK stackAnirvan Chakraborty

An Introduction to Distributed Search with Datastax Enterprise SearchPatricia Gorla

Sa introduction to big data pipelining with cassandra & spark west mins...Simon Ambridge

Real-Time Anomaly Detection with Spark MLlib, Akka and CassandraNatalino Busa

Reactive dashboard’s using apache sparkRahul Kumar

Using the SDACK Architecture to Build a Big Data ProductEvans Ye

Analyzing Time Series Data with Apache Spark and CassandraPatrick McFadin

Reactive app using actor model & apache sparkRahul Kumar

The How and Why of Fast Data Analytics with Apache SparkLegacy Typesafe (now Lightbend)

Real Time Data Processing Using Spark StreamingHari Shreedharan

Getting Started Running Apache Spark on Apache MesosPaco Nathan

Apache cassandra & apache spark for time series dataPatrick McFadin

Re-envisioning the Lambda Architecture : Web Services & Real-time Analytics ...Brian O'Neill

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...DataStax Academy

Lambda Architecture Using SQLSATOSHI TAGOMORI

Alpine academy apache spark series #1 introduction to cluster computing wit...Holden Karau

Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...Lucidworks

Was ist angesagt? (20)

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

Akka in Production - ScalaDays 2015

Streaming Big Data & Analytics For Scale

Real-time personal trainer on the SMACK stack

An Introduction to Distributed Search with Datastax Enterprise Search

Sa introduction to big data pipelining with cassandra & spark west mins...

Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra

Reactive dashboard’s using apache spark

Using the SDACK Architecture to Build a Big Data Product

Analyzing Time Series Data with Apache Spark and Cassandra

Reactive app using actor model & apache spark

The How and Why of Fast Data Analytics with Apache Spark

Real Time Data Processing Using Spark Streaming

Getting Started Running Apache Spark on Apache Mesos

Apache cassandra & apache spark for time series data

Re-envisioning the Lambda Architecture : Web Services & Real-time Analytics ...

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...

Lambda Architecture Using SQL

Alpine academy apache spark series #1 introduction to cluster computing wit...

Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...

Andere mochten auch

Big Data and Fast Data - Lambda Architecture in ActionGuido Schmutz

Demystifying salesforce for developersHeitor Souza

Extreme Salesforce Data Volumes WebinarSalesforce Developers

How Apache Kafka is transforming Hadoop, Spark and StormEdureka!

Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...Martin Zapletal

7가지 동시성 모델 람다아키텍처Sunggon Song

Handling of Large Data by SalesforceThinqloud

Big Data Day LA 2015 - Event Driven Architecture for Web Analytics by Peyman ...Data Con LA

Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.

Machine learning at Scale with Apache SparkMartin Zapletal

Salesforce REST API Bohdan Dovhań

Understanding the Salesforce Architecture: How We Do the Magic We DoSalesforce Developers

Big Data Architectural Patterns and Best Practices on AWSAmazon Web Services

Salesforce API Series: Fast Parallel Data Loading with the Bulk API WebinarSalesforce Developers

Building a Lambda Architecture with Elasticsearch at Yieldbotyieldbot

Introduction to Apache NiFi - Seattle Scalability MeetupSaptak Sen

Microservice-based Architecture on the Salesforce App Cloudpbattisson

Large Data Management StrategiesSalesforce Developers

람다아키텍처HyeonSeok Choi

Kafka at Scale: Multi-Tier ArchitecturesTodd Palino

Andere mochten auch (20)

Big Data and Fast Data - Lambda Architecture in Action

Demystifying salesforce for developers

Extreme Salesforce Data Volumes Webinar

How Apache Kafka is transforming Hadoop, Spark and Storm

Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...

7가지 동시성 모델 람다아키텍처

Handling of Large Data by Salesforce

Big Data Day LA 2015 - Event Driven Architecture for Web Analytics by Peyman ...

Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud

Machine learning at Scale with Apache Spark

Salesforce REST API

Understanding the Salesforce Architecture: How We Do the Magic We Do

Big Data Architectural Patterns and Best Practices on AWS

Salesforce API Series: Fast Parallel Data Loading with the Bulk API Webinar

Building a Lambda Architecture with Elasticsearch at Yieldbot

Introduction to Apache NiFi - Seattle Scalability Meetup

Microservice-based Architecture on the Salesforce App Cloud

Large Data Management Strategies

람다아키텍처

Kafka at Scale: Multi-Tier Architectures

Kafka Lambda architecture with mirroring

1. Hadoop Cluster Hadoop Cluster Speed Layer Storm/Spark (Real time processing) Batch Layer Analytics ELT Kafka Cluster (Mirror) Mirroring Kafka Cluster Hadoop Client for Camus Hive/Pig DWH HBase Master Plan (Lambda Architecture) Variety Sources REST/IS

2. Kafka Cluster A REST/IS REST/IS Kafka Cluster B Mirroring Kafka Cluster C Kafka Cluster D Topic : “Data” Topic : “Data” Kafka (Mirroring) Configuration Integration of all the sources coming to cluster A and B with topic Name “Data” Integration of all the sources coming to cluster A and B with topic Name “Data” Host Configuration -Quad-Core AMD Opteron(TM) Processor -8GB RAM -320Gb Batch Layer (Writing the data into hadoop cluster) Speed Layer (Feeding data from Kafka to Storm )