Cassandra + Spark + Elk

•

8 gefällt mir•6,834 views

Vasil Remeniuk

Presentation made at scalaby #13 ~> http://scala.by

Technologie

Cassandra+Spark+ELK
Dmitriy Kalyada @ 2015

What is Spark?
• Master: Driver program
• Workers: Executors
• High Availability
• Standby Masters with
ZooKeeper
• Single-Node Recovery with
Local File System

Under the hood
• Resilient Distributed Dataset
(RDD)
• Scala + Akka Framework
• Java, Scala, Python API
• Spark SQL, MLib, Spark
Streaming, GraphX

Our particular case
Devices
Cassandra
Spark
ELK

Data ﬂow
Fetcher Transformer Saver
Input Source(s)
x-RDD x-RDD
Output Source

Spark Cassandra Connector
• Represents Cassandra tables as Spark RDDs
• Write Spark RDDs to Cassandra tables
• Execute CQL queries in Spark applications
https://github.com/datastax/spark-cassandra-connector

CassandraRDD settings
• Connection params
• Fetching params
1. input.split.size: C* partitions in a Spark
Partition.
2. input.page.row.size: number of CQL rows
fetched per roundtrip.

Fetching essentials
…
…
…
…
-968391295277638458 … -893783532241185833
-968391295277638458, -893783532241185833
-7378580094811526501, -7340240117176401239
6426215139012569257, 6428979455828914106
-6094480671546553265, -6016282219056649738
-7259249675596554667, -7237838231745167324
-6734336817058726139, -6684208157211348972
-3891103372671105499, -3822513456325086923
4453206019575747361,4462441725813855391
7855385326468991461,7906589648045207141
-129433796439502583,-101280166181350027
-2233788032218452383,-2066644620711092198
3248662132571799756,3396129453515776704
7744134136205124749,7812918342246679728
-1408208314239486033,-1403736406052004344
• Support
Murmur3Partitioner
and
RandomPartitioner
• Retrieve token ranges
from Cassandra
• Prediction on base of 16
random token ranges

Data to RDD
…
Tokens Per RDD
[input.split.size]
Token Range #N
Slurp amount
[input.page.row.size]

What to do?
• Change read strategy
• Split data on a smaller pieces
• Increase cluster strength
• Reorganize Cassandra schema

Elastic Search & Kibana
• Index initialization: TransportClient
• Create/Delete Index
• Setup Mappings
• Indexing: ScalaEsRDD
• Data presentation: Kibana

Deployment
• Build package: Spark Job + Dependent Jars +
Conﬁgs
• Upload to the Spark Master Node
• Start job submit script

Thank you
dkaliada@exadel.com
Dmitriy Kalyada @ 2015

Empfohlen

Elassandra: Elasticsearch as a Cassandra Secondary Index (Rémi Trouville, Vin...DataStax

Apache Spark RDDsDean Chen

Cassandra spark connectorDuyhai Doan

An Introduction to Distributed Search with Datastax Enterprise SearchPatricia Gorla

Feeding Cassandra with Spark-Streaming and KafkaDataStax Academy

Analytics with Cassandra & SparkMatthias Niehoff

Simplifying Big Data Analytics with Apache SparkDatabricks

SMACK Stack 1.1Joe Stein

Empfohlen

Elassandra: Elasticsearch as a Cassandra Secondary Index (Rémi Trouville, Vin...DataStax

Apache Spark RDDsDean Chen

Cassandra spark connectorDuyhai Doan

An Introduction to Distributed Search with Datastax Enterprise SearchPatricia Gorla

Feeding Cassandra with Spark-Streaming and KafkaDataStax Academy

Analytics with Cassandra & SparkMatthias Niehoff

Simplifying Big Data Analytics with Apache SparkDatabricks

SMACK Stack 1.1Joe Stein

Cassandra & Spark for IoTMatthias Niehoff

Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...Аліна Шепшелей

Reactive dashboard’s using apache sparkRahul Kumar

Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...Data Con LA

Analyzing Time Series Data with Apache Spark and CassandraPatrick McFadin

DataEngConf SF16 - Spark SQL WorkshopHakka Labs

Real time data pipeline with spark streaming and cassandra with mesosRahul Kumar

Tuning and Debugging in Apache SparkPatrick Wendell

Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...Anton Kirillov

How We Used Cassandra/Solr to Build Real-Time Analytics PlatformDataStax Academy

The How and Why of Fast Data Analytics with Apache SparkLegacy Typesafe (now Lightbend)

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

Spark Sql for TrainingBryan Yang

Spark + Cassandra = Real Time Analytics on Operational DataVictor Coustenoble

Spark And Cassandra: 2 Fast, 2 FuriousJen Aman

Building a Lambda Architecture with Elasticsearch at Yieldbotyieldbot

Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...Spark Summit

Developing a Real-time Engine with Akka, Cassandra, and SprayJacob Park

Fast NoSQL from HDDs? ScyllaDB

Kindling: Getting Started with Spark and CassandraDataStax Academy

SMARTSTUDY Django 오픈 세션 2012-08Hyun-woo Park

BI, Reporting and Analytics on Apache CassandraVictor Coustenoble

Weitere ähnliche Inhalte

Was ist angesagt?

Cassandra & Spark for IoTMatthias Niehoff

Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...Аліна Шепшелей

Reactive dashboard’s using apache sparkRahul Kumar

Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...Data Con LA

Analyzing Time Series Data with Apache Spark and CassandraPatrick McFadin

DataEngConf SF16 - Spark SQL WorkshopHakka Labs

Real time data pipeline with spark streaming and cassandra with mesosRahul Kumar

Tuning and Debugging in Apache SparkPatrick Wendell

Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...Anton Kirillov

How We Used Cassandra/Solr to Build Real-Time Analytics PlatformDataStax Academy

The How and Why of Fast Data Analytics with Apache SparkLegacy Typesafe (now Lightbend)

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...Helena Edelson

Spark Sql for TrainingBryan Yang

Spark + Cassandra = Real Time Analytics on Operational DataVictor Coustenoble

Spark And Cassandra: 2 Fast, 2 FuriousJen Aman

Building a Lambda Architecture with Elasticsearch at Yieldbotyieldbot

Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...Spark Summit

Developing a Real-time Engine with Akka, Cassandra, and SprayJacob Park

Fast NoSQL from HDDs? ScyllaDB

Kindling: Getting Started with Spark and CassandraDataStax Academy

Was ist angesagt? (20)

Cassandra & Spark for IoT

Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...

Reactive dashboard’s using apache spark

Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...

Analyzing Time Series Data with Apache Spark and Cassandra

DataEngConf SF16 - Spark SQL Workshop

Real time data pipeline with spark streaming and cassandra with mesos

Tuning and Debugging in Apache Spark

Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...

How We Used Cassandra/Solr to Build Real-Time Analytics Platform

The How and Why of Fast Data Analytics with Apache Spark

Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...

Spark Sql for Training

Spark + Cassandra = Real Time Analytics on Operational Data

Spark And Cassandra: 2 Fast, 2 Furious

Building a Lambda Architecture with Elasticsearch at Yieldbot

Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...

Developing a Real-time Engine with Akka, Cassandra, and Spray

Fast NoSQL from HDDs?

Kindling: Getting Started with Spark and Cassandra

Andere mochten auch

SMARTSTUDY Django 오픈 세션 2012-08Hyun-woo Park

BI, Reporting and Analytics on Apache CassandraVictor Coustenoble

TensorFrames: Google Tensorflow on Apache SparkDatabricks

An Introduction to Distributed Search with Cassandra and SolrDataStax Academy

SASI: Cassandra on the Full Text Search Ride (DuyHai DOAN, DataStax) | C* Sum...DataStax

Spark to Production @WindwardDemi Ben-Ari

Cassandra vs. MongoDBScaleGrid.io

Solr & Cassandra: Searching Cassandra with DataStax EnterpriseDataStax Academy

Stratio's Cassandra Lucene index: Geospatial Use Cases (Andrés de la Peña & J...DataStax

Storm and Cassandra T Jake Luciani

Continuous Deployment: Beyond Continuous DeliveryTimothy Fitz

A Cassandra + Solr + Spark Love Triangle Using DataStax EnterprisePatrick McFadin

Realtime Analytics and Anomalities Detection using Elasticsearch, Hadoop and ...DataWorks Summit

Real Time Business Intelligence with Cassandra, Kafka and Hadoop - A Real Sto...DataStax

Using Event-Driven Architectures with CassandraDataStax Academy

SASI, Cassandra on the full text search ride - DuyHai Doan - Codemotion Amste...Codemotion

Enabling Search in your Cassandra Application with DataStax EnterpriseDataStax Academy

Advanced Data Modeling with Apache CassandraDataStax Academy

Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...DataStax

Real time analytics using Hadoop and ElasticsearchAbhishek Andhavarapu

Andere mochten auch (20)

SMARTSTUDY Django 오픈 세션 2012-08

BI, Reporting and Analytics on Apache Cassandra

TensorFrames: Google Tensorflow on Apache Spark

An Introduction to Distributed Search with Cassandra and Solr

SASI: Cassandra on the Full Text Search Ride (DuyHai DOAN, DataStax) | C* Sum...

Spark to Production @Windward

Cassandra vs. MongoDB

Solr & Cassandra: Searching Cassandra with DataStax Enterprise

Stratio's Cassandra Lucene index: Geospatial Use Cases (Andrés de la Peña & J...

Storm and Cassandra

Continuous Deployment: Beyond Continuous Delivery

A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise

Realtime Analytics and Anomalities Detection using Elasticsearch, Hadoop and ...

Real Time Business Intelligence with Cassandra, Kafka and Hadoop - A Real Sto...

Using Event-Driven Architectures with Cassandra

SASI, Cassandra on the full text search ride - DuyHai Doan - Codemotion Amste...

Enabling Search in your Cassandra Application with DataStax Enterprise

Advanced Data Modeling with Apache Cassandra

Building a Pluggable Analytics Stack with Cassandra (Jim Peregord, Element Co...

Real time analytics using Hadoop and Elasticsearch

Ähnlich wie Cassandra + Spark + Elk

Migrating ETL Workflow to Apache Spark at Scale in PinterestDatabricks

Apache Spark and DataStax EnablementVincent Poncet

5 Ways to Use Spark to Enrich your Cassandra EnvironmentJim Hatcher

Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Chris Fregly

A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks

Jump Start on Apache® Spark™ 2.x with Databricks Databricks

Jumpstart on Apache Spark 2.2 on DatabricksDatabricks

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3Databricks

Apache Spark on HDinsight TrainingSynergetics Learning and Cloud Consulting

Intro to Apache SparkRobert Sanders

Intro to Apache Sparkclairvoyantllc

Cassandra and Spark SQLRussell Spitzer

Spark IntroductionDataStax Academy

Spark Summit EU talk by Miklos Christine paddling up the streamSpark Summit

Paris Data Geek - Spark Streaming Djamel Zouaoui

Apache cassandra and spark. you got the the lighter, let's start the firePatrick McFadin

Homologous Apache Spark Clusters Using Nomad with Alex DadgarDatabricks

Introduction to Spark - DataFactZDataFactZ

11. From Hadoop to Spark 2/2Fabio Fumarola

Spark Summit East 2015 Advanced Devops Student SlidesDatabricks

Ähnlich wie Cassandra + Spark + Elk (20)

Migrating ETL Workflow to Apache Spark at Scale in Pinterest

Apache Spark and DataStax Enablement

5 Ways to Use Spark to Enrich your Cassandra Environment

Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...

A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...

Jump Start on Apache® Spark™ 2.x with Databricks

Jumpstart on Apache Spark 2.2 on Databricks

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

Apache Spark on HDinsight Training

Intro to Apache Spark

Cassandra and Spark SQL

Spark Introduction

Spark Summit EU talk by Miklos Christine paddling up the stream

Paris Data Geek - Spark Streaming

Apache cassandra and spark. you got the the lighter, let's start the fire

Homologous Apache Spark Clusters Using Nomad with Alex Dadgar

Introduction to Spark - DataFactZ

11. From Hadoop to Spark 2/2

Spark Summit East 2015 Advanced Devops Student Slides

Mehr von Vasil Remeniuk

Product Minsk - РТБ и ПрограмматикVasil Remeniuk

Работа с Akka Сluster, @afiskon, scalaby#14Vasil Remeniuk

Cake pattern. Presentation by Alex Famin at scalaby#14Vasil Remeniuk

Scala laboratory: Globus. iteration #3Vasil Remeniuk

Testing in Scala by Adform researchVasil Remeniuk

Spark Intro by Adform ResearchVasil Remeniuk

Types by Adform Research, Saulius ValatkaVasil Remeniuk

Types by Adform ResearchVasil Remeniuk

Scalding by Adform Research, Alex GryzlovVasil Remeniuk

Spark by Adform Research, PauliusVasil Remeniuk

Scala Style by Adform Research (Saulius Valatka)Vasil Remeniuk

Spark intro by Adform ResearchVasil Remeniuk

SBT by Aform Research, Saulius ValatkaVasil Remeniuk

Scala laboratory: Globus. iteration #2Vasil Remeniuk

Testing in Scala. Adform ResearchVasil Remeniuk

Scala laboratory. Globus. iteration #1Vasil Remeniuk

Опыт использования Spark, Основано на реальных событияхVasil Remeniuk

ETL со SparkVasil Remeniuk

Funtional Reactive Programming with Examples in Scala + GWTVasil Remeniuk

Mehr von Vasil Remeniuk (20)

Product Minsk - РТБ и Программатик

Работа с Akka Сluster, @afiskon, scalaby#14

Cake pattern. Presentation by Alex Famin at scalaby#14

Scala laboratory: Globus. iteration #3

Testing in Scala by Adform research

Spark Intro by Adform Research

Types by Adform Research, Saulius Valatka

Types by Adform Research

Scalding by Adform Research, Alex Gryzlov

Spark by Adform Research, Paulius

Scala Style by Adform Research (Saulius Valatka)

Spark intro by Adform Research

SBT by Aform Research, Saulius Valatka

Scala laboratory: Globus. iteration #2

Testing in Scala. Adform Research

Scala laboratory. Globus. iteration #1

Опыт использования Spark, Основано на реальных событиях

ETL со Spark

Funtional Reactive Programming with Examples in Scala + GWT

Kürzlich hochgeladen

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

Pigging Solutions in Pet Food ManufacturingPigging Solutions

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

APIForce Zurich 5 April Automation LPDGMarianaLemus7

Pigging Solutions Piggable Sweeping ElbowsPigging Solutions

Unlocking the Potential of the Cloud for IBM Power SystemsPrecisely

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community

Understanding the Laravel MVC ArchitecturePixlogix Infotech

Artificial intelligence in the post-deep learning eraDeakin University

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

Kürzlich hochgeladen (20)

Presentation on how to chat with PDF using ChatGPT code interpreter

Connect Wave/ connectwave Pitch Deck Presentation

Human Factors of XR: Using Human Factors to Design XR Systems

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Pigging Solutions in Pet Food Manufacturing

Benefits Of Flutter Compared To Other Frameworks

My Hashitalk Indonesia April 2024 Presentation

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...

APIForce Zurich 5 April Automation LPDG

Pigging Solutions Piggable Sweeping Elbows

Unlocking the Potential of the Cloud for IBM Power Systems

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx

Understanding the Laravel MVC Architecture

Artificial intelligence in the post-deep learning era

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365

Unblocking The Main Thread Solving ANRs and Frozen Frames

Advanced Test Driven-Development @ php[tek] 2024

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation

Cassandra + Spark + Elk

1. Cassandra+Spark+ELK Dmitriy Kalyada @ 2015

2. What is Spark? • Master: Driver program • Workers: Executors • High Availability • Standby Masters with ZooKeeper • Single-Node Recovery with Local File System

3. Under the hood • Resilient Distributed Dataset (RDD) • Scala + Akka Framework • Java, Scala, Python API • Spark SQL, MLib, Spark Streaming, GraphX

4. Our particular case Devices Cassandra Spark ELK

5. Data ﬂow Fetcher Transformer Saver Input Source(s) x-RDD x-RDD Output Source

6. Spark Cassandra Connector • Represents Cassandra tables as Spark RDDs • Write Spark RDDs to Cassandra tables • Execute CQL queries in Spark applications https://github.com/datastax/spark-cassandra-connector

7. CassandraRDD settings • Connection params • Fetching params 1. input.split.size: C* partitions in a Spark Partition. 2. input.page.row.size: number of CQL rows fetched per roundtrip.

8. Fetching essentials … … … … -968391295277638458 … -893783532241185833 -968391295277638458, -893783532241185833 -7378580094811526501, -7340240117176401239 6426215139012569257, 6428979455828914106 -6094480671546553265, -6016282219056649738 -7259249675596554667, -7237838231745167324 -6734336817058726139, -6684208157211348972 -3891103372671105499, -3822513456325086923 4453206019575747361,4462441725813855391 7855385326468991461,7906589648045207141 -129433796439502583,-101280166181350027 -2233788032218452383,-2066644620711092198 3248662132571799756,3396129453515776704 7744134136205124749,7812918342246679728 -1408208314239486033,-1403736406052004344 • Support Murmur3Partitioner and RandomPartitioner • Retrieve token ranges from Cassandra • Prediction on base of 16 random token ranges

9. Data to RDD … Tokens Per RDD [input.split.size] Token Range #N Slurp amount [input.page.row.size]

10. Token range vs rows number

11. What to do? • Change read strategy • Split data on a smaller pieces • Increase cluster strength • Reorganize Cassandra schema

12. Elastic Search

13. Elastic Search & Kibana • Index initialization: TransportClient • Create/Delete Index • Setup Mappings • Indexing: ScalaEsRDD • Data presentation: Kibana

14. Kibana

15. Deployment • Build package: Spark Job + Dependent Jars + Conﬁgs • Upload to the Spark Master Node • Start job submit script

16. Thank you dkaliada@exadel.com Dmitriy Kalyada @ 2015