Interactive Apache Spark in Your Browser

•

12 gefällt mir•3,388 views

Cloudera, Inc.

Presented by Romain Rigaux at Spark Lightning Night 2015

Software

INTERACTIVE
SPARK IN YOUR
BROWSER
Spark
Lightning
Night
-‐
Hadoop
World
September
2015

GOAL 
OF HUE
WEB INTERFACE FOR ANALYZING DATA
WITH APACHE HADOOP

SIMPLIFY AND INTEGRATE 
 
FREE AND OPEN SOURCE
—> “BIG NOTEBOOK” FOR HADOOP

SPARK 
ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Backend
partWeb
part

SPARK 
ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Spark
ServerUI

• Married
with
full
ecosystem

• File,
Job
browsers

• Create
table
wizards

• Any
language
(Hive,
Spark...)

• Graphing

• Export/Import/Sharing

• MulT
users

• ImpersonaTon

HADOOP WITH SPARK 
NOTEBOOK

• REST
Web
server
in
Scala

• InteracTve
Spark
Sessions
and
Batch
Jobs

• Type
IntrospecTon
for
VisualizaTon

• Running
sessions
in
YARN

local

• Backends:
Scala,
Python,
R

• Open
Source:
h[ps://github.com/cloudera/hue/
tree/master/apps/spark/java

• Play
with
Curl
h[p://gethue.com/how-‐to-‐use-‐the-‐
LIVY 
SPARK SERVER

LIVY WEB SERVER 
ARCHITECTURE YARN

Master
Spark

Client
YARN 
Node
Spark 
Interpreter
Spark 
Context
YARN 
Node
Spark 
Worker
YARN 
Node
Spark 
Worker
Livy
Server
Scalatra
Session
Manager
Session

LIVY WEB SERVER 
ARCHITECTURE
Livy
Server
YARN

Master
Scalatra
Spark

Client
Session
Manager
Session
YARN 
Node
Spark 
Interpreter
Spark 
Context
YARN 
Node
Spark 
Worker
YARN 
Node
Spark 
Worker
1

$SESSION CREATION  AND EXECUTION % curl -‐XPOST localhost:8998/sessions -‐d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -‐XPOST localhost:8998/sessions/0/statements -‐d '{"code": " { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }$

• Stability
and
Scaling

• Security

• iPython/Jupyter
interpreter

and
ﬁle
format

COMING SOON

TWITTER
@gethue
USER GROUP
hue-‐user@
WEBSITE
h[p://gethue.com
LEARN
h[p://learn.gethue.com
THANKS!

Empfohlen

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...gethue

Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Evan Chan

Spark Summit 2014: Spark Job Server TalkEvan Chan

How to build your query engine in sparkPeng Cheng

APACHE TOREE: A JUPYTER KERNEL FOR SPARK by Marius van NiekerkSpark Summit

spark-kafka_modVritika Godara

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...Spark Summit

Mobius: C# Language Binding For SparkSpark Summit

Empfohlen

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...gethue

Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Evan Chan

Spark Summit 2014: Spark Job Server TalkEvan Chan

How to build your query engine in sparkPeng Cheng

APACHE TOREE: A JUPYTER KERNEL FOR SPARK by Marius van NiekerkSpark Summit

spark-kafka_modVritika Godara

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...Spark Summit

Mobius: C# Language Binding For SparkSpark Summit

Spark Summit EU talk by William BentonSpark Summit

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)Helena Edelson

Debugging Apache SparkJoey Echeverria

Reactive app using actor model & apache sparkRahul Kumar

Building a High-Performance Database with Scala, Akka, and SparkEvan Chan

SF Big Analytics_20190612: Scaling Apache Spark on Kubernetes at LyftChester Chen

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Chris Fregly

Spark Working Environment in Windows OSUniversiti Technologi Malaysia (UTM)

Apache Spark Introduction - CloudxLabAbhinav Singh

Kafka & Hadoop - for NYC Kafka MeetupGwen (Chen) Shapira

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - Jam...JAXLondon2014

Spark Summit EU talk by Steve LoughranSpark Summit

Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...Spark Summit

Stream Processing using Apache Spark and Apache KafkaAbhinav Singh

Spark Streaming & Kafka-The Future of Stream ProcessingJack Gudenkauf

Lambda Architecture in PracticeNavneet kumar

Building production spark streaming applicationsJoey Echeverria

Scaling spark on kubernetes at LyftLi Gao

Emerging technologies /frameworks in Big DataRahul Jain

Livy: A REST Web Service for SparkAshish kumar

What is SparkBruno Faria

Weitere ähnliche Inhalte

Was ist angesagt?

Spark Summit EU talk by William BentonSpark Summit

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)Helena Edelson

Debugging Apache SparkJoey Echeverria

Reactive app using actor model & apache sparkRahul Kumar

Building a High-Performance Database with Scala, Akka, and SparkEvan Chan

SF Big Analytics_20190612: Scaling Apache Spark on Kubernetes at LyftChester Chen

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Chris Fregly

Spark Working Environment in Windows OSUniversiti Technologi Malaysia (UTM)

Apache Spark Introduction - CloudxLabAbhinav Singh

Kafka & Hadoop - for NYC Kafka MeetupGwen (Chen) Shapira

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - Jam...JAXLondon2014

Spark Summit EU talk by Steve LoughranSpark Summit

Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...Spark Summit

Stream Processing using Apache Spark and Apache KafkaAbhinav Singh

Spark Streaming & Kafka-The Future of Stream ProcessingJack Gudenkauf

Lambda Architecture in PracticeNavneet kumar

Building production spark streaming applicationsJoey Echeverria

Scaling spark on kubernetes at LyftLi Gao

Emerging technologies /frameworks in Big DataRahul Jain

Was ist angesagt? (20)

Spark Summit EU talk by William Benton

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Debugging Apache Spark

Reactive app using actor model & apache spark

Building a High-Performance Database with Scala, Akka, and Spark

SF Big Analytics_20190612: Scaling Apache Spark on Kubernetes at Lyft

HDFS on Kubernetes—Lessons Learned with Kimoon Kim

Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...

Spark Working Environment in Windows OS

Apache Spark Introduction - CloudxLab

Kafka & Hadoop - for NYC Kafka Meetup

Detecting Events on the Web in Real Time with Java, Kafka and ZooKeeper - Jam...

Spark Summit EU talk by Steve Loughran

Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...

Stream Processing using Apache Spark and Apache Kafka

Spark Streaming & Kafka-The Future of Stream Processing

Lambda Architecture in Practice

Building production spark streaming applications

Scaling spark on kubernetes at Lyft

Emerging technologies /frameworks in Big Data

Ähnlich wie Interactive Apache Spark in Your Browser

Livy: A REST Web Service for SparkAshish kumar

What is SparkBruno Faria

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...Alex Zeltov

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim DowlingDatabricks

Learn Apache Spark: A Comprehensive GuideWhizlabs

Mining public datasets using opensource tools: Zeppelin, Spark and Jujuseoul_engineer

Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015Mike Broberg

Writing Apache Spark and Apache Flink Applications Using Apache BahirLuciano Resende

A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende

Spark Streamingによるリアルタイムユーザ属性推定Yoshiyasu SAEKI

Building a REST Job Server for interactive Spark as a service by Romain Rigau...Spark Summit

Building a REST Job Server for Interactive Spark as a ServiceCloudera, Inc.

Migrating structured data between Hadoop and RDBMSBouquet

SparkKoushik Mondal

Building data pipelines for modern data warehouse with Apache® Spark™ and .NE...Michael Rys

Big Data Processing with .NET and Spark (SQLBits 2020)Michael Rys

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...Simplilearn

Intro to Big Data Analytics using Apache Spark and Apache ZeppelinAlex Zeltov

Bringing the Power and Familiarity of .NET, C# and F# to Big Data Processing ...Michael Rys

H2O World - H2O Rains with Databricks CloudSri Ambati

Ähnlich wie Interactive Apache Spark in Your Browser (20)

Livy: A REST Web Service for Spark

What is Spark

Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Learn Apache Spark: A Comprehensive Guide

Mining public datasets using opensource tools: Zeppelin, Spark and Juju

Apache Spark™ + IBM Watson + Twitter DataPalooza SF 2015

Writing Apache Spark and Apache Flink Applications Using Apache Bahir

A Jupyter kernel for Scala and Apache Spark.pdf

Spark Streamingによるリアルタイムユーザ属性推定

Building a REST Job Server for interactive Spark as a service by Romain Rigau...

Building a REST Job Server for Interactive Spark as a Service

Migrating structured data between Hadoop and RDBMS

Spark

Building data pipelines for modern data warehouse with Apache® Spark™ and .NE...

Big Data Processing with .NET and Spark (SQLBits 2020)

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Intro to Big Data Analytics using Apache Spark and Apache Zeppelin

Bringing the Power and Familiarity of .NET, C# and F# to Big Data Processing ...

H2O World - H2O Rains with Databricks Cloud

Mehr von Cloudera, Inc.

Partner Briefing_January 25 (FINAL).pptxCloudera, Inc.

Cloudera Data Impact Awards 2021 - Finalists Cloudera, Inc.

2020 Cloudera Data Impact Awards FinalistsCloudera, Inc.

Edc event vienna presentation 1 oct 2019Cloudera, Inc.

Machine Learning with Limited Labeled Data 4/3/19Cloudera, Inc.

Data Driven With the Cloudera Modern Data Warehouse 3.19.19Cloudera, Inc.

Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.

Introducing Cloudera Data Science Workbench for HDP 2.12.19Cloudera, Inc.

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19Cloudera, Inc.

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 3Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 1Cloudera, Inc.

Extending Cloudera SDX beyond the PlatformCloudera, Inc.

Federated Learning: ML with Privacy on the Edge 11.15.18Cloudera, Inc.

Analyst Webinar: Doing a 180 on Customer 360Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18Cloudera, Inc.

Introducing the data science sandbox as a service 8.30.18Cloudera, Inc.

Mehr von Cloudera, Inc. (20)

Partner Briefing_January 25 (FINAL).pptx

Cloudera Data Impact Awards 2021 - Finalists

2020 Cloudera Data Impact Awards Finalists

Edc event vienna presentation 1 oct 2019

Machine Learning with Limited Labeled Data 4/3/19

Data Driven With the Cloudera Modern Data Warehouse 3.19.19

Introducing Cloudera DataFlow (CDF) 2.13.19

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19

Leveraging the cloud for analytics and machine learning 1.29.19

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19

Leveraging the Cloud for Big Data Analytics 12.11.18

Modern Data Warehouse Fundamentals Part 3

Modern Data Warehouse Fundamentals Part 2

Modern Data Warehouse Fundamentals Part 1

Extending Cloudera SDX beyond the Platform

Federated Learning: ML with Privacy on the Edge 11.15.18

Analyst Webinar: Doing a 180 on Customer 360

Build a modern platform for anti-money laundering 9.19.18

Introducing the data science sandbox as a service 8.30.18

Kürzlich hochgeladen

Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesKrzysztofKkol1

Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptxRTS corp

VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics

Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea

Introduction to Firebase Workshop Slidesvaideheekore1

2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin

Powering Real-Time Decisions with Continuous Data StreamsSafe Software

OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan

JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver

UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz

Precise and Complete Requirements? An Elusive GoalLionel Briand

SAM Training Session - How to use EXCEL ?Alexandre Beguel

Salesforce Implementation Services PPT By ABSYZABSYZ Inc

Comparing Linux OS Image Update Models - EOSS 2024.pdfDrew Moseley

2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin

Osi security architecture in network.pptxVinzoCenzo

Large Language Models for Test Case Evolution and RepairLionel Briand

Post Quantum Cryptography – The Impact on Identityteam-WIBU

What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics

Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden

Kürzlich hochgeladen (20)

Amazon Bedrock in Action - presentation of the Bedrock's capabilities

Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx

VictoriaMetrics Q1 Meet Up '24 - Community & News Update

Keeping your build tool updated in a multi repository world

Introduction to Firebase Workshop Slides

2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf

Powering Real-Time Decisions with Continuous Data Streams

OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording

JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...

UI5ers live - Custom Controls wrapping 3rd-party libs.pptx

Precise and Complete Requirements? An Elusive Goal

SAM Training Session - How to use EXCEL ?

Salesforce Implementation Services PPT By ABSYZ

Comparing Linux OS Image Update Models - EOSS 2024.pdf

2024 DevNexus Patterns for Resiliency: Shuffle shards

Osi security architecture in network.pptx

Large Language Models for Test Case Evolution and Repair

Post Quantum Cryptography – The Impact on Identity

What’s New in VictoriaMetrics: Q1 2024 Updates

Simplifying Microservices & Apps - The art of effortless development - Meetup...

Interactive Apache Spark in Your Browser

1. INTERACTIVE SPARK IN YOUR BROWSER Spark Lightning Night -‐ Hadoop World September 2015

4. GOAL  OF HUE WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP SIMPLIFY AND INTEGRATE    FREE AND OPEN SOURCE —> “BIG NOTEBOOK” FOR HADOOP

5. SPARK  ARCHITECTURE Livy Spark Spark Spark YARN Backend partWeb part

6. SPARK  ARCHITECTURE Livy Spark Spark Spark YARN Spark ServerUI

7. HADOOP NOTEBOOK UI

8. • Married with full ecosystem • File, Job browsers • Create table wizards • Any language (Hive, Spark...) • Graphing • Export/Import/Sharing • MulT users • ImpersonaTon HADOOP WITH SPARK  NOTEBOOK

9. LIVY SPARK SERVER Backend

10. • REST Web server in Scala • InteracTve Spark Sessions and Batch Jobs • Type IntrospecTon for VisualizaTon • Running sessions in YARN local • Backends: Scala, Python, R • Open Source: h[ps://github.com/cloudera/hue/ tree/master/apps/spark/java • Play with Curl h[p://gethue.com/how-‐to-‐use-‐the-‐ LIVY  SPARK SERVER

11. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker Livy Server Scalatra Session Manager Session

12. LIVY WEB SERVER  ARCHITECTURE Livy Server YARN Master Scalatra Spark Client Session Manager Session YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1

13. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 2 Livy Server Scalatra Session Manager Session

14. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 2 3 Livy Server Scalatra Session Manager Session

15. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 2 3 4 Livy Server Scalatra Session Manager Session

16. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 2 3 4 5 Livy Server Scalatra Session Manager Session

17. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 2 3 4 5 6 Livy Server Scalatra Session Manager Session

18. LIVY WEB SERVER  ARCHITECTURE YARN Master Spark Client YARN  Node Spark  Interpreter Spark  Context YARN  Node Spark  Worker YARN  Node Spark  Worker 1 7 2 3 4 5 6 Livy Server Scalatra Session Manager Session

19. SESSION CREATION  AND EXECUTION % curl -‐XPOST localhost:8998/sessions -‐d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -‐XPOST localhost:8998/sessions/0/statements -‐d '{"code": " { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }

20. DEMO TIME 

21. • Stability and Scaling • Security • iPython/Jupyter interpreter and ﬁle format COMING SOON

22. TWITTER @gethue USER GROUP hue-‐user@ WEBSITE h[p://gethue.com LEARN h[p://learn.gethue.com THANKS! 