Automated Hadoop Cluster Construction on EC2

•

1 gefällt mir•930 views

Mark Kerzner

Presented at Houston Hadoop Meetup

Technologie

Automated Hadoop
Clusters on EC2
Mark Kerzner
SHMsoft

What is Hadoop? :) :) :)
Everybody knows that

... What is your definition?

What is a cloud?
Everybody knows that, but

1. Elastic resources
2. Internet delivery
3. SAAS
4. Virtualization
5. Device-enabled
6. Only (1) or all of the above

You are the Hadoop programmer
... and you need tools

What are your alternatives?
● IDE
● Local "cluster"
● Pseudo-distributed cluster
● EC2

You are the Hadoop programmer
... and you need tools

What are your alternatives?
● IDE - compile and run the code
● Local "cluster" - local file system
● Pseudo-distributed cluster - test outside
● EC2 - test on the cluster, test for scale

What are your resources
● Tom White, "Hadoop, the Definitive Guide"
● www.hadoopilluminated.com

Hadoop+ (oh, by the way...)
HBase, Cassandra, MongoDB, NoSQL,
Dynamo, BigTable, Dryad (MS), Azure (MS),
MapReduce, MapR (EMC), Cloudera
distribution, EMC distribution, IBM distribution...

Whirr
Setup

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

Install
curl -O http://www.apache.org/dist/whirr/whirr-0.7.1/whirr-0.7.1.tar.gz
tar zxf whirr-0.7.1.tar.gz; cd whirr-0.7.1

Generate key

sssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr

Run
bin/whirr launch-cluster --config recipes/zookeeper-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr

Whirr limitations
● No EBS
● All or nothing
● Generates configuration artifacts
● Takes over your computer, no more local
development - uses proxy
● Hard to customize

EMR limitations
● No choice of image
● Fixed architecture
● Hard to debug
● Hard to customize

You do it
Repeat the manual procedure, only automate it

Prepare
AMI, Java, Hadoop

On-the-fly
Start AMI, login, configure, start services,
verify, run test jobs

You do it - advanced

On startup

Under-provision, over-provision, progress

On-the-fly

Monitor, run test jobs, watch for cluster
deterioration

On the large scale
Hadoop 0.20 - up to 4,000 nodes
Hadoop 0.23 - up to 20,000
GridGain - 100's of 1,000's

Empfohlen

Everyone Loves a SausageNick Jones

Bare metal Hadoop provisioningGoDataDriven

Cloud foundry on kubernetes상준 윤

Build an affordable Cloud StroageAlex Lau

New microsoft power point presentationrajsandhu1989

Alluxio in MOMOAlluxio, Inc.

Functional Hostnames and Why they are BadPuppet

The Practice of Alluxio in Near Real-Time Data Platform at VIPShop [Chinese]Alluxio, Inc.

Empfohlen

Everyone Loves a SausageNick Jones

Bare metal Hadoop provisioningGoDataDriven

Cloud foundry on kubernetes상준 윤

Build an affordable Cloud StroageAlex Lau

New microsoft power point presentationrajsandhu1989

Alluxio in MOMOAlluxio, Inc.

Functional Hostnames and Why they are BadPuppet

The Practice of Alluxio in Near Real-Time Data Platform at VIPShop [Chinese]Alluxio, Inc.

Guava Overview Part 2 Bucharest JUG #2 Andrei Savu

MSST-2013 Openstack in the Land of GuilderJoshua McKenty

Modern Cassandra for DevelopersJeremy Hanna

Heroku Dockerの使い所Yusuke Kon

Guavafbenault

Redis for .NET DevelopersYuriy Guts

A site in 15 minutes with yiiAndy Kelk

A Developer Overview of RedisYuriy Guts

Terraform 9Jerry Singh

Etcd terraform by Alex SomesanMaarten van der Hoef

Haskell Tooling WhirlwindSteven Shaw

openSUSE storage workshop 2016Alex Lau

Hostvn ceph in production v1.1 dungtqViet Stack

Terraform, Ansible or pure CloudFormationgeekQ

Environment for training modelsFlyElephant

Terraform, Ansible, or pure CloudFormation?geekQ

Dev ops meetupBigdata Meetup Kochi

Hadoop enhancements using next gen IA technologiesBigdata Meetup Kochi

Sharding with spider solutions 20160721Kentoku

NetBSD on Google Compute Engine (en)Ryo ONODERA

Postgres the hardwayDave Pitts

Cobbler, Func and Puppet: Tools for Large Scale EnvironmentsMichael Zhang

Weitere ähnliche Inhalte

Was ist angesagt?

Guava Overview Part 2 Bucharest JUG #2 Andrei Savu

MSST-2013 Openstack in the Land of GuilderJoshua McKenty

Modern Cassandra for DevelopersJeremy Hanna

Heroku Dockerの使い所Yusuke Kon

Guavafbenault

Redis for .NET DevelopersYuriy Guts

A site in 15 minutes with yiiAndy Kelk

A Developer Overview of RedisYuriy Guts

Terraform 9Jerry Singh

Etcd terraform by Alex SomesanMaarten van der Hoef

Haskell Tooling WhirlwindSteven Shaw

openSUSE storage workshop 2016Alex Lau

Hostvn ceph in production v1.1 dungtqViet Stack

Terraform, Ansible or pure CloudFormationgeekQ

Environment for training modelsFlyElephant

Terraform, Ansible, or pure CloudFormation?geekQ

Dev ops meetupBigdata Meetup Kochi

Hadoop enhancements using next gen IA technologiesBigdata Meetup Kochi

Sharding with spider solutions 20160721Kentoku

NetBSD on Google Compute Engine (en)Ryo ONODERA

Was ist angesagt? (20)

Guava Overview Part 2 Bucharest JUG #2

MSST-2013 Openstack in the Land of Guilder

Modern Cassandra for Developers

Heroku Dockerの使い所

Guava

Redis for .NET Developers

A site in 15 minutes with yii

A Developer Overview of Redis

Terraform 9

Etcd terraform by Alex Somesan

Haskell Tooling Whirlwind

openSUSE storage workshop 2016

Hostvn ceph in production v1.1 dungtq

Terraform, Ansible or pure CloudFormation

Environment for training models

Terraform, Ansible, or pure CloudFormation?

Dev ops meetup

Hadoop enhancements using next gen IA technologies

Sharding with spider solutions 20160721

NetBSD on Google Compute Engine (en)

Ähnlich wie Automated Hadoop Cluster Construction on EC2

Postgres the hardwayDave Pitts

Cobbler, Func and Puppet: Tools for Large Scale EnvironmentsMichael Zhang

Cobbler, Func and Puppet: Tools for Large Scale EnvironmentsViSenze - Artificial Intelligence for the Visual Web

Using GPUs to handle Big Data with Java by Adam Roberts.J On The Beach

Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....Jeffrey Breen

Lightweight Virtualization with Linux Containers and Docker | YaC 2013dotCloud

Lightweight Virtualization with Linux Containers and Docker I YaC 2013Docker, Inc.

Puppet and Apache CloudStackPuppet

Infrastructure as code with Puppet and Apache CloudStackke4qqq

Introduction to Docker and ContainersDocker, Inc.

Introduction to Docker at SF Peninsula Software Development Meetup @GuidewiredotCloud

Puppet and CloudStackke4qqq

Take care of hundred containers and not go crazyHonza Horák

Facing enterprise specific challenges – utility programming in hadoopfann wu

Let's Containerize New York with Docker!Jérôme Petazzoni

Running Applications on the NetBSD Rump Kernel by Justin Cormack eurobsdcon

syzkaller: the next gen kernel fuzzerDmitry Vyukov

Big Data in Container; Hadoop Spark in Docker and MesosHeiko Loewe

Lessons from Driverless AI going to ProductionSri Ambati

"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...Yandex

Ähnlich wie Automated Hadoop Cluster Construction on EC2 (20)

Postgres the hardway

Cobbler, Func and Puppet: Tools for Large Scale Environments

Using GPUs to handle Big Data with Java by Adam Roberts.

Big Data Step-by-Step: Infrastructure 3/3: Taking it to the cloud... easily.....

Lightweight Virtualization with Linux Containers and Docker | YaC 2013

Lightweight Virtualization with Linux Containers and Docker I YaC 2013

Puppet and Apache CloudStack

Infrastructure as code with Puppet and Apache CloudStack

Introduction to Docker and Containers

Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire

Puppet and CloudStack

Take care of hundred containers and not go crazy

Facing enterprise specific challenges – utility programming in hadoop

Let's Containerize New York with Docker!

Running Applications on the NetBSD Rump Kernel by Justin Cormack

syzkaller: the next gen kernel fuzzer

Big Data in Container; Hadoop Spark in Docker and Mesos

Lessons from Driverless AI going to Production

"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...

Mehr von Mark Kerzner

IBM Strategy for SparkMark Kerzner

Toorcamp 2016Mark Kerzner

Witsml data processing with kafka and spark streamingMark Kerzner

Hadoop as a service presented by Ajay Jha at Houston Hadoop MeetupMark Kerzner

Hadoop Hadoop & Spark meetup - AltiscaleMark Kerzner

Oil and gas big data editionMark Kerzner

Cloudera searchMark Kerzner

Joe Witt presentation on Apache NiFiMark Kerzner

FreeEed popcorn overviewMark Kerzner

FreeEed presentationMark Kerzner

Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)Mark Kerzner

Night owl by Boyd Meyer of PROS Mark Kerzner

SHMcloud visionMark Kerzner

Porting your hadoop app to horton works hdpMark Kerzner

Hadoop on ec2Mark Kerzner

Open source e_discoveryMark Kerzner

FreEed - Open Source eDiscoveryMark Kerzner

Houston Hadoop Meetup Presentation by Vikram Oberoi of ClouderaMark Kerzner

Google Office in Zurich, SwitzerlandMark Kerzner

Fun art with fruit and vegetableMark Kerzner

Mehr von Mark Kerzner (20)

IBM Strategy for Spark

Toorcamp 2016

Witsml data processing with kafka and spark streaming

Hadoop as a service presented by Ajay Jha at Houston Hadoop Meetup

Hadoop Hadoop & Spark meetup - Altiscale

Oil and gas big data edition

Cloudera search

Joe Witt presentation on Apache NiFi

FreeEed popcorn overview

FreeEed presentation

Nutch + Hadoop scaled, for crawling protected web sites (hint: Selenium)

Night owl by Boyd Meyer of PROS

SHMcloud vision

Porting your hadoop app to horton works hdp

Hadoop on ec2

Open source e_discovery

FreEed - Open Source eDiscovery

Houston Hadoop Meetup Presentation by Vikram Oberoi of Cloudera

Google Office in Zurich, Switzerland

Fun art with fruit and vegetable

Kürzlich hochgeladen

CNIC Information System with Pakdata Cf In Pakistandanishmna97

Apidays New York 2024 - The value of a flexible API Management solution for O...apidays

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood

Exploring Multimodal Embeddings with MilvusZilliz

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

Corporate and higher education May webinar.pptxRustici Software

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc

Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya

[BuildWithAI] Introduction to Gemini.pdfSandro Moreira

DBX First Quarter 2024 Investor PresentationDropbox

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz

Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Kürzlich hochgeladen (20)

CNIC Information System with Pakdata Cf In Pakistan

Apidays New York 2024 - The value of a flexible API Management solution for O...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Exploring Multimodal Embeddings with Milvus

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Corporate and higher education May webinar.pptx

Boost Fertility New Invention Ups Success Rates.pdf

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Axa Assurance Maroc - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Artificial Intelligence Chap.5 : Uncertainty

[BuildWithAI] Introduction to Gemini.pdf

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

presentation ICT roal in 21st century education

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Automated Hadoop Cluster Construction on EC2

1. Automated Hadoop Clusters on EC2 Mark Kerzner SHMsoft

2. What is Hadoop? :) :) :) Everybody knows that ... What is your definition?

3. What is a cloud? Everybody knows that, but 1. Elastic resources 2. Internet delivery 3. SAAS 4. Virtualization 5. Device-enabled 6. Only (1) or all of the above

4. You are the Hadoop programmer ... and you need tools What are your alternatives? ● IDE ● Local "cluster" ● Pseudo-distributed cluster ● EC2

5. You are the Hadoop programmer ... and you need tools What are your alternatives? ● IDE - compile and run the code ● Local "cluster" - local file system ● Pseudo-distributed cluster - test outside ● EC2 - test on the cluster, test for scale

6. What are your resources ● Tom White, "Hadoop, the Definitive Guide" ● www.hadoopilluminated.com

7. For real play, you need a cluster

8. Hadoop+ (oh, by the way...) HBase, Cassandra, MongoDB, NoSQL, Dynamo, BigTable, Dryad (MS), Azure (MS), MapReduce, MapR (EMC), Cloudera distribution, EMC distribution, IBM distribution...

9. Whirr Setup export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... Install curl -O http://www.apache.org/dist/whirr/whirr-0.7.1/whirr-0.7.1.tar.gz tar zxf whirr-0.7.1.tar.gz; cd whirr-0.7.1 Generate key sssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr Run bin/whirr launch-cluster --config recipes/zookeeper-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr

10. Whirr limitations ● No EBS ● All or nothing ● Generates configuration artifacts ● Takes over your computer, no more local development - uses proxy ● Hard to customize

11. Amazon EMR

12. EMR limitations ● No choice of image ● Fixed architecture ● Hard to debug ● Hard to customize

13. You do it Repeat the manual procedure, only automate it Prepare AMI, Java, Hadoop On-the-fly Start AMI, login, configure, start services, verify, run test jobs

14. You do it - advanced On startup Under-provision, over-provision, progress On-the-fly Monitor, run test jobs, watch for cluster deterioration

15. Cloudera Manager

16. MapR Manager

17. On the large scale Hadoop 0.20 - up to 4,000 nodes Hadoop 0.23 - up to 20,000 GridGain - 100's of 1,000's

18. Thank you Questions?