Suche senden
Hochladen
Hadoop operations
•
3 gefällt mir
•
2,008 views
Marc Cluet
Folgen
Lynx Consultants training about Hadoop Operations
Weniger lesen
Mehr lesen
Technologie
News & Politik
Melden
Teilen
Melden
Teilen
1 von 32
Jetzt herunterladen
Downloaden Sie, um offline zu lesen
Empfohlen
Hadoop Operations: How to Secure and Control Cluster Access
Hadoop Operations: How to Secure and Control Cluster Access
Cloudera, Inc.
April 2014 HUG : Apache Sentry
April 2014 HUG : Apache Sentry
Yahoo Developer Network
Apache Sentry for Hadoop security
Apache Sentry for Hadoop security
bigdatagurus_meetup
Securing the Hadoop Ecosystem
Securing the Hadoop Ecosystem
DataWorks Summit
Hadoop Security Features That make your risk officer happy
Hadoop Security Features That make your risk officer happy
DataWorks Summit
Hadoop Security Features that make your risk officer happy
Hadoop Security Features that make your risk officer happy
Anurag Shrivastava
Hadoop REST API Security with Apache Knox Gateway
Hadoop REST API Security with Apache Knox Gateway
DataWorks Summit
Hadoop Security Architecture
Hadoop Security Architecture
Owen O'Malley
Empfohlen
Hadoop Operations: How to Secure and Control Cluster Access
Hadoop Operations: How to Secure and Control Cluster Access
Cloudera, Inc.
April 2014 HUG : Apache Sentry
April 2014 HUG : Apache Sentry
Yahoo Developer Network
Apache Sentry for Hadoop security
Apache Sentry for Hadoop security
bigdatagurus_meetup
Securing the Hadoop Ecosystem
Securing the Hadoop Ecosystem
DataWorks Summit
Hadoop Security Features That make your risk officer happy
Hadoop Security Features That make your risk officer happy
DataWorks Summit
Hadoop Security Features that make your risk officer happy
Hadoop Security Features that make your risk officer happy
Anurag Shrivastava
Hadoop REST API Security with Apache Knox Gateway
Hadoop REST API Security with Apache Knox Gateway
DataWorks Summit
Hadoop Security Architecture
Hadoop Security Architecture
Owen O'Malley
Hadoop Security: Overview
Hadoop Security: Overview
Cloudera, Inc.
Deploying Enterprise-grade Security for Hadoop
Deploying Enterprise-grade Security for Hadoop
Cloudera, Inc.
Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop security @ Philly Hadoop Meetup May 2015
Shravan (Sean) Pabba
Hadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, Future
Uwe Printz
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Big Data Spain
Overview of HDFS Transparent Encryption
Overview of HDFS Transparent Encryption
Cloudera, Inc.
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Abhiraj Butala
Hadoop Security and Compliance - StampedeCon 2016
Hadoop Security and Compliance - StampedeCon 2016
StampedeCon
Hadoop security overview_hit2012_1117rev
Hadoop security overview_hit2012_1117rev
Jason Shih
Hadoop Security Today and Tomorrow
Hadoop Security Today and Tomorrow
DataWorks Summit
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Cloudera, Inc.
Hadoop security
Hadoop security
Shivaji Dutta
Data protection for hadoop environments
Data protection for hadoop environments
DataWorks Summit
The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014
Cloudera, Inc.
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache Knox
Vinay Shukla
Sentry - An Introduction
Sentry - An Introduction
Alexander Alten
Managing enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystem
DataWorks Summit
Hadoop Security Now and Future
Hadoop Security Now and Future
tcloudcomputing-tw
Structor - Automated Building of Virtual Hadoop Clusters
Structor - Automated Building of Virtual Hadoop Clusters
Owen O'Malley
Introducing Node.js in an Oracle technology environment (including hands-on)
Introducing Node.js in an Oracle technology environment (including hands-on)
Lucas Jellema
HDFS presented by VIJAY
HDFS presented by VIJAY
thevijayps
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
VMware Tanzu
Weitere ähnliche Inhalte
Was ist angesagt?
Hadoop Security: Overview
Hadoop Security: Overview
Cloudera, Inc.
Deploying Enterprise-grade Security for Hadoop
Deploying Enterprise-grade Security for Hadoop
Cloudera, Inc.
Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop security @ Philly Hadoop Meetup May 2015
Shravan (Sean) Pabba
Hadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, Future
Uwe Printz
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Big Data Spain
Overview of HDFS Transparent Encryption
Overview of HDFS Transparent Encryption
Cloudera, Inc.
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Abhiraj Butala
Hadoop Security and Compliance - StampedeCon 2016
Hadoop Security and Compliance - StampedeCon 2016
StampedeCon
Hadoop security overview_hit2012_1117rev
Hadoop security overview_hit2012_1117rev
Jason Shih
Hadoop Security Today and Tomorrow
Hadoop Security Today and Tomorrow
DataWorks Summit
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Cloudera, Inc.
Hadoop security
Hadoop security
Shivaji Dutta
Data protection for hadoop environments
Data protection for hadoop environments
DataWorks Summit
The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014
Cloudera, Inc.
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache Knox
Vinay Shukla
Sentry - An Introduction
Sentry - An Introduction
Alexander Alten
Managing enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystem
DataWorks Summit
Hadoop Security Now and Future
Hadoop Security Now and Future
tcloudcomputing-tw
Structor - Automated Building of Virtual Hadoop Clusters
Structor - Automated Building of Virtual Hadoop Clusters
Owen O'Malley
Introducing Node.js in an Oracle technology environment (including hands-on)
Introducing Node.js in an Oracle technology environment (including hands-on)
Lucas Jellema
Was ist angesagt?
(20)
Hadoop Security: Overview
Hadoop Security: Overview
Deploying Enterprise-grade Security for Hadoop
Deploying Enterprise-grade Security for Hadoop
Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop security @ Philly Hadoop Meetup May 2015
Hadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, Future
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...
Overview of HDFS Transparent Encryption
Overview of HDFS Transparent Encryption
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...
Hadoop Security and Compliance - StampedeCon 2016
Hadoop Security and Compliance - StampedeCon 2016
Hadoop security overview_hit2012_1117rev
Hadoop security overview_hit2012_1117rev
Hadoop Security Today and Tomorrow
Hadoop Security Today and Tomorrow
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Hadoop Security, Cloudera - Todd Lipcon and Aaron Myers - Hadoop World 2010
Hadoop security
Hadoop security
Data protection for hadoop environments
Data protection for hadoop environments
The Future of Hadoop Security - Hadoop Summit 2014
The Future of Hadoop Security - Hadoop Summit 2014
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache Knox
Sentry - An Introduction
Sentry - An Introduction
Managing enterprise users in Hadoop ecosystem
Managing enterprise users in Hadoop ecosystem
Hadoop Security Now and Future
Hadoop Security Now and Future
Structor - Automated Building of Virtual Hadoop Clusters
Structor - Automated Building of Virtual Hadoop Clusters
Introducing Node.js in an Oracle technology environment (including hands-on)
Introducing Node.js in an Oracle technology environment (including hands-on)
Ähnlich wie Hadoop operations
HDFS presented by VIJAY
HDFS presented by VIJAY
thevijayps
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
VMware Tanzu
Session 01 - Into to Hadoop
Session 01 - Into to Hadoop
AnandMHadoop
Introduction to Hadoop - The Essentials
Introduction to Hadoop - The Essentials
Fadi Yousuf
Visual Mapping of Clickstream Data
Visual Mapping of Clickstream Data
DataWorks Summit
HBase with MapR
HBase with MapR
Tomer Shiran
Farming hadoop in_the_cloud
Farming hadoop in_the_cloud
Steve Loughran
Hadoop - HDFS
Hadoop - HDFS
KavyaGo
Ozone - Evolution of hdfs scalability
Ozone - Evolution of hdfs scalability
Dinesh Chitlangia
Introduction to HDFS and MapReduce
Introduction to HDFS and MapReduce
Derek Chen
Hadoop Interview Questions and Answers by rohit kapa
Hadoop Interview Questions and Answers by rohit kapa
kapa rohit
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Lucidworks
H2O on Hadoop Dec 12
H2O on Hadoop Dec 12
Sri Ambati
SQL Server 2012 and Big Data
SQL Server 2012 and Big Data
Microsoft TechNet - Belgium and Luxembourg
Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...
Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...
Data Con LA
Tom Kraljevic presents H2O on Hadoop- how it works and what we've learned
Tom Kraljevic presents H2O on Hadoop- how it works and what we've learned
Sri Ambati
Apache Tez - A unifying Framework for Hadoop Data Processing
Apache Tez - A unifying Framework for Hadoop Data Processing
DataWorks Summit
SpringPeople Introduction to Apache Hadoop
SpringPeople Introduction to Apache Hadoop
SpringPeople
Introduction to hadoop
Introduction to hadoop
Marc Cluet
Apache hadoop, hdfs and map reduce Overview
Apache hadoop, hdfs and map reduce Overview
Nisanth Simon
Ähnlich wie Hadoop operations
(20)
HDFS presented by VIJAY
HDFS presented by VIJAY
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)
Session 01 - Into to Hadoop
Session 01 - Into to Hadoop
Introduction to Hadoop - The Essentials
Introduction to Hadoop - The Essentials
Visual Mapping of Clickstream Data
Visual Mapping of Clickstream Data
HBase with MapR
HBase with MapR
Farming hadoop in_the_cloud
Farming hadoop in_the_cloud
Hadoop - HDFS
Hadoop - HDFS
Ozone - Evolution of hdfs scalability
Ozone - Evolution of hdfs scalability
Introduction to HDFS and MapReduce
Introduction to HDFS and MapReduce
Hadoop Interview Questions and Answers by rohit kapa
Hadoop Interview Questions and Answers by rohit kapa
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
H2O on Hadoop Dec 12
H2O on Hadoop Dec 12
SQL Server 2012 and Big Data
SQL Server 2012 and Big Data
Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...
Big Data Day LA 2015 - What's new and next in Apache Tez by Bikas Saha of Hor...
Tom Kraljevic presents H2O on Hadoop- how it works and what we've learned
Tom Kraljevic presents H2O on Hadoop- how it works and what we've learned
Apache Tez - A unifying Framework for Hadoop Data Processing
Apache Tez - A unifying Framework for Hadoop Data Processing
SpringPeople Introduction to Apache Hadoop
SpringPeople Introduction to Apache Hadoop
Introduction to hadoop
Introduction to hadoop
Apache hadoop, hdfs and map reduce Overview
Apache hadoop, hdfs and map reduce Overview
Mehr von Marc Cluet
Your Kernel and You
Your Kernel and You
Marc Cluet
Managing DevOps teams, staying alive
Managing DevOps teams, staying alive
Marc Cluet
The DevOps journey - How to get there painlessly
The DevOps journey - How to get there painlessly
Marc Cluet
Elastic Beanstalk, usos prácticos y conceptos
Elastic Beanstalk, usos prácticos y conceptos
Marc Cluet
Service discovery and puppet
Service discovery and puppet
Marc Cluet
Puppet Camp London Fall 2015 - Service Discovery and Puppet
Puppet Camp London Fall 2015 - Service Discovery and Puppet
Marc Cluet
Puppet and your Metadata - PuppetCamp London 2015
Puppet and your Metadata - PuppetCamp London 2015
Marc Cluet
Consul First Steps
Consul First Steps
Marc Cluet
Autoscaling Best Practices - WebPerf Barcelona Oct 2014
Autoscaling Best Practices - WebPerf Barcelona Oct 2014
Marc Cluet
Microservices and the Cloud - DevOps Cardiff Meetup
Microservices and the Cloud - DevOps Cardiff Meetup
Marc Cluet
Microservices and the Cloud
Microservices and the Cloud
Marc Cluet
How to implement microservices
How to implement microservices
Marc Cluet
A Metadata Ocean in Chef and Puppet
A Metadata Ocean in Chef and Puppet
Marc Cluet
Autoscaling Best Practices
Autoscaling Best Practices
Marc Cluet
Rackspace Hack Night - Vagrant & Packer
Rackspace Hack Night - Vagrant & Packer
Marc Cluet
Innovation in the Cloud - Rackspace Zurich Event
Innovation in the Cloud - Rackspace Zurich Event
Marc Cluet
Introduction to DevOps - Rackspace tech night
Introduction to DevOps - Rackspace tech night
Marc Cluet
Ssh that wonderful thing
Ssh that wonderful thing
Marc Cluet
Networking & dns 101
Networking & dns 101
Marc Cluet
Juju + Puppet (Puppetconf 2011)
Juju + Puppet (Puppetconf 2011)
Marc Cluet
Mehr von Marc Cluet
(20)
Your Kernel and You
Your Kernel and You
Managing DevOps teams, staying alive
Managing DevOps teams, staying alive
The DevOps journey - How to get there painlessly
The DevOps journey - How to get there painlessly
Elastic Beanstalk, usos prácticos y conceptos
Elastic Beanstalk, usos prácticos y conceptos
Service discovery and puppet
Service discovery and puppet
Puppet Camp London Fall 2015 - Service Discovery and Puppet
Puppet Camp London Fall 2015 - Service Discovery and Puppet
Puppet and your Metadata - PuppetCamp London 2015
Puppet and your Metadata - PuppetCamp London 2015
Consul First Steps
Consul First Steps
Autoscaling Best Practices - WebPerf Barcelona Oct 2014
Autoscaling Best Practices - WebPerf Barcelona Oct 2014
Microservices and the Cloud - DevOps Cardiff Meetup
Microservices and the Cloud - DevOps Cardiff Meetup
Microservices and the Cloud
Microservices and the Cloud
How to implement microservices
How to implement microservices
A Metadata Ocean in Chef and Puppet
A Metadata Ocean in Chef and Puppet
Autoscaling Best Practices
Autoscaling Best Practices
Rackspace Hack Night - Vagrant & Packer
Rackspace Hack Night - Vagrant & Packer
Innovation in the Cloud - Rackspace Zurich Event
Innovation in the Cloud - Rackspace Zurich Event
Introduction to DevOps - Rackspace tech night
Introduction to DevOps - Rackspace tech night
Ssh that wonderful thing
Ssh that wonderful thing
Networking & dns 101
Networking & dns 101
Juju + Puppet (Puppetconf 2011)
Juju + Puppet (Puppetconf 2011)
Kürzlich hochgeladen
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Remote DBA Services
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
MadyBayot
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
sammart93
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
Khem
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Andrey Devyatkin
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
The Digital Insurer
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
DianaGray10
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
jfdjdjcjdnsjd
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Anna Loughnan Colquhoun
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
apidays
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Edi Saputra
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
Zilliz
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
Zilliz
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Deepika Singh
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
Kürzlich hochgeladen
(20)
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
presentation ICT roal in 21st century education
presentation ICT roal in 21st century education
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
Hadoop operations
1.
Marc Cluet –
Lynx Consultants How Hadoop Works
2.
What we’ll cover? ¡
Understand Hadoop in detail ¡ See how Hadoop works operationally ¡ Be able to start asking the right questions from your data Lynx Consultants © 2013
3.
Hadoop Distributions ¡ Cloudera
CDH ¡ Hortonworks ¡ MapR Lynx Consultants © 2013
4.
Hadoop Components ¡ HDFS
¡ Hbase ¡ MapRed ¡ YARN Lynx Consultants © 2013
5.
Hadoop Components ¡ HDFS
§ Hadoop Distributed File System § Everything sits on top of it § Has 3 copies by default of every block ¡ Hbase ¡ MapRed ¡ YARN Lynx Consultants © 2013
6.
Hadoop Components ¡ HDFS
¡ Hbase § Hadoop Schemaless Database § Key value Store § Sits on top of HDFS ¡ MapRed ¡ YARN Lynx Consultants © 2013
7.
Hadoop Components ¡ HDFS
¡ Hbase ¡ MapRed § Hadoop Map/Reduce § Non-‐pluggable, archaic § Requires HDFS for temp storage ¡ YARN Lynx Consultants © 2013
8.
Hadoop Components ¡ HDFS
¡ Hbase ¡ MapRed ¡ YARN § Hadoop Map/Reduce version 2.0 § Pluggable, you can add your own § Fast and not so much memory hungry Lynx Consultants © 2013
9.
Hadoop Component Breakdown ¡
All these components divide themselves in § client/server § master/slave scenarios ¡ We will now check each individual component breakdown Lynx Consultants © 2013
10.
Hadoop Components Breakdown ¡
HDFS § Master Namenode ▪ Keeps track of all file allocation on Datanodes ▪ Rebalances data if one of the namenodes goes down ▪ Is Rack aware § Secondary Namenode ▪ Does cleanup services for the namenode ▪ Not necessarily two different servers § Datanode ▪ Stores the data ▪ Good to have not RAID disks for extra I/O speed Lynx Consultants © 2013
11.
Hadoop Components Breakdown ¡
HDFS § How to access ▪ Client can connect with hadoop client to hdfs://namenode:8020 ▪ Supports all basic Unix commands § Configuration files ▪ /etc/hadoop/conf/core-‐site.xml ▪ Defines major configuration as hdfs namenode and default parameters ▪ /etc/hadoop/conf/hdfs-‐site.xml ▪ Defines configuration specific to namenode or datanode on file locations ▪ /etc/hadoop/conf/slaves ▪ Defines the list of servers that are available in this cluster Lynx Consultants © 2013
12.
Hadoop Components Breakdown ¡
Hbase § Master ▪ Controls the Hbase cluster, knows where the data is allocated and provides a client listening socket using Thrift and/or a RESTful API § Regionserver ▪ Hbase node, stores some of the information in one of the regions, it’d be equivalent to sharding § Thrift / REST ▪ Interface to connect to HBase Lynx Consultants © 2013
13.
Hadoop Components Breakdown ¡
Hbase § How to access ▪ Through the Hbase client (using Thrift) ▪ Through the RESTful API § Configuration files ▪ /etc/hbase/conf/hbase-‐site.xml ▪ Defines all the basic configuration for accessing hbase ▪ /etc/hbase/conf/hbase-‐policy.xml ▪ Defines all the security (ACL) and all the hbase memory tweaks ▪ /etc/hbase/conf/regionservers ▪ List all the regionservers available to this cluster Lynx Consultants © 2013
14.
Hadoop Components Breakdown ¡
MapRed § JobTracker ▪ Creates the Map/Reduce jobs ▪ Stores all the intermediate data ▪ Keeps track of all the previous results through the HistoryServer § TaskTracker ▪ Executed Tasks related to the Map/Reduce job ▪ Very CPU and memory intensive ▪ Stores intermediate results which then are pushed to JobTracker Lynx Consultants © 2013
15.
Hadoop Components Breakdown ¡
MapRed § How to access ▪ Through the Hadoop Client ▪ Through any MapRed client like Pig or Hive ▪ Own Java code § Configuration files ▪ /etc/hadoop/conf/mapred-‐site.xml ▪ Defines how to contact this MapRed Cluster ▪ /etc/hadoop/conf/mapred-‐queue-‐acls.xml ▪ Defines ACL structure for accessing MapRed, normally not necessary ▪ /etc/hadoop/conf/slaves ▪ Defines the list of TaskTrackers in this cluster Lynx Consultants © 2013
16.
Hadoop Components Breakdown ¡
YARN § Same structure as MapRed (lives on top of it) § Configuration files ▪ /etc/hadoop/conf/yarn-‐site.xml ▪ All required configuration for YARN Lynx Consultants © 2013
17.
Hadoop Cluster Breakdown ¡
Namenode Server § HDFS Namenode § Hbase Master ¡ Secondary Namenode Server § HDFS Secondary Namenode ¡ JobTracker Server § MapRed JobTracker § MapRed History Server Lynx Consultants © 2013
18.
Hadoop Cluster Breakdown ¡
Datanode Server § HDFS Datanode § Hbase RegionServer § MapRed TaskTracker Lynx Consultants © 2013
19.
Hadoop Hardware Requirements ¡
Namenode Server § Redundant power supplies § RAID1 Drives § Enough memory (16Gb) ¡ Secondary Namenode Server § Almost none Lynx Consultants © 2013
20.
Hadoop Hardware Requirements ¡
Jobtracker Server § Redundant power supplies § RAID1 Drives § Enough memory (16Gb) ¡ Datanode Server § Lots of cheap disk (no RAID) § Lots of memory (32Gb) § Lots of CPU Lynx Consultants © 2013
21.
Hadoop Default Ports ¡
HDFS § 8020: HDFS Namenode § 50010: HDFS Datanode FS transfer ¡ MapRed § No defaults ¡ Hbase § 60010: Master § 60020: Regionserver Lynx Consultants © 2013
22.
Hadoop HDFS Workflow Lynx
Consultants © 2013
23.
Hadoop MapRed Workflow Lynx
Consultants © 2013
24.
Hadoop MapRed Workflow Lynx
Consultants © 2013
25.
Flume ¡ Transports streams
of data from point A to point B ¡ Source § Where the data is read from ¡ Channel § How the data is buffered ¡ Sink § Where the data is written Lynx Consultants © 2013
26.
Flume ¡ Flume is
fault tolerant ¡ Sources are pointer kept § With some exceptions, but most sources are in a known state ¡ Channels can be fault tolerant § Channel written to disk can recover from where it left ¡ Sinks can be redundant § More than one sink for the same data § Data is serialised and deduplicated using AVRO Lynx Consultants © 2013
27.
Flume Lynx Consultants ©
2013
28.
Flume ¡ Configuration files
§ /etc/flume-‐ng/conf/flume.conf ▪ Defines the agent configuration with source, channel, sink Lynx Consultants © 2013
29.
Flume Lynx Consultants ©
2013
30.
Hadoop Recommended Reads Lynx
Consultants © 2013
31.
Hadoop References ¡ Hadoop
§ http://hadoop.apache.org/docs/stable/cluster_setup.html § http://rc.cloudera.com/cdh/4/hadoop/hadoop-‐yarn/hadoop-‐yarn-‐site/ ClusterSetup.html § http://pig.apache.org/docs/r0.7.0/setup.html § http://wiki.apache.org/hadoop/NameNodeFailover ¡ Hbase § http://hbase.apache.org/book/book.html ¡ Flume § http://archive.cloudera.com/cdh4/cdh/4/flume-‐ng/ FlumeUserGuide.html Lynx Consultants © 2013
32.
Questions? Lynx Consultants ©
2013
Jetzt herunterladen