Suche senden
Hochladen
48a tuning
•
2 gefällt mir
•
1,304 views
M
mapr-academy
Folgen
Unterhaltung & Humor
Business
Melden
Teilen
Melden
Teilen
1 von 18
Empfohlen
Taming Latency: Case Studies in MapReduce Data Analytics
Taming Latency: Case Studies in MapReduce Data Analytics
EMC
52 nfs
52 nfs
mapr-academy
50a volumes
50a volumes
mapr-academy
071410 sun a_1515_feldman_stephen
071410 sun a_1515_feldman_stephen
Steve Feldman
RAMinate Invited Talk at NII
RAMinate Invited Talk at NII
Takahiro Hirofuchi
Optimizing MapReduce Job performance
Optimizing MapReduce Job performance
DataWorks Summit
D3 d12 a-new-meaning-for-efficiency-and-performance
D3 d12 a-new-meaning-for-efficiency-and-performance
mistercteam
RAMinate ACM SoCC 2016 Talk
RAMinate ACM SoCC 2016 Talk
Takahiro Hirofuchi
Empfohlen
Taming Latency: Case Studies in MapReduce Data Analytics
Taming Latency: Case Studies in MapReduce Data Analytics
EMC
52 nfs
52 nfs
mapr-academy
50a volumes
50a volumes
mapr-academy
071410 sun a_1515_feldman_stephen
071410 sun a_1515_feldman_stephen
Steve Feldman
RAMinate Invited Talk at NII
RAMinate Invited Talk at NII
Takahiro Hirofuchi
Optimizing MapReduce Job performance
Optimizing MapReduce Job performance
DataWorks Summit
D3 d12 a-new-meaning-for-efficiency-and-performance
D3 d12 a-new-meaning-for-efficiency-and-performance
mistercteam
RAMinate ACM SoCC 2016 Talk
RAMinate ACM SoCC 2016 Talk
Takahiro Hirofuchi
High Availability Options for DB2 Data Centre
High Availability Options for DB2 Data Centre
terraborealis
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Yahoo Developer Network
IT Platform Selection by Economic Factors and Information Security Requiremen...
IT Platform Selection by Economic Factors and Information Security Requiremen...
ECLeasing
MapReduce presentation
MapReduce presentation
Vu Thi Trang
Solaris Linux Performance, Tools and Tuning
Solaris Linux Performance, Tools and Tuning
Adrian Cockcroft
Virtualization for Emerging Memory Devices
Virtualization for Emerging Memory Devices
Takahiro Hirofuchi
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
Vijay Rayapati
AutoDOPandRest
AutoDOPandRest
Rick van Ek
UNIT 4 B.docx
UNIT 4 B.docx
Nagendrababu Vasa
Probabilistic consolidation of virtual machines in self organizing cloud data...
Probabilistic consolidation of virtual machines in self organizing cloud data...
Papitha Velumani
Autonomous control in Big Data platforms: and experience with Cassandra
Autonomous control in Big Data platforms: and experience with Cassandra
Emiliano
Understanding DB2 Optimizer
Understanding DB2 Optimizer
terraborealis
06340356
06340356
Narava Vivek
70a monitoring & troubleshooting
70a monitoring & troubleshooting
mapr-academy
13c planning
13c planning
mapr-academy
Hadoop mapreduce and yarn frame work- unit5
Hadoop mapreduce and yarn frame work- unit5
RojaT4
14 lab-planing
14 lab-planing
mapr-academy
14 lab-planing
14 lab-planing
Inyoung Cho
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
IRJET Journal
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Impetus Technologies
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Hortonworks
Enhancing Performance and Fault Tolerance of Hadoop Cluster
Enhancing Performance and Fault Tolerance of Hadoop Cluster
IRJET Journal
Weitere ähnliche Inhalte
Was ist angesagt?
High Availability Options for DB2 Data Centre
High Availability Options for DB2 Data Centre
terraborealis
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Yahoo Developer Network
IT Platform Selection by Economic Factors and Information Security Requiremen...
IT Platform Selection by Economic Factors and Information Security Requiremen...
ECLeasing
MapReduce presentation
MapReduce presentation
Vu Thi Trang
Solaris Linux Performance, Tools and Tuning
Solaris Linux Performance, Tools and Tuning
Adrian Cockcroft
Virtualization for Emerging Memory Devices
Virtualization for Emerging Memory Devices
Takahiro Hirofuchi
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
Vijay Rayapati
AutoDOPandRest
AutoDOPandRest
Rick van Ek
UNIT 4 B.docx
UNIT 4 B.docx
Nagendrababu Vasa
Probabilistic consolidation of virtual machines in self organizing cloud data...
Probabilistic consolidation of virtual machines in self organizing cloud data...
Papitha Velumani
Autonomous control in Big Data platforms: and experience with Cassandra
Autonomous control in Big Data platforms: and experience with Cassandra
Emiliano
Understanding DB2 Optimizer
Understanding DB2 Optimizer
terraborealis
06340356
06340356
Narava Vivek
Was ist angesagt?
(13)
High Availability Options for DB2 Data Centre
High Availability Options for DB2 Data Centre
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
Hadoop Summit 2010 Tuning Hadoop To Deliver Performance To Your Application
IT Platform Selection by Economic Factors and Information Security Requiremen...
IT Platform Selection by Economic Factors and Information Security Requiremen...
MapReduce presentation
MapReduce presentation
Solaris Linux Performance, Tools and Tuning
Solaris Linux Performance, Tools and Tuning
Virtualization for Emerging Memory Devices
Virtualization for Emerging Memory Devices
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
Big Data and Hadoop in Cloud - Leveraging Amazon EMR
AutoDOPandRest
AutoDOPandRest
UNIT 4 B.docx
UNIT 4 B.docx
Probabilistic consolidation of virtual machines in self organizing cloud data...
Probabilistic consolidation of virtual machines in self organizing cloud data...
Autonomous control in Big Data platforms: and experience with Cassandra
Autonomous control in Big Data platforms: and experience with Cassandra
Understanding DB2 Optimizer
Understanding DB2 Optimizer
06340356
06340356
Ähnlich wie 48a tuning
70a monitoring & troubleshooting
70a monitoring & troubleshooting
mapr-academy
13c planning
13c planning
mapr-academy
Hadoop mapreduce and yarn frame work- unit5
Hadoop mapreduce and yarn frame work- unit5
RojaT4
14 lab-planing
14 lab-planing
mapr-academy
14 lab-planing
14 lab-planing
Inyoung Cho
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
IRJET Journal
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Impetus Technologies
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Hortonworks
Enhancing Performance and Fault Tolerance of Hadoop Cluster
Enhancing Performance and Fault Tolerance of Hadoop Cluster
IRJET Journal
Introduction to map reduce
Introduction to map reduce
M Baddar
Managing Big data Module 3 (1st part)
Managing Big data Module 3 (1st part)
Soumee Maschatak
Hadoop fault tolerance
Hadoop fault tolerance
Pallav Jha
Report Hadoop Map Reduce
Report Hadoop Map Reduce
Urvashi Kataria
IEEE CLOUD \'11
IEEE CLOUD \'11
David Ribeiro Alves
MapReduce
MapReduce
Surinder Kaur
22 configuration
22 configuration
mapr-academy
Introduction to Spark
Introduction to Spark
Carol McDonald
Yarns About Yarn
Yarns About Yarn
Cloudera, Inc.
YARN (2).pptx
YARN (2).pptx
Raja Ram Dutta
Apache Spark - Santa Barbara Scala Meetup Dec 18th 2014
Apache Spark - Santa Barbara Scala Meetup Dec 18th 2014
cdmaxime
Ähnlich wie 48a tuning
(20)
70a monitoring & troubleshooting
70a monitoring & troubleshooting
13c planning
13c planning
Hadoop mapreduce and yarn frame work- unit5
Hadoop mapreduce and yarn frame work- unit5
14 lab-planing
14 lab-planing
14 lab-planing
14 lab-planing
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
Novel Scheduling Algorithms for Efficient Deployment of Map Reduce Applicatio...
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Advanced Hadoop Tuning and Optimization - Hadoop Consulting
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Enhancing Performance and Fault Tolerance of Hadoop Cluster
Enhancing Performance and Fault Tolerance of Hadoop Cluster
Introduction to map reduce
Introduction to map reduce
Managing Big data Module 3 (1st part)
Managing Big data Module 3 (1st part)
Hadoop fault tolerance
Hadoop fault tolerance
Report Hadoop Map Reduce
Report Hadoop Map Reduce
IEEE CLOUD \'11
IEEE CLOUD \'11
MapReduce
MapReduce
22 configuration
22 configuration
Introduction to Spark
Introduction to Spark
Yarns About Yarn
Yarns About Yarn
YARN (2).pptx
YARN (2).pptx
Apache Spark - Santa Barbara Scala Meetup Dec 18th 2014
Apache Spark - Santa Barbara Scala Meetup Dec 18th 2014
Mehr von mapr-academy
80a disaster recovery
80a disaster recovery
mapr-academy
58a migration
58a migration
mapr-academy
55a remote cluster
55a remote cluster
mapr-academy
53 lab-nfs
53 lab-nfs
mapr-academy
51 lab-volumes
51 lab-volumes
mapr-academy
42 lab-managing services
42 lab-managing services
mapr-academy
41a managing services
41a managing services
mapr-academy
30a accessing your cluster
30a accessing your cluster
mapr-academy
20a installation
20a installation
mapr-academy
12a architecture
12a architecture
mapr-academy
10c introduction
10c introduction
mapr-academy
3 map r installation & setup administration course description
3 map r installation & setup administration course description
mapr-academy
Mehr von mapr-academy
(12)
80a disaster recovery
80a disaster recovery
58a migration
58a migration
55a remote cluster
55a remote cluster
53 lab-nfs
53 lab-nfs
51 lab-volumes
51 lab-volumes
42 lab-managing services
42 lab-managing services
41a managing services
41a managing services
30a accessing your cluster
30a accessing your cluster
20a installation
20a installation
12a architecture
12a architecture
10c introduction
10c introduction
3 map r installation & setup administration course description
3 map r installation & setup administration course description
Kürzlich hochgeladen
Hotel And Home Service Available Kolkata Call Girls Sonagachi ✔ 6297143586 ✔C...
Hotel And Home Service Available Kolkata Call Girls Sonagachi ✔ 6297143586 ✔C...
ritikasharma
Independent Garulia Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Independent Garulia Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Riya Pathan
Independent Sonagachi Escorts ✔ 9332606886✔ Full Night With Room Online Booki...
Independent Sonagachi Escorts ✔ 9332606886✔ Full Night With Room Online Booki...
Riya Pathan
Top Rated Pune Call Girls Dhayari ⟟ 6297143586 ⟟ Call Me For Genuine Sex Ser...
Top Rated Pune Call Girls Dhayari ⟟ 6297143586 ⟟ Call Me For Genuine Sex Ser...
Call Girls in Nagpur High Profile
Behala ( Call Girls ) Kolkata ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Ready ...
Behala ( Call Girls ) Kolkata ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Ready ...
ritikasharma
↑Top Model (Kolkata) Call Girls Howrah ⟟ 8250192130 ⟟ High Class Call Girl In...
↑Top Model (Kolkata) Call Girls Howrah ⟟ 8250192130 ⟟ High Class Call Girl In...
noor ahmed
Call Girls in Barasat | 7001035870 At Low Cost Cash Payment Booking
Call Girls in Barasat | 7001035870 At Low Cost Cash Payment Booking
noor ahmed
Model Call Girls In Velappanchavadi WhatsApp Booking 7427069034 call girl ser...
Model Call Girls In Velappanchavadi WhatsApp Booking 7427069034 call girl ser...
Shivani Pandey
Independent Joka Escorts ✔ 8250192130 ✔ Full Night With Room Online Booking 2...
Independent Joka Escorts ✔ 8250192130 ✔ Full Night With Room Online Booking 2...
noor ahmed
Kanpur call girls 📞 8617697112 At Low Cost Cash Payment Booking
Kanpur call girls 📞 8617697112 At Low Cost Cash Payment Booking
Nitya salvi
Russian ℂall gIRLS In Goa 9316020077 ℂall gIRLS Service In Goa
Russian ℂall gIRLS In Goa 9316020077 ℂall gIRLS Service In Goa
russian goa call girl and escorts service
↑Top Model (Kolkata) Call Girls Sonagachi ⟟ 8250192130 ⟟ High Class Call Girl...
↑Top Model (Kolkata) Call Girls Sonagachi ⟟ 8250192130 ⟟ High Class Call Girl...
noor ahmed
❤Personal Whatsapp Number Keylong Call Girls 8617697112 💦✅.
❤Personal Whatsapp Number Keylong Call Girls 8617697112 💦✅.
Nitya salvi
Borum Call Girls Service ☎ ️93326-06886 ❤️🔥 Enjoy 24/7 Escort Service
Borum Call Girls Service ☎ ️93326-06886 ❤️🔥 Enjoy 24/7 Escort Service
Damini Dixit
Verified Trusted Call Girls Tambaram Chennai ✔✔7427069034 Independent Chenna...
Verified Trusted Call Girls Tambaram Chennai ✔✔7427069034 Independent Chenna...
Shivani Pandey
𓀤Call On 6297143586 𓀤 Sonagachi Call Girls In All Kolkata 24/7 Provide Call W...
𓀤Call On 6297143586 𓀤 Sonagachi Call Girls In All Kolkata 24/7 Provide Call W...
rahim quresi
Independent Hatiara Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Independent Hatiara Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Riya Pathan
Dakshineswar Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Se...
Dakshineswar Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Se...
aamir
Goa Call "Girls Service 9316020077 Call "Girls in Goa
Goa Call "Girls Service 9316020077 Call "Girls in Goa
sexy call girls service in goa
Nayabad Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Sex At ...
Nayabad Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Sex At ...
aamir
Kürzlich hochgeladen
(20)
Hotel And Home Service Available Kolkata Call Girls Sonagachi ✔ 6297143586 ✔C...
Hotel And Home Service Available Kolkata Call Girls Sonagachi ✔ 6297143586 ✔C...
Independent Garulia Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Independent Garulia Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Independent Sonagachi Escorts ✔ 9332606886✔ Full Night With Room Online Booki...
Independent Sonagachi Escorts ✔ 9332606886✔ Full Night With Room Online Booki...
Top Rated Pune Call Girls Dhayari ⟟ 6297143586 ⟟ Call Me For Genuine Sex Ser...
Top Rated Pune Call Girls Dhayari ⟟ 6297143586 ⟟ Call Me For Genuine Sex Ser...
Behala ( Call Girls ) Kolkata ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Ready ...
Behala ( Call Girls ) Kolkata ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Ready ...
↑Top Model (Kolkata) Call Girls Howrah ⟟ 8250192130 ⟟ High Class Call Girl In...
↑Top Model (Kolkata) Call Girls Howrah ⟟ 8250192130 ⟟ High Class Call Girl In...
Call Girls in Barasat | 7001035870 At Low Cost Cash Payment Booking
Call Girls in Barasat | 7001035870 At Low Cost Cash Payment Booking
Model Call Girls In Velappanchavadi WhatsApp Booking 7427069034 call girl ser...
Model Call Girls In Velappanchavadi WhatsApp Booking 7427069034 call girl ser...
Independent Joka Escorts ✔ 8250192130 ✔ Full Night With Room Online Booking 2...
Independent Joka Escorts ✔ 8250192130 ✔ Full Night With Room Online Booking 2...
Kanpur call girls 📞 8617697112 At Low Cost Cash Payment Booking
Kanpur call girls 📞 8617697112 At Low Cost Cash Payment Booking
Russian ℂall gIRLS In Goa 9316020077 ℂall gIRLS Service In Goa
Russian ℂall gIRLS In Goa 9316020077 ℂall gIRLS Service In Goa
↑Top Model (Kolkata) Call Girls Sonagachi ⟟ 8250192130 ⟟ High Class Call Girl...
↑Top Model (Kolkata) Call Girls Sonagachi ⟟ 8250192130 ⟟ High Class Call Girl...
❤Personal Whatsapp Number Keylong Call Girls 8617697112 💦✅.
❤Personal Whatsapp Number Keylong Call Girls 8617697112 💦✅.
Borum Call Girls Service ☎ ️93326-06886 ❤️🔥 Enjoy 24/7 Escort Service
Borum Call Girls Service ☎ ️93326-06886 ❤️🔥 Enjoy 24/7 Escort Service
Verified Trusted Call Girls Tambaram Chennai ✔✔7427069034 Independent Chenna...
Verified Trusted Call Girls Tambaram Chennai ✔✔7427069034 Independent Chenna...
𓀤Call On 6297143586 𓀤 Sonagachi Call Girls In All Kolkata 24/7 Provide Call W...
𓀤Call On 6297143586 𓀤 Sonagachi Call Girls In All Kolkata 24/7 Provide Call W...
Independent Hatiara Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Independent Hatiara Escorts ✔ 9332606886✔ Full Night With Room Online Booking...
Dakshineswar Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Se...
Dakshineswar Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Se...
Goa Call "Girls Service 9316020077 Call "Girls in Goa
Goa Call "Girls Service 9316020077 Call "Girls in Goa
Nayabad Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Sex At ...
Nayabad Call Girls ✔ 8005736733 ✔ Hot Model With Sexy Bhabi Ready For Sex At ...
48a tuning
1.
Tuning MapReduce
7/6/2012 © 2012 MapR Technologies Tuning 1
2.
Tuning MapReduce
Agenda • Tuning MapReduce • ExpressLane™ • Label-Based Scheduling © 2012 MapR Technologies Tuning 2
3.
Tuning MapReduce
Objectives At the end of this module you will be able to: • Effectively tune your MapReduce jobs • Explain how ExpressLane works and what jobs it applies to in your cluster • Configure label-based scheduling © 2012 MapR Technologies Tuning 3
4.
Tuning MapReduce © 2012
MapR Technologies Tuning 4
5.
Important Parameters
Number of task slots per node Number of task slots on the cluster Memory buffer size JVM size Speculative execution © 2012 MapR Technologies Tuning 5
6.
Number of Task
Slots per Node Number of concurrent map and reduce tasks on a node In mapred-site.xml – mapred.tasktracker.map.tasks.maximum – mapred.tasktracker.reduce.tasks.maximum Recommendations: – Map slots: 0.75 * # of cores (minimum 1) – Reduce slots: 0.5 * # of cores (minimum 1) Decrease map and reduce slots on CLDB nodes Increase slots on nodes with more memory, disk, network bandwidth – E.g. reducers are bandwidth-intensive © 2012 MapR Technologies Tuning 6
7.
Number of Task
Slots on the Cluster How many concurrent map and reduce tasks can run In mapred-site.xml – mapred.map.tasks – mapred.reduce.tasks • Best parameter to tune © 2012 MapR Technologies Tuning 7
8.
Memory Buffer Size
Memory used by map task for output during shuffle – io.sort.mb Set to about 2x block size – Use hadoop mfs to see block size If set too low, spills will result in lower performance – Visible in MapR Metrics © 2012 MapR Technologies Tuning 8
9.
JVM Size
Size of child JVM that runs a map or reduce task – mapred.map.child.java.opts – set to about 2x io.sort.mb – mapred.reduce.child.java.opts – leave at default setting © 2012 MapR Technologies Tuning 9
10.
Speculative Execution
Set to true: – mapred.map.tasks.speculative.execution – mapred.reduce.tasks.speculative.execution © 2012 MapR Technologies Tuning 10
11.
ExpressLane™ © 2012 MapR
Technologies Tuning 11
12.
ExpressLane™
Allow a small job to run when all slots are occupied Only applies when cluster is busy and if job meets criteria specified in mapred-site.xml Check the documentation for ExpressLane criteria – http://mapr.com/doc/display/MapR/ExpressLane Note: jobs that fit the small job definition, but are in fact larger than anticipated are killed and re-queued for normal execution © 2012 MapR Technologies Tuning 12
13.
Label-Based Scheduling © 2012
MapR Technologies Tuning 13
14.
Label-Based Scheduling
Restrict job execution to a set of nodes within the cluster – By hardware config, department, etc. Admin applies label(s) to nodes User specifies label when submitting job Admin can specify default/override label per queue © 2012 MapR Technologies Tuning 14
15.
Label-Based Scheduling
On a jobtracker node in mapred-site.xml mapreduce.jobtracker.node.labels.file = <path to node-label mapping file> – Within the mapping file, each line uses the format <node pattern/regex> <labels> – Examples hadoop-prod-0* qa /hadoop-prod-1.*/ sales, product, 4_disks hadoop-prod-2 12_disks, engineering hadoop-prod-3 big_ram, support © 2012 MapR Technologies Tuning 15
16.
Label-Based Scheduling
Specify a label when submitting a job in hadoop command line mapred.job.label = <label> © 2012 MapR Technologies Tuning 16
17.
Label-Based Scheduling
Default label per queue – Examples mapred.queue.<queue-name>.label = <label> mapred.queue.<queue-name>.label.policy = <PREFER_QUEUE | PREFER_JOB | AND | OR> © 2012 MapR Technologies Tuning 17
18.
Questions © 2012 MapR
Technologies Tuning 18