1. An Introduction to
MapReduce 2 and
YARN
Tom White
April 25, 2012
Seattle Hadoop / Scalability / NoSQL Meetup
Wednesday, April 25, 2012

2. First, whatʼs
MapReduce 1?
Wednesday, April 25, 2012

3. Wednesday, April 25, 2012

4. Whatʼs wrong with
MR1?
Wednesday, April 25, 2012

5. Motivation
•Scaling >4000 nodes
•HA of Job Tracker
•Poor resource utilization
Wednesday, April 25, 2012

6. Yet Another Resource Negotiator
Wednesday, April 25, 2012

7. Wednesday, April 25, 2012

8. Wednesday, April 25, 2012

9. Node Manager
is a generalized Task Tracker
• Task Tracker
• fixed number of map or reduce
slots
• Node Manager
• containers with variable resource
limits
Wednesday, April 25, 2012

10. Wednesday, April 25, 2012

11. Wednesday, April 25, 2012

12. MR is user space
YARN is kernel
Wednesday, April 25, 2012

13. Bonus Apps
•Distributed shell
•MPI (MAPREDUCE-2911)
•Master-worker(MAPREDUCE-3315)
•Apache Giraph, Hama
Wednesday, April 25, 2012

14. Wednesday, April 25, 2012

15. Wednesday, April 25, 2012

16. Old API ≠ MR1
New API ≠ MR2
Wednesday, April 25, 2012

17. Old API
o.a.h.mapred
New API
o.a.h.mapreduce
MR1 ✓ ✓
MR2 ✓ ✓
Wednesday, April 25, 2012

18. Wednesday, April 25, 2012

19. Try out MR2
•Apache Hadoop 0.23.1
•CDH4 Beta 2
Wednesday, April 25, 2012

20. <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>0.23.1</version>
</dependency>
MR1
MR2
Wednesday, April 25, 2012

21. TODO
• Still alpha status
• Performance tuning
• Usability bug fixes
• RM recovery
• Security in MR2 not complete
Wednesday, April 25, 2012

22. Further Reading
Wednesday, April 25, 2012

23. Thank You!
Wednesday, April 25, 2012