Map Reduce In 5 Minutes

1. MapReduce

2. MapReduce
reduced

3. PetaMengurangi

4. *
PetaMengurangi
*google translate

5. the problem

6. lots of data

7. e.g.
the entire interwebs

8. single computer
not going to work

9. lots of computers

10. we have that

11. cluster programming

12. cluster programming
= suck

13. MapReduce

14. makes the pain go away

15. 2 main stages

19. map

20. map
process data on hosts

21. reduce

22. reduce
summarise the results

23. example
count words on lines

24. >>> reduce(operator.add, map(countWords, lines))

25. >>> reduce(operator.add, map(countWords, lines))

26. >>> reduce(operator.add, map(countWords, lines))

27. except in this case

28. lots of machines

29. typical cluster

31. O(103) machines
each 2‐8Gb RAM
local IDE disks

32. GFS distributes the data

33. process data on hosts
summarise results

34. split data into chunks
process data on hosts
summarise results

35. split data into chunks
allocate machines
process data on hosts
summarise results

36. split data into chunks
allocate machines
start processes
process data on hosts
summarise results

37. split data into chunks
allocate machines
start processes
send data to mappers
process data on hosts
summarise results

38. split data into chunks
allocate machines
start processes
send data to mappers
process data on hosts
monitor hosts
summarise results

39. split data into chunks
allocate machines
start processes
send data to mappers
process data on hosts
monitor hosts
send results to reducers
summarise results

40. split data into chunks
allocate machines
start processes
send data to mappers
process data on hosts
monitor hosts
redo failed and stragglers
send results to reducers
summarise results

41. split data into chunks
allocate machines
start processes
send data to mappers
process data on hosts
monitor hosts
redo failed and stragglers
send results to reducers
summarise results
output final results

42. MapReduce does the
yukky stuff

43. split data into chunks
allocate machines
start processes
send data to mappers
MapReduce
process data on hosts
monitor hosts programmer
redo failed and stragglers
send results to reducers
summarise results
output final results

44. handles failures

45. handles stragglers

46. a vanity search

47. % of refs to Anthony

48. % of refs to Anthony
Baxter

49. count(‘Anthony Baxter’)
count(‘Anthony’)

50. C++ library

51. ... with Python bindings,
yay!

52. class AnthonyMapper(mrpython.Mapper):
def Map(self, map_input):
meCount = otherCount = 0
docId = map_input.key() # ignored ‐ doc id
src = map_input.value() # document source
text = ExtractText(src).split()
seenAnthony = False
for word in text:
if not seenAnthony:
if word.lower() == 'anthony':
seenAnthony = True
else:
if word.lower() == 'baxter':
meCount += 1
else:
otherCount += 1
seenAnthony = False
yield 'me', meCount
yield 'other', otherCount

53. class AnthonyReducer(mrpython.Reducer):
def Reducer(self, reduce_input):
''' Passed a key (either 'me' or 'other') and a list
of counts. Adds the counts and returns them.
'''
count = 0
for val in reduce_input.values():
sum += int(val)
yield count

54. the result:

55. the result:
about 1 in 4000

56. other uses for MapReduce

57. web link graphs

58. access logs

59. text analysis

60. google news clustering

61. local search

62. road traffic

63. take speed samples

65. group by road segment

67. take the average

68. once per minute

69. output to a map layer

71. limitation: availability of
data

72. MapReduce is pretty cool

73. for more information

74. “mapreduce paper”
“gfs paper”
“google papers”

75. if you’d like to play

76. hadoop.apache.org

77. open source

78. java implementation

79. HDFS