This document discusses Hadoop and its use on Amazon Web Services. It describes how Hadoop can be used to process large amounts of data in parallel across clusters of computers. Specifically, it outlines how to run Hadoop jobs on an Elastic Compute Cloud (EC2) cluster configured with Hadoop and store data in Amazon Simple Storage Service (S3). The document also provides examples of using Hadoop Streaming to run MapReduce jobs written in Ruby on an EC2 Hadoop cluster.

1. Hadoop

2. • sasata299 ( )
• Hadoop
• NoSQL
•
• http://blog.livedoor.jp/sasata299/

3. Hadoop
• Google MapReduce OSS
•
•
•
•

4. • Hadoop
• Hadoop
•
• & !?

5. • Hadoop
• Hadoop
•
• & !?

7. 896
30 3 1

8. 896
30 3 1

9. 896
30 3 1
” ”

13. GROUP BY !!
MySQL
( 3.5 )

14. GROUP BY !!
MySQL
( 3.5 )
7000
≒292 ……orz

15. MySQL

16. …
※

17. !?

19. Hadoop

20. …

22. 7000
30

24. • Hadoop
• Hadoop
•
• & !?

25. Hadoop
• Hadoop Streaming
• Ruby
• Cloudera CDH1 (0.18.3)
• EC2 Hadoop
• 10-50
• Hadoop S3

26. Hadoop
• Hadoop Streaming
• Ruby
• Cloudera CDH1 (0.18.3)
• EC2 Hadoop
• 10-50
• Hadoop S3

32. ○○
××
○○
××

33. Hadoop

36. Hadoop AWS

38. Hadoop

39. Hadoop
Hadoop
(EC2)

40. hadoop-ec2 push [cluster] mapper.rb
hadoop-ec2 push [cluster] reducer.rb
Hadoop
(EC2)

41. hadoop-ec2 push [cluster] mapper.rb
hadoop-ec2 push [cluster] reducer.rb
Hadoop
(EC2)

42. Hadoop
hadoop-ec2 exec [cluster] [command]
Hadoop S3
(EC2)

43. Hadoop
hadoop-ec2 exec [cluster] [command]
Hadoop S3
(EC2)

44. Hadoop
hadoop-ec2 exec [cluster] [command]
Hadoop S3
(EC2)

45. Hadoop
hadoop-ec2 exec [cluster] [command]
Hadoop S3
(EC2)

46. S3

47. S3

48. Hadoop
Hadoop
(EC2)

49. Hadoop

50. !!

51. Hadoop

52. 1)
2) Hadoop
3)

53. • Hadoop
• Hadoop
•
• & !?

54. target_ids # [21310,12902,15321,..]
ARGF.each do |log|
log.chomp!
id, foo, bar, ... = log.split(/,/)
next if target_ids.include?(id)
end
target_ids 5
…

55. :-)

56. # 1000
hash = Hash.new {|h,k| h[k] = []}
target_ids.each do |_id|
hash[_id.to_s[0,3]] << _id
end
ARGF.each do |log|
log.chomp!
id, foo, bar, ... = log.split(/,/)
#
next if hash[id[0,3]].include?(id)
end

58. Mapper
Reducer
- Mapper
-
Reducer

59. • Hadoop
• Hadoop
•
• & !?

60. EC2
-> AZ
->
JobTracker
-> 50030
-> hadoop job -list

61. 10h 8h
JobTracker

62. !?

63. Amazon Elastic MapReduce

64. Elastic MapReduce