961万人の食卓を支えるデータ解析

•Als KEY, PDF herunterladen•

36 gefällt mir•5,248 views

Tatsuya Sasaki

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Technologie Seele & Geist

•
•
• Hadoop (Cloudera)

• Elastic MapReduce
•
•

• (@sasata299)

• 2009 8 JOIN

•
• Hadoop

• Hadoop MySQL

•
- GROUP BY

• 7000 …

• (´Д` )

Hadoop
• Google MapReduce OSS

•
-
-
-
-

Hadoop
master ( )

slave ( )

<key,value>
Map
Shufﬂe & Sort

Hadoop
master ( )

slave ( )

<key,value>
Map Reduce
Shufﬂe & Sort

• Hadoop Streaming (Ruby )

• EC2 Cloudera Hadoop
- Cloudera CDH1
- Hadoop 0.18.3

• S3

MySQL → Hadoop
•
• GROUP BY MapReduce
- ( )
- key

• JOIN MapReduce
•

(1) master
master slave scp

(2) S3

S3 slave scp

MySQL vs Hadoop

7000

MySQL Hadoop
MySQL Hadoop

MySQL vs Hadoop

( Д )
7000
30

MySQL Hadoop
MySQL Hadoop

• Hadoop
-
• Hadoop (HADOOP-6254)
- S3
- SocketTimeoutException

• EMR (Elastic MapReduce)
- Amazon Hadoop

• Cloudera CDH2
-

AMI
(Amazon Machine
UP Image)

EMR

CDH2

EMR
BootStrap Action

Step (Hadoop Job)

Job Flow ( )

•
-
- --alive
• AMI
- AMI

- BootStrap Action

•
- mapred.child.java.opts
- streaming

•
-
- ElasticMapReduce-master 5100

Weitere ähnliche Inhalte

Ähnlich wie 961万人の食卓を支えるデータ解析

COOKPADでのHadoop利用Tatsuya Sasaki

Hadoop導入事例 in クックパッドTatsuya Sasaki

Amebaサービスのログ解析基盤Toshihiro Suzuki

Hadoop入門とクラウド利用Naoki Yanai

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop londonYahoo Developer Network

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)npinto

HadoopThe Hadoop Java Software FrameworkThoughtWorks

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコードShinpei Ohtani

HadoopRajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012designandanalytics

Hadoop and MapReduceHemanth Kumar Mantri

Azure Data WarehouseNikolay Stanev

Ähnlich wie 961万人の食卓を支えるデータ解析 (20)

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Hadoop london

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

HadoopThe Hadoop Java Software Framework

Qubole Overview at the Fifth Elephant Conference

Hadoop Overview & Architecture

Hadoop Conference Japan 2011 Fall

Hadoop Overview kdd2011

データ解析技術入門(Hadoop編)

サンプルから見るMap reduceコード

サンプルから見るMapReduceコード

Hadoop

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Mehr von Tatsuya Sasaki

からあげエンジニアについてTatsuya Sasaki

クックパッドでのemr利用事例Tatsuya Sasaki

からあげとビーチと私Tatsuya Sasaki

メタプログラミングでDSLを書こうTatsuya Sasaki

NoSQLデータベースが登場した背景と特徴Tatsuya Sasaki

Hadoopをemr経由で利用する方法Tatsuya Sasaki

YUITatsuya Sasaki

Mehr von Tatsuya Sasaki (7)

からあげエンジニアについて

クックパッドでのemr利用事例

からあげとビーチと私

メタプログラミングでDSLを書こう

NoSQLデータベースが登場した背景と特徴

Hadoopをemr経由で利用する方法

YUI

Kürzlich hochgeladen

Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya

Architecting Cloud Native ApplicationsWSO2

DBX First Quarter 2024 Investor PresentationDropbox

Elevate Developer Efficiency & build GenAI Application with Amazon QBhuvaneswari Subramani

Exploring Multimodal Embeddings with MilvusZilliz

Understanding the FAA Part 107 License ..Christopher Logan Kennedy

[BuildWithAI] Introduction to Gemini.pdfSandro Moreira

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra

MS Copilot expands with MS Graph connectorsNanddeep Nachan

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Orbitshub

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub

WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz

Apidays New York 2024 - The value of a flexible API Management solution for O...apidays

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity

ICT role in 21st century education and its challengesrafiqahmad00786416

CNIC Information System with Pakdata Cf In Pakistandanishmna97

Kürzlich hochgeladen (20)

Artificial Intelligence Chap.5 : Uncertainty

Architecting Cloud Native Applications

DBX First Quarter 2024 Investor Presentation

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Exploring Multimodal Embeddings with Milvus

Understanding the FAA Part 107 License ..

[BuildWithAI] Introduction to Gemini.pdf

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

MS Copilot expands with MS Graph connectors

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

WSO2's API Vision: Unifying Control, Empowering Developers

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Apidays New York 2024 - The value of a flexible API Management solution for O...

presentation ICT roal in 21st century education

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

ICT role in 21st century education and its challenges

CNIC Information System with Pakdata Cf In Pakistan

961万人の食卓を支えるデータ解析

1. 961

2. • • • Hadoop (Cloudera) • Elastic MapReduce • •

3. • • • Hadoop (Cloudera) • Elastic MapReduce • •

4. • (@sasata299) • 2009 8 JOIN • • Hadoop

5. • • • Hadoop (Cloudera) • Elastic MapReduce • •

7. • 961 • 30 3 1 • - -

8. • • ( , , ...) - - ( , , , …) -

10.

11.

12.

13.

14. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

15. • MySQL - • - -

16.

17. • • • Hadoop (Cloudera) • Elastic MapReduce • •

18. Hadoop • Google MapReduce OSS • - - - -

19. Hadoop master ( ) slave ( )

20. Hadoop master ( ) slave ( ) Map

21. Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

22. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

23. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

24. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

25. (1) master (2) S3

26. (1) master (2) S3

27. (1) master master slave scp (2) S3

28. (1) master master slave scp (2) S3 S3 slave scp

29. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

30. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

31. Hadoop++ ←Hadoop ↓MySQL

32. • • • Hadoop (Cloudera) • Elastic MapReduce • •

33. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

34. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

35. AMI (Amazon Machine UP Image) EMR CDH2

36. AMI (Amazon Machine UP Image) EMR CDH2

37. EMR Job Flow ( )

38. EMR BootStrap Action Job Flow ( )

39. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

40. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

41. • - - --alive • AMI - AMI - BootStrap Action

42.

43. Created job ﬂow j-8IXS98OW1WEE ID

44. Hadoop

45.

46. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

47. • • • Hadoop (Cloudera) • Elastic MapReduce • •

48. • Map - • Reduce - key Reduce -

49. UU Map Reduce

50. UU Map ID Reduce

51. UU Map Reduce ID

52. UU Map Reduce ID

53. Map Reduce

54. Map ID key Reduce

55. Map Reduce key Reduce

56. Map 100 100 Reduce key Reduce

57. × Map 100 × 100 Reduce key Reduce

58. × Map 100 ×100 Reduce Reduce key sort

59. × Map 100 ×100 Reduce Reduce key sort

60. Hadoop • - Hadoop

61. Hadoop • - Hadoop

62. Hadoop • - Hadoop

63. Hadoop • - Hadoop

64. Hadoop • - Hadoop

65. • • • Hadoop (Cloudera) • Elastic MapReduce • •

66. • Hadoop - - - Reduce

961万人の食卓を支えるデータ解析

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie 961万人の食卓を支えるデータ解析

Ähnlich wie 961万人の食卓を支えるデータ解析 (20)

Mehr von Tatsuya Sasaki

Mehr von Tatsuya Sasaki (7)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

961万人の食卓を支えるデータ解析

Hinweis der Redaktion