Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Ähnlich wie Hadoop Conference Japan 2011 Fallに行ってきました
Ähnlich wie Hadoop Conference Japan 2011 Fallに行ってきました (20)
Mehr von moai kids
Mehr von moai kids (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Hadoop Conference Japan 2011 Fallに行ってきました
- 1. @just_do_neet 1
- 2. Hadoop Conference Japan 2011 fall • 2011/9/26( ) http://hadoop-conference-japan-2011-fall.eventbrite.com/ • Hadoop 1000 • Cloudera HortonWorks MapR • NTT mixi Yahoo 2
- 4. 4
- 5. 5
- 6. Hadoop T 6
- 7. Hadoop 7
- 8. Hadoop • Apache Software Foundation OSS • • :MapReduce • :HDFS/HBase 8
- 9. Hadoop • Google GFS(ACM ’03) “The Google File System” → HDFS MapReduce(OSDI ’04) “Simplified Data Processing on Large Clusters” → MapReduce BigTable(OSDI ’06) “A Distributed Storage System for Structured Data” → HBase • Yahoo! Inc. Cloudera Doug Cutting Google 2004 Java OSS ”Hadoop” Doug 9
- 10. Hadoop • index index etc... ※Google ※ Doug Lucene Nutch OSS • Hadoop • Hadoop 10
- 11. Hadoop • HBase BigTable • ZooKeeper Chubby • Hive HDFS SQL • Flume • and more.... 11
- 12. • sorry, confidential... 12
- 13. 13
- 14. Hadoop Conference Japan 2011 Fall • Hadoop 0.23 • • 14
- 15. Hadoop 0.23 • 2011 Q4 beta Integration Test Pig • Hadoop NameNode SPoF (NameNode • MapReduce v2 : MapReduce MPI(C Giraph Hama Graph Spark etc • MapReduce shuffle 30% / 10,000 4000 / etc 15
- 16. Giraph • Hadoop https://github.com/aching/Giraph Graph → Google PageRank Google Pregel(SIGMOD ’10) • BSP(Bulk Synchronous Parallel 16
- 17. Spark • Scala http://www.spark-project.org/ → ex. • Mesos http://www.mesosproject.org/ Hadoop 17
- 18. H • Cloudera http://www.cloudera.com/ OSS (CDH Sqoop Hue • HortonWorks http://www.hortonworks.com/ Yahoo! Inc. Hadoop • MapR http://www.mapr.com/ Hadoop • Hadoop 18
- 19. H Cloudera • CDH Linux • OSS Sqoop SQL HDFS Hue Web Console etc.. • SCM Express Web Hadoop 50 free • Cloudera Enterprise Cloudera Management Suite • 19
- 20. H HortonWorks • Yahoo! Inc. Hadoop 22 Hadoop ※ ( 500,000 • 2011/7 • Yahoo! Inc. Yahoo! Mail 42,000 Hadoop • Horton the Elephant http://en.wikipedia.org/wiki/Horton_the_Elephant 20
- 21. H MapR • Hadoop EMC OEM Greenplum http://www.greenplum.com/ • HDFS C++ HDFS random read/write NIC bonding snapshot etc.. NFS Hadoop R • OSS Hadoop 21
- 22. • Hadoop LSP etc.. • Asakusa Framework OSS https://github.com/asakusafw DSL 22
- 23. Yahoo! Japan • Hadoop Geohash Hadoop Iterable Twitter ※ MPI ... 23
- 24. 24
- 25. • Hadoop BtoC Hadoop 0.23 MapReduce v2 Hadoop • Hadoop Hadoop Map/Reduce + Graph BI 25
- 26. 26