Als PDF, PPTX herunterladen
![WordCount Example
reduce (word, count[]) {
total = 0;
foreach number in count[] {
total += number
}
emit (word, total)
}
Montag, 6. Juni 2011](https://image.slidesharecdn.com/hadoopintroduction-berlinbuzzwords2011-110622060240-phpapp02/75/Hadoop-introduction-berlin-buzzwords-2011-9-2048.jpg)
![[译]No sql生态系统](https://cdn.slidesharecdn.com/ss_thumbnails/nosql-110624114358-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Hadoop introduction berlin buzzwords 2011
- 1. An Introduction Kai Voigt, Cloudera Inc Berlin Buzzwords, June 6th 2011 Montag, 6. Juni 2011
- 2. Big Data • Capture • Storage • Search • Analytics Montag, 6. Juni 2011
- 3. hadoop.apache.org Google Filesystem Hadoop Distributed (GFS) Filesystem (HDFS) MapReduce MapReduce Montag, 6. Juni 2011
- 4. Hadoop Distributed Filesystem • Easy Access • Distributed • Redundant • Scalable Montag, 6. Juni 2011
- 5. File Splits Large File 6440M Block Block Block Block Block Block Block Block 1 2 3 4 5 6 ... 100 101 64M 64M 64M 64M 64M 64M 64M 40M Montag, 6. Juni 2011
- 6. Block Placement Block Block Block Block Block 1 2 3 2 2 Block Block Block 3 1 3 Block 1 Node 1 Node 2 Node 3 Node 4 Node 5 Montag, 6. Juni 2011
- 7. MapReduce Block Block Block Block Input 1 2 3 4 map() Inter- mediate Data reduce() Output Montag, 6. Juni 2011
- 8. WordCount Example map (offset, line) { foreach word in line { emit (word, 1) } } Montag, 6. Juni 2011
- 9. WordCount Example reduce (word, count[]) { total = 0; foreach number in count[] { total += number } emit (word, total) } Montag, 6. Juni 2011
- 10. map() ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) Montag, 6. Juni 2011
- 11. Sort & Shuffle ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , 1) ( , (1,1)) ( , (1,1,1,1)) ( , (1,1)) ( , (1,1)) ( , (1)) Montag, 6. Juni 2011
- 12. reduce() ( , (1,1)) ( , (1,1,1,1)) ( , (1,1)) ( , (1,1)) ( , (1)) ( , 2) ( , 4) ( , 2) ( , 2) ( , 1) Montag, 6. Juni 2011
- 13. Use Case: Recommendations • "People looking as this article also looked at these articles" • "You might also know these people" • iTunes Genius Playlist • Banner Placement Montag, 6. Juni 2011
- 14. Use Case: Text Processing • Document Indexing • Semantic Analytics Montag, 6. Juni 2011
- 15. Use Case: Machine Learning Data Data Data • Spam vs No Spam • Credit Card Fraud • "People of Interest" Information Montag, 6. Juni 2011
- 16. Use Case: Graphs • Shortest Paths • Bottleneck Nodes • Flow Optimization • Spanning Trees • Spanning Routes Montag, 6. Juni 2011
- 17. Hadoop Ecosystem Hive & Pig High Level Language Access HBase Real Time Access Sqoop SQL to/from Hadoop Flume Distributed Data Collection Oozie Job Workflow Mahout Machine Learning Library many more Montag, 6. Juni 2011
- 18. Homework • Cloudera's Distribution including Hadoop (CDH3) • Online Tutorials • WordCount Example • Conference Demo Cluster Montag, 6. Juni 2011
- 19. Quick Demo! Montag, 6.Juni 2011
- 20. Thank you! • Kai Voigt • kai@cloudera.com • http://www.cloudera.com/ • http://apache.hadoop.org/ Montag, 6. Juni 2011