140614 bigdatacamp-la-keynote-jon hsieh

1. Headline Goes Here Speaker Name or Subhead Goes Here DO NOT USE PUBLICLY PRIOR TO 10/23/12 Evolution of the Big Data Stack Jonathan Hsieh| Tech Lead / Software engineer @ Cloudera BigDataCamp LA ‘14 June 14, 2014

2. Who Am I? • Cloudera since 2009 • Tech Lead HBase Team • Software Engineer • Apache HBase committer / PMC • Apache Flume founder / PMC • U of Washington: • Research in Distributed Systems 6/14/14 BigDataCamp LA '14 - Hsieh2

3. Big Data Stack Evolution •Inspiration •Imitation •Innovation 6/14/14 BigDataCamp LA '14 - Hsieh3

4. Big Data Stack Evolution •Inspiration •Imitation •Innovation 6/14/14 BigDataCamp LA '14 - Hsieh4

5. Emergence of Big Data Inspiration 6/14/14 BigDataCamp LA '14 - Hsieh5

6. 6/14/14 BigDataCamp LA '14 - Hsieh6

7. 6/14/14 BigDataCamp LA '14 - Hsieh7

8. The brute force solution 1. Collect all the data 2. Analyze all the data 3. Serve the results 6/14/14 BigDataCamp LA '14 - Hsieh8

9. End of free MHz coincides with Rise of Big Data 6/14/14 BigDataCamp LA '14 - Hsieh http://cacm.acm.org/magazines/2012/4/147359-cpu-db-recording-microprocessor-history/abstract 9

10. A Move towards Distributed Systems • Scaling Horizontally instead of Vertically • Challenges: • Reliability • Fault tolerance • Atomicity / Consistency / Isolation / Durability • High-Availability • Latency Predictability 6/14/14 BigDataCamp LA '14 - Hsieh10

11. Google built a Big Data Stack Sawzall MapReduce GFS 6/14/14 BigDataCamp LA '14 - Hsieh11

12. Google built a Big Data Stack Sawzall MapReduce MySql Gateway Big Table GFS Chubby Evenflow Protobufs 6/14/14 BigDataCamp LA '14 - Hsieh12

13. The core of a Big Data Stack • . Query Processing Data Integration Fast Read / Write access File System Distributed Coordination Workflow and Scheduling Metadata 6/14/14 BigDataCamp LA '14 - Hsieh13

14. Big Data for the rest of us Imitation 6/14/14 BigDataCamp LA '14 - Hsieh14

15. 6/14/14 BigDataCamp LA '14 - Hsieh15

16. The core of a Hadoop stack Query Processing Data Integration Fast Read / Write access File System Distributed Coordination Workflow and Scheduling Metadata 6/14/14 BigDataCamp LA '14 - Hsieh16

17. built a Big Data stack • Donated Hadoop + Friends to the Apache Software Foundation Pig / Hive HadoopData Highway* HBase HDFS ZooKeeper Oozie Hive 6/14/14 BigDataCamp LA '14 - Hsieh17

18. Parallel Components 6/14/14 BigDataCamp LA '14 - Hsieh Function Google Yahoo! Facebook The Rest of Us File system GFS => Colossus HDFS HDFS HDFS Low latency Data store (NoSQL) BigTable => Megastore => Spanner PNUTS => Hbase HBase Hbase Batch processing Google MapReduce Hadoop MapReduce Hadoop MapReduce Hadoop MapReduce Spark Batch query Sawzall, Tenzing, FlumeJava Pig Hive Pig, Hive, Impala, Drill, Crunch Resource Management Borg => Omega => YARN => Corona YARN Mesos Ingest EvenFlow Custom MySQL Proxy Custom Scribe / Calligraphus Custom proxy Sqoop Flume Kafka Coordination Chubby Zookeeper Zookeeper Zookeeper Graph Processing Pregel Giraph Giraph, Golden orb Hama, Titan Stream processing MillWheel S3 => Storm Puma/PTail Storm, Spark 18

19. Simplify and remove features to enable scaling • Scalable and simple first • Focus only on needed features. Exclude others. • Re-add them later. • Ex: NoSQL • No transactions • No Schema 6/14/14 BigDataCamp LA '14 - Hsieh19

20. Big Data industry steps up Innovation 6/14/14 BigDataCamp LA '14 - Hsieh20

21. Nov ’06: Google BigTable, Chubby OSDI ‘06 Mar’10: Cloudera Founded Big Data Stack Timeline 6/14/14 BigDataCamp LA '14 - Hsieh 20142006 2007 2008 2009 2010 2011 20132012 Apr’11: CDH3 GA with HBase, Flume, Sqoop, Oozie Feb’12: CDH4 GA with HDFS NN HA, and YARN preview Mar’10: CDH2 GA with CM (manager) 2009: CDH1 GA (first hadoop distro) Mar ’04: Google MapReduce OSDI ‘04 Oct ’03: Google GFS SOSP ‘03 2008: Google Tenzing Pub (VLDB’11) 2008: Facebook Hive ICMD ‘08: Pig Latin 21

22. Nov ’06: Google BigTable, Chubby OSDI ‘06 Mar’10: Cloudera Founded Big Data Stack Timeline 6/14/14 BigDataCamp LA '14 - Hsieh 20142006 2007 2008 2009 2010 2011 20132012 Apr’11: CDH3 GA with HBase, Flume, Sqoop, Oozie Feb’12: CDH4 GA with HDFS NN HA, and YARN preview Mar’10: CDH2 GA with CM (manager) 2009: CDH1 GA (first hadoop distro) Apr’14: CDH5 GA with Impala, Spark, Solr, Navigator Mar ’04: Google MapReduce OSDI ‘04 Oct ’03: Google GFS SOSP ‘03 2008: Google Tenzing Pub (VLDB’11) 2008: Google Spanner OSDI ‘12 2008: Facebook Hive 2014: Facebook discusses HydraBase ICMD ‘08: Pig Latin 2011: Google Megastore CIDR ‘11 2010: Google Percolator OSDI’10 22

23. Usability 6/14/14 BigDataCamp LA '14 - Hsieh23

24. Security + Integration 6/14/14 BigDataCamp LA '14 - Hsieh24

25. New directions 6/14/14 BigDataCamp LA '14 - Hsieh oryx 25

26. 6/14/14 BigDataCamp LA '14 - Hsieh26

27. Thanks! @jmhsieh 6/14/14 BigDataCamp LA '14 - Hsieh27