Hw09 Hadoop Development At Facebook Hive And Hdfs
•Als PPT, PDF herunterladen•
10 gefällt mir•6,290 views
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Hw09 Hadoop Development At Facebook Hive And Hdfs
Ähnlich wie Hw09 Hadoop Development At Facebook Hive And Hdfs (20)
Mehr von Cloudera, Inc.
Mehr von Cloudera, Inc. (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Hw09 Hadoop Development At Facebook Hive And Hdfs
- 1. Hadoop and Hive Development at Facebook Dhruba Borthakur Zheng Shao {dhruba, zshao}@facebook.com Presented at Hadoop World, New York October 2, 2009
- 6. Data Flow into Hadoop Cloud Web Servers Scribe MidTier Network Storage and Servers Hadoop Hive Warehouse Oracle RAC MySQL
- 7. Hadoop Scribe: Avoid Costly Filers Web Servers Scribe Writers Realtime Hadoop Cluster Hadoop Hive Warehouse Oracle RAC MySQL Scribe MidTier http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html
- 9. Archival: Move old data to cheap storage Cheap NAS Hadoop Archival Cluster Hadoop Archive Node NFS Hive Query Hadoop Warehouse http://issues.apache.org/jira/browse/HDFS-220
- 13. Hive Architecture HDFS Map Reduce Web UI + Hive CLI + JDBC/ODBC Browse, Query, DDL Hive QL Parser Planner Optimizer Execution SerDe CSV Thrift Regex UDF/UDAF substr sum average FileFormats TextFile SequenceFile RCFile User-defined Map-reduce Scripts
Hinweis der Redaktion
- 1GB connectivity within a rack, 100MB across racks? Are all disks 7200 SATA?
- This is the architecture of our backend data warehouing system. This system provides important information on the usage of our website, including but not limited to the number page views of each page, the number of active users in each country, etc. We generate 4TB of compressed log data every day. All these data are stored and processed by the hadoop cluster which consists of over 600 machines. The summary of the log data is then copied to Oracle and MySQL databases, to make sure it is easy for people to access.
- The Network attached storage is replaced by a hadoop cluster. This hadoop cluster has real-time characteristics and is configured as failfast. ThebBenefits are no storage fragmentation and easier to manage. Overlapped clusters for HA.
- The Network attached storage is replaced by a hadoop cluster. This hadoop cluster has real-time characteristics and is configured as failfast. ThebBenefits are no storage fragmentation and easier to manage.