Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Big data ppt

  1. 1. Introduction toBIG DATA Thiru
  2. 2. What is BIGDATA? http://www.forbes.com/sites/oreillymedia/2012/01/19/volume-velocity- variety-what-you-need-to-know-about-big-data
  3. 3.  Search EngineData Scalability  10KB / doc * 20B docs = 200TBProblems  Reindex every 30 days: 200TB/30days = 6 TB/day  Log Processing / Data Warehousing  0.5KB/events * 3B pageview events/day = 1.5TB/day  100M users * 5 events * 100 feed/event * 0.1KB/feed = 5TB/day  Multipliers: 3 copies of data, 3-10 passes of raw data  Processing Speed (Single Machine)  2-20MB/second * 100K seconds/day = 0.2-2 TB/day
  4. 4. What’s the social sentiment How do I better predict futurefor my brand or products outcomes? How do I optimize my fleet based on weather and traffic patterns?
  5. 5. Traditional E-Commerce Data Flow
  6. 6. New E-Commerce Big Data Flow
  7. 7. Introduction to Hadoop
  8. 8. Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which mayHADOOP be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.
  9. 9. HadoopHistory Jan 2006 – Doug Cutting joins Yahoo Feb 2006 – Hadoop splits out of Nutch and Yahoo starts using it. Dec 2006 –Yahoo creating 100-node Webmap with Hadoop Apr 2007 –Yahoo on 1000-node cluster Jan 2008 – Hadoop made a top-level Apache project Dec 2007 –Yahoo creating 1000-node Webmap with Hadoop Sep 2008 – Hive added to Hadoop as a contrib project
  10. 10. • Commodity hardwareBIG DATA compatibility ECONOMICS • Reduction in storage costEconomics • Open source system • The Web economy
  11. 11. Column Store Database
  12. 12. Row Store andColumn Store
  13. 13.  Can be significantly faster than row stores for some applications  Fetch only required columns for a query  Better cache effects  Better compression (similar attribute values within a column)Why Column  But can be slower for other applications  OLTP with many row inserts, ..Store?  Long war between the column store and row store camps :-)
  14. 14. So How Does It Work?
  15. 15. So How Does It Work?
  16. 16. The Hadoop Ecosystem
  17. 17. Traditional RDBMS vs. MapReduceComparisons
  18. 18. HDFS, the storage layer of Hadoop, is a distributed, scalable, Java-based file system adept at storing large volumes of unstructured data. MapReduce is a software framework that serves as the compute layer of Hadoop. MapReduce jobs are divided into two (obviously named) parts. The “Map” function divides a query into multiple parts and processes data at the node level. The “Reduce” function aggregates the results of the “Map” functionHadoop to determine the “answer” to the query.Ecosystem Hive is a Hadoop-based data warehouse developed by Facebook. It allows users to write queries in SQL, which are then converted to MapReduce. This allows SQL programmers with no MapReduce experience to use the warehouse and makes it easier to integrate with business intelligence and visualization tools such as Microstrategy, Tableau, Revolutions Analytics, etc. Pig Latin is a Hadoop-based language developed by Yahoo. It is relatively easy to learn and is adept at very deep, very long data pipelines (a limitation of SQL.)
  19. 19. HBase is a non-relational database that allows for low-latency, quick lookups in Hadoop. It adds transactional capabilities to Hadoop, allowing users to conduct updates, inserts and deletes. EBay and Facebook use HBase heavily. . Flume is a framework for populating Hadoop with data. Agents are populated throughout ones IT infrastructure – inside web servers, application servers and mobile devices, for example – to collect data and integrate it into Hadoop.HadoopEcosystem Oozie is a workflow processing system that lets users define a series of jobs written in multiple languages – such as Map Reduce, Pig and Hive -- then intelligently link them to one another. Oozie allows users to specify, for example, that a particular query is only to be initiated after specified previous jobs on which it relies for data are completed Whirr is a set of libraries that allows users to easily spin-up Hadoop clusters on top of Amazon EC2, Rackspace or any virtual infrastructure. It supports all major virtualized infrastructure vendors on the market.
  20. 20. Avro is a data serialization system that allows for encoding the schema of Hadoop files. It is adept at parsing data and performing removed procedure calls Mahout is a data mining library. It takes the most popular data mining algorithms for performing clustering, regression testing and statistical modeling and implements them using the Map Reduce modelHadoopEcosystem Sqoop is a connectivity tool for moving data from non-Hadoop data stores – such as relational databases and data warehouses – into Hadoop. It allows users to specify the target location inside of Hadoop and instruct Sqoop to move data from Oracle, Teradata or other relational databases to the target. BigTop is an effort to create a more formal process or framework for packaging and interoperability testing of Hadoops sub-projects and related components with the goal improving the Hadoop platform as a whole.
  21. 21. Microsoft & Hadoop
  22. 22. Insights to allusers byactivating newtypes of data
  23. 23. Microsoft BI
  24. 24. Stats Machine Legend Graph Pipeline / Workflow processing Learning (Pegasus) Red = Core Hadoop (RHadoop) (Mahout) Blue = Data (Oozie) Metadata processing (HCatalog) Purple = Microsoft ( ODBC / SQOOP/ REST) integration points Scripting Query Data Integration NoSQL Database and value adds (Pig) (Hive) Yellow = DataMicrosoft (HBase) Movement Distributed Processing Event PipelineHadoop Stack (MapReduce) Green = Packages (Flume) White = Coming Soon Distributed Storage (HDFS) Monitoring & Active Directory Deployment (Security) (System Center)
  25. 25. Others
  26. 26. HadoopCommercialDistributors
  27. 27. Other Big DataWorlds
  28. 28. Other Big DataWorlds
  29. 29. Big DataIntegrations,Visualizations& Analytics
  30. 30. Thank You
  • Sakshisohane1

    Jun. 25, 2021
  • solomonNegasa

    Jun. 10, 2021
  • RajPatel948

    Apr. 13, 2021
  • SwatiBogalkar

    Feb. 2, 2021
  • AnithaKrishnan5

    Jan. 31, 2021
  • HuddaSoni

    Sep. 30, 2020
  • anikhas

    Jan. 11, 2020
  • MonaliSaste

    Oct. 15, 2019
  • AinagulBekezhanova

    Sep. 2, 2019
  • HebaEssam11

    Sep. 2, 2019
  • HaiLinLu

    Jul. 11, 2019
  • vishwamaniya

    Mar. 31, 2019
  • KyawKyaw56

    Dec. 27, 2018
  • ahmedsaree3

    Dec. 18, 2018
  • vasuntanwandee7

    Sep. 25, 2018
  • ssuser68bcf6

    Sep. 13, 2018
  • bss79

    Aug. 4, 2018
  • ama0017

    Jul. 31, 2018
  • SibonginkosiSShongwe

    May. 24, 2018
  • JOYROY29

    May. 1, 2018


Aufrufe insgesamt


Auf Slideshare


Aus Einbettungen


Anzahl der Einbettungen