Hadoop vs. RDBMS for Advanced Analytics

•

8 gefällt mir•3,955 views

joshwills

Technologie Bildung

About Me

• jwills@cloudera.com
• Formerly of Google (2008 – 2011)
• Worked on the ad auction
• Led the team that build the data infrastructure for Google+
• Before that: a bunch of startups
• Sometimes as a software engineer, sometimes as a statistician
• Math degree from Duke and a half-finished PhD from The
University of Texas at Austin
• Now: Director of Data Science at Cloudera

Copyright 2012 Cloudera Inc. All rights reserved

Getting Started with Hadoop: Apache Hive

• Stick with the relational
models that you are
used to working with

• Great for the common
starter use cases
• Logs processing
• Online data archival
• ETL/ELT

Copyright 2012 Cloudera Inc. All rights reserved

Hadoop for Advanced Analytics

When Should I Use Hadoop instead of an RDBMS?

Copyright 2012 Cloudera Inc. All rights reserved

First Symptom: COUNT DISTINCT

Copyright 2012 Cloudera Inc. All rights reserved

Second Symptom: Cursors

Copyright 2012 Cloudera Inc. All rights reserved

Third Symptom: ALTER TABLE OF_DOOM

Copyright 2012 Cloudera Inc. All rights reserved

The Unit of Analysis Problem

• Data warehouses are
optimized to analyze
transactions
• Awesome for finance
and ERP
• Not ideal for product
and marketing
• A function of what
databases are good at

Copyright 2012 Cloudera Inc. All rights reserved

What Are You Trying to Analyze?

Simple Entities Complex Entities
• Static attributes • Evolving attributes
• Flat data structure • Hierarchical data structure
• Transient • Persistent
• Examples • Examples
• SKUs • Customers
• Line items from an invoice • Suppliers
• Log messages • Website visitors

Copyright 2011 Cloudera Inc. All rights reserved

Rods and Cones vs. Facial Recognition

Copyright 2012 Cloudera Inc. All rights reserved

Structure the Data to Fit the Problem

• HDFS Lets Us Store Our
Data However We Want
• We can choose storage
schemas that are:
• Flexible
• Evolvable
• Compact
• Fast
serialization/deserializati
on

Copyright 2012 Cloudera Inc. All rights reserved

Advaned Analytics: Use Cases

Copyright 2012 Cloudera Inc. All rights reserved

Simple Counts on Complex Objects

Copyright 2012 Cloudera Inc. All rights reserved

Self-Self-Self-Joins

Copyright 2012 Cloudera Inc. All rights reserved

Matching Problems

Copyright 2012 Cloudera Inc. All rights reserved

Empfohlen

Hadoop and Hive in Enterprisesmarkgrover

Schema-on-Read vs Schema-on-WriteAmr Awadallah

SQL-on-Hadoop TutorialDaniel Abadi

Big Data Architecture Workshop - Vahid Amiridatastack

Big Data IntroductionDurga Gadiraju

Big data hadoop rdbmsArjen de Vries

Big Data in the Real WorldMark Kromer

Introduction to Kudu - StampedeCon 2016StampedeCon

Empfohlen

Hadoop and Hive in Enterprisesmarkgrover

Schema-on-Read vs Schema-on-WriteAmr Awadallah

SQL-on-Hadoop TutorialDaniel Abadi

Big Data Architecture Workshop - Vahid Amiridatastack

Big Data IntroductionDurga Gadiraju

Big data hadoop rdbmsArjen de Vries

Big Data in the Real WorldMark Kromer

Introduction to Kudu - StampedeCon 2016StampedeCon

Innovation in the Data Warehouse - StampedeCon 2016StampedeCon

What is hadoopAsis Mohanty

Introduction to PolyBaseJames Serra

Data lake-itweekend-sharif university-vahid amirydatastack

Boston Hadoop Meetup, April 26 2012Daniel Abadi

Integrating hadoop - Big Data TechCon 2013Jonathan Seidman

Big data architecture on cloud computing infrastructuredatastack

Big data vahidamiri-datastack.irdatastack

Hd insight essentials quick viewRajesh Nadipalli

Introduction to Apache Hadoop Eco-SystemMd. Hasan Basri (Angel)

Big Data Platforms: An OverviewC. Scyphers

Integrated Data Warehouse with Hadoop and Oracle DatabaseGwen (Chen) Shapira

Planing and optimizing data lake architectureMilos Milovanovic

Data warehousing with Hadoophadooparchbook

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...The Hive

Data lake – On Premise VS CloudIdan Tohami

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"DataConf

Introduction to HadoopDr. C.V. Suresh Babu

Running cost effective big data workloads with Azure Synapse and Azure Data L...Michael Rys

Bi on Big Data - Strata 2016 in LondonDremio Corporation

Hadoop World 2011: Hadoop vs. RDBMS for Big Data Analytics...Why Choose?Cloudera, Inc.

Relational databases vs Non-relational databasesJames Serra

Weitere ähnliche Inhalte

Was ist angesagt?

Innovation in the Data Warehouse - StampedeCon 2016StampedeCon

What is hadoopAsis Mohanty

Introduction to PolyBaseJames Serra

Data lake-itweekend-sharif university-vahid amirydatastack

Boston Hadoop Meetup, April 26 2012Daniel Abadi

Integrating hadoop - Big Data TechCon 2013Jonathan Seidman

Big data architecture on cloud computing infrastructuredatastack

Big data vahidamiri-datastack.irdatastack

Hd insight essentials quick viewRajesh Nadipalli

Introduction to Apache Hadoop Eco-SystemMd. Hasan Basri (Angel)

Big Data Platforms: An OverviewC. Scyphers

Integrated Data Warehouse with Hadoop and Oracle DatabaseGwen (Chen) Shapira

Planing and optimizing data lake architectureMilos Milovanovic

Data warehousing with Hadoophadooparchbook

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...The Hive

Data lake – On Premise VS CloudIdan Tohami

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"DataConf

Introduction to HadoopDr. C.V. Suresh Babu

Running cost effective big data workloads with Azure Synapse and Azure Data L...Michael Rys

Bi on Big Data - Strata 2016 in LondonDremio Corporation

Was ist angesagt? (20)

Innovation in the Data Warehouse - StampedeCon 2016

What is hadoop

Introduction to PolyBase

Data lake-itweekend-sharif university-vahid amiry

Boston Hadoop Meetup, April 26 2012

Integrating hadoop - Big Data TechCon 2013

Big data architecture on cloud computing infrastructure

Big data vahidamiri-datastack.ir

Hd insight essentials quick view

Introduction to Apache Hadoop Eco-System

Big Data Platforms: An Overview

Integrated Data Warehouse with Hadoop and Oracle Database

Planing and optimizing data lake architecture

Data warehousing with Hadoop

The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...

Data lake – On Premise VS Cloud

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"

Introduction to Hadoop

Running cost effective big data workloads with Azure Synapse and Azure Data L...

Bi on Big Data - Strata 2016 in London

Andere mochten auch

Hadoop World 2011: Hadoop vs. RDBMS for Big Data Analytics...Why Choose?Cloudera, Inc.

Relational databases vs Non-relational databasesJames Serra

Big data, Hadoop, NoSQL DB - introductionkvaderlipa

Don't be Hadooped when looking for Big Data ROIDataWorks Summit

Big data meets big analyticsDeepak Ramanathan

BDSA Solutions Comparison sheetKanikkannan L

Distributed RDBMS: Challenges, Solutions & Trade-offsAhmed Magdy Ezzeldin, MSc.

Big Data Course - BigData HUBAhmed Salman

Co-existence or competition - RDBMS and HadoopFlytxt

Life history of frogbhargavnarayana

Abc ile kosztuje ogrzewanie pompa cieplaabc-kotly

Abc jak zbudowany jest kolektor slonecznyabc-kotly

Compuandresjae10

Javascript4mozks

Protección de las mujeres contra la violencia de genero en la argentinaMaría Isabel Sanchez

La tecnologia esperanzadoraMaría Isabel Sanchez

Papi 2008 buku 2citra Joni

Barriers to insulin therapyDr Joozer Rangwala

Abc czy potrzebna mi wentylacja mechaniczna z odzyskiem cieplaabc-kotly

Abc jakie grzejniki dla kotla kondensacyjnegoabc-kotly

Andere mochten auch (20)

Hadoop World 2011: Hadoop vs. RDBMS for Big Data Analytics...Why Choose?

Relational databases vs Non-relational databases

Big data, Hadoop, NoSQL DB - introduction

Don't be Hadooped when looking for Big Data ROI

Big data meets big analytics

BDSA Solutions Comparison sheet

Distributed RDBMS: Challenges, Solutions & Trade-offs

Big Data Course - BigData HUB

Co-existence or competition - RDBMS and Hadoop

Life history of frog

Abc ile kosztuje ogrzewanie pompa ciepla

Abc jak zbudowany jest kolektor sloneczny

Compu

Javascript4

Protección de las mujeres contra la violencia de genero en la argentina

La tecnologia esperanzadora

Papi 2008 buku 2

Barriers to insulin therapy

Abc czy potrzebna mi wentylacja mechaniczna z odzyskiem ciepla

Abc jakie grzejniki dla kotla kondensacyjnego

Ähnlich wie Hadoop vs. RDBMS for Advanced Analytics

Hadoop and Machine Learningjoshwills

Machine Learning and Hadoop: Present and futureCloudera, Inc.

Building a Modern Analytic Database with Cloudera 5.8Cloudera, Inc.

MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, ClouderaMongoDB

Hadoop and the Data Warehouse: Point/Counter PointInside Analysis

Machine Learning and Hadoop: Present and FutureData Science London

Hadoop Essentials -- The What, Why and How to Meet Agency ObjectivesCloudera, Inc.

Architecting a datalakeLaurent Leturgez

Big Data Use CasesInSemble

Beyond TCODataWorks Summit/Hadoop Summit

Options for Data Prep - A Survey of the Current MarketDremio Corporation

Enterprise Architecture in the Era of Big Data and Quantum ComputingKnowledgent

Hadoop and the Data Warehouse: When to Use Which DataWorks Summit

Simplifying Real-Time Architectures for IoT with Apache KuduCloudera, Inc.

Choosing the Right Big Data Tools for the Job - A Polyglot ApproachDATAVERSITY

Retail & CPGTata Consultancy Services

Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017StampedeCon

How to use Big Data and Data Lake concept in business using Hadoop and Spark...Institute of Contemporary Sciences

Building great dashboards by Mrunal Shridhar - Tableau Customer Conference 2012Mrunal Shridhar

2012 Tableau Customer Conference - Building Great Dashboards by Mrunal ShridharMrunal Shridhar

Ähnlich wie Hadoop vs. RDBMS for Advanced Analytics (20)

Hadoop and Machine Learning

Machine Learning and Hadoop: Present and future

Building a Modern Analytic Database with Cloudera 5.8

MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera

Hadoop and the Data Warehouse: Point/Counter Point

Machine Learning and Hadoop: Present and Future

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

Architecting a datalake

Big Data Use Cases

Beyond TCO

Options for Data Prep - A Survey of the Current Market

Enterprise Architecture in the Era of Big Data and Quantum Computing

Hadoop and the Data Warehouse: When to Use Which

Simplifying Real-Time Architectures for IoT with Apache Kudu

Choosing the Right Big Data Tools for the Job - A Polyglot Approach

Retail & CPG

Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

Building great dashboards by Mrunal Shridhar - Tableau Customer Conference 2012

2012 Tableau Customer Conference - Building Great Dashboards by Mrunal Shridhar

Kürzlich hochgeladen

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

Slack Application Development 101 Slidespraypatel2

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4

Understanding the Laravel MVC ArchitecturePixlogix Infotech

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

Pigging Solutions Piggable Sweeping ElbowsPigging Solutions

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

Pigging Solutions in Pet Food ManufacturingPigging Solutions

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Kürzlich hochgeladen (20)

[2024]Digital Global Overview Report 2024 Meltwater.pdf

IAC 2024 - IA Fast Track to Search Focused AI Solutions

SIEMENS: RAPUNZEL – A Tale About Knowledge Graph

Handwritten Text Recognition for manuscripts and early printed texts

Slack Application Development 101 Slides

Scaling API-first – The story of a global engineering organization

Azure Monitor & Application Insight to monitor Infrastructure & Application

Understanding the Laravel MVC Architecture

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Enhancing Worker Digital Experience: A Hands-on Workshop for Partners

Swan(sea) Song – personal research during my six years at Swansea ... and bey...

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation

Unblocking The Main Thread Solving ANRs and Frozen Frames

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service

Pigging Solutions Piggable Sweeping Elbows

Injustice - Developers Among Us (SciFiDevCon 2024)

Pigging Solutions in Pet Food Manufacturing

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

Hadoop vs. RDBMS for Advanced Analytics

1. Hadoop vs. RDBMS for Advanced Analytics Josh Wills April 26th, 2012

2. About Me • jwills@cloudera.com • Formerly of Google (2008 – 2011) • Worked on the ad auction • Led the team that build the data infrastructure for Google+ • Before that: a bunch of startups • Sometimes as a software engineer, sometimes as a statistician • Math degree from Duke and a half-finished PhD from The University of Texas at Austin • Now: Director of Data Science at Cloudera Copyright 2012 Cloudera Inc. All rights reserved

3. Getting Started with Hadoop: Apache Hive • Stick with the relational models that you are used to working with • Great for the common starter use cases • Logs processing • Online data archival • ETL/ELT Copyright 2012 Cloudera Inc. All rights reserved

8. The Unit of Analysis Problem • Data warehouses are optimized to analyze transactions • Awesome for finance and ERP • Not ideal for product and marketing • A function of what databases are good at Copyright 2012 Cloudera Inc. All rights reserved

9. What Are You Trying to Analyze? Simple Entities Complex Entities • Static attributes • Evolving attributes • Flat data structure • Hierarchical data structure • Transient • Persistent • Examples • Examples • SKUs • Customers • Line items from an invoice • Suppliers • Log messages • Website visitors Copyright 2011 Cloudera Inc. All rights reserved

11. Structure the Data to Fit the Problem • HDFS Lets Us Store Our Data However We Want • We can choose storage schemas that are: • Flexible • Evolvable • Compact • Fast serialization/deserializati on Copyright 2012 Cloudera Inc. All rights reserved

16. We’re Hiring. jwills@cloudera.com

Hinweis der Redaktion

How do you know you have a unit of analysis problem? You’re doing a bunch of COUNT DISTINCT queries. You’re doing LAG/LEAD-style queries, or using a cursor.