Hadoop and Pig at Twitter__HadoopSummit2010

•

25 gefällt mir•2,429 views

Yahoo Developer Network

Hadoop Summit 2010 - Developers Track Hadoop and Pig at Twitter Kevin Weil, Twitter

Technologie

Hadoop at Twitter ,[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object],1 Community Open Source 2 Twitter Open Source (or soon)

My Background ,[object Object],[object Object],[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object],1 Community Open Source 2 Twitter Open Source

What Data? ,[object Object],[object Object],[object Object]

Logs ,[object Object],[object Object],[object Object],[object Object]

Scribe ,[object Object],[object Object],[object Object],[object Object],FE FE FE Agg Agg HDFS File

Scribe at Twitter ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Scribe at Twitter ,[object Object],[object Object],[object Object],[object Object],[object Object],1 http://github.com/traviscrawford/scribe

Tabular Data ,[object Object],[object Object],[object Object],[object Object],[object Object]

Crane Driver Configuration/Batch Management Extract Load Transform Protobuf P1 Protobuf P2 Source Sink ZooKeeper Registration

Crane ,[object Object],[object Object],[object Object],[object Object]

Crane ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Crane ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Storage Basics ,[object Object],[object Object],[object Object],[object Object],[object Object],1 http://www.github.com/kevinweil/hadoop-lzo

Elephant Bird http://www.flickr.com/photos/jagadish/3072134867/ 1 http://github.com/kevinweil/elephant-bird

Elephant Bird ,[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object],[object Object]

HBase ,[object Object],[object Object],[object Object]

HBase at Twitter ,[object Object],[object Object],[object Object],[object Object]

Enter Pig ,[object Object],[object Object],[object Object],[object Object],[object Object]

Pig Democratizes Large-scale Data Analysis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Pig Examples ,[object Object],[object Object]

Pig Workflow ,[object Object],[object Object],[object Object],[object Object],1 http://github.com/ningliang/piglet

Counting Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Correlating Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Research on Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Research on Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Products ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Questions ? Follow me at twitter.com/kevinweil TM ,[object Object]

Weitere ähnliche Inhalte

Was ist angesagt?

Pig workshopSudar Muthu

Apache Pig: A big data processorTushar B Kute

apache pig performance optimizations talk at apachecon 2010Thejas Nair

Nov HUG 2009: Hadoop Record Reader In PythonYahoo Developer Network

Practical pigtrihug

January 2011 HUG: Howl PresentationYahoo Developer Network

Hands on Hadoop and pigSudar Muthu

Karmasphere hadoop-productivity-toolsHadoop User Group

Apache PIGPrashant Gupta

03 pig introSubhas Kumar Ghosh

Pig and Pig Latin - Module 5Rohit Agrawal

Practical Hadoop using PigDavid Wellman

TriHUG November Pig Talk by Alan Gatestrihug

Apache Pig: Making data transformation easyVictor Sanchez Anguix

Introduction to PigPrashanth Babu

Apache spark sessionknowbigdata

Apache pigSadiq Basha

Hive integration: HBase and Rcfile__HadoopSummit2010Yahoo Developer Network

An Introduction to Apache PigSachin Vakkund

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Alex Levenson

Was ist angesagt? (20)

Pig workshop

Apache Pig: A big data processor

apache pig performance optimizations talk at apachecon 2010

Nov HUG 2009: Hadoop Record Reader In Python

Practical pig

January 2011 HUG: Howl Presentation

Hands on Hadoop and pig

Karmasphere hadoop-productivity-tools

Apache PIG

03 pig intro

Pig and Pig Latin - Module 5

Practical Hadoop using Pig

TriHUG November Pig Talk by Alan Gates

Apache Pig: Making data transformation easy

Introduction to Pig

Apache spark session

Apache pig

Hive integration: HBase and Rcfile__HadoopSummit2010

An Introduction to Apache Pig

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...

Andere mochten auch

Hadoop Pig: MapReduce the easy way!Nathan Bijnens

Hadoop and pig at twitter (oscon 2010)Kevin Weil

Un introduction à PigModern Data Stack France

Apache pigMudassir Khan Pathan

Pig statementsGanesh Sanap

Hadoop, Pig, and Python (PyData NYC 2012)mortardata

Hadoop Ecosystem at Twitter - Kevin Weil - Hadoop World 2010Cloudera, Inc.

High-level Programming Languages: Apache Pig and Pig LatinPietro Michiardi

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...Hadoop User Group

Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | EdurekaEdureka!

introduction to data processing using Hadoop and PigRicardo Varela

Practical Problem Solving with Apache Hadoop & PigMilind Bhandarkar

Hadoop, Pig, and Twitter (NoSQL East 2009)Kevin Weil

Hadoop et son écosystèmeKhanh Maudoux

Big Data: Concepts, techniques et démonstration de Apache Hadoophajlaoui jaleleddine

Pig and Python to Process Big DataShawn Hermans

Solr + Hadoop - Fouillez facilement dans votre système Big Datafrancelabs

Apache Cassandra - Concepts et fonctionnalitésRomain Hardouin

Andere mochten auch (18)

Hadoop Pig: MapReduce the easy way!

Hadoop and pig at twitter (oscon 2010)

Un introduction à Pig

Apache pig

Pig statements

Hadoop, Pig, and Python (PyData NYC 2012)

Hadoop Ecosystem at Twitter - Kevin Weil - Hadoop World 2010

High-level Programming Languages: Apache Pig and Pig Latin

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...

Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka

introduction to data processing using Hadoop and Pig

Practical Problem Solving with Apache Hadoop & Pig

Hadoop, Pig, and Twitter (NoSQL East 2009)

Hadoop et son écosystème

Big Data: Concepts, techniques et démonstration de Apache Hadoop

Pig and Python to Process Big Data

Solr + Hadoop - Fouillez facilement dans votre système Big Data

Apache Cassandra - Concepts et fonctionnalités

Ähnlich wie Hadoop and Pig at Twitter__HadoopSummit2010

Hadoop at Twitter (Hadoop Summit 2010)Kevin Weil

Data Infrastructure for a World of MusicLars Albertsson

Eric Baldeschwieler Keynote from Storage Developers ConferenceHortonworks

Hadoop basicsAntonio Silveira

Hive @ Hadoop day seattle_2010nzhang

Data infrastructure at Facebook AhmedDoukh

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Big data and APIs for PHP developers - SXSW 2011Eli White

Finding the needles in the haystack. An Overview of Analyzing Big Data with H...Chris Baglieri

Bigdata and Hadoop BootcampSpotle.ai

Overview of big data & hadoop version 1 - Tony NguyenThanh Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1Thanh Nguyen

Large Scale Data With Hadoopguest27e6764

Big data-at-detikk4ndar

How Hadoop Revolutionized Data Warehousing at Yahoo and FacebookAmr Awadallah

Hadoop Frameworks Panel__HadoopSummit2010Yahoo Developer Network

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune amrutupre

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...Cloudera, Inc.

Architecting the Future of Big Data and SearchHortonworks

Big data or big dealeduarderwee

Ähnlich wie Hadoop and Pig at Twitter__HadoopSummit2010 (20)

Hadoop at Twitter (Hadoop Summit 2010)

Data Infrastructure for a World of Music

Eric Baldeschwieler Keynote from Storage Developers Conference

Hadoop basics

Hive @ Hadoop day seattle_2010

Data infrastructure at Facebook

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...

Big data and APIs for PHP developers - SXSW 2011

Finding the needles in the haystack. An Overview of Analyzing Big Data with H...

Bigdata and Hadoop Bootcamp

Overview of big data & hadoop version 1 - Tony Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1

Large Scale Data With Hadoop

Big data-at-detik

How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook

Hadoop Frameworks Panel__HadoopSummit2010

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...

Architecting the Future of Big Data and Search

Big data or big deal

Mehr von Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Mehr von Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Kürzlich hochgeladen

Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada

Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma

Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos

Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh

Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani

Decarbonising Buildings: Making a net-zero built environment a realityIES VE

Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica

Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

QCon London: Mastering long-running processes in modern architecturesBernd Ruecker

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3

Top 10 Hubspot Development Companies in 2024TopCSSGallery

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes

Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen

Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers

A Glance At The Java Performance ToolboxAna-Maria Mihalceanu

Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar

Kürzlich hochgeladen (20)

Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...

Microservices, Docker deploy and Microservices source code in C#

Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)

Generative AI - Gitex v1Generative AI - Gitex v1.pptx

Potential of AI (Generative AI) in Business: Learnings and Insights

Decarbonising Buildings: Making a net-zero built environment a reality

Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...

Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger

[Webinar] SpiraTest - Setting New Standards in Quality Assurance

QCon London: Mastering long-running processes in modern architectures

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx

Top 10 Hubspot Development Companies in 2024

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

Testing tools and AI - ideas what to try with some tool examples

Design pattern talk by Kaya Weers - 2024 (v2)

A Glance At The Java Performance Toolbox

Long journey of Ruby standard library at RubyConf AU 2024

The State of Passkeys with FIDO Alliance.pptx

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes