Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

•

8 gefällt mir•1,482 views

Hadoop Summit 2010 - Research Track Efficient Parallel Set-Similarity Joins Using Hadoop Chen Li, University of California, Irvine

Technologie Unterhaltung & Humor

Efficient Parallel Set-Similarity Joins Using Hadoop ,[object Object],Joint work with Michael Carey and Rares Vernica

Motivation: Data Cleaning Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime Find movies starring Tom Hanks

Movies starring S..warz…ne…ger? Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

Similarity Search Find movies with a star “ similar to ” Schwarrzenger . Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Samuel Jackson Iron man 2008 Sci-Fi Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

Record linkage Table R Table S Star Keanu Reeves Samuel Jackson Schwarzenegger … Star Keanu Reeves Samuel L. Jackson Schwarzenegger …

Two-step solution Table R Table S Step 2: Verification Star … Star … Step 1: Similarity Join

[object Object],[object Object],[object Object],[object Object],Focus of this talk

[object Object],[object Object],[object Object],[object Object],Talk Outline

Set-Similarity Join Finding pairs of records with a similarity on their join attributes > t

Why this formulation? ,[object Object],[object Object],“ Samuel L. Jackson”  {Samuel, L., Jackson} “ Samuel Jackson”  {Samuel, Jackson} S c h w a r z e n e g g e r

Set-similarity functions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Talk Outline

[object Object],[object Object],[object Object],[object Object],[object Object],Why Hadoop?

[object Object],A naïve solution ,[object Object],[object Object],[object Object]

Solving frequency skew: prefix filtering ,[object Object],[object Object],[object Object],prefix r1 r2 Sorted by frequency Chaudhuri, Ganti, Kaushik: A Primitive Operator for Similarity Joins in Data Cleaning. ICDE 2006: 5

Prefix filtering: example ,[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Hadoop Solution: Overview

Stage 1: Sort tokens by frequency ,[object Object],[object Object],MapReduce phase 1 MapReduce phase 2

Stage 2: Find “similar” id pairs ,[object Object],[object Object]

Stage 3: id pairs  record pairs (phase 1) ,[object Object]

Stage 3: id pairs  record pairs (phase 2) ,[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Experimental Setting

Running time ,[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Additional results

[object Object],[object Object],[object Object],Summary

Thank you ,[object Object],[object Object],[object Object],Acknowledgements: NSF, Google, IBM.

Weitere ähnliche Inhalte

Ähnlich wie Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Deduplication on large amounts of codesource{d}

Tokenamooool2000

Tg noh jeju_workshopTae-Gil Noh

Class 9: Consistent HashingDavid Evans

R language introductionShashwat Shriparv

Hands on Mahout!OSCON Byrum

Dcn 20170823 yjy재연 윤

(Talk in Powerpoint Format)butest

Lambdas: Myths and MistakesRichardWarburton

Language Technology Enhanced Learningtelss09

PyGotham NY 2017: Natural Language Processing from ScratchNoemi Derzsy

GhostJhih-Ming Chen

R Analytics in the CloudDataMine Lab

Stack_Overflow-Network_GraphYaopeng (Gyoho) Wu

Clusterguest1babda

P3 2018 python_regexesProf. Wim Van Criekinge

Machine Learning Applications in NLP.pptbutest

Schemaless DatabasesDan Gunter

Дмитрий Селиванов, OK.RU. Finding Similar Items in high-dimensional spaces: L...Mail.ru Group

So Far (Schematically) yet So Near (Semantically)Amit Sheth

Ähnlich wie Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010 (20)

Deduplication on large amounts of code

Token

Tg noh jeju_workshop

Class 9: Consistent Hashing

R language introduction

Hands on Mahout!

Dcn 20170823 yjy

(Talk in Powerpoint Format)

Lambdas: Myths and Mistakes

Language Technology Enhanced Learning

PyGotham NY 2017: Natural Language Processing from Scratch

Ghost

R Analytics in the Cloud

Stack_Overflow-Network_Graph

Cluster

P3 2018 python_regexes

Machine Learning Applications in NLP.ppt

Schemaless Databases

Дмитрий Селиванов, OK.RU. Finding Similar Items in high-dimensional spaces: L...

So Far (Schematically) yet So Near (Semantically)

Mehr von Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Mehr von Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Kürzlich hochgeladen

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

GenCyber Cyber Security Day PresentationMichael W. Hawkins

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Tech Trends Report 2024 Future Today Institute.pdfhans926745

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j

A Domino Admins Adventures (Engage 2024)Gabriella Davis

GenAI Risks & Security Meetup 01052024.pdflior mazor

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous

Kürzlich hochgeladen (20)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Handwritten Text Recognition for manuscripts and early printed texts

08448380779 Call Girls In Friends Colony Women Seeking Men

Data Cloud, More than a CDP by Matt Robison

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

08448380779 Call Girls In Civil Lines Women Seeking Men

Finology Group – Insurtech Innovation Award 2024

GenCyber Cyber Security Day Presentation

[2024]Digital Global Overview Report 2024 Meltwater.pdf

How to Troubleshoot Apps for the Modern Connected Worker

Tech Trends Report 2024 Future Today Institute.pdf

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

A Domino Admins Adventures (Engage 2024)

GenAI Risks & Security Meetup 01052024.pdf

Exploring the Future Potential of AI-Enabled Smartphone Processors

Strategies for Landing an Oracle DBA Job as a Fresher

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

2. Motivation: Data Cleaning Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime Find movies starring Tom Hanks

3. Movies starring S..warz…ne…ger? Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

4. Similarity Search Find movies with a star “ similar to ” Schwarrzenger . Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Samuel Jackson Iron man 2008 Sci-Fi Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

5. Record linkage Table R Table S Star Keanu Reeves Samuel Jackson Schwarzenegger … Star Keanu Reeves Samuel L. Jackson Schwarzenegger …

6. Two-step solution Table R Table S Step 2: Verification Star … Star … Step 1: Similarity Join

9. Set-Similarity Join Finding pairs of records with a similarity on their join attributes > t

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25. Speedup

26.

27.

28.

29.

30.

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Ähnlich wie Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010 (20)

Mehr von Yahoo Developer Network

Mehr von Yahoo Developer Network (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010