Quick Doctor In Kuwait +2773`7758`557 Kuwait Doha Qatar Dubai Abu Dhabi Sharj...
Pick a Crowd
1. Pick-A-Crowd: Tell Me What You Like,
and I’ll Tell You What to Do
A Crowdsourcing Platform for Personalized
Human Intelligence Task Assignment Based on Social
Networks
Djellel E. Difallah, GianlucaDemartini, Philippe Cudré-Mauroux
eXascaleInfolab
University of Fribourg, Switzerland
15th May 2013, WWW 2013 - Rio De Janeiro, Brazil
1
2. Crowdsourcing
• Exploit human intelligence to solve tasks that
are simple for Humans and complex for
machines
• Examples:
– Wikipedia, reCaptcha, Duolingo
• Incentives
– Financial, fun, visibility
2
3. Motivation
• The Pull Methodology is suboptimal
Actual workers
Max Overlap
Effective workers
3
5. Contribution and Claim
• Pick-A-Crowd: A system architecture that uses
Task-to-Worker matching:
– The worker’s social profile
– The task context
• Workers can provide higher quality answers
on tasks they relate to
5
7. Problem Definition (1)The Human Intelligence Task (HIT)
Categorization
Survey
Image Tagging
Data Collection
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
8
8. ProblemDefinition (2)The Worker
Completed HITs: 256
Approval Rate: 96%
Qualification Types
Generic Qualifications
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
9
9. Problem Definition (3) –
Task-to-Worker Matching
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
1- Task-to-Page Matching Function
- Category
- Expert finding
- Semantic
2- Worker Ranking
10
10. Matching Models (1/3)–
Category Based
• The requester provides a list of categories related to the batch
• We create a subset of pages whose category is in the category
list of the batch
• Rank the workers by the number of liked pages in the subset
11
11. Matching Models (2/3) –
Expert Finding
•
•
•
Build an inverted index on the pages’ titles and description
Use the title/description of the tasks as a key word query on the
inverted index and get a subset of pages
Rank the workers by the number of liked pages in the subset
12
12. Matching Models (3/3) –
Semantic Based
•
•
Link the context to an external knowledge base (e.g., DBPedia)
Exploit the underlying graph structure to determine the Hits and Pages similarity
– Assumption that a worker who likes a page is able to answer questions about related entities
– Worker who likes a page is able to answer questions about entities of the same type
•
Rank the workers by the number of liked pages in the subset
Similarity
Relatedness
HIT
FB Pages
Type-Similarity
13
14. Experimental Evaluation
• The Facebook app OpenTurkimplements part
of the Pick-A-Crowd architecture:
– More than 170 registered workers participated
– Over 12k pages crawled
• Covered both multiple answer questions as
well as open-ended questions
– 50 images with multiple choice question and 5 candidate answers
(Soccer, Actors, Music, Authors,Movies, Animes)
– Answer 20 open-ended questions related to the topic (Cricket)
16
22. Conclusions and Future Work
• Pull vs. Pushmethodologies in Crowdsourcing
• Pick-A-Crowd system architecture with Taskto-Worker recommendation
• Experimental comparison with AMT shows a
consistent quality improvement
“Workers Know what they Like”
• Exploit more of the social activity, and handle
content-less tasks
25
23. Next Step
• We are building a Crowdsourcing platform for
the research community
• Pre-register on:
www.openturk.com
Thank You!
26