SlideShare ist ein Scribd-Unternehmen logo
1 von 31
Downloaden Sie, um offline zu lesen
Pattern-Based Specification of
Crowdsourcing Applications
Alessandro Bozzon (TU Delft)
Marco Brambilla (Politecnico di Milano)

Stefano Ceri (Politecnico di Milano)

Andrea Mauri (Politecnico di Milano)

Riccardo Volonterio (Politecnico di Milano)
Crowdsourcing and Human Computation
It works like magic!
Endless Applications
Endless Success Stories
2008 Olympics Opening Ceremony
Actually…
Often a Try&HopeError process
Task Design Matters
Crowd can be unreliable
($) Incentives Matter
Quality Control Matters
Platform of Execution Matters
Pattern-Based Specification of Crowdsourcing Applications
Setting Nº1
Global Annotations with
simple counting
Setting Nº2
Local Annotations
with Bounding Boxes
Pattern-Based Specification of Crowdsourcing Applications
Pattern-Based Specification of Crowdsourcing Applications
Setting Nº3
Local Annotations with Verified Bounding Boxes
Setting Nº3
Local Annotations with Verified Bounding Boxes
Setting Nº3
Local Annotations with Verified Bounding Boxes
Ok, so what?
#Workers
#Useful
Workers
#Executions Cost $
Time
(hours)
Precision
Setting
Nº1
732
44
(6%)
488 72 ~40 ~67%
Setting
Nº2
498
25
(5%)
547 48 ~169 ~67%
Setting
Nº3
1420
464

(32%)
3387 83 ~184 ~75%
Total 2152 508 4422 203 ~16 days
Our study
Our study
Our Contribution
GOAL
!
Simplify and systematize the design, deploy, and
monitoring of applications (including experiments)
Contributions
An Abstract
Model of
Crowdsourcing
Activities
A Composition
model for
Crowdsourcing
Activities
A Library of
crowdsourcing
Patterns
• A conceptual framework
• A specification paradigm
• A reactive execution control
environment
Models
DEMO VIDEO
Crowd Task
[T operation types]
(intra-task patterns)
Object Type
block size
min #obj
(cons)
input buffer
batch flow (on closed task)
stream flow (on closed object)
MicroTask
[MT operation types]
r
data manipulator
Case Study: Movie Scene Analysis
Scenario 1: Scene Positioning
Spoiler Alert!
Order Scenes
Scene in Beg/Mid/End
Scenario 2: Actor Identification
Find Actor
Validate Actor
Pattern-Based Specification of Crowdsourcing Applications
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
5
Order Scenes
[Order] (SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
[Class=E]
[Class=B OR M]
Example of Scenario 1 Model
Patterns
Intra-Task Auxiliary Workflow
Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
• Consensus
• Join
• Sort
• Grouping
• Performer Control
• Planning
• Assignment
• Aggregation
• Quality & Performer
Auxiliary Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
• Pruning • Tie Breaking
• Operations before or
after the execution
Workflow
• Set of heterogeneous
tasks
Create Decide
Improve
Compare
/ Verify
Find Fix
(a)
(b)
(c)
Auxiliary
Task
Create/Decide
Improve/Compare
Find/Fix/Verify
Experiments
1700 Executions
39$
September 2013
Streaming Vs. Batch Execution
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
7
Order Scenes
[Order]
(SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
[Class=E]
[Class=B OR M]
(P1)
5
3
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
7
Order Scenes
[Order]
(SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
Cons.
[Class=E]
[Class=B OR M]
(P2)
5
3
P.Beg P.Mid P.End
P1 0.5 1 0.11
P2 0.5 0.8 0.33
Spear.Beg Spear. Mid

.MidP1 0.5 0.54
P2 0.9 0.51
Position
Order
P1 P
b) Elapsed Tim
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5
Position
Order
P1
Position
Order
P2
b) Elapsed Time (Mins)
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5 60 120 180 240 300 360
Position
Order
P1 P
b) Elapsed Tim
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5
Intra-Task Consensus Vs. Workflow Decision
A4 A5 A6
Precision
 0.99 0.95 0.89
Recall 0.90 1 0.96
F-Score 0.93 0.97 0.90
Find Actors
[Tag]
(Static Agreement@3)
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
MicroTask [Like]
Scene+Actor
block All
min 1
5
(A4)
Find Actors
[Tag]
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
(Majority Voting@2)
MicroTask [Like]
Scene+Actor
block All
min 1
5 3
(A5)
(A6)
Find Actors
[Tag]
(Static Agreement@3)
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
(Majority Voting@2)
MicroTask [Like]
Scene+Actor
block All
min 1
5 3
count(Actor.Like)<=1
Actor
Validate
A3
Actor
Validate
A4 A5
a) Elapsed Time (
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5 30 60 90 120 160 5
Actor
Validate
Actor
Validate
A4
Actor
Validate
A5
a) Elapsed Time (Mins)
30 60 90 120 160 5 30 60 90 120 160 5 60 120 180 240 60
Actor
Validate
Actor
Validate
A5
a) Elapsed Time (Mins)
Actor
Validate
A6
30 60 90 120 160 5 60 120 180 240 60 300 540 780
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5
Take-Home Message
Engineering approaches should be
applied to people management too
In crowdsourcing, control and
monitoring are key for successful
outcomes
Questions?

Weitere ähnliche Inhalte

Mehr von Alessandro Bozzon

Weaving the Web of People and Things for Intelligent Cities
Weaving the Web of People and Things for Intelligent CitiesWeaving the Web of People and Things for Intelligent Cities
Weaving the Web of People and Things for Intelligent CitiesAlessandro Bozzon
 
Trustworthy Micro-task Crowdsourcing: Challenges and Opportunities
Trustworthy Micro-task Crowdsourcing: Challenges and OpportunitiesTrustworthy Micro-task Crowdsourcing: Challenges and Opportunities
Trustworthy Micro-task Crowdsourcing: Challenges and OpportunitiesAlessandro Bozzon
 
Social Data Science For Intelligent Cities
Social Data Science For Intelligent CitiesSocial Data Science For Intelligent Cities
Social Data Science For Intelligent CitiesAlessandro Bozzon
 
ICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionAlessandro Bozzon
 
An Introduction to Human Computation and Games With A Purpose - Part I
An Introduction to Human Computation and Games With A Purpose - Part IAn Introduction to Human Computation and Games With A Purpose - Part I
An Introduction to Human Computation and Games With A Purpose - Part IAlessandro Bozzon
 
A Service-Based Architecture for Multi-domain Search on the Web
A Service-Based Architecture for Multi-domain Search on the WebA Service-Based Architecture for Multi-domain Search on the Web
A Service-Based Architecture for Multi-domain Search on the WebAlessandro Bozzon
 
Engineering Web Search Applications
Engineering Web Search ApplicationsEngineering Web Search Applications
Engineering Web Search ApplicationsAlessandro Bozzon
 
Liquid Query: Multi-domain Exploratory Search on the Web
Liquid Query: Multi-domain Exploratory Search on the WebLiquid Query: Multi-domain Exploratory Search on the Web
Liquid Query: Multi-domain Exploratory Search on the WebAlessandro Bozzon
 

Mehr von Alessandro Bozzon (9)

Weaving the Web of People and Things for Intelligent Cities
Weaving the Web of People and Things for Intelligent CitiesWeaving the Web of People and Things for Intelligent Cities
Weaving the Web of People and Things for Intelligent Cities
 
Trustworthy Micro-task Crowdsourcing: Challenges and Opportunities
Trustworthy Micro-task Crowdsourcing: Challenges and OpportunitiesTrustworthy Micro-task Crowdsourcing: Challenges and Opportunities
Trustworthy Micro-task Crowdsourcing: Challenges and Opportunities
 
Social Data Science For Intelligent Cities
Social Data Science For Intelligent CitiesSocial Data Science For Intelligent Cities
Social Data Science For Intelligent Cities
 
ICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo Session
 
An Introduction to Human Computation and Games With A Purpose - Part I
An Introduction to Human Computation and Games With A Purpose - Part IAn Introduction to Human Computation and Games With A Purpose - Part I
An Introduction to Human Computation and Games With A Purpose - Part I
 
A Service-Based Architecture for Multi-domain Search on the Web
A Service-Based Architecture for Multi-domain Search on the WebA Service-Based Architecture for Multi-domain Search on the Web
A Service-Based Architecture for Multi-domain Search on the Web
 
Engineering Web Search Applications
Engineering Web Search ApplicationsEngineering Web Search Applications
Engineering Web Search Applications
 
Search Computing
Search ComputingSearch Computing
Search Computing
 
Liquid Query: Multi-domain Exploratory Search on the Web
Liquid Query: Multi-domain Exploratory Search on the WebLiquid Query: Multi-domain Exploratory Search on the Web
Liquid Query: Multi-domain Exploratory Search on the Web
 

Kürzlich hochgeladen

Krishi Vigyan Kendras - कृषि विज्ञान केंद्र
Krishi Vigyan Kendras - कृषि विज्ञान केंद्रKrishi Vigyan Kendras - कृषि विज्ञान केंद्र
Krishi Vigyan Kendras - कृषि विज्ञान केंद्रKrashi Coaching
 
Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.ShwetaHattimare
 
KeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceKeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceLayne Sadler
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...marwaahmad357
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docxmarwaahmad357
 
Main Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearMain Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearmarwaahmad357
 
RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024suelcarter1
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxAkinrotimiOluwadunsi
 
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdf
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdfPests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdf
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdfPirithiRaju
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptxVijayaKumarR28
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptSachin Teotia
 
Substances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestSubstances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestAkashDTejwani
 
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxQ3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxArdeniel
 
soft skills question paper set for bba ca
soft skills question paper set for bba casoft skills question paper set for bba ca
soft skills question paper set for bba caohsadfeeling
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfsantiagojoderickdoma
 
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTMARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTjipexe1248
 
M.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsM.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsSumathi Arumugam
 

Kürzlich hochgeladen (20)

Krishi Vigyan Kendras - कृषि विज्ञान केंद्र
Krishi Vigyan Kendras - कृषि विज्ञान केंद्रKrishi Vigyan Kendras - कृषि विज्ञान केंद्र
Krishi Vigyan Kendras - कृषि विज्ञान केंद्र
 
Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.Role of Herbs in Cosmetics in Cosmetic Science.
Role of Herbs in Cosmetics in Cosmetic Science.
 
KeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data scienceKeyBio pipeline for bioinformatics and data science
KeyBio pipeline for bioinformatics and data science
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
 
PSP3 employability assessment form .docx
PSP3 employability assessment form .docxPSP3 employability assessment form .docx
PSP3 employability assessment form .docx
 
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
 
Main Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearMain Exam Applied biochemistry final year
Main Exam Applied biochemistry final year
 
RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
 
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdf
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdfPests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdf
Pests of cumbu_Identification, Binomics, Integrated ManagementDr.UPR.pdf
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptx
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.ppt
 
Substances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestSubstances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening Test
 
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxQ3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
 
soft skills question paper set for bba ca
soft skills question paper set for bba casoft skills question paper set for bba ca
soft skills question paper set for bba ca
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
 
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
Data delivery from the US-EPA Center for Computational Toxicology and Exposur...
 
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENTMARKER ASSISTED SELECTION IN CROP IMPROVEMENT
MARKER ASSISTED SELECTION IN CROP IMPROVEMENT
 
Cheminformatics tools supporting dissemination of data associated with US EPA...
Cheminformatics tools supporting dissemination of data associated with US EPA...Cheminformatics tools supporting dissemination of data associated with US EPA...
Cheminformatics tools supporting dissemination of data associated with US EPA...
 
M.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery SystemsM.Pharm - Question Bank - Drug Delivery Systems
M.Pharm - Question Bank - Drug Delivery Systems
 

Pattern-Based Specification of Crowdsourcing Applications