Slides of the presentation given at the 22nd International Conference on the World Wide Web.
URL: http://www2013.org/program/561-reactive-crowdsourcing/
More information on the Crowdsearcher project available at
crowdsearcher.search-computing.com
2. Crowd Control is tough…
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
Wednesday, May 15 Reactive Crowdsourcing 2
http://xkcd.com/1060/
3. Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing 3
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
• Goal: taming the crowd
• Cost
• Time
• Quality
4. Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing 4
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
• Goal: taming the crowd
• Cost
• Time
• Quality
• Motivation!
• Need for higher level abstractions and tools
• CONTROL as first-class citizen
5. Reactive Crowdsourcing
• A conceptual framework for modeling crowdsourcing
computations and control requirements
• Task Design
• Reactive Control Design
• Active Rule programming framework
• Declarative rule language
• A reactive execution environment for requirement
enforcement and reactive execution
• Based on the CrowdSearcher approach
Wednesday, May 15 Reactive Crowdsourcing 5
6. Why Active Rules?
• Crowdsourcing control typically focuses on task data
• Execution results, agreement on truth value, workers performance
• An active rule approach can provide
• Ease of Use: control is easily expressible
• Simple control data structures
• Familiar formalism
• Power: support to arbitrarily complex controls
• Extensibility mechanisms
• Automation: most active rules can be system-generated
• Well-defined semantic
• Flexibility: simple control variants have localized impact on the
rules set
• Control isolation
Wednesday, May 15 Reactive Crowdsourcing 6
7. The CrowdSearcher Approach
• Human-Enhanced data management with social networks
and Q&A systems as crowdsourcing platforms
• Example: search task (WWW2012)
Wednesday, May 15 Reactive Crowdsourcing 7
Human Interaction Management
Social
Networks
Human
Computation
Platforms
Q&A
Search Execution
Engine
raction
ent
Query Interface
Social
Networks
ery Answer
Search Execution
Engine
anInteraction
anagement
Human
Query Interface
Local
Social
Networks
Q&A
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Data Management System
Human Access
Interface
Remote Data
Access
Local Data
Access
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Socia
Networ
Q&A
Crowd
source
platform
Query Answer
Task
Human-Enhanced
Data
Query
Results
8. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Wednesday, May 15 Reactive Crowdsourcing 8
I O
9. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 9
• Task Spec: task operations, objects, and performers Dimension Tables
10. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 10
• Task Spec: task operations, objects, and performers Dimension Tables
• Task Planning: work distribution Execution Table for task monitoring
11. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 11
• Task Spec: task operations, objects, and performers Dimension Tables
• Task Planning: work distribution Execution Table for task monitoring
• Control Specification: task control policies Control Mart
13. Task Specification_2/3
Wednesday, May 15 Reactive Crowdsourcing 13
Politician
classifiedParty
lastName
photo
oID
• Input Objects Schema: typed attributes
• Output Attributes (according to task type)
Task
tID
opType
categories
Task Specification Task Planning Control Specification
Task Configuration
Object
Specification
o1
Obama
http://….
?????
14. Task Specification_3/3
Wednesday, May 15 Reactive Crowdsourcing 14
Politician
classifiedParty
lastName
photo
oID
Task Configuration
Object
Specification
Performer
Specification
• Execution platform(s)
• Qualifications, etc.
Task
tID
opType
categories Performer
name
pID
platform
Task Specification Task Planning Control Specification
p1
Alessandro
Facebook
15. Task Planning_1/2
• Organize the task in MicroTasks, and allocate input objects
• μTaskObjectExecution Designed for execution monitoring
• Track performers response
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Wednesday, May 15 Reactive Crowdsourcing 15
Politician
classifiedParty
lastName
photo
oID
Task
tID
opType
categories Performer
name
pID
platform
Splitting
Task Specification Task Planning Control Specification
mt1
O1
…
…
…
…
Facebook
16. Task Planning_2/2
• Assign performers to MicroTasks on platforms
• Pull: dynamic assignment (First come - First served / Choice of the
performer)
• Push: static assignment (Performers’ priority / Performer matching)
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Wednesday, May 15 Reactive Crowdsourcing 16
Politician
classifiedParty
lastName
photo
oID
Task
tID
opType
categories Performer
name
pID
platform
Splitting Assignment
Task Specification Task Planning Control Specification
mt1
O1
P1
Republican
00:00:01
00:00:10
Facebook
17. Control Specification_1/4
Wednesday, May 15 Reactive Crowdsourcing 17
Task Specification Task Planning Control Specification
• Status Variable: tracking task and performers status
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status Trusted/SpammerCreated/Planned/Closed
18. Control Specification_2/4
Wednesday, May 15 Reactive Crowdsourcing 18
Task Specification Task Planning Control Specification
• Object : tracking objects status
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
19. Control Specification_3/4
Wednesday, May 15 Reactive Crowdsourcing 19
Task Specification Task Planning Control Specification
• Object : tracking object responses
• Performer: tracking performer behavior (e.g. spammers)
Performer
Control #right
pID
#eval
#wrong
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
20. Control Specification_4/4
• Object : tracking object responses
• Performer: tracking performer behavior (e.g. spammers)
• Task: tracking task status: closing @completion, re-plan
Wednesday, May 15 Reactive Crowdsourcing 20
Task
Control#compObj
tID
#compExec
Performer
Control #right
pID
#eval
#wrong
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
Task Specification Task Planning Control Specification
21. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
Wednesday, May 15 Reactive Crowdsourcing 21
22. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
Wednesday, May 15 Reactive Crowdsourcing 22
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
23. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
Wednesday, May 15 Reactive Crowdsourcing 23
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
24. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
• Actions: updates on data structures (e.g. change attribute
value, create new instances), special functions (e.g. replan)
Wednesday, May 15 Reactive Crowdsourcing 24
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
25. Wednesday, May 15 Reactive Crowdsourcing 25
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tIDEvent
26. Wednesday, May 15 Reactive Crowdsourcing 26
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Condition
27. Wednesday, May 15 Reactive Crowdsourcing 27
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Action
28. Wednesday, May 15 Reactive Crowdsourcing 28
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Action
29. Rule Programming Best Practice
• We define three classes of rules
Wednesday, May 15 Reactive Crowdsourcing 29
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
30. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 30
• We define three classes of rules
• Control rules: modifying the control tables;
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
31. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 31
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
32. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 32
• Top-to-bottom, left-to-right, evaluation
• Guaranteed termination
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
33. Rule Programming Best Practice
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
• Execution rules: modifying the execution table, either directly or through re-planning
Wednesday, May 15 Reactive Crowdsourcing 33
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
• Termination must be proven (Rule precedence graph has cycles)
34. Experimental Evaluation
• GOAL: demonstrate the flexibility and expressive power
of reactive crowdsourcing
• 3 experiments, focused on Italian politicians
• Parties: Human Computation affiliation classification
• Law: Game With a Purpose guess the convicted politician
• Order: Pure Game hot or not
• 1 week (November 2012)
• 284 distinct performers
• Recruited through public mailing lists and social networks
announcements
• 3500 Micro Tasks
Wednesday, May 15 Reactive Crowdsourcing 34
35. Politician Affiliation
• Given the picture and name of a politician, specify his/her political
affiliation
• No time limit
• Performers are encouraged to look up online
• 2 set of rules
• Majority Evaluation
• Spammer Detection
Wednesday, May 15 Reactive Crowdsourcing 35
36. Results – Majority Evaluation_1/3
Wednesday, May 15 Reactive Crowdsourcing 36
30 object; object redundancy = 9;
Final object classification as simple majority after 7 evaluations
37. Results - Majority Evaluation_2/3
Wednesday, May 15 Reactive Crowdsourcing 37
Majority @7
Early Majority @3 R @7
-27% executions
-18% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
Final object classification as total majority after 3 evaluations
Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7
38. Results - Majority Evaluation_3/3
Wednesday, May 15 Reactive Crowdsourcing 38
Majority @7
Early Majority @3 R @7
Majority @3 R @5 R @7
-23% executions
+26% precision
+50% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
Final object classification as total majority after 3 evaluations
Otherwise, simple majority at 5 or at 7 (with replan)
39. Results – Spammer Detection_1/2
Wednesday, May 15 Reactive Crowdsourcing 39
New rule for spammer detection without ground truth
Performer correctness on final majority. Spammer if > 50% wrong classifications
Majority @3 R @5 R @7
Majority @3 R @5 R @7 - Spammer Detection
+46% executions
+1.5% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
41. Summary
• Results
• An integrated framework for crowdsourcing task design and control
• Well-structured control rules with some guarantees of termination
• Support for cross-platform crowd interoperability
• A working prototype crowdsearcher.search-computing.org
• Forthcoming
• Exploitation of interoperability
• Expertise finding
• Dynamic planning
• Integration with other social-networks and human computation
platforms
Wednesday, May 15 Reactive Crowdsourcing 41
Good afternoonI’m very happy to be here today and discuss with you our work, named reactive crowdsourcing
But first allow me to contextualize a bit its scope. I’m sure we are all very familiar with crowdsourcing and human computation, which is a based on a very neat idea: organize humans to accomplish tasks. Unfortunately, or luckily for us , such organization is a complex matter. When designing systems and experiments, there are several things to consider: how do you split your tasks in microtask, which are the best performers for it, how do you scout them, etc.
It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
This paper wepropose a conceptual framework and a reactive execution environment for modelling and controlling crowdsourcing computationsWit the ultimate goal of minimizing the effort required for control specification, we propose:a simple task design processA rule specification language,whose properties (e.g., termination) can be easily proved in the context of a well-organized computational framework
But why we decided to go for an approach based on active rules?The choice stemmed from the observation that crowdsourcing control is typically driven by data, like the status of the HIT executions, the worker performance, the current agreement on the truth value of some object. Therefore, it came almost natural for us to turn on a data-driven approach, that proven very effective for the definition of control in several contexts, including database systemsActive rules are actually relatively easy to use, when expressed on well-define data structures. They allow the definition of arbitrary complex control logic, most of which can be easily automated thanks to a well-defined syntax and semantic. Also, they allow for a great flexibilty, since changes in the control logic of the application can be well-isolated into localized changes of the rule set
This work capitalizes on the results of our www2012 paper, where we an approach for data management that integrates SN, QA and traditional HC platform for the execution of human computation and crowdsourcing tasks.This work specifically addressed crowd-enhanced search, a now very popular trend. However, the approach we proposed can be very easily generalized to any kind of human-enhanced data management system, and we believe this crowd-interoperability can be exploited in very interesting ways. LOCAL SOURCE: sorgentidatilocalisfruttate dal Search Execution Engine, magariaccedutedallo Human Interaction Management per configurare / gestirei task. La suaesistenza e’ accessoriarispettoaglialtri, e codificainformazioni applicative specificheICONE DI DX, DALL’ALTO a SX(social networks) Facebook, Twitter, Google +(Q&A systems) StackOverlflow, Yahoo Answers, Quora(HC Platforms) Freebase, Amazon Mechanical Turk, ODesk
The design process we propose is based on a very simple data-driven abstract model for task execution: data in, workers do stuff, data outCoherently with this vision, the task design and deployment process is based upon specific data structures, created in an automated way trough model driven transformations. Unfortunately we don’t have time to go too deep in this matter, so I refer you to the paper for further details.
The process composes of three simple steps. The first, task specification, is devoted to the definition of basic task information, such as the data-driven operations to perform (e.g. classify a picture, add new instances of a given object, etc.), the shape of the actual objects to evaluate, and the charactersticis of the targeted set of performers. All those aspect are encoded in what we call Dimension Tables
Then, the second step is devoted to task planning, i.e. specify how works should be partitioned and distributed among performers. Those planning aspects are encoded into an Execution Table, specifically devoted to task monitoring
Finally, the developer must specify the control logic for the task, and it does so by defining active rules upon control-specific data structures contained in what we call the Control Mart.
To clarify the process, let us show visually how the supporting data structure are created in a simple case study: a picture classification task, where the picture contain picture of politicians, and the task is to specify to which party they belong. We will use color codes to identify data structures types, and we highlight in bold those attributes which depend on the specifically designed tasks. All the other are, somehow, standard. The first dimension table is the Task table, which contains info about the performed operations (a classification, in this case), and the categories to which politicians can be assigned (e.g. republican and democtrats)
Then, we have a dimension table for the actual objects of the task, in this case, politicians. Notice that the classifiedPartyattribute is actually produced by the crowd at the end of the task
Finally, a Performer dimension table represents the population of performer currently available for task execution. The table can be pre-filled, if the performers are known in advance, or empty, in case of openly available tasks.
The second activity of the process deals with the planning of the task and it also composes of two phases. In the first phase, suited algorithms will decide such things as how which objects should appear in each MicroTask? (e.g. ground truth assessment), how many objects in each MicroTask, how often an object should appear in MicroTasks, etc. TheμTaskObjectExecution table keeps track of this organization, and it contains one tuple for each object/microTask association. A performer, when executing a microTask, will fill in the data value required by the specified operation (e.g. the party of a given politician)
The second phase of task planning deals with the assignment of microTasks to performers. This can be done according to several policies (e.g. pull or push). In the example, the assignment is performed in a pull fashion, and attributes are given value on performer arrival
The third part of the process is devoted to the specification of control. Please notice here that I added to an addition color code to denote attributes in the Dimension Tables which serve task control purposes. So for instance, a performer can be classified as trusted or spammer. A task as “created/planned/closed” and so on
Bin addition to status variable, control is defined upon the Control Mart, which composes of three data structures. The first one, called ObjectControl contains control variable related to the object of the task, like the number of evaluation it received, the number of classifications it received for each class, and the current truth value.
The second one, called PerformerControl contains control variable related to the performer of the task, like the number of evaluation she performerd, and the number of times she provided a right or wrong answer (assuming the existance of a ground truth)
Finally, the thirdone, called TaskControl contains control variable related to the task, like the number total number of objects currently evaluated, or the number of executed hits.
The previous data structure provide in a very simple yet complete way the control variables that are needed to define the task control policies. But how can control be specified? We rely on a language based on the classic ECA paradigm
Where events are updated on the data structure values. We decided for a row-level update granularity, so to easily track the before and after states of rows. In the example, the rule triggers when the ClassifiedParty attribute of a tuple in the μTaskObjectExecution table changes
Conditions are expressed as conditions on data attributes (e.g., the value specified by the performer)
And actions are updated performed on the same, or other data structures. Such updates can be done directly or trough special functions, devoted to such operations as replanning’Of course there is no time to show the syntax of the language, but you can find more on the paper.
The second rule is a bit more complex, and it is used to assess the truth value of an object trough majority voting. For instance, here we assume that as soon as a Politician gets 2 evaluation as Rep or Dem, the object can be deemed as completed. triggers when the ObjectControl table updates
It is quite known that active rule programming can be rather subtle and unstable, as the behavior of a set of rules may change dramatically as a consequence of small changes in the rules To simply, and better control rule programming we devise three classes of rules which, as I will show soon, have interesting properties
The first class are named control rules, and are meant to modify control tables. Arrows represents rules triggering on a table (the source of the arrow) and affecting another table (the destination). As you can see, not all the possible source-target couple are adimissible, and I will explain soon why.
The second class of rules modify the dimension tables, and are the one devoted to changing the status of the main task entities (e.g. setting a perforomer as spammer when she makes too many bad classifications)
Note that, since we suggest a very well-defined top-to-bottom, left-to-right semantic, no cycles are allowed, and therefore rules are guardanteed to terminate. Those cycles in the object control – performer control etc. still bases on precise rules also on the attributes of the table
Finally, execution rules are responsivble for the modification of the execution table, and are therefore responsible for modifying the set?distribution/assignment of the currently defined microtasks. Those rules introduce cycles and, therefore, might cause unconvergence (and termination must be verified)
To demonstrate the flexibily and expressive power of reactive crowdsourcing we advised the experiments, conducted during one week of November 2012. We developed three very different scenario, all programmed with our approach. Unfortunately we don’t have enough time to describe them all, so we focus on just one. A classification task similar to the one used as example in the presentation