Tutorial at the Web Science summer school in Koblenz, Germany

1. SOCIAL MACHINES
TECHNOLOGY DESIGN AND INCENTIVES
Elena Simperl
e.simperl@soton.ac.uk
@esimperl
July 4th, 2016
1

2. TUTORIAL OVERVIEW
• Social machines as engine of
creativity, growth, and sociality
• This tutorial covers
 Introduction to social machines
 How-to crowdsourcing design
 How-to study citizen science projects
 Examples from current research
TUTORIAL@ISWC2013

3. THE ORDER OF SOCIAL MACHINES
‘Real life is and must be full of all kinds of social constraint – the very processes
from which society arises. Computers can help if we use them to create abstract
social machines on the Web: processes in which the people do the creative
work and the machine does the administration […]
The stage is set for an evolutionary growth of new social engines.’
Tim Berners-Lee, Weaving the Web, 1999
3

4. PEOPLE SUPPLY, CURATE AND ANALYZE DATA

5. OR CREATE CONTENT FROM SCRATCH

6. FOR FREE OR AGAINST A REWARD

7. PEOPLE ARE (ELEMENTARY) PROBLEM SOLVERS

8. SOMETIMES ON ANY GIVEN TOPIC

9. PEOPLE LIKE TO SHARE THEIR VIEWS
9

10. SOMETIMES THEY WORK TOGETHER
10

11. SOMETIMES THEY COMPETE
11

12. IT DOESN’T GO WELL ALL THE TIME
12

13. BUT THE ONES THAT DO WELL SEEM TO HAVE THINGS IN
COMMON
i. Problems solved by the scale of
human participation
ii. Timely mobilisation of people,
technology, and information
resources
iii. Access to (or else the ability to
generate) large amounts of
relevant data
iv. Confidence in the quality of the
data
v. Efficient, effective and equitable
vi. Trust in the agents and process
vii. Aligned incentives and network
effects
viii. Intuitive, user-centred interfaces
ix. Cross platform support
x. Exploit the power of the open

14. sociam.org
@project_sociam

15. WHAT IS A SOCIAL MACHINE? Exercise 1
15

16. Based on the definition and examples mentioned so far,
discuss examples of social machines you are familiar with.
When is a social machine successful?
How do social machines compare with: human computation,
crowdsourcing, collective intelligence, social computing,
wisdom of the crowds, open innovation, socio-technical
systems etc.?
16

17. SOCIAL MACHINES AS
CROWDSOURCING
17

18. CROWDSOURCING:
PROBLEM SOLVING VIA OPEN CALLS
"Simply defined, crowdsourcing represents the act of a company or institution taking a
function once performed by employees and outsourcing it to an undefined (and
generally large) network of people in the form of an open call. This can take the form
of peer-production (when the job is performed collaboratively), but is also often
undertaken by sole individuals. The crucial prerequisite is the use of the open call
format and the large network of potential laborers.“
[Howe, 2006]
7/4/2016 18

19. DIFFERENT FORMS AND PLATFORMS TO CHOOSE
FROM
19
Macrotasks
Microtasks
Challenges
Self-organized crowds
Crowdfunding
Source:
[Prpić et al., 2015]

20. FOR EXAMPLE: MICROTASK CROWDSOURCING
7/4/2016 20
Work is broken down into smaller pieces that can be solved independently

21. Are social
machines just
crowdsourcing?

22. CROWDSOURCING & SOCIALITY

23. CROWDSOURCING CREATIVE TASKS

24. CROWDSOURCING & ETHICS
[Difallah et al, 2015]

25. MANY QUESTIONS TO ANSWER
TASK DESIGN
WORKFLOW
DESIGN AND
EXECUTION
TASK INTERFACES QUALITY
ASSURANCE
TASK
ASSIGNMENT
CROWD
TRAINING AND
FEEDBACK
INCENTIVES
ENGINEERING
COLLABORATION,
COMPETITION,
SELF-
ORGANIZATION
REAL-TIME
DELIVERY NICHESOURCING EXTENSIONS TO
TECHNOLOGIES
SOCIAL
MACHINES
ENGINEERING

26. LUNCH BREAK
e.simperl@soton.ac.uk
@esimperl
sociam.org
26

27. CROWDSOURCING DESIGN
27

28. CROWDSOURCING DESIGN
101
What: Goal
Who: Contributors
How: Process
Why: Motivation and incentives
http://sloanreview.mit.edu/article/the-
collective-intelligence-genome
28
Image at
http://jasonbiv.files.wordpress.co
m/2012/08/collectiveintelligence_
genome.jpg

29. EXAMPLE: ZOONIVERSE
WHAT IS OUTSOURCED
• Object recognition, labeling, categorization in media
content
WHO IS THE CROWD
• Anyone
HOW IS THE TASK OUTSOURCED
• Highly parallelizable tasks
• Every item is handled by multiple annotators
• Every annotator provides an answer
• Consolidated answers solve scientific problems
WHY DO PEOPLE CONTRIBUTE
• Fun, interest in science, desire to contribute to science
7/4/2016 29
zooniverse.org

30. DESIGN YOUR OWN Exercise 2
30

31. IN THIS TALK: CROWDSOURCING DATA CITATION
‘The USEWOD experiment ‘
 Goal: collect information about the usage of Linked
Data sets in research papers
 Explore different crowdsourcing methods
 Online tool to link publications to data sets (and
their versions)
 1st feasibility study with 10 researchers in May
2014
7/4/2016 31
http://prov.usewod.org/
9650 publications

32. Discuss how you would crowdsource the problem illustrated on the
previous slide.
Consider using the schema presented earlier.
Think about how you would ensure that the data is accurate.
(15 minutes discussions in groups, then plenary discussion of
results)
32

33. DIMENSIONS OF CROWDSOURCING
33

34. WHAT
In general: something you cannot do (fully) automatically
In our example: annotating research papers with data set information
 Alternative representations of the domain
 What if the paper is not available?
 What if the domain is not known in advance or is infinite?
 Do we know the list of potential answers?
 Is there only one correct solution to each atomic task?
 How many people would solve the same task?
7/4/2016 34
What Who
How Why

35. BROAD RANGE OF TASKS
7/4/2016 35
What Who
How Why

36. WHO
In general
 An open (‘unknown’) crowd
 Scale helps
 Qualifications might be a pre-requisite
In our example
• People who know the papers or the data sets
• Experts in the (broader) field
• Casual gamers
• Librarians
• Anyone (knowledgeable of English, with a computer/cell phone etc)
• Combinations thereof
7/4/2016 36
What Who
How Why

37. Country of residence
• United States: 46.80%
• India: 34.00%
• Miscellaneous: 19.20%
7/4/2016 37
A LARGE, BUT NOT ALWAYS
DIVERSE CROWD
What Who
How Why

38. SIGNIFICANT AMOUNT OF RESOURCES AND
TIMELY DELIVERY
7/4/2016 38
What Who
How Why

39. SIGNIFICANT AMOUNT OF RESOURCES AND TIMELY DELIVERY (2)
39
http://www.mturk-tracker.com/
60% of
workers spend
more than 4
hours a week
on MTurk
What Who
How Why

40. HOW: PROCESS
In general:
 Explicit vs. implicit participation
 Affects motivation
 Tasks broken down into smaller units undertaken in parallel by different people
 Does not apply to all forms of crowdsourcing
 Coordination required to handle cases with more complex workflows
 Sometimes using different crowds for workflow parts
 Example: text summarization
 Task assignment to match skills, preferences, and contribution history
 Example: random assignment vs meritocracy vs full autonomy
 Partial or independent answers consolidated and aggregated into complete
solution
 Example: challenges (e.g., Netflix) vs aggregation (e.g., Wikipedia)
7/4/2016 40
See also [Quinn & Bederson, 2012]
What Who
How Why

41. HOW: PROCESS (2)
In our example:
 Use the data collected here to train a IE algorithm
 Use paid microtask workers to go a first screening, then expert crowd to sort out
challenging cases
 What if you have very long documents potentially mentioning different/unknown
data sets?
 Competition via Twitter
 ‘Which version of DBpedia does this paper use?’
 One question a day, prizes
 Needs golden standard to bootstrap and redundancy
 Involve the authors
 Use crowdsourcing to find out Twitter accounts, then launch campaign on
Twitter
 Write an email to the authors…
 Change the task
 Which papers use Dbpedia 3.X?
 Competition to find all papers
7/4/2016 41
What Who
How Why

42. HOW: VALIDATION
In general:
 Solutions space closed vs. open
 Redundancy vs. iterations/peer-review
 Performance measurements/ground truth
 Available and accepted
 Automated techniques employed to predict accurate solutions
In our example:
 Domain is fairly restricted
 Spam and obvious wrong answers can be detected easily
 When are two answers the same?
 Can there be more than one correct answer per question?
 Redundancy may not be the final answer
 Most people will be able to identify the data set, but sometimes the actual
version is not trivial to reproduce
 Make educated version guess based on time intervals and other features
7/4/2016 42
What Who
How Why

43. WHY Money (rewards)
Love
Glory
Love and glory reduce costs
Money and glory make the crowd move
faster
Intrinsic vs extrinsic motivation
Rewards/incentives influence
motivation.
 They are granted by an external ‘judge’
Successful volunteer
crowdsourcing is difficult to
predict or replicate
 Highly context-specific
 Not applicable to arbitrary tasks
Reward models often easier to
study and control (if performance
can be reliably measured)
 Not always easy to abstract from social
aspects (free-riding, social pressure)
 May undermine intrinsic motivation
43
What Who
How Why

44. WHY (2)
In our example:
Who benefits from the results
Who owns the results
Money
 Different models: pay-per-time, pay-per-unit, winner-takes-it-all
 Define the rewards, analyze trade-offs accuracy vs costs, avoid spam
Love
 Authors, researchers, Dbpedia community, open access advocates, publishers, casual gamers etc.
Glory
 Competitions, mentions, awards
7/4/2016 44
Image from Nature.com
What Who
How Why

45. PRICING ON MTURK
7/4/2016 45
[Ipeirotis, 2008]

46. IT‘S NOT ALWAYS JUST ABOUT MONEY
7/4/2016 46
http://www.crowdsourcing.org/editorial/how-to-motivate-the-crowd-infographic/
http://www.oneskyapp.com/blog/tips-to-motivate-participants-of-crowdsourced-
translation/
[Kaufmann, Schulze, Viet, 2011]

47. CURRENT RESEARCH
47

48. CROWDSOURCING RESEARCH
TASK DESIGN TASK ASSIGNMENT QUALITY
ASSURANCE
INCENTIVES
ENGINEERING
WORKFLOW
DESIGN AND
EXECUTION
CROWD TRAINING SELF-ORGANIZING
CROWDS
COLLABORATIVE
CROWDSOURCING
REAL-TIME DELIVERY EXTENSIONS TO
TECHNOLOGIES
PROGRAMMING
LANGUAGES
APPLICATION
SCENARIOS

49. IMPROVING PAID MICROTASKS @WWW15
Compared effectivity of microtasks on CrowdFlower vs self-
developed game
 Image labelling on ESP data set as gold standard
 Evaluated
 accuracy
 #labels
 cost per label
 avg/max #labels/contributor
 For three types of tasks
 Nano: 1 image
 Micro: 11 images
 Small: up to 2000 images
 Probabilistic reasoning to personalize furtherance incentives
Findings
 Gamification and payments work well together
 Furtherance incentives particularly interesting for top contributors

50. CROWDSOURCING DBPEDIA CURATION @ISWC13

51. ENTITY CLASSIFICATION IN DBPEDIA
@SEMANTICWEBJOURNAL15
Study different ways to crowdsource entity typing using paid microtasks.
Three workflows
 Free associations
 Validating the machine
 Exploring the DBpedia ontology
Findings
 Shortlists are easy & fast
 Popular classes are not enough
 Alternative ways to explore the taxonomy
 Freedom comes with a price
 Unclassified entities might be unclassifiable
 Different human data interfaces
 Working at the basic level of abstraction achieves greatest precision
 But when given the freedom to choose, users suggest more specific classes
4.58M
things

52. HYBRID NER ON TWITTER @ESWC15
Identified content and crowd factors that impact effectivity
 #entities in post
 types of entities
 content sentiment
 skipped TP posts
 avg. time/task
 UI interaction
Findings
 Shorter tweets with fewer entities work better
 Crowd is more familiar with people and places from recent news
 MISC as a NER category sometimes confusing but useful to identify partial and
implicitly named entities

53. OBSERVE AND IMPROVE*
Elena Simperl, University of
Southampton, UK
*with slides by Ramine Tinati, Markus
Luczak-Rösch, and others

54. OVERVIEW
Background to citizen science
Citizen science & human computation
Citizen science & online communities
Citizen science & science
Methods and design guidelines

55. WHAT IS CITIZEN SCIENCE

56. EXAMPLE: ZOONIVERSE

57. EXAMPLE: PYBOSSA

58. EXAMPLE: VOLUNTEER SCIENCE

59. EXAMPLE: EYEWIRE

60. STUDYING CITIZEN SCIENCE: HUMAN COMPUTATION
Task design
Task assignment
Answer validation and aggregation
Contributors’ performance
Motivation and incentives

61. STUDYING CITIZEN SCIENCE: ONLINE COMMUNITY
Roles and activities
Patterns of participation
Motivation and incentives
Evolution in time

62. STUDYING CITIZEN SCIENCE: ESCIENCE
New workflows
New best practices
New publishing models

63. A FOOTER
When is a citizen science project
successful?

64. LEVELS OF ENGAGEMENT
0
2
4
6
8
10
12
14
16
18
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
Activeusersin%
Month since registration
~1% of participants contribute 72% of Talk & 29% of Task

65. BEYOND DATA COLLECTION AND ANALYSIS
Discussion and community engagement are integral part of the
experience

66. WORK VS TALK
40.5
%
Classifications
Talkcontributions
Classifications

67. CHAT AND INSTANT MESSAGING
Microposts
PH SG SW NN GZ CC PF SF AP WS
91%
2
0
6
4
10
8

68. DISCUSSION PROFILES
Deeply engaged
volunteers, few
threads but
multiple posts
within them
9 0.1
%
Content
producers,
posting across
many boards
and threads
7
0.1
%
Thread
followers and
PM (one-to-
one) talkers
8 0.4
%
First to respond
and question
answerers
4 1%
Highly active
thread starters
and answerers
across a wide
range of topics
1 2.8
%
Infrequent
volunteers, single
thread posts, no
personal
messages
5 5.5
%
Watcher and
starter of many
threads, but not
first to reply
3
6.5
%
Highly active
thread starters
and first to
reply back
2 14.6
%
Long active
volunteers (the
core group),
posting
sporadically
6 69.0
%

69. TALKING DOES NOT MEAN LESS WORK
How long do games take to complete depending on when the chat messages are made?
Games took longer to complete when chat messages were being made during the
gaming session, compare to games that had messages before or after.

70. ECOSYSTEM EFFECTS
Project A
Project B
Project C
Participant X
Part. Y

71. DESIGNING PLATFORMS
Task
specificity
Community
development
Task design PR and
engagement
Bootstrapping the community
Serendipitous scientific
discovery
Engaging with people, supporting
profession team
Supporting individuals, finding new
scientific discoveries
Obtaining new citizen
scientists
Retaining people
Supporting people, improving
task completion
Obtaining new citizen scientists
Reinvigorating old users

72. WHAT’S NEXT?
Human computation
 Task assignment: what tasks are interesting/relevant for whom?
 Peer review, collaborative approaches
 The role of gamification: is science a game?
Online community
 Mapping design patterns to online community design frameworks: do CS form a community?
 Making discussions more effective
Science
 Citizen science platforms that everyone can use

73. PUBLICATIONS
R. Tinati, M. Van Kleek, E. Simperl, M. Luczak-Rösch, R. Simpson and N. Shadbolt.
Designing for citizen data analysis: a cross-sectional case study of a multi-domain
citizen science platform. Proceedings of ACM CHI Conference on Human Factors in
Computing Systems 2014 (CHI2015), pages 1-10, 2015.
M. Luczak-Rösch, R. Tinati, E. Simperl, M. Van Kleek, N. Shadbolt and R. Simpson. Why
Won't Aliens Talk to Us? Content and Community Dynamics in Online Citizen Science.
Proceedings of the International Conference on Weblogs and Social Media
(ICWSM2014), 2014.
A FOOTER

74. E.SIMPERL@SOTON.AC.UK
@ESIMPERL
WWW.SOCIAM.ORG
WWW.STARS4ALL.EU
7/4/2016 74

75. FURTHER READING
7/4/2016 75