Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
A WHOLE NEW ZOONIVERSE
GUIDELINES AND TOOLS FOR
CROWDSOURCED SCIENCE
Elena Simperl
e.simperl@soton.ac.uk
@esimperl
Novembe...
OVERVIEW
• Citizen science is a fascinating subject
for Web science research
• Our work helps system designers with
• Fram...
CITIZEN SCIENCE@WAIS
3
WHAT IS CITIZEN SCIENCE
CITIZEN SCIENCE PROJECTS
5
CITIZEN SCIENCE PLATFORMS
6
STUDYING CITIZEN
SCIENCE: HUMAN
COMPUTATION &
CROWDSOURCING
Task design
Task
assignment
Answer
validation and
aggregation
...
STUDYING CITIZEN
SCIENCE: ONLINE
COMMUNITY
Roles and activities
Patterns of
participation
Community health
Motivation and
...
STUDYING CITIZEN
SCIENCE: OPEN
SCIENCE
Scientific workflows
Scientific practice
Publishing, citation, and
peer-review mode...
STUDYING CITIZEN
SCIENCE: EDUCATION
AND SCICOMM
Teaching
methods and
assessment
Tutorial design
Learning
analytics
Engagem...
WHEN MAKES
CITIZEN
SCIENCE
SUCCESSFUL
11
tasks people time
quality science
communi
ty
learning
social
media
…
WHAT MAKES CITIZEN SCIENCE
SUCCESSFUL (2)
12
[Cox et al., 2015]
LEVELS OF ENGAGEMENT
0
2
4
6
8
10
12
14
16
18
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
Activeusersin%
Month since reg...
DATA QUALITY
Existing quality inference
algorithms are limited to binary
classification
Conceptualise the problem for
real...
DATA QUALITY (2)
Majority Voting
 Find the annotation with the top vote as
the true label for each object
Message Passing...
GAMIFICATION
Survey of 27
papers &
31 VCS projects
16
SOCIALITY
Discussions and engagement
with volunteers are integral part
of the experience
Leads to serendipitous scientific...
WORK VS TALK
40.5
%
Classifications
Talkcontributions
Classifications
[Luczak-Roesch et al., 2014; Tinati et
CHAT AND INSTANT MESSAGING
Microposts
PH SG SW NN GZ CC PF SF AP WS
91%
2
0
6
4
10
8
[Luczak-Roesch et al., 2014; Tinati e...
DISCUSSION PROFILES
Deeply
engaged
volunteers,
few threads
but multiple
posts within
them
9 0.
1%
Content
producers,
posti...
FROM
CROWD TO
COMMUNITY
Survey of 48 projects and 150
publications
Identifying affordances from
online community themes
wi...
FROM PROJECTS TO ECOSYSTEMS
Project A
Project B
Project C
Participant X
Part. Y
[Luczak-Roesch et al., 2014]
DESIGNING PLATFORMS
Task
specificity
Community
development
Task design
PR and
engagement
Bootstrapping the
community
Seren...
WHAT’S NEXT?
Human computation & crowdsourcing
 Task assignment: what tasks are interesting/relevant for whom?
 Data qua...
E.SIMPERL@SOTON.AC.UK
@ESIMPERL
WWW.SOCIAM.ORG
WWW.STARS4ALL.EU
11/18/2016 25
All publications available at http://dblp.un...
Nächste SlideShare
Wird geladen in …5
×

A whole new Zooniverse: guidelines and tools for crowdsourced science

151 Aufrufe

Veröffentlicht am

Overview of research in citizen science and crowdsourcing. See also recording at https://youtu.be/afikivaWxCM.

Veröffentlicht in: Bildung
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

A whole new Zooniverse: guidelines and tools for crowdsourced science

  1. 1. A WHOLE NEW ZOONIVERSE GUIDELINES AND TOOLS FOR CROWDSOURCED SCIENCE Elena Simperl e.simperl@soton.ac.uk @esimperl November 16th, 2016 1
  2. 2. OVERVIEW • Citizen science is a fascinating subject for Web science research • Our work helps system designers with • Frameworks of motivations and incentives engineering • Design guidelines and recommendations • Methods to make crowdsourced tasks more effective • Methods to study engagement and community health TUTORIAL@ISWC2013
  3. 3. CITIZEN SCIENCE@WAIS 3
  4. 4. WHAT IS CITIZEN SCIENCE
  5. 5. CITIZEN SCIENCE PROJECTS 5
  6. 6. CITIZEN SCIENCE PLATFORMS 6
  7. 7. STUDYING CITIZEN SCIENCE: HUMAN COMPUTATION & CROWDSOURCING Task design Task assignment Answer validation and aggregation Contributors’ performance Motivation and incentives
  8. 8. STUDYING CITIZEN SCIENCE: ONLINE COMMUNITY Roles and activities Patterns of participation Community health Motivation and incentives
  9. 9. STUDYING CITIZEN SCIENCE: OPEN SCIENCE Scientific workflows Scientific practice Publishing, citation, and peer-review models 9
  10. 10. STUDYING CITIZEN SCIENCE: EDUCATION AND SCICOMM Teaching methods and assessment Tutorial design Learning analytics Engagement strategy 10
  11. 11. WHEN MAKES CITIZEN SCIENCE SUCCESSFUL 11 tasks people time quality science communi ty learning social media …
  12. 12. WHAT MAKES CITIZEN SCIENCE SUCCESSFUL (2) 12 [Cox et al., 2015]
  13. 13. LEVELS OF ENGAGEMENT 0 2 4 6 8 10 12 14 16 18 20 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 Activeusersin% Month since registration ~1% of participants contribute 72% of Talk & 29% of Task [Luczak-Roesch et al., 2014]
  14. 14. DATA QUALITY Existing quality inference algorithms are limited to binary classification Conceptualise the problem for realistic workflows Develop efficient implementations of algorithms Compare them 14
  15. 15. DATA QUALITY (2) Majority Voting  Find the annotation with the top vote as the true label for each object Message Passing  Use object-specific worker messages to represent how reliable a worker is in labelling each specific object. Expectation Maximization  Infer the true label for each object, using annotations from all users, accounting for the error rates of each user;  Estimates the error rates of each user by comparing their annotations with inferred true labels. Results measured in terms of different accuracy metrics and time Experiments still ongoing Preview  Majority voting performs exceptionally well for large numbers of annotations  If less data is available, one could explore message passing (possibly in combination with majority voting) 15
  16. 16. GAMIFICATION Survey of 27 papers & 31 VCS projects 16
  17. 17. SOCIALITY Discussions and engagement with volunteers are integral part of the experience Leads to serendipitous scientific discoveries Encourages autonomy and helps with community building 17
  18. 18. WORK VS TALK 40.5 % Classifications Talkcontributions Classifications [Luczak-Roesch et al., 2014; Tinati et
  19. 19. CHAT AND INSTANT MESSAGING Microposts PH SG SW NN GZ CC PF SF AP WS 91% 2 0 6 4 10 8 [Luczak-Roesch et al., 2014; Tinati et al., 2015,
  20. 20. DISCUSSION PROFILES Deeply engaged volunteers, few threads but multiple posts within them 9 0. 1% Content producers, posting across many boards and threads 7 0. 1% Thread followers and PM (one-to-one) talkers 8 0.4 % First to respond and question answerers 4 1% Highly active thread starters and answerers across a wide range of topics 1 2. 8% Infrequent volunteers, single thread posts, no personal messages 5 5.5 % Watcher and starter of many threads, but not first to reply 3 6. 5% Highly active thread starters and first to reply back 2 14. 6% Long active volunteers (the core group), posting sporadically 6 69.0 % [Tinati et al., 2015, WebSci]
  21. 21. FROM CROWD TO COMMUNITY Survey of 48 projects and 150 publications Identifying affordances from online community themes within literature  Task visibility  Goals  Feedback  Rewards Community features found to have greater role than previously considered  Encourage task completion, discussions etc. Themes align to key success factors of volunteer engagement, task completion and submission accuracy 21[Reeves et al., 2017]
  22. 22. FROM PROJECTS TO ECOSYSTEMS Project A Project B Project C Participant X Part. Y [Luczak-Roesch et al., 2014]
  23. 23. DESIGNING PLATFORMS Task specificity Community development Task design PR and engagement Bootstrapping the community Serendipitous scientific discovery Engaging with people, supporting profession team Supporting individuals, finding new scientific discoveries Obtaining new citizen scientists Retaining people Supporting people, improving task completion Obtaining new citizen scientists Reinvigorating old users [Tinati et al., 2015, CHI]
  24. 24. WHAT’S NEXT? Human computation & crowdsourcing  Task assignment: what tasks are interesting/relevant for whom?  Data quality: scalable and in real-time  Peer review, collaborative approaches  The role of gamification: is science a game? Online community  Making discussions more effective Science  Citizen science platforms that everyone can use  New forms of publishing, citation, reproducibility, and replication
  25. 25. E.SIMPERL@SOTON.AC.UK @ESIMPERL WWW.SOCIAM.ORG WWW.STARS4ALL.EU 11/18/2016 25 All publications available at http://dblp.uni- trier.de/pers/hd/s/Simperl:Elena_Paslaru_Bontas

×