1. Social Computing @ Know-Center
1
Crowd-Based Evaluations
Christoph Trattner
Know-Center
ctrattner@know-center.at
@Graz University of Technology, Austria
. Christoph Trattner 15.12.2015 – Graz, Austria
2. Social Computing @ Know-Center
2
Before I will start in this class, I will talk a bit about
myself (background), what I am doing, etc.
. Christoph Trattner 15.12.2015 – Graz, Austria
3. Social Computing @ Know-Center
3
Where do I come from (Austria)?
. Christoph Trattner 15.12.2015 – Graz, Austria
4. Social Computing @ Know-Center
4
Graz
. Christoph Trattner 15.12.2015 – Graz, Austria
5. Social Computing @ Know-Center
5
Trondheim
. Christoph Trattner 15.12.2015 – Graz, Austria
6. Social Computing @ Know-Center
6
Academic Back-Ground?
Studies Computer Science at Graz University of
Technology & University of Pittsburgh
Worked since 2009 as scientific researcher at the KMI &
IICM (BSc 2008, MSc 2009)
My PhD thesis was on the Search & Navigation in Social
Tagging Systems (defended 2012)
Since Feb. 2013 @ Know-Center
Leading the SC Area @ KC
@TUG
WebScience
Semantic Technologies
@NTNU
Datamining
. Christoph Trattner 15.12.2015 – Graz, Austria
7. Social Computing @ Know-Center
7
What is my group doing?
… we research on novel methods and tools that exploit
social data to generate a greater value for the
individual, communities, companies and the society as
whole.
Our competences:
• Network & Web Science
• Science 2.0
• Crowdsourcing
• Predictive Modeling
• Social Network Analysis
• Information Quality Assessment
• User Modeling
• Machine Learning and Data Mining
• Collaborative Systems
Our Services:
• Social Analytics: Hub-, Expert -, Community -
, Influencer -, Information Flow-, Trend
(Event) Detection, etc.
• Information Quality Assessment
• Social & Location-based Recommander
Systems
• Customer Segmentation
• Social Systems Design
. Christoph Trattner 15.12.2015 – Graz, Austria
8. Social Computing @ Know-Center
8
Ok, lets start….
. Christoph Trattner 15.12.2015 – Graz, Austria
9. Social Computing @ Know-Center
9
Crowd-Based Evaluations
Definition
Crowd
1. A large number of people gathering together.
2. A group of people.
3. A large number of things positioned or considered together.
Source: http://www.thefreedictionary.com/crowd
. Christoph Trattner 15.12.2015 – Graz, Austria
10. Social Computing @ Know-Center
10
Crowd-Based Evaluations
Definition
Crowdsourcing
is the process of obtaining needed services, ideas, or
content by soliciting contributions from a large group
of people, and especially from an online community,
rather than from traditional employees or suppliers.
Source: http://en.wikipedia.org/wiki/Crowdsourcing
. Christoph Trattner 15.12.2015 – Graz, Austria
11. Social Computing @ Know-Center
11
Crowd-Based Evaluation
And what is crowd-based evaluation?
• Evaluation with a bunch (a lot) of people
• In the literature also often referred as:
• „crowdsourcing“
• Evaluation with crowdsourcing platforms
. Christoph Trattner 15.12.2015 – Graz, Austria
12. Social Computing @ Know-Center
12
Crowd-Based Studies vs. Lab Studies
Crowdsourcing
• Study-type: Controlled
• Payment: 0.1 USD
• Design: Between
subject
• Duration: Short
(seconds)
• Users: > 100
• Duration: minutes
Lab Studies
• Study-type: Controlled
• Payment: > 30USD
• Design: Within subject
• Duration: Long (hours)
• Users: < 100
• Duration: days/weeks
. Christoph Trattner 15.12.2015 – Graz, Austria
13. Social Computing @ Know-Center
13
Is crowdsourcing something new?
1714 – The Longitude Prize: When the British government was
trying to find a way to measure a ship’s longitude, they offered
the public a monetary prize to whomever came up with the best
solution. (2014 http://longitudeprize.org/)
1884 – Oxford English Dictionary (OED): 800 volunteers
catalogued words to create the first fascicle of the OED
2001 – Launch of Wikipedia
2004 – Toyota’s first “Dream car art” contest: Children were asked
globally to draw their ‘dream car of the future.’
2006 – Jeff Howe coined the term crowdsourcing in Wired
(magazine).
. Christoph Trattner 15.12.2015 – Graz, Austria
14. Social Computing @ Know-Center
14
Crowdsourcing Plattforms
. Christoph Trattner 15.12.2015 – Graz, Austria
15. Social Computing @ Know-Center
15
. Christoph Trattner 15.12.2015 – Graz, Austria
16. Social Computing @ Know-Center
16
Crowdsourcing Platforms for Evaluation
Amazon Mechanical Turk
URL: https://www.mturk.com/mturk/welcome
• Founded March 2007- 100,000 workers in over 100
countries.
• 2011 - over 500,000 workers from over 190 countries
in January 2011.
• Tasks = Hits
• Workers = Turkers
. Christoph Trattner 15.12.2015 – Graz, Austria
17. Social Computing @ Know-Center
17
Mturk - workflow
. Christoph Trattner 15.12.2015 – Graz, Austria
18. Social Computing @ Know-Center
18
What do I have to do
...as a turker?
. Christoph Trattner 15.12.2015 – Graz, Austria
19. Social Computing @ Know-Center
19
Mtruk.com
. Christoph Trattner 15.12.2015 – Graz, Austria
20. Social Computing @ Know-Center
20
. Christoph Trattner 15.12.2015 – Graz, Austria
21. Social Computing @ Know-Center
21
. Christoph Trattner 15.12.2015 – Graz, Austria
22. Social Computing @ Know-Center
22
. Christoph Trattner 15.12.2015 – Graz, Austria
23. Social Computing @ Know-Center
23
What do I have to do
...as a hit submitter?
. Christoph Trattner 15.12.2015 – Graz, Austria
24. Social Computing @ Know-Center
24
. Christoph Trattner 15.12.2015 – Graz, Austria
25. Social Computing @ Know-Center
25
. Christoph Trattner 15.12.2015 – Graz, Austria
26. Social Computing @ Know-Center
26
. Christoph Trattner 15.12.2015 – Graz, Austria
27. Social Computing @ Know-Center
27
Source: http://dev.globis.ethz.ch/crowdstudy/res/eics2013.pdf
. Christoph Trattner 15.12.2015 – Graz, Austria
28. Social Computing @ Know-Center
28
. Christoph Trattner 15.12.2015 – Graz, Austria
29. Social Computing @ Know-Center
29
. Christoph Trattner 15.12.2015 – Graz, Austria
30. Social Computing @ Know-Center
30
. Christoph Trattner 15.12.2015 – Graz, Austria
31. Social Computing @ Know-Center
31
. Christoph Trattner 15.12.2015 – Graz, Austria
32. Social Computing @ Know-Center
32
Who are the Turkers?
. Christoph Trattner 15.12.2015 – Graz, Austria
33. Social Computing @ Know-Center
33
Mturk Study
Survey of 573 people
Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B.
(2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon
Mechanical Turk". In: CHI EA 2010. (2863-2872)
. Christoph Trattner 15.12.2015 – Graz, Austria
34. Social Computing @ Know-Center
34
Mturk Study
. Christoph Trattner 15.12.2015 – Graz, Austria
35. Social Computing @ Know-Center
35
Mturk Study
. Christoph Trattner 15.12.2015 – Graz, Austria
36. Social Computing @ Know-Center
36
Mturk Stats
Source: http://mturk-tracker.com/general/general-data/
. Christoph Trattner 15.12.2015 – Graz, Austria
37. Social Computing @ Know-Center
37
Mturk Alternatives
Biggest Limitation:
„Officially“ only available (as Hit submitter) in the US!
Alternatives:
• CrowdFlower
• Founded in 2007 in SF
• Over 153 other countries
• Jobs = Hits
• except from that more or less identical to MTurk
. Christoph Trattner 15.12.2015 – Graz, Austria
38. Social Computing @ Know-Center
38
CrowdFlower.com
. Christoph Trattner 15.12.2015 – Graz, Austria
39. Social Computing @ Know-Center
39
CrowdFlower - Stats
Source: http://venturebeat.com/2013/11/14/how-enterprises-use-crowdsourcing-infographic/
. Christoph Trattner 15.12.2015 – Graz, Austria
40. Social Computing @ Know-Center
40
CrowdFlower - Stats
. Christoph Trattner 15.12.2015 – Graz, Austria
41. Social Computing @ Know-Center
41
CrowdFlower - Stats
. Christoph Trattner 15.12.2015 – Graz, Austria
42. Social Computing @ Know-Center
42
CrowdFlower - Stats
. Christoph Trattner 15.12.2015 – Graz, Austria
43. Social Computing @ Know-Center
43
CrowdFlower - Stats
. Christoph Trattner 15.12.2015 – Graz, Austria
44. Social Computing @ Know-Center
44
Mturk: Alternatives
Clickworker:
• Qualification: Little Knowledge – 2 pre-assessment
needed (language check)
• Registration Fee: free
• Min. Payment: 1USD
Example Tasks:
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
. Christoph Trattner 15.12.2015 – Graz, Austria
http://clickworker.com
45. Social Computing @ Know-Center
45
Mturk: Alternatives
Microworkers:
Qualification: Little Knowledge
Registration Fee: free
Min. Payment: 0.10 USD
Example Tasks:
• Twitter follower tasks
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
. Christoph Trattner 15.12.2015 – Graz, Austria
http://microworkers.com
46. Social Computing @ Know-Center
46
...is crowdsourcing a reasonable alternative for „real“
user studies?
. Christoph Trattner 15.12.2015 – Graz, Austria
47. Social Computing @ Know-Center
47
Crowdsourcing User Studies
• In 2008 Kittur et al. conducted 2 experiments to test
the utility of Mechanical Turk as a user study platform
• In both experiments workers had to rate quality of
Wikipedia articles
• Comparative study – expert raters vs. turkers
Kittur, A., Chi, E. H., & Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk.
In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456).
ACM.
. Christoph Trattner 15.12.2015 – Graz, Austria
48. Social Computing @ Know-Center
48
Experiment 1
Study Design
• 14 radomly chosen articles
• 7-point likert scale
• writing, factually accuracy, neutrality, structure...
• Questions - Wikipedia “Featured article criteria“
Results
• 58 users provided 210 ratings
• 93 ratings within 24hrs (very fast)
• Correlation was only marginally significant
r = 0.50, p = .07
. Christoph Trattner 15.12.2015 – Graz, Austria
49. Social Computing @ Know-Center
49
„The results from Experiment 1 provided only weak
support for the utility of Mechanical Turk as a user
measurement tool.“
. Christoph Trattner 15.12.2015 – Graz, Austria
50. Social Computing @ Know-Center
50
Experiment 2
Study Design
• More or less indendical to Experiment 1
• Contrary to Exp. 1, users had to input how many
references, images, and sections the article had.
• After that they were asked the overall article quality
question
Results
• 124 users provided 277 ratings
• Correlation was higher AND statistically significant
r = 0.66, p = 0.01
. Christoph Trattner 15.12.2015 – Graz, Austria
51. Social Computing @ Know-Center
51
Stats Mturk Experiment
. Christoph Trattner 15.12.2015 – Graz, Austria
52. Social Computing @ Know-Center
52
Other Applications
CrowdSearch
Yan, T., Kumar, V., & Ganesan, D. (2010, June). Crowdsearch: exploiting crowds for accurate
real-time image search on mobile phones. In Proceedings of the 8th international conference on
Mobile systems, applications, and services (pp. 77-90). ACM.
. Christoph Trattner 15.12.2015 – Graz, Austria
53. Social Computing @ Know-Center
53
Eval search index
. Christoph Trattner 15.12.2015 – Graz, Austria
54. Social Computing @ Know-Center
54
Eval Search index
. Christoph Trattner 15.12.2015 – Graz, Austria
55. Social Computing @ Know-Center
55
Eval Search Index
Lin, Y., Trattner, C., Brusilovsky, P. and He, D.: The Impact of Image Descriptions on User Tagging
Behavior: A Study of the Nature and Functionality of Crowdsourced Tags, Journal of the Association for
Information Science and Technology, 2015. (in press)
. Christoph Trattner 15.12.2015 – Graz, Austria
56. Social Computing @ Know-Center
56
...ok that‘s basically it
. Christoph Trattner 15.12.2015 – Graz, Austria
57. Social Computing @ Know-Center
57
Thank you!
Christoph Trattner
Email: ctrattner@know-center.at
Web: christophtrattner.info
Twitter: @ctrattner
Sponsors:
. Christoph Trattner 15.12.2015 – Graz, Austria
58. Social Computing @ Know-Center
58
Any questions?
. Christoph Trattner 15.12.2015 – Graz, Austria