Keynote talk at South African Institute for Computer Scientists and Information Technologists (SAICSIT) Oct 2012

1. Social Participation:
How Collective Activity Can Make Change
Benjamin B. Bederson
Computer Science Department
Human-Computer Interaction Lab
Institute for Advanced Computer Studies
University of Maryland
www.cs.umd.edu/~bederson
@bederson
Wednesday, October 3, 12

2. Approach
Let people collaborate with computers
in a way that can be aggregated to provide value to a greater goal
=> sometimes called Human Computation
Blog commenting
Q&A Sites
Twitter & Facebook
Wednesday, October 3, 12

3. Approach
Let people collaborate with computers
in a way that can be aggregated to provide value to a greater goal
Translation
Photo tagging
Things Face recognition Things
COMPUTERS Human detection HUMANS
can do Speech recognition can do
Text analysis
Planning
Human Computation
Wednesday, October 3, 12

4. Human Computation Taxonomy
Replace computers
with humans
Human
Computation
Replace humans
with humans Social
Crowdsourcing
Computing
Collective Intelligence
Data
Mining
Wednesday, October 3, 12

5. Trade-off Space
Computers
Speed, Affordability
Human
Computation
Human Workers
(traditional)
Quality
Wednesday, October 3, 12

6. Online Labor Markets
Wednesday, October 3, 12

7. Active Area
ESP Game - www.gwap.com
Wednesday, October 3, 12

8. Active Area
Fold It - fold.it
Wednesday, October 3, 12

9. Active Area
VizWiz - www.vizwiz.org
Wednesday, October 3, 12

10. Active Area
Color Name Models - vis.stanford.edu/color-names/
Wednesday, October 3, 12

11. A Real-World Problem: ICDL
Now: Goal:
– 4,386 books 10,000 books
– 54 languages 100 languages
– Translations in a few languages Every book in every
– 100K unique visitors/month language!
www.childrenslibrary.org
Wednesday, October 3, 12

12. Machine
Transla,on
Speed / Affordability
Amateur0Bilingual0Human0
Par,cipa,on
Professional0Bilingual0Human0
Par,cipa,on
Quality
Wednesday, October 3, 12

13. The key idea
Wednesday, October 3, 12

14. Translation with the Crowd
Translate with the Monolingual Crowd
Wikipedia: 900 translators vs. 1,200,000 contributors
Wednesday, October 3, 12

15. Machine
Translation
Speed / Affordability
Philip Resnik
Linguistics
Monolingual
Chang Hu, Ph.D.
Human
=> Microsoft
Par,cipa,on
Amateur Bilingual Human
Participation
Professional Bilingual
Human Participation
Quality
Wednesday, October 3, 12

16. The Protocol
Stopping Condition
Machine Translation
Annotation Projection
Source Best
Sentence Translation
Machine Back-Translation
Annotation Projection
Wednesday, October 3, 12

17. La0sonda0...
Wednesday, October 3, 12

18. Wednesday, October 3, 12

19. Accuracy
Sentences"per"Accuracy"Category"
160"
140" Google"
120" MonoTrans2"
#"of"Sentences"
100"
80"
60"
40"
20"
0"
1=None" 2" 3" 4" 5=All"
Google: 10% of sentences
MonoTrans2: 68% of sentences
24
Wednesday, October 3, 12

20. MonoTrans2 UI Complexity
Hard to get casual users
Too many sentences
Too many tasks
Wednesday, October 3, 12

21. MonoTrans Widgets
Too many sentences
=> One sentence at a time
Too many tasks
=> One task at a time
Wednesday, October 3, 12

22. 27
Wednesday, October 3, 12

23. 28
Wednesday, October 3, 12

24. Julio0say0that0the0 Julio0said0that0the0
mouse0needed0the0 mouse0needed0the0
tooth0for0his0secret0 tooth0for0his0secret0
formula. formula.
29
Wednesday, October 3, 12

25. Deployment
Aug 2011 – May 2012
155,000 visits
11 books (6 languages)
1,282 sentences
31,000 tasks
Wednesday, October 3, 12

26. Widgets Evaluation
2 books 739 users
Spanish 14 days
English 54 sentences
Wednesday, October 3, 12

27. Sentences"per"Accuracy"Category"
Accuracy
50"
Google"
40"
Widgets"
#"of"Sentences"
30"
20"
10"
0"
1=None" 2" 3" 4" 5=All"
Google: 31% of sentences
Widgets: 52% of sentences
Wednesday, October 3, 12

28. How0could0a0crowd Alex Quinn
CS Ph.D. student
help0you0make0an
important
decision?
Wednesday, October 3, 12

29. Wednesday, October 3, 12

30. AskSheet Example Problems
Pick a grad
school
Where to buy
groceries?
Where/when go
on vacation?
Which papers should
be accepted?
Wednesday, October 3, 12

31. Decision modeling process
Define the Problem
Develop a Model
Acquire Input Data
Develop a Solution
Test the Solution
Wednesday, October 3, 12

32. Decision modeling process
Frugal input acquisition
Research focus Acquire Input Data
Wednesday, October 3, 12

33. Create Model
Create a blank decision model with ASK formulas in cells
C2:E6. AskSheet highlights priority cells in dark green.
Wednesday, October 3, 12

34. Configure Task
Set up the task by specifying who will receive the HITs, instructions, and other
details. The "Prioritization" slider controls how many inputs to include in each batch.
Wednesday, October 3, 12

35. Preview Task
At the bottom of the setup screen is a dynamically updated preview of what workers
will see. The input form is automatically generated from the spreadsheet model.
Wednesday, October 3, 12

36. Data is Collected
The system prioritizes the inputs. Obtaining the dark green cells first
would provide the greatest opportunity to eliminate other cells. Braces
expressions show the possible range of output values for each cell.
Wednesday, October 3, 12

37. Task Completed
The model is now complete. Store C wins because its maximum possible
price is $68, is lower than the minimum possible price of either Store A ($85)
or Store B. ($71). The system avoided requesting 11 of the 27 inputs.
Wednesday, October 3, 12

38. “ASK” formula
A1: =ASK(1, 1, 5)
cost, min, max
B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
Wednesday, October 3, 12

39. “ASK” formula
A1: =ASK(1, 1, 5)
B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
Task Prioritization
A1
B1
1 2 3 4 5 6 7 8 9 10
Wednesday, October 3, 12

40. AskSheet Trial
Task: Find pediatrician that is rated well,
accepted by insurance, and sufficiently close
Answer rcvd
in < 9 hrs
Eliminated 74% of work
Wednesday, October 3, 12

41. Open Problems
•Evaluation of student work in MOOCs
•Aggregation of comments on blogs / class forums
•Assuring quality of work (i.e., product reviews)
Wednesday, October 3, 12

42. Design Lessons
Assuring quality at a viable cost remains hard
•Assume adversarial participants.
•Best practices: redundancy, gold standards, inform you
are watching, treat workers as employees
The "motivation" problem remains hard
•Don’t forget "what's in it for me".
Make it fundamental to primary task
Ethics
Wednesday, October 3, 12

43. Ethics of Human Computation
$$$ Anonymity
Wednesday, October 3, 12

44. Money Problems
• Bulk of problems are money/quality
related
• Workers complain about
– Low wages
– Not getting paid
– Slow payment
• Requesters (who are less anonymous) complain about
– Low quality work
• Also, significant other issues
– Decontextualization
– Tasks that are illegal or unacceptable
– Privacy / anonymity
Wednesday, October 3, 12

45. Current Situation
• Workers and Requesters alike build their own
reputation/quality mgmt systems
– Turker Nation
Wednesday, October 3, 12

46. Current Situation
• Workers and Requesters alike build their own
reputation/quality mgmt systems
– Turker Nation
– Turkopticon
Wednesday, October 3, 12

47. Current Situation
• Workers and Requesters alike build their own
reputation/quality mgmt systems
– Turker Nation
– Turkopticon
– CrowdFlower
Wednesday, October 3, 12

48. Questions?
MonoTrans0supported0by0Na,onal0Science0Founda,on
MonoTrans0&0SearchParty0supported0by0Google
www.cs.umd.edu/~bederson
@bederson
Wednesday, October 3, 12