Presentation held at ACM Workshop on Serious Games at ACM Multimedia 2014 to introduce techniques that can be used to match Human Computation Tasks to Game Mechanics. The development process of a traditional digital game is described to put the basis on which the development of GWAP can be described. The concept of Human Computation Task is provided along with the most common multimedia refinement tasks that can be found in literature and everyday scenarios. Finally game mechanics that could be matched to particular task instances are shown, along with examples. The entire process has been applied to a real Game with a Purpose scenario.

1. MATCHING GAME MECHANICS
AND HUMAN COMPUTATION
TASKS IN GAMES WITH A
PURPOSE
LUCA GALLI
POLITECNICO DI MILANO
ACM International Workshop on Serious Games

2. ABOUT MYSELF
LUCA GALLI
Ph.D. Student - Politecnico di Milano
lgalli@elet.polimi.it
2
• RESEARCH BACKGROUND AND INTERESTS
• Crowdsourcing and Human Computation
• Game Design
• Robotics & AI
ACM International Workshop on Serious Games

3. OUTLINE
• Brief recap on Games with a Purpose
• Current Issues
• Development Process
• Task Design
• Game Mechanics & Operation Matching
• Use Case Study: Sketchness
ACM International Workshop on Serious Games
3

4. GAMES WITH A PURPOSE
ACM International Workshop on Serious Games
4
Games with a Purpose (GWAPs) are digital games that generate
useful data as a by-product of play. [vA06,LvA09]
The design of a GWAP requires to create a game so that its
structure encourages computation, correctness of the output and
players retention.

5. ARE THEY WORKING?
ACM International Workshop on Serious Games
5
ESP Game
Purposes:
Let players determine the
contents of images by
submitting meaningful labels
they can agree on.
Results:
A total of 13,630 people
played the game during the
test phase, generating
1,271,451 labels for
293,760 different images.
Luis von Ahn and Laura Dabbish. 2004.
Labeling images with a computer game.

6. GAME… MECHANICS?
ACM International Workshop on Serious Games
6
Input Agreement Output Agreement Inversion problem
“These game mechanics are the building blocks with which one could
create games with a purpose to solve computational tasks”.
Designing Games with a Purpose - Luis Von Ahn

7. NO!
THEY ARE
VALIDATION
TEMPLATES!
ACM International Workshop on Serious Games
7

8. ACM International Workshop on Serious Games
8
Katharina Siorpaes and Martin Hepp. 2008. OntoGame:
weaving the semantic web by online games.

9. GAMES WITH A PURPOSE:
CURRENT ISSUES
ACM International Workshop on Serious Games
9
● “Ad-Hoc” Design
● Lacking of mapping guidelines from task to game
mechanics
E.G. Real user comments about OntoGalaxy

10. ISSUE 1: «AD HOC» DESIGN
ACM International Workshop on Serious Games
10
Game design is primarily an artistic process
and reliance on formal procedures is
inimical to creativity
Yet a GWAP have a lot in
common with traditional
software due to
their very nature…

11. GAME DEVELOPMENT PROCESS
ACM International Workshop on Serious Games
11
Derived by modeling the guidelines and best practices found in:
Fullerton, T.; Swain, C. & Hoffman, S.
Game Design Workshop: A playcentric
approach to creating innovative games,
2008
Crowford, C. The Art of Computer
Game Design, 1984

12. 12
Taken from the
Book: Level
Up, By Scott
Rogers
ACM International Workshop on Serious Games

13. ACM International Workshop on Serious Games
13
Taken from the
book:
Game
Mechanics:
Joris Dorman,
Ernest Adams

14. ACM International Workshop on Serious Games
14
Board
Prototype for
IGAM
Challenge

15. ACM International Workshop on Serious Games
15
Prototype, Alex Mercer

16. ACM International Workshop on Serious Games
16
Alpha version
of Unreal
Tournament

17. GWAP DEVELOPMENT
PROCESS
ACM International Workshop on Serious Games
17

18. TASK DESIGN
A Human Computation Task (HCT) is a “unit of work“ assigned to a user of a
Human Computation system.
Input Objects
ACM International Workshop on Serious Games
18
Task Task
Description
Output Objects
Execution Interface
Operations
Aggregation
Strategy

19. TASK DESIGN: INPUT OBJECT
Input object for a HC task may include texts, images, audio segments, video
segments, other unstructured data, structured data
Input Objects
ACM International Workshop on Serious Games
19
Task Task
Description
Output Objects
Operations
Aggregation
Strategy

20. TASK DESIGN: DESCRIPTION
A description details the goal that has to be reached and a set of admissible
operations that represent the mean with which this particular task can be
accomplished by the user.
Input Objects
ACM International Workshop on Serious Games
20
Task Task
Description
Output Objects
Operations
Aggregation
Strategy
Execution Interface

21. TASK DESIGN: AGGREGATION
Usually one single contribution from a user does not suffice for obtaining
meaningful results, thus the contribution of several users is aggregated using
techniques such as ranking, clustering, majority voting…
Input Objects
ACM International Workshop on Serious Games
21
Task Task
Description
Output Objects
Operations
Aggregation
Strategy
Execution Interface

22. DEFINING OPERATIONS
Input Objects
ACM International Workshop on Serious Games
22
Execution Interface
Task Output Objects
Operations
Task
Description
Aggregation
Strategy

23. OPERATION TYPES
ACM International Workshop on Serious Games
23
They fall in two broad categories:
Generative tasks
Decision tasks
The media refining tasks that are suited to benefit from
human contributions are:
• Object Recognition/Identification/Detection
• Clustering
• Ordering
• Natural Language Processing
• State Space Exploration
• Content Generation/Submission
• User preference/opinion elicitation

24. OBJECT RECOGNITION
IDENTIFICATION, DETECTION
ACM International Workshop on Serious Games
24
Given a specific object, identify it in the image or
environment with an annotation which selects a
subset of samples with a particular meaning.
(E.G. Tracing a bounding box)

25. CLUSTERING
ACM International Workshop on Serious Games
25
Define a (subjective) similarity measure to compare
the input data with and group objects into clusters
based on it.

26. ORDERING
ACM International Workshop on Serious Games
26
Define a (subjective) evaluation criteria to compare
the input data and order the objects based on the
chosen criteria.

27. NATURAL LANGUAGE PROCESSING
ACM International Workshop on Serious Games
27
Solving various Natural Language Processing
tasks by exploiting humans' ability to understand
natural language
And so on…

28. GWAP DEVELOPMENT
PROCESS
ACM International Workshop on Serious Games
28

29. «ISSUE 2»: TASK TO GAME
MECHANICS MATCHING
How to match the task to proper game mechanics in
order to design a GWAP?
ACM International Workshop on Serious Games
29
Does a game with game
mechanics similar to the task
exists?
If so, integrate the task within
the game
If not, custom game mechanics
has to be implemented.

30. SOLUTION MECHANICS
Game mechanics that were similar to the proposed
multimedia refinement tasks has been defined from the
analysis of manuals about game design and specialized
websites such as BoardGameGeek
Agreement
Betting/Wagering
Memory
ACM International Workshop on Serious Games
30
Trivia
Tile (Resource) -
Placement
Pattern Building
Line Drawing
Bluffing
Area Enclosure

31. AGREEMENT
ACM International Workshop on Serious Games
31
Players are requested to reach an agreement over a
question or a topic based on some hints provided by
the game. Agreement is one of the most widely used
mechanics in GWAP, being the foundation on which
templates like input-agreement and output-agreement
rely on to be able to automatically validate the
contributions of different players.

32. AGREEMENT APPLIED
TO GWAPS
ACM International Workshop on Serious Games
32
Purpose: Collect “common-sense” fact for
specific words.
Game Mechanic: players must agree on the same tags
by relying on an image as their only common hint

33. TILE PLACEMENT
ACM International Workshop on Serious Games
33
Tile Placement games feature placing a piece to
score points, based on adjacent pieces or pieces in
the same group/cluster, taking into consideration
non-spatial properties like color, "feature
completion", cluster size etc.

34. TILE PLACEMENT APPLIED TO GWAPS
Game Mechanic: align the sequence contained in each row in
order to obtain the greatest number of columns with matching
colors.
ACM International Workshop on Serious Games
34
Purpose: Used to
arrange the sequences
of D.N.A, R.N.A or
proteins to identify
regions of similarity
Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, et al. (2012) Phylo: A Citizen
Science Approach for Improving Multiple Sequence Alignment

35. MEMORY
ACM International Workshop on Serious Games
35
Games that use the Memory mechanic
require players to recall previous game
events or information in order to reach
an objective.

36. MEMORY APPLIED TO
GWAPS
ACM International Workshop on Serious Games
36
Purpose: Find similar Images
Game Mechanic: Exploit the visual memory of
players to find images which are similar.
And so on…

37. GAME TO MECHANICS
MATCHING
ACM International Workshop on Serious Games
37
Game Mechanics Task Type Significative Examples
Agreement Object Recognition,
Clustering, Ordering, NLP
ESP Game, TagATune
Tile (Resource)- Placement Clustering, Ordering Phylo
Line Drawing Object Recognition Sketchness, Squigl
Betting/Wagering User Preferences /
Opinion Elicitation
N/A
Memory Clustering FlipIt
Pattern Building State Exploration FoldIt, Eyewire
Bluffing Ordering, Object
Identification
Disguise, Search War
Trivia Natural Language
Processing
Verbosity, WebPardy
Area Enclosure Object Recognition PeekABoom, Ask’nSeek

38. CASE STUDY: FASHION TREND MINING
Problem statement: segment fashion images for mining trends based
on visual features of garments (e.g. color and texture)
Use case: identifying trends in collections of images of people
and garments
Applications: retrieving similar garments, inspect clothing trends
in image collections, analyzing trends change in the years
ACM International Workshop on Serious Games
38
Color
descriptors
Texture
descriptors
coarse
(sub-)image
similarity

39. THE FASHION TREND MINING PIPELINE
ACM International Workshop on Serious Games
39
Male, 24 Female, 22Female?, ??

40. REQUIREMENTS SPECIFICATION
Input Objects: A fashion image, an optional
tag defining the garment to identify.
Output Data: For each submitted task the game
has to provide the contour of the garment within
the image (Polyline) and a tag defining the
garment that has been segmented
ACM International Workshop on Serious Games
40
Task Description: Recognize if a particular
garment is present within a picture or define a
new one and outline its contours.
Aggregation Strategy: assign a value of 0 to
each pixel outside the contour and 1 to each
pixel contained within the contour, sum all the
contribution and apply a threshold based on the
number of players.

41. LINE DRAWING
ACM International Workshop on Serious Games
41
Games that make use of this mechanics
involve drawing drawing of lines in one
way or another.

42. LINE DRAWING
APPLIED TO GWAPS
ACM International Workshop on Serious Games
42
Purpose: Segment images
Game Mechanic: draw the shapes of objects in a provided image
in order to make the other players guess the underlying object.

43. GWAP DESIGN
Solution Mechanics:
Line Drawing
Agreement
Established genre:
Draw and Guess
Inversion Problem Mechanic
PLAYERS:Number >=2
Multilateral Competition
Two different roles:
Sketcher: has to draw the contours
of the stated garment
Guesser: has to guess the garment
drawn by the sketcher
ACM International Workshop on Serious Games
43

44. PLAYER ROLE: SKETCHER
ACM International Workshop on Serious Games
44
● The only player to see the
low confidence image
● “May” be asked to provide a
tag for the image
● Is asked to draw the contour
of the object for which the
tag is provided within the
allotted time
● Goal of the Sketcher is to
let the other players guess
the tag within a time slot
without providing
any other hints than the
contour

45. PLAYER ROLE: GUESSER
ACM International Workshop on Serious Games
45
● Any other player in the game
● His/Her goal is to guess the
object for which the Sketcher
has provided the contour
● Not allowed to draw on the
whiteboard, just to type
in the chat box the probable
answer as fast as possible
● Scoring:
● Sketcher: 10 pts + 1 for each guesser
● Guesser: 10 pts to the first, then
decreasing down to five

46. RESULTS:
ACM International Workshop on Serious Games

47. RESULTS: PIXEL ACCURACY
u: 95.4 std: 1.3 (whole dataset) u: 89.0 std:0.8
Sketchness Aggregator
Kota Yamaguchi, M Hadi Kiapour, Luis E
Ortiz, Tamara L Berg, "Parsing Clothing in
Fashion Photographs", CVPR 2012
ACM International Workshop on Serious Games

48. BUT CONTROLLED DATASET!
ACM International Workshop on Serious Games

49. RESULTS:
USER SUBMITTED CONTENT
Baseline: Background
Algorithm Mean Standard Deviation
Fashionista 81.7901 13.555
Paperdoll 78.2486 13.5549
Sketchness 93.2669 9.6266
Baseline: against manually annotated groundtruth
Algorithm Mean Standard Deviation
Fashionista 26.7811 31.4905
Paperdoll 54.2095 37.056
Sketchness 81.3803 24.4377
ACM International Workshop on Serious Games

50. THE CUBRIK PROJECT
50
● CUbRIK is a research project
financed by the European Union
● Goals:
● Advance the architecture of
multimedia search
● Exploit the human
contribution in multimedia
search
● Use open-source
components provided by
the community
● Start up a search business
ecosystem
● http://www.cubrikproject.eu/
ACM International Workshop on Serious Games