(How) does Gamification work?

Juho Hamari
http://juhohamari.com

http://www.homesliceapp.com/wp-content/
uploads/2014/05/lifehacker.png

“Does
gamification
work?”
or why we should not ask the question
Citation: Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does Gamification
Work? – A Literature Review of Empirical Studies on Gamification. In
Proceedings of the 47th Hawaii International Conference on System
Sciences, Hawaii, USA, January 6-9, 2014.

Authors
Jonna Koivisto
http://jonnakoivisto.com
jonna.koivisto@uta.fi
Harri Sarsa
harri.sarsa@aalto.fi

Gamification -
a trending topic
● Gartner predicts: By 2015, 50% of
organizations have gamified their processes
● “Silverbullet” for customer/user engagement
- because games are fun, gamified services
are also fun
● Companies emerging that provide
gamification services/systems
● Everyone wants to know “does gamification
really work?”

Gamification -
an untrending topic?
● “just a marketing gimmick!”
● “exploitative!”
● “bullshit!”
● “extrinsic rewards!”
● “nothing new!”
● “paternalism!”
● “manipulation!”

Gamification -
an untrending topic ?
HYPE UP:
Gartner (2011) & IEEE (2014):
Most organizations adopt !
HYPE DOWN:
Gartner (2012): Most
implementations fail !
Again: the idea is that
gamification will succeed or fail
as a whole

Research problem
● Clearly gamification is booming and we have
very polarized opinions: either a
“Silverbullet” or “Bullshit”
● But what does the research say? “Does it
actually work?”
● What kind of research has been done under
the flag of gamification?

… whatever the case; a mindblowing
increase in hits in the literature

Step 1: Search hits per db
Database Total number of results Peer-reviewed papers
EBSCOHost 399 17
Proquest 3423 124
Web of Science 56 56
Scopus 330 293
ScienceDirect 93 93
Google Scholar 3480 N/A
ACM Digital library 239 196
AISel 30 30

Step 2: Focused searches
● Search words: gamification, gameful, gamif*,
motivational affordances from all the
databases
● Selection criteria
○ Empirical study
○ Peer-reviewed
○ Full research paper
○ in an international venue
● 24 papers found

Motivational affordances
● The list of
affordances
rather well
corresponds
to what can be
considered the
blueprint of
gamification

Psychological outcomes
● Measured in 9 papers
○ 6 inferential, 3 descriptive
● Such as: Engagement, enjoyment, attitude,
social influence, flow, task involvement, etc.

Behavioural outcomes
● Measured in 21 papers
○ 15 inferential, 6 descriptive
● Such as: Amount and quality of
use/contributions, learning outcomes,
intentions to use, task performance

Overview of variables
Motivational affordances Psychological outcomes Behavioral outcomes
Points
Leaderboard
Badges/Achievements
Levels
Story/theme
Clear goals
Feedback
Rewards
Progress
Challenge
Social comparison
Clear goals
Intrinsic motivations
Extrinsic motivations
Social influence
Recognition
Reciprocal benefits
Attitude
Motivation
Happiness
Flow
Task involvement
Social motivation
Perceived added-value
Fun
Engagement
Enjoyment
Satisfaction
Amount of use
Quality of use
Quality of completed tasks
Task completion speed
Amount of contributions
Change in type of contributions
Amount of social interaction
Amount on contributions
Intentions to continue using
Intentions to recommend to others
Change in energy consumption
Learning effectiveness
Task performance
Time management
Carefulness
Learning
Response patterns
Increased knowledge

Results in qualitative
studies
● Both positive and negative perceptions
● Positive experiences in all of the studies
● The same aspects were most often disliked
by some respondents in the study such as
competition → individual differences.

Results in quantitative
studies

“Does gamification work?”
● Simple answer: According to the literature
review, most studies find positive effects from
gamification to some dependent variables
● Realistic long answer: Well …
○ The studies are quite different (different variables,
methods, etc.)
○ Methodological shortcomings
○ Relatively small amount of studies
○ Conflicting results
● Most studies report both positive and null
results

Methodological shortcomings
- to be avoided
1) Small sample sizes
2) No validated psychometric variables
3) Lack of control groups and thus inability to infer whether
gamification affected user behavior
4) Lack of control between implemented motivational
affordances
5) Experiment timeframes often short
6) None of the papers MEASURED variables from all 3
stages of gamification (affordance -> psych -> behavior)
7)Publication bias

“Does gamification work?”
● Further conclusion:
● its a horrible question to ask - “through
the disillusionment”
● We should not care whether gamification
works

Gamification - what is it?
Points
Leaderboard
Badges/Achievements
Levels
Story/theme
Clear goals
Feedback
Rewards
Progress
Challenge
Social comparison
Clear goals
Intrinsic motivations
Extrinsic motivations
Social influence
Recognition
Reciprocal benefits
Attitude
Motivation
Happiness
Flow
Task involvement
Social motivation
Perceived added-value
Fun
Engagement
Enjoyment
Satisfaction
Amount of use
Quality of use
Quality of completed tasks
Task completion speed
Amount of contributions
Change in type of contributions
Amount of social interaction
Amount on contributions
Intentions to continue using
Intentions to recommend to others
Change in energy consumption
Learning effectiveness
Task performance
Time management
Carefulness
Learning
Response patterns
Increased knowledge

Future(through “Scope of enlightenment” to “Plateau of
Productivity”)
● Future looks bright
● Moving from “disillusionment”, through
“enlightenment” to realistic “productivity”
○ No longer questions whether gamification is “bullshit”
or a “silverbullet”
○ Not a singular technology
○ There are no sure things that either work or don’t
○ It is not about the GAME MECHANICS (alone)

Future - in practice(through “Scope of
enlightenment” to “Plateau of Productivity”)
● Read specific (quality) empirical literature
related to your own case
● Execute well → “make a good game”
● Think about all the steps
○ Which design is used
○ Which psychological aspects
○ Which behavioral outcomes
○ In which context
○ To whom

Future - in practice(through “Scope of
enlightenment” to “Plateau of Productivity”)
● Create your theory about how gamification
works in your case
● and test it by yourself (avoid methodological
pitfalls)
Badges
Likes
Enjoyment
Flow
Social influence
Effective work
Sustainable work
Friendly working
environment

Thanks!
Workshop tomorrow with more about designing
gamification, theorizing about it and testing it !
More research on the topic:
http://juhohamari.com
juho.hamari@uta.fi

● Social factors related to social influence,
recognition and reciprocal benefits were
strong predictors for attitude towards using
gamification to enhance physical exercise as
well as for continuing using it
● The results show that the mere implementation of
gamification mechanics (social comparison and clear goals)
does not automatically lead to significant increases in use
activity in the studied utilitarian service, however, those
users who actively monitored their own badges and those
of others showed increased user activity.

● Perceptions of usefulness, playfulness and
enjoyment are higher in the beginning but start to
decline with prolonged tenure

● Effects of a gamification plugin deployed in a learning management
system compared to effects of a social networking site in the same
educational setting.
● Both resulted in better performance than a traditional e-learning
approach in terms of academic achievement, but for assessing
knowledge, the traditional e-learning approach was better.
● However, participation rates and scores remained low with the new
tools, although students’ attitudes were positive.

● A large-scale (n > 1000) randomized, controlled experiment on the impact of
incorporating a badge-based achievement system within an online learning tool
● Significant positive effect on the quantity of students’ contributions, without a
corresponding reduction in their quality, as well as on the period of time over
which students engaged with the tool

● Gamification works for a while
o Probably due to novelty effects
● Declines quickly

HOWEVER:
● Removing gamification results in drop of wanted
behavior in an enterprise social networking service

Crowding-out-effect
● The initial (intrinsic motivations) are replaced
and overpowered by “surrogate” motivations
● Undermining the initial intrinsic motivations
● “The rewards become more important than
the activity”

● Game-based rewards related to a fictional narrative read by teachers
were provided when the school as a whole met a fruit or vegetable
consumption goal in the alternating-treatments design.
● On intervention days, fruit and vegetable consumption increased by
39% and 33.

● Female undergraduates took a math test in a virtual classroom after
being exposed to one of three leaderboard conditions: leaderboard
with men in top positions, leaderboard with women in top positions,
and no leaderboard condition
● Participants in the female majority leaderboard condition performed
more poorly on the test than those in the male leaderboard condition
● However, they showed a higher level of academic identification than
those in the male and control conditions

● One course received a gamified curriculum, featuring a leaderboard
and badges, whereas the other course received the same curriculum
without the gamified elements.
● Students in the gamified course showed less motivation,
satisfaction, and empowerment over time than those in the non-gamified
class.
● Effect on final exam scores was mediated by students' levels of
intrinsic motivation, with students in the gamified course showing
less motivation and lower final exam scores than the non-gamified
class.

● Findings suggest that points, levels and leaderboards by
themselves neither make nor break users’ intrinsic motivation
in non-game contexts. Instead, it is assumed that they act as
progress indicators, guiding and enhancing user performance.

● “providing feedback” and “designing for optimal
challenge” into the collaboration environment, ...
significant performance gains were realized.

Hamari, J., & Koivisto, J. (2013). Social motivations to use gamification:
an empirical study of gamifying exercise. In Proceedings of the 21st European
Conference on Information Systems, Utrecht, Netherlands, June 5–8, 2013.
● Network exposure,
● Social influence,
● Recognition,
● Reciprocal benefits
... increased favorable attitude and use
intentions and WOM
-> social aspects also
important for adoption
of gamification

Social aspects are also important
●Social features and community are important in
supporting the gamification (Hamari & Koivisto
2013)

Novelty effects
● Intuitively gamification seems interesting for
users
● People are curious and will explore the
gamified elements which results in a
temporary usage increase
● However, perceptions of usefulness and
enjoyment decline with tenure (Koivisto &
Hamari, forthcoming) as well as use (Farzan
et al. 2008).

Context
● Utilitarian vs Hedonic services (Davis et al.
1992)
● Involvement of the user: Cognitive vs.
affective (Zaichowsky 1994)
● Is the service used for intrinsic or extrinsic
reason (Deci & Ryan 1985)

If the system is predominantly
utilitarian, people might have an
incompatible attitude with ”fun”

Mere addition of ”game-like”
elements does not automatically
transform the experience

The goal of the designer differs from what
games are essentially about – voluntariness,
autonomy, free choice

Types of studies
Type of study Affordances Relationship Psychological Relationship Behavioral
Full
Aff->psych
Psych->beh
Aff->beh ---------------------------------------------------------------------
>
Aff->beh, psych
hypo ( )
Descriptive

Gamification - what is it?
● Deterding et al. (2011)
○ The use of game design in non-game contexts
● Huotari & Hamari (2012)
○ The process of enhancing services/systems with
(motivational) affordances for gameful experiences
…. as means to increase X behavior
Motivational
affordances
Psychological
outcomes
Behavioral
outcomes

Take-aways (through “Scope of enlightenment” to
“Plateau of Productivity”)
● Gamification should not be considered as one technology
● Don’t give in to the hype!
○ Who cares if its “bullshit” or a “silverbullet” !?
● “Does gamification work?” is a bad question
○ Variety of different technologies
○ Variety of sought after psychological outcomes
○ Variety of sought after behavioral outcomes
○ Variety of contexts
○ Variety of users
● Use common sense, read specific (quality)
empirical literature, execute well, test yourself.

Does gamification work?
● Example studies on badges:
○ Dominguez et al. (2013): positive effect on practical assignments,
possible negative on written assignments
○ Hakulinen et al. (2013): results depend on badge type
○ Denny (2013): Only positive effects on level on contributions
○ Hamari (2013): No automatic effects from being able to compare nor
from clear goals, BUT for those with higher engagement with badges
were also more likely to be active
○ Hamari (2014): Positive effect from badges overall
○ Fitz-Walter et al. (2011) & Montola et al. (2009) both positive and
negative consequences

Future(through “Scope of enlightenment” to “Plateau of
Productivity”)
● Context of gamification
○ Hedonic or utilitarian
○ Sporadic use
○ Voluntariness / autonomy
● User traits
o Gaming motivations
o Orientation towards goals
● Psychological mediators
o How people actually experience gamification
o Do users’ motivations actually change

(How) does Gamification work?

Recommended

Recommended

More Related Content

What's hot

What's hot (9)

Viewers also liked

Viewers also liked (20)

Similar to (How) does Gamification work?

Similar to (How) does Gamification work? (20)

More from CyberForumKA

More from CyberForumKA (7)

Recently uploaded

Recently uploaded (20)

(How) does Gamification work?