This document discusses multiple perspectives on evaluating serious games. It describes an experiment using a role-playing court game workshop to explore the perspectives of an economics viewpoint (represented by a game industry association) and a scientific viewpoint (represented by a game company). The workshop revealed three main perspectives on evaluation: an economic perspective focused on effects and outcomes, a scientific perspective focused on the design process, and an arts perspective focused on the value of evaluation itself. The document concludes that considering multiple evaluation perspectives aligned with different parts of society can influence research on serious games.

1. All rise!
Multiple perspectives on serious game
evaluation
Heide Lukosch, Harald Warmelink, Rens Kortmann
International Simulation and Gaming Association Conference 2012 10-07-12
Delft
University of
Technology
Challenge the future

2. 10-07-12
Delft
University of
Technology
Challenge the future

3. Why should we evaluate
such a game?
How can we evaluate
such a game?
3 | 17

4. Structure
Starting points
Understanding of Serious Games
The Evaluation of Serious Games
Our Court Game Workshop
Multiple Perspectives on Serious Game Evaluation
The Contribution of Social Systems Theory
Conclusions and Discussion
4 | 17

5. Starting Points
Image: forresterblogs.seriousgaming
Disagreements on a proper
evaluation method within the community
Evaluation is needed to increase
the quality of serious games
Which perspectives on serious game evaluation do exist?
5 | 17

6. Understanding of (Serious) Games
Games as tools to support
processes of
Organizational change
Decision Making
Learning
Policy making
What are preconditions and effects?
Images: CharliePapa simulation Game; Craft Game, | 17
6 Kenteq, NL

7. The Evaluation of (Serious) Games
Influenced by social sciences
Theory based evaluation
Design perspective
Analytical perspective
Focus on learning effects of a game
Images: CharliePapa simulation Game; TU Delft game17
7 | session

8. The evaluation of (Serious) Games
Increasing interest in Serious Games Development
A lot of money is spent
Still less is known about
proper evaluation methods
Often pre- and post-
assesments are conducted
to find proof for effectiveness
Input – Proces – Output model
Images: CharliePapa simulation Game; TU Delft game17
8 | session

9. Our Court Game Workshop
Role-play to facilitate a discussion
Two opposite perspectives
(plaintiff and defendant)
representing our hypothesis:
existence of economic and
scientific perspective
Group discussions
Image: DIGRA Court role play 9 | 17
workshop

10. Our Court Game Workshop
Structured approach, applied to a role-play
e
Introduction Presentation Facilitated f rol
Alternatives
Identification
Analysis
of
of the group
positions discussion
Summary Presentation
le
of results o n ro
Conclusions
ole De-Briefing
on r n ro
le e
o
off rol
Based on Kolfschoten/De Vreede, 17
10 | 2009

11. Our Court Game Workshop
Dutch Civil Law
Game industry association
(plaintiff)
Game company
(defendant)
Expert Witnesses
Additional Material
Image: DIGRA Court role play workshop
11 | 17

12. Our Court Game Workshop
Items to discuss:
Types of measured variables
Number of measuring
instruments
Number of respondents
Use of theory
Objectivity of evaluation
Image: DIGRA Court role play workshop
12 | 17

13. Multiple perspectives on Evaluation
Evaluation of effects problematic
Evaluation of the design
possible
Value of Evaluation itself?
A third, „arts“ perspective emerged
Image: Dijk Patrouille game 13 | 17

14. The Contribution of Social Systems
Theory
Economics
Arts
Sciences
observe
Serious Games
14 | 17

15. Conclusions
The way we look at evaluation
of games and simulations in our research group
Tools for change
How can games
influence change
processes?
Image: Craft Game, 15 | 17 NL
Kenteq,

16. Conclusions
Multiple perspectives are related
to distinct parts of modern society:
Science
Economics
Arts
Understood as paradigms,
could they influence research
on serious games?
Image: www.designingasociety.net
16 | 17

17. Discussion
Types of measured variables
Number of measuring instruments
Number of respondents
Use of theory
Objectivity of evaluation
more items ?
Any more perspectives to consider?
17 | 17