Games User Research & User Testing 101 - talk on games and user research at IGCMG in 2015

1. A D VA N C I N G G A M E D E S I G N W I T H
E Y E - T R A C K I N G A N D T H I N K - A L O U D
S T U D I E S
J O H A N N A P I R K E R
J P I R K E R @ M I T. E D U
G R A Z U N I V E R S I T Y O F T E C H N O L O G Y
C H I P L AY - G U R T O O L S

2. • GUR is not QA and not Marketing! -> UX
• Methods sometimes similar
• E.g. Playtesting
http://www.gamasutra.com/view/feature/168114/understanding_user_research_its_.php

3. G A M E S A R E M A D E F O R
T H E P L AY E R S
http://www.gamasutra.com/view/feature/168114/understanding_user_research_its_.php

4. • Understanding player behaviour
• Determine issues and difficulties
• Object awareness and usability insights
• Does the user understand how to play the game?
• Does the user understand how to interact with the game
objects and the environment?
• Is the user interface easy to learn?
• Does the user enjoy the experience?

5. M E T H O D S
• Think Aloud Protocol (TA)
• Retrospective Testing
• Heuristic Evaluation (HE)
• Playtesting
• Game Analytics
• Physiology-based Playtesting / Biometrics
http://gameuserr.editme.com/TESTING-METHODS

6. M E T H O D S + D ATA
• Think Aloud Protocol (TA)
• Notes, Audio, Video, ..
• Retrospective Testing
• Annotations in video, notes, audio,..
• Heuristic Evaluation (HE)
• Notes
• Playtesting
• Surveys, Interviews, Game Logs
• Game Analytics
• Quantitative data <3 e.g. engagement
• Physiology-based Playtesting / Biometrics
• Biodata, “feelings”, phyiscal states, emotions,

7. Christian Rohrer - http://www.nngroup.com/articles/which-ux-research-methods/

8. T H E G A M E I S T E S T E D
N O T T H E U S E R

11. T H I N K - A L O U D ( TA ) P R O T O C O L S
• Users share thoughts and feelings loudly while
performing specific tasks
• Data is collected by noting or recording what users say
while testing
• Data = explanations & reasoning

12. E Y E - T R A C K I N G
• User’s gaze while playing the game is recorded
• What does the user see?
• What does he miss?

13. E Y E - T R A C K I N G
• Live Eye Gaze (real-time)
• Analyze users intention and their behaviour
in areas where users would expect specific
interactions
• Quantitative data (Fixation counts, revisits, ..)
• Visualisations (heat map)
• Illustrate areas of interest
• Analyze more complex data (find unnoticed
areas, needlessly complex interactions)

14. C O M B I N I N G E T A N D TA
•Qualitative TA-Data (User speaks out loud)
• Analysis of verbal in-play probes (+/-)*
• Analysis in-play responses (Bored, Frustrated, Lost,
Annoyed, Puzzled,...)*
•ET- Data (User’s gaze)
• Analysis of eye movement data
• Quantitative Analysis of eye movement data:
Fixations, Variation of mean fixations*

20. Jeanne H. Brockmyer, Christine M. Fox, Kathleen A. Curtiss, Evan McBroom, Kimberly M. Burkhart, Jacquelyn N. Pidruzny. (2009). The development of the Game Engagement Questionaire: A
measure of engagement in video game playing. Journal of Experimental Social Psychology. Vol. 45, pp. 624-634.

21. Usability / Learning Experience and Motivation
Testing
• U S A B I L I T Y, M O T I VAT I O N , L E A R N I N G
P R O G R E S S
• C O M PA R E D I F F E R E N T M E T H O D S
• T H I N K I N G A L O U D T E S T
• E Y E T R A C K I N G

22. I T E R AT I V E G A M E D E V E L O P M E N T

23. B I G D ATA ! ! !
http://insights.dice.com/2012/12/07/for-riot-games-big-data-is-serious-business/

24. C H I P L AY
• What Drives People: Creating Engagement Profiles of
Players from Game Log Data
• 120 mio race entries from 1.2 mil players
http://chiplay.org/acm-dl-table-of-contents-chi-2015/
http://delivery.acm.org/10.1145/2800000/2793114/p369-harpstead.pdf?
=80.110.92.122&id=2793114&acc=OPENTOC&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E9F04A3A78F7D3B8D&CFID=551694141&CFTOKEN=62044429&__acm__=1444761920_df0bd79fd99
9812ced8096f35d127f3a

25. J O B S ?
• UX/Usability:
• Psychology / Design / Computer Science
• Data Analyst / Engineer
• Math, Statistics, CS (Machine Learning)
• http://gamesuserresearchsig.org/career/games-user-
research-jobs/

26. T H A N K Y O U F O R Y O U R
AT T E N T I O N .
J O H A N N A P I R K E R , J P I R K E R @ M I T. E D U , W W W. J P I R K E R . C O M , @ J O E Y P R I N K