User Experience 6: Qualitative Methods, Playtesting and Interviews

Unit 6: Qualitative Methods:
Playtesting and Interviews
Second term, January 2019 Dr. Marc Miquel Ribé
Course in User Experience
Bachelor Degree in Video Game Design and Production
Computer Engineering for Information System Management

Goal of the Session
Learn to plan and conduct playtesting sessions and point out
its strengths and weaknesses.

Overview of the Session
6.1 Playtesting / Usability testing
6.1.1 Purpose and problems (What)
6.1.2 Production pipeline (When)
6.1.3 Participants (Who)
6.1.4 Method (How)
6.1.5 Cost
6.1.6 Conclusions and Report
6.1.7 Pros / Cons
Playtesting is king

6.2 Interviews
6.2.1 Purpose and problems (What)
6.2.3 Environment / production pipeline (When
and Where)
6.2.4 Conducting the interview (How)
6.2.5 Interview structure (How)
6.2.6 Question creation
6.2.5 Typical Questions
6.2.6 Data transcription and interpretation
6.2.7 Pros / Cons
Listening to the user talk
First goal: learn to plan and conduct playtesting sessions and point out its
strengths and weaknesses.
Second goal: learn to how to plan for an interview and interpret its results.

5.1 Playtesting
The Art of Game Design: A Book of Lenses. Jesse Schell. Chapter 25, p. 390.
The truth is… I hate playtesting - Jesse Schell
"My work is not ready to show it; People just won't get it”
“They are going to give me feedback that I already know”
You need to overcome this fear!

Playtesting is the single most important activity in a game design process, and ironically, it is one
the one designers understand the least about. The common misconception is that playtesting is
simple—just play the game and gather feedback. It is more than that, and one who conducts best
a playtesting session is the games user researcher.
Playtesting involves systematic observation under controlled conditions to determine how
people play the game. The way a good playtesting session is conducted is very similar to a
usability testing, but paying attention to many more aspects involving the player user experience,
the game’s usability issues, game functioning, among others.
Playtesting, as an exploratory technique, should help answering the question:
“How would a final player respond to our game under realistic conditions?”
Extra Credits - Playtesting - How to Get Good Feedback on Your Game
[https://www.youtube.com/watch?v=on7endO4lPY]
For Halo 3, Microsoft Games User Research conducted over 3000 hours of playtesting with more than
600 players in one of the most sophisticated playtesting facilities in the world.

a) What are we studying with playtesting?
• User experience – How does this make her feel now?
• User behaviour / interactions – What is she doing?
Can she use the user interface properly?
• Obviously, we also pay attention to the game elements and their functioning
(involuntarily, by doing playtesting we can also find QA aspects such as bugs).
On the contrary, we cannot obtain any data in regards of the recalled experience or the
physiological reactions.
5.1.1 Purpose and problems (what)
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)

a) What are we studying with usability?
• User experience – How does this make her feel now?
• User behaviour / interactions – What is she doing?
Can she use the user interface properly?
• Obviously, we also pay attention to the game elements and their functioning
(involuntarily, by doing playtesting we can also find QA aspects such as bugs).
On the contrary, we cannot obtain any data in regards of the recalled experience or the
physiological reactions.
5.1.1 Purpose and problems (what)
Recalled UX (later)
Now
UX

b) What kind of method it is mainly qualitative and behavioural
Datasource
playtesting
usability
eye tracking
data analysis
Interviews & focus groups surveys
physiological data
playtesting/usability
think-aloud
Playtesting answers both what and why questions but not how much

Remember that playtesting is one of the few methods that can work both ways: theory-driven
(i.e. question-driven) and data-driven.
This is the theory-driven. We start with a question, we find a concept (theory) that may help us
understanding it, then we design the method to find an answer to the problem.
1. Context/Object
2. Problem/Question/Hypothesis
3. Concept / Theoretical Framework
4. Methods (Methodology)
5. Measurements (Methodology)
6. Results analysis
7. Conclusions
8. Reporting
We can also perform data-driven research with playtesting. That means letting the player play
the game and observe what happens with no questions in mind (problem: higher costs).
c) Experiment structure

In playtesting, researchers need to look for a very broad set of potential problems. These are the
concepts that might arise some research questions:
• Mechanics and balance. The mechanics must work as expected. The game must be fair for
the different types of players in case they have different attributes.
• Aesthetics and narrative. Is the story and its elements too pretentious? Are the characters
designed to appear as real people?
• Visual interface usability. As we all know, usability is the extent to which players understand
the visual interface and menus, and are able to successfully operate it to achieve intended
tasks.
• Controls usability. Each game has its own custom mapping of controls. They essential to
deliver a smooth experience during the gameplay.
• Affect/Emotions. Are players frustrated, bored or amused? What emotions can we see in
their faces?
• Motor. How much physical skill the game demands of the player?
• Cognition. Players must be able to solve the cognitive challenges that the game presents.
Playful Design. Chapter 8. John Ferrara. Classes of problems. p.100.
d) Most common concepts we should study in a playtesting session

The reason? It is a matter of productivity: you need to be sure that the playtesting is giving you
important data, and this is the one related to the KEY GAME AREAS.
• General setup. Tutorials or (even better) initial levels are fundamental to test.
All the things the player needs to learn (mechanics, etc.).
Test the first hour of play. This is the most important hour for a video game.
• Custom game setup. You want to skip to specific save points (scenarios), or shortcuts (do not
forget to take them out in the final version!), or test a custom version (e.g. a specific feature).
Test control situations. Sometimes you want to look at the same situation from all the possible
angles with all sort of people. E.g. You want to see how very different people enjoy a final battle
(after playing the previous part, without, etc.) Fullerton. Game Design Workshop. p.265-268.
The First Hour Experience: How the Initial Play Can Engage (or Lose) New Players. Cheung et al. (2014).
We can learn by just playtesting… without any question in mind.
But it is more important to look first for the things you are looking for (your research questions).

5.1.2 Production pipeline (when)
As early as possible. It is never too early: as soon as you have a prototype (squares moving
around the screen). GO GO GO.
The development team, all the time!
These would be the ‘heuristics experts’
QA

We may never stop doing playtesting, while we do not need to close the game.
Or we can stop when data we obtain is very similar.
We are here

5.1.3 Participants (who)
There are three different takes on who to test, and they are all right:
1. Most common and wise: test your target player, the one who is going to play, who is going to
spend the money. Find out how old they are, genre, previous gaming knowledge, etc. (Ferrara,
2012). Do not forget that identifying the target player or players can be difficult.
Usually, game fans are no good playtesters. If you recruit exclusively from eager volunteers, maybe
you are oversampling players who are skilled, knowledgeable, or positively biased toward gaming.
This could distort the picture of the game that emerges from testing.
2. The circles or classes: developers, team, mates, friends, family, formal playtesters.
Playtesting with confidants, playtesting with people you do not know, playtesting with your
target audience (Fullerton, 2014)
3. Everyone and judge its value according to the distance to the target. Everyone and stop
playtesting when the comments get repeated.
ANECDOTE: Age of Empires 2 was even tested with grandmas. They did not enjoyed much, but they could
operate properly with the first stages of the game. Conclusion: The game was very usable!
(Fullerton, Game Design Workshop, p. 267,)
Survey to select playtesters for VALVE: [http://www.valvesoftware.com/gamersurvey.php?action=survey]

In its purest form, playtesting is a simple process: have people play the game, and then ask them
what they thought. The real trick is getting as many opinions as possible.
• Usability issues:
Nielsen states that around five users already give many insights on what the problems are.
[https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/]
Other authors like Faulkner affirm that for a 82% of the usability problems you need around 10
users. [http://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-
participants-for-usability-studies.php] The answer to how many users oscillates between 5-10.
• User Experience aspects:
You may never stop doing playtesting as long as the project has funding and no firm release date.
Remember, Halo 3 was tested by 600 playtesters. However, it is reasonable to stop if you are
receiving very equivalent data.
How many participants

• Each playtester receives a reward in
compensation for the time and effort.
This may vary depending on who is
behind the test. Microsoft offers free
software, while consultancy companies
usually pay 30€ for an hour.
• The real cost is in terms of organization,
resources, and material evaluation (time
more than money).
[https://www.activision.com/company/playtest]
By completing the sign-up survey, you may be eligible to
participate in future studies, where you will typically come
out to our offices and try out unreleased games, apps, or
websites and help by giving us your thoughts on them.
We are always looking for players of ALL types so please
be as honest as possible when answering these
questions. As testing opportunities arise, we may reach
out to you with follow-up questions via e-mail or a phone
call. As a token of our appreciation for participating in a
study, you may receive swag, gift cards or games.
Activision survey to become a playtester
Playtester reward

5.1.4 Conducting the session (where)
There are three different takes on who to test, and they are all right:
1. In your studio: (or whatever you call the place you actually make the games)
Pros: Developers are there, the game is there. Super convenient. Everyone from the team
can get convinced of the right or wrong direction.
Cons: Playtesters might not feel completely comfortable (pressures, people looking, etc.).
2. In a playtesting lab: large companies offer labs set aside for playtesting.
Pros: It probably has all the things you need: mirrors, cameras, etc. The right testers.
Cons: It is expensive. Depending on the target player, it might not be the best place.
3. At some public venue: could be a shopping mall, an event on a college campus, etc.
Pros: It does not cost much, and you may find many testers if you choose well the venue.
Cons: You may have a hard time finding the “right” testers.
4. At the playtester home: Why not let them play in their real environment?
Pros: They will have their friends over and you will see real interaction.
Cons: Playtesting will be limited. Not able to record any data.
5. On the Internet/Remote: Why restrict your playtesting to the confines of the Internet?
Pros: Lots of people with different configurations. Ideal for multiplayer.
Cons: Quantity of playtesting comes at the price of quality of playtesting.

Microsoft Games Studios Playtest lab
[https://www.microsoft.com/en-us/playtest/]
You should try it!

5.1.5 Conducting the Session (How)
Test your test before you test: Set aside some time for everyone to ensure things run as smoothly
as possible. Test the player’s experience from the moment they enter your door. Check out
speakers, mouse, etcetera. Brief all playtesters helpers on your process, etcetera.
Take the pressure off: Make clear that “the game is being tested, not the user”. The goal is to “not
interfere”. Don’t tell people what to do. Don’t explain the story. Don’t explain what is missing.
Don’t explain the controls. Don’t explain the interface. Don’t explain anything. You can tell the
game isn’t done yet.
Playtesting + Interview at the same time? This is not the best option, as it is easy to mislead
(introduce bias), to create a different experience, among other problems. But some people do it, as
they can collect more data in a faster way.
Some people introduce questions in the playtesting. Fullerton (2014) suggests that short questions
are possible. “Why did you make that choice?”, “Does that rule seem X?”, “What did you think that
would do?”.
In case you do the interview at the same time of the playtesting, just embrace the comments as
they are. You do not need to justify anything to anyone.
Watch and listen. Take notes. Watch their screen and face, how they laugh and what they are doing.

Typical problems with quick playthroughs with friends, family and other team devs.
Typical UX tests Why are they flawed?
Game played in developer’s office Leads to positive bias towards game.
Sessions led by company employees Leads to positive bias towards game, as
employees working on the game for ages
are emotionally invested in the game.
Asking questions in the wrong manner Leads to game evaluated higher than it
should be.
Asking the wrong questions Leads to game evaluated higher than it
should be.
[http://www.gamasutra.com/blogs/OmTandon/20151029/257417/UX_Walkthrough_Anatomy_of_a_Usability
_Test_in_Video_Games_Part1.php]
Biases and flawes (playtesting and playtesting + interview)

Experimenter Bias: This happens when the moderator influences the result with the questions.
Sampling Bias: This is where the selection of people to take part in your research can impact on
your results.
"People whom you personally know will be heavily biased in their opinion towards your game."
Biases and flawes (playtesting and playtesting + interview)

Response Bias: This is where research participants either consciously or unconsciously feel a need
to tell the researcher what they want to hear.
Procedural Bias: This is where the environment in which the research takes place can also
influence participants.

Confirmation Bias: This is where you search for, interpret and remember information that
confirms your own beliefs.
Sunk Cost Fallacy: This is the inability to see potential problems or failures once a venture has
been heavily invested in.

Recording the playtesting sessions and team involvement
You should usually conduct one-on-one sessions but may conduct paired sessions only when we
are looking to test social elements.

An insight of how even presence of moderator can alter the players test behaviour.
Reason for segregating observers, players and moderator.

Camera Setup for Play Test
Developers Observing in the Next Room
• The session begins with about 10 min of getting to know the participant.
• We get them to play the test game for 20-30 min (which is about how often they’re likely to play
the game in their first sitting).
• During the sessions, developers are not only viewing the players from a different room but also
taking down notes. It is very positive to involve developers so they understand the process. Schell
(2009) hints that developers should not be *next to the user/player* - they are too attached to it.

People are oblivious and sometimes say things contradictory to their actions!
Importance of video recordings of play session!
• It eliminates a number of inherent biases with UX testing;
• It focuses on direct observation for deeper insights;
• It is more collaborative for the development team;
• It leads to actionable next steps for everyone.

Two mistakes you can make
• Talking between users: Users can have a huge influence on each other, so you should
discourage them discussing even during breaks. (And generally expect players to need
breaks every hour)
• Priming: Saying a particular word or categorizing a part of the game or software will
affect the way users will approach the test “Yes, first try the tutorial because it is a bit
messy”.

Methodological variation. In the think aloud technique, the gamer sits down to play the video
game while a user experience team member is seated nearby to listen and take notes. The user
is given specific instructions that as they play the game, they are to say out loud the reason they
took each action. This allows the team member to document both their actions, and what the
user was thinking as they took them. The team repeats this with multiple gamers to get multiple
perspectives and viewpoints.
Data collected within think aloud sessions are notes on what users said and where user said it
within the gameplay session.
Think-Aloud Technique (Playtesting variation)
[http://www.gamasutra.com/view/feature/6130/usability_breakthroughs_four_.php?print=1]
In think-aloud, a player verbally expresses thoughts, feelings, actions, and experiences during
gameplay. The researcher conducts the sessions, probing the player when necessary to share
experiences aloud. It is even important for researchers, not the game designers, to conduct
these studies.
People describe their actions as they play. They are unprompted and uncorrected.
Think-aloud is very used in web usability studies because tasks are much clearer and
challenge is always something to avoid.

Positive aspects of the Think-Aloud Technique
• Team members are able to understand what a player is thinking.
• This technique works well in an iterative design and testing process.
• Enables real-time glimpse into player thoughts, feelings, and motivations.
• Bring up unnoticed details.
• Effective for “why” questions.
Problems with the Think-Aloud Technique
• Many players have no problem talking while they play, but some will have problems
with this. Uncooperative players (intentional or not) can cause frustration and lost time.
• "Thinking aloud" doesn't come naturally to players. In a natural setting, gamers rarely
verbalize all of things they are doing in a game. It's also possible that in trying to explain
what they're doing, they may make up reasons that are different from the actual cause
of a behavior.
• If it's hard for some people to walk and chew gum at the same time, you can imagine
that it's hard for many people to play a game well and talk about it at the same time.
Often they focus on one or the other, with less than ideal results. You can try to fix this
by asking them what they're doing or thinking if they stop talking at some point.
• Talking about what you are doing modifies what you are doing. It interferes with
gameplay/create an artificial experience/distracting.
• Inaccurate and biased.
Let the Game Tester Do the Talking: Think Aloud and Interviewing to Learn About the Game Experience. C.M Hoonhout.

Think-Aloud Technique or Protocol (Usability testing variation)
[http://uxpajournal.org/intervene-think-aloud-protocols-usability-testing]
[https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool]

User testing facilitation techniques.
Facilitate better user tests with these 3 techniques: Echo, Boomerang, and Columbo. These
methods allow you to get clarification from participants with minimal disruption or bias.
[https://www.youtube.com/watch?v=V-An4Jfplew]

Contextual Inquiry (Playtesting variation)
[http://www.gamasutra.com/view/feature/170332/?print=1]
This is essentially fieldwork where you test X game and go out and observe your players in their
natural habitat. The idea being that you can observe them in their normal play setting going
about their normal routines when using your game, and therefore get insights that may not be
available in your own test situation.
Contextual inquiries also tend to be more useful for improving existing games, or observing
similar games to yours to learn about how people play them in the real world. As such, testing
prototypes or games in development in this fashion can be less useful, as players haven't had an
opportunity to develop a routine with them. Still, it is certainly nice to see how your game will
be used (and abused) in the wild.
Contextual inquiry (CI) is a research method in which the researcher watches the user in the
course of the user’s normal activities and discusses them with the user.
This is sometimes referred as Naturalistic Observation Technique.

The player will be noticed by e-mail or phone on how long the playtesting session will take.
The first playtesting can take as much as 30 minutes, although in other occasions 1 hour is very
standard. For board games, it is usual to play the entire game. It always depends on the type of
games, platform, etcetera.
In Playtesting, players are told that they
can leave whenever they want.
Playtesting duration

You need to analyze the aggregated data. You should not interpret a player entirely (this would
lead to bias), but the same gameplay segment/question/problem from all the players you
obtained data from.
If possible, try interpreting playtesting data and other sources for the same issue (surveys, data
analytics or interviews when available).
• Interpreting comments the right way is crucial. You should not listen to idiot comments as they
are, but as what they imply.
• People tend to explain solutions to problems, instead of describing the problem itself.
For instance, they may tell you that a weapon should be stronger, and instead what they mean is
that they could not kill other players. There are many other solutions: more bullets per second?
Playtesting can be complemented with surveys and interviews right after, or even days after. It
can also be complemented with analytical data tracked into a server.
Data analysis (or interpretation)

5.1.6 Conclusions and Report
Reporting playtesting to dev team (in case they were not involved)
• When you report your findings, the difference between getting cooperation and
opposition from a dev team member can be as simple as the words you use, or the way
you explain the problem.
• Using negative words like "broken," "problem," "unusable," "confusing," or "odd" can
upset the devs. responsible for those features and lead to a lack of cooperation or even
long drawn-out arguments.
• It's often better to talk about features that "need attention" or "could be improved." Silly
as it sounds, poor word choices can lead to internal conflict on teams, and conflict can
lead to delays in the development process, costing the team time and money.
• When describing usability issues, it's almost always better to "show" the problem rather
than "tell" about it. If you have video of users making a mistake, that can be the best
possible way to get a developer to recognize a problem and get behind fixing it.
Be careful with your language, you need the devs. in!

Usability problems found
Priority is a compound variable calculated from
1. Frequency – how often the issue occurs
2. Impact – how serious is the issue if it occurs (e.g. showstopper, or minor annoyance)?
Give playtesting usability issue list by priority
After giving the list you can provide the report with the problem headline, description,
audiovisual content, rating, possible fix or solution.
[https://www.gamasutra.com/view/feature/130745/better_games_through_usability_.php?print=1]

UX problems/insights found
Things that tell us that players are having a hard time for the right reasons.
Things that tell us that players see the challenges engaging or discouraging.
Things that tell us that the challenge level is appropriate for the current stage of the game.
• E.g. Level loading is something that is annoying, repetitive actions, people get tired of
certain controls, etc.
We aim at distinguishing between real UX problems and appropiate challenges.
Sometimes we find real confirmation to something we thought it was a problem. But we can also
find that what we thought it was the real fun in the game it is not, but something else
unexpected is much more fun. This is probably one of the most valuable things you can take from
playtesting.

Pros:
• Effective for the ‘what’ and ‘why’ questions (you know what was fun, what was
overpowered, why something is attractive, etc.).
• Good and abundant playtesting is definitive.
Cons:
• It can be biased by the moderator (questions or just the presence) and setting.
• It is very time-consuming.
• It cannot answer with certainty ‘how much’ something happens.
Remember: Observation as an evaluation method is objective (it's clear WHAT the player did,
not subjective) but keep in mind its interpretation is still qualitative rather than quantitative.
[https://www.activision.com/company/playtest]
Nothing beats direct gameplay observation. (Valve Team dixit)
It combines very well with an interview (after testing), surveys and analytical data.
5.1.7 Pros / Cons

[http://www.rollingstone.com/glixel/features/splatoon-2-hideo-kojima-nintendo-japanese-games-w501322]
You’ve suggested that Western devs playtest too much. Why?
It’s the designer’s job to make playtests as unnecessary as possible. It’s a cheeky statement, but
it’s true. When you hear what Ubisoft, Naughty Dog, Valve, all those guys are doing--they track
your eyes, they do it for months with hundreds of players--that’s a waste of money. If you feel
that you need that much playtesting, and if playtesting results in significant fixes to your game,
something went wrong before the playtests.
Interview with Jordan Amaro, game designer.
What do you think about his suggestion
about western devs and playtesting?

[https://www.youtube.com/watch?v=on7endO4lPY&t=2s]
Playtesting - How to Get Good Feedback on Your Game - Extra Credits
Watch this video. Do they explain anything that has not been explained in these slides?

Developing Dominion. What Game Development Is All About, by Dale Yu. (p. 74)
I’m always quick to point out that there is a huge difference between a game developer and a
game designer. The developer takes these ideas and creates a finished product from them.
- The game developer (the publishing company agent) writes the questions while he learns the
game in an initial playtesting session. Write down each of the mechanics as a list.
- Then, look for redundant elements. Look for highly improbable scenarios - the perfect storm of
card draws, for example. Playtesting is aimed at proving the game is “not broken”.
- Then, it is time for the blind playtesting. The producer is present during the session but does not
say anything during it. First impressions are very useful. At the end of the session, ask a lot of
questions!
It’s Not Done Till They Say It’s Done. The Who, What, Where, When, and Why of Playtesting by
Teeuwynn Woodruff (p. 99)
- Testing the rules as they are written is really important. Development playtest is a playtest
before the rules are finalized.
- Blind playtest involves a number of people in the target demographic, with no association to the
game or its creators.
- In your post-playtest survey, in addition to quizzing the testers about specific aspects of your
game and your rules, ask the testers to name their favorite games again. The most important
thing to remember is to stay neutral.
5.1.8 Discussion: Playtesting with Video games and Board games
“The Kobold Guide to Board Game Design” (Book)
Conducting playtesting sessions with board games is similar to with video games

5.1.8 Onboarding Plan with Playtesting
An onboarding plan aims at engaging with your players during the first minutes. This is especially
important in free-to-play games (about 20% are gone after one hour of player). The first hour is
fundamental.
You need to create an onboarding plan so you can teach the basis of the game. You need to teach
it without restrictions, with context and meaning – this is how good tutorials work and help the
player stick to what matters (and they do not look like tutorials).
The onboarding plan needs to help the players process the ‘core gameplay loop’ and ‘complex
mechanics’ in a way that they remember it. Not in a superficial way.
You want to structure small ‘tasks’ or ’activities’ and identify the mechanics that are linked to
them – so they practice 2 mechanics, they learn 2 more and they have an informational wrapper
that makes sense.
Players must become competent and not overwhelmed in the first hours of playing.

UX Expert Celia Hodent explains us in her book ‘The Gamer’s Brain’ her approach to the
onboarding plan analysis. She proposes classifying each ‘learning’ in the following labels:
• Category: (this is, the mechanic)
• Priority level: (to define the importance, you can use 0-3, being 0 critical and 3 accessory)
• When: (mission, level, hour of play, etc.)
• Tutorial order: (the order in which this takes place)
• Difficulty: (anticipated difficulty)
• Why: (why is this important: you need to learn to take weapons to be able to fight when…)
• How: (how to teach: learn by doing, GUI step-by-step, dynamic text, voice,….)
• Narrative wrapper: (this is the story that makes sense of the tutorial order)
• UX feedback: (if you already have some test results)
If you want to learn more, check the Chapter 13 in Hodent’s book. But here’s this is the general
idea. Be analytical.
As soon as you have your onboarding plan set on an excel file, it is time to conduct the
playtesting.

This is an onboarding plan example by Hodent (2017) in the Gamer’s Brain

Could you identify the elements in the onboarding plan that supposedly used The Legend of
Zelda Breath of the Wild?
This game has an onboarding plan where everything is ‘learn by doing’.
Within the first hour you:
• Learn all the tools you need (weapons, armor and bombs)
• Climb a mountain: you do not consume stamina when you stop.
• Learn to keep wooden weapon, shield and bow by hand
[https://www.theguardian.com/technology/2017/feb/24/the-legend-of-zelda-breath-of-the-wild-
nintendo-switch-console]
[https://www.forbes.com/sites/erikkain/2017/03/03/10-tips-for-playing-the-legend-of-zelda-breath-of-
the-wild/#e790628650e8]
Link dies quite frequently, weapons tend to fall apart, and the game looks beautiful.

Case Study: World of Tanks GUI
1. Research Question: How usable are World of Tanks graphic user interfaces?
2. Concepts/Theoretical Framework: Usability Nielsen’s dimensions, Information Architecture
3. Methodology: playtesting, data analysis, card sorting
[https://www.youtube.com/watch?v=Abdy1Cmueyg&t=1172s]

Case Study: Learning The Legend of Zelda Breath of the Wild
1. Research Question: How easy to learn are BOTW mechanics and controls?
2. Concepts/Theoretical Framework: usability, learning theories, affordances, natural mapping.
3. Methodology: playtesting with boarding plan.

Case Study: Destiny pre-alpha
1. Research Question: How enjoyable are game areas and multiplayer activities in Destiny pre-
alpha version?
2. Concepts/Theoretical Framework: Ekman’s emotion theory, Usability, Game Design.
3. Methodology: playtesting, interviews, surveys.
4. Results: They tested correctly mainly environments: what is space used for, planet
destination, colors, etc. (preventing confusion). Results showed user satisfaction and
understanding of the game elements.
Game UX Summit 2016 - Jennifer Ash - UX Lessons Learned on Destiny
[https://www.youtube.com/watch?v=bWYplcb3fgo]

Case Study:
Research Question: Does bigger products help in finding faster ‘bok choy’ from fresh
vegetables to the shopping cart?
Research Question: Does more products help in finding faster ‘bok choy’ from fresh
vegetables to the shopping cart?
Context: Designing an e-commerce website.

Case Study:
Research Question: What are the habits, expectations and beliefs of a regular sports
shopper? What is the profile?
Research Question: How can you classify the items in a sports shop?
Context: sports shop e-commerce website.

Case Study:
Research Question: Which are the most important tasks both for the user and the
business in the Amazon app?
Research Question: Can we improve and validate design changes in order to make
some features more usable such as filtering reviews, the shipping address button,
voice feature?
Context: all kind of products e-commerce app.

Key Questions and Concepts (TakeAways)
• Either if you want to become a UX professional or a game designer, you want
good playtesting. Because that is where you learn from. Sooner or later, you
are going to get feedback. The more you learn from it, the better you will
become. However, you will always need to test.
• We can reduce the bias by recording the player instead of being present
during the playtesting. We can do more playtesting to obtain more data (more
data, although it has bias, equals better data).
• We know that data can be biased according to the type of player we do
playtest with (our relationship with them, previous experience, etc.). We can
improve that by obtaining participants closer to our target players, so they
provide us better feedback. In general we need to pay attention to all the
biases that we may introduce into the session.
We can combine our playtesting with other methods to obtain different sorts of data.

References and Bibliography:
• All the references provided in the Powerpoint are valuable.
• Games for Change / Classcraft 2016 Interview - Celia Hodent of Epic Games - #G4C16 - Part 1 and 2 (UX,
Serious Games, Interfaces, etc.)
[https://www.youtube.com/watch?v=KD2f_f24vNs https://www.youtube.com/watch?v=Phkp8nyxI3Y]
• Full Sail on Air Celia Hodent (Interview) [https://www.youtube.com/watch?v=PMTyHn1lZfY&t=207s]
• What Should a Playtest Smell Like?
[http://www.gamasutra.com/blogs/SebastianLong/20170221/291752/What_Should_a_Playtest_Smell_Like.ph
p]
• Methods for Game User Research. Studying Player Behavior to Enhance Game Design (Paper). Desurvire and El-
Nasr. 2013. Interesting case study involving ‘heuristics’, ’playtesting’ and ’think aloud’.
• Game User Research. Miguel Angel Garcia-Ruiz. 2016
• Game Usability: Advancing the Player Experience. Katherine Isbister, Noah Schaffer. 2015
• Game Design Workshop. Chapter 9. Tracy Fullerton. 2014
• Playful Design. Chapter 8. John Ferrara. 2012
• The Art of Game Design: A Book of Lenses. Chapter 25. Jesse Schell. 2015
• Kobold Board Game Design. Selinker et al. 2012
– Developing Dominion. What Game is All About, by Dale Yu (p.74)
– It’s not Don’e Till they Say It’s Done. The Who, What, Where, When, and Why of Playtesting by Teeuwynn Woodruff (p.99)

5.2 Interviews
Interviewing is the most usual method in Social Sciences to obtain an idea of what people think,
feel or motivate in any activity of life by asking questions, recording/writing down the answers
and analyzing them carefully.
Most of the playtesting include some sort of interview during or after the session. It is preferable
to do it after, since during the game it would distort their experience (if we have to do it, we
already explained that questions must be concise and show no hint on the answer).
An interview after a playing experience can be useful to discover what the player found most fun,
most exciting, among others.

a) What are we studying with an interview?
In an interview, we only study the recalled Experience
– What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
5.2.1 Purpose and Problems (What)
UX
Recalled UX (later)
We cannot know about the real gameplay (game functioning and usability) with just the interview.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes interviews are conducted after the playtesting – as a complementary but
not essential or as ‘core’ as playtesting or data analytics.

a) What are we studying with an interview?
In an interview, we only study the recalled Experience
– What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
We cannot know about the real gameplay (game functioning and usability) with just the interview.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes interviews are conducted after the playtesting – as a complementary but
not essential or as ‘core’ as playtesting or data analytics.
UX
Now
Recalled UX (later)

“Conducting an interview is a flexible means of gathering information about the experience the
participants just had, about their opinion regarding the application, about previous experiences
and how this one compares to earlier ones, about their perceptions, preferences, opinions,
attitudes, thoughts, ideas, etc. Interviews allow the researcher to collect potentially a rich set of
qualitative data on the attitudinal side.”
Let the Game Tester Do the Talking: Think Aloud and Interviewing to Learn About the Game Experience. Game
Usability. Katherine Isbister and Noah Schaffer (p. 73)
An interview is not a good method to understand how usability affects the player.
For example, in an interview with shooter players, it could be that they say a target is difficult to
aim and shot and this may lead us to wrong conclusions about controls or enemy AI, while
instead, by watching a playtesting session we see it is a matter of perception.
We can find out about aspects of the perceived balance (i.e. challenge), their preferences, their
emotions, among others, but not usability or game interaction.

b) What kind of method it is Qualitative and attitudinal
It answers well ‘what’ and ‘why’ questionsDatasource
playtesting eye tracking
data analysis
Interviews & focus groups surveys
physiological data
playtesting
think-aloud

An interview follows the research structure. It is recommendable to have good research
questions and the right concepts to analyse them, but we can let the interviewed player explain
and dig into specific topics.
1. Context/Object
2. Problem/Question/Hypothesis
3. Concept / Theoretical Framework
4. Measurements (Methodology)
5. Methods (Methodology)
6. Results analysis
7. Conclusions
8. Reporting
Unlike other methods, the interview is not a rigid methodology per se.
It depends on the topic:
a) We want to explore general aspects about the interaction, the realism, the narrative, among
others. It does not need to be rigid. We let the player wander with their talking.
b) We want to explore the motivation. It needs to be rigid.
Conclusions for the first topic can be extracted from few participants, while the second may
require something like a sample (i.e. some more).
c) Experiment structure

Most common and wise: test your target player, the one who is going to play, who is going to
spend the money. Find out how old they are, genre, previous gaming knowledge, etcetera.
You need to interview at least 6-12 players from your target player in order to extract general
conclusions.
You can have an interview with other profiles but only at the very initial phase in order to obtain
‘open ideas’.
You can interview any player (the circles), but be aware of their profile and possible bias.
Remember, the different biases we explained in the previous sections. Be careful with the
sampling bias.
How many simultaneously?
Plan to generally only interview one (or two) people at a time. One is usually best, so as to avoid
one person's opinion dominating, but sometimes two people can have a nice dynamic and
prompt each other -- particularly if discussing a cooperative or multiplayer game.

5.2.3 Environment/production Pipeline (When)
• Post-game interview is a way to get a sense of how they really felt about the game, since
you can see emotion in their faces and hear it in their voices.
Post-game interviews usually happen within few minutes after the playtesting session, but they
can be resumed or extended through a phone call one day or several days after.
The advantage of this second interview is that the data refers to the “memory” of the
playtesting session, and so it can be rich in dimensions different than the original.
For instance, the sensations of ”fear” can be considered in a better way than during the
playtesting time. As a game designer and a company, you are interested in what “can be said
about the game, especially those people who are not users of it”.
It depends on the time it has taken from the playtesting session, we may transit from a
recalled experience into reconstructed experience (memory) influenced by many other
factors such as the other games the player played, among others.
In a general way: we remember emotions, we do not remember well interactions and we
tend to distort our memories. Be aware of this.

5.2.3 Environment/production Pipeline (Where)
Ideally, it should be done in their environment, so it may reduce some sort biases (pressure,
intimidation, fan-effect). Remember the procedural bias.
If you are going to be interviewing people in their own play environment, this gets harder, but
still do you best to make sure extra distractions are not around (e.g. ask politely if the door
can be closed, etc.).
In case in their environment it is not possible, you need to minimize all the possible
distractions in the lab. Generally speaking, it should be comfortable.

Other modalities and their strengths
1. Video (ex. Skype, Google Hangouts). This may be useful to obtain material without all the
logistics of bringing the participant. When combined with playtesting, this is often called
‘remote testing’.
2. Audio (ex. Skype, VoIP software). Tone and emotion are still clear. Frequently used by
many online players while gaming.
3. Text Chat (in-game). Location within the game environment can lead to more detailed
answers. It does not require transcription.
4. Text Chat (out-of-game, ex. Google Chat). It does not require transcription. This is hardly
used.
When conducting interviews through the Internet we may lose some dimensions of the
human communication and perceive less clearly (or not perceive) the emotional amplitude.
However, this may not matter when we are asking about their ‘weapon preferences’ in a
shooter in order to balance the game.
In-depth interviews for games research. Amanda Cote and Julia G. Raz. Game Research Methods: An Overview.
Patri Lankoski et al. 2015 (p. 93)

5.2.4 Conducting the Interview (How)
The first and one of the most difficult aspects in conducting the interview is to maintain the right
tone and pose. The moderator must maintain a certain distance. He is not creating a relationship
with the participant. He must be in positive mood but not too excited. Cordiality.
Sometimes, it can be very difficult to stay “neutral” instead of defensive. In some cases, it might
even be a good idea not to involve any of the developers in the interview session, but let a
neutral, “outside” person conduct the interview session like a user researcher.
Do not let a tired or angry developer/producer conduct the interview

• Playtesters may avoid hurting your feelings, particularly if they know (or think) you helped
make the game. Sometimes, staying objective is not enough. It may be useful to make a big
show of saying “I need your help. This game has some real problems, but we’re not sure
what they are. Please, if there is anything at all you don’t like about this game, it will be a
great help to me if you let me know”. This gives a tester permission to speak honestly.
• You need to alert the person you are interviewing to the recording and get their permission
(privacy). You also have to explain to them what you are interested in asking them about,
and make it clear that this interview is not about evaluating their knowledge or qualities
in a personal way, but to look at their experience of, and the performance of, the game.
• You should also make an interview script; i.e. write down the questions you want to ask,
and the order in which you want to ask them. However you should also be prepared to ask
follow-up questions, although take care and don't be confrontational when you do. It may
be your game they are badmouthing, but you need to play it cool.
Remember the response and experimenter bias.
The Art of Game Design: A Book of Lenses. Jesse Schell 2012 (p. 400)

Special conditions for a good interview
• You need to tell participants in advance how long the interview is going to be (even if you
said how long the previous playtesting was). The interview should preferably not be too
long; thirty minutes will in most cases be enough. Depending a bit on the topic, and the
rapport the researcher has established with the participant, the interview could extend to
approximately sixty minutes. This is what is called an in-depth interview.
• Also it is useful to set up a way to record the interview; this doesn't have to be video, but it
is important that you at least record what is said. You will not remember everything, and
even if you take great notes it is a good idea to have the recording to refer back to. This is
also preferable to making notes constantly, as this tends to distract the person you are
interviewing and can create a feeling that they are being assessed personally -- rather than
the game.

5.2.5 Interview structure (How)
Prepare a script. Cote and Raz suggest to structure the interview in different parts: introductory
script, warm-up questions, substantive questions and demographic questions.
• Introductory script: you need to explain the purpose and goal of the study. Discuss consent form
and procedures.
• Warm-up questions. Put the participant at ease and build rapport. Focus on things the
participant can answer more easily. General questions like “How long have you been playing
videogames for?”, “what’s one of your favorite gaming memories?”.
• Substantive Questions. This is the part in which you want to take the most out of the
participants. Open-ended questions. Here you want answers to your research questions.
Prepare potential follow-ups to prompt elaboration or help you respond to surprising answers.
• Demographic Questions. In case you need it, you should complete your data.

Some tips to prepare and conduct the interview by Jesse Schell
The Art of Game Design: A Book of Lenses. Jesse Schell 2012 (p. 400)
• Ask for more than you need. Instead of asking “what was your least favorite part?”, why
not ask “what were your three least favorite parts?”. You get more data, and you can sort it
by priority. The thing that stands out it will come to player’s mind first.
• Avoid memory tests. Asking players questions like “On level three, when you got to the
yellow butterflies, you flew left instead of right. Why?” will generally get you blank stares.
This questions can only be asked while the game is being played.
• Don’t expect playtesters to be game designers. Questions like “Would the game have been
better if level three was harder?” may not get the results you want. In general, players think
they want the game to be easier, so they are likely to say “no” to that question”.
• Set your ego aside. It can be very hard to sit and listen to someone to tell you how bad
your game is. You may be tempted to defend the game but instead, you should steel
yourself and ask objective questions like “what don’t you like about it?” and “can you tell
me more about that?”.

Some postgame questions by Tracy Fullerton
General questions
What was your first impression?
How did that impression change as you played?
Was there anything you found frustrating?
Did the game drag at any point?
Were there particular aspects that you found satisfying?
What was the most exciting moment in the game?
Did the game feel too long, too short, or just about
right?
Formal elements (game design)
Describe the objective of the game.
Was the objective clear at all times?
What types of choices did you make during the game?
What was the most important decision you made?
What was your strategy for winning?
Did you find any loopholes in the system?
How would you describe the conflict?
In what way did you interact with other players?
Do you prefer to play alone or with human opponents?
What elements do you think could be improved?
Dramatic elements (narrative)
Was the game’s premise appealing to you?
Did the story enhance or detract from the game?
As you played, did the story evolve with the game?
On a piece of paper, graph your emotional involvement
over the course of the game.
Did you feel a sense of dramatic climax as the game
progressed?
How would you make the story and game work better as
a whole?
Game Design Workshop. Tracy Fullerton. (p. 263)
5.2.6 Question creation (How)
Do not ask questions with insinuations: ‘are you sure’,
‘don’t you mean’,.

Procedures, rules, interface, and controls
1. Were the procedures and rules easy to understand?
2. How did the controls feel? Did they make sense?
3. Could you find the information you needed on the interface?
4. Was there anything about the interface you would change?
5. Did anything feel clunky, awkward, or confusing?
6. Are there any controls or interface features you would like to see added?
End of session
1. Overall, how would you describe this game’s appeal?
2. Would you purchase this game?
3. What elements of the game attracted you?
4. What was missing from the game?
5. If you could change just one thing, what would it be?
6. Who do you think is the target audience for this game?
7. If you were to give this game as a gift, who would you give it to?
When coming up with these questions
many of the same rules creating
questionnaires apply. In other words, try
to be clear with your questions, and
check they are not leading, loaded, and
only have one meaning.
Clear does not mean too specific. You
can be broad and general. Take care to
avoid questions that can be answered by
a simple yes or no; if you are after that
kind of information use a questionnaire
instead.
Tips to create your own questions

Case Study: Life is Strange Ending
1. Research Questions:
• Do our target players like the Life is Strange ending? Which one do they like most?
• What emotions do they feel during the entire game?
• Does the prequel of the game named ’Before the Storm’ provide an experience with
very different emotional range than the original?
2. Concepts/Theoretical Framework: Ekman’s basic emotions, narrative theory.
3. Methodology: Interview, Survey, Biometrics.
EXERCISE: Try preparing some questions to the participants in order to answer your research
questions about Life is Strange series. This could be useful to design a third game in the series!

Research Questions:
• Do our target players like the Life is Strange ending? Which one do they like most?
• What emotions do they feel during the entire game?
• Does the prequel of the game named ’Before the Storm’ provide an experience with
very different emotional range than the original?
You can use Plutchik’s wheel as
an emotional map.
Or the Ekman’s six emotions and
face expressions:
Anger, fear, disgust, surprise,
happiness and sadness.

5.2.6 Data transcription and Interpretation
Make a transcript. Be aware this will take some time. Then, if you conducted multiple
interviews, look for common themes and threads in what people are saying. It is necessary to
organize the data from each interview according to ”main” and “repeated ideas”. It is
important to find out which are the most important “themes”.
You should analyze and interpret the data of different interviews at the same time. Only this
way it is possible to obtain a general view. Aggregate by topics.
Important: if you are studying emotion, motivation or belief, try identifying what links to the
concepts in order to extract general conclusions and answer the research question.
One good thing about interviews (which is also a good thing about gameplay observation and
think out loud methods) is that they can generate great sound bites or quotes that can be
quite convincing when given to others on the development (or even management) team.
Be careful with interpretation! People tend to explain solutions instead of stating the
problems as they are (e.g. there are not enough power ups). When they do give you solution,
dig in and try to establish the underlying problem they are trying to explain, because you will
discover something else like “my gun doesn’t feel powerful enough”, which may require a
totally different solution than the one they offered.

Case Study:
Research Question: What actions can Facebook suggest to people with eating
disorders in order to support them?
Context: social media App.

[http://www.rollingstone.com/glixel/features/splatoon-2-hideo-kojima-nintendo-japanese-games-w501322]
Question: With Splatoon 2, there definitely are people who want to know why they can’t play
the Salmon Run mode all the time.
Answer: I’m not allowed to speak on it, because I’m not the game director. What I can say, and
what I think can be said, is that there are lots of reasons. You have to trust us that if you could
play Salmon Run online anytime, that would result in a worse experience for you and
everybody.
Interview with Jordan Amaro, game designer.
His answer shows that ‘listening to the
player’ as obeying the player is not always
the best choice. Data is data. Data is not a
rulebook to follow to obtain success with
your game.

5.2.7 Pros / Cons
Pros
• Rich data source for subjective impressions (opinions, thoughts, emotions, etc.).
• Good for exploratory ideas in a prototype phase (alpha) to take general decisions.
• You can ask follow up questions that would have never appeared.
• You may obtain very useful “pitch lines” from participants useful to convince developers and
management.
Cons
• Not easy to quantify the importance of each insight (hard to code into main themes).
• Time-consuming.
• Attitudinal is attitudinal (it may provide contradicting data with what they just did in playtesting).
Economical costs
• The real cost of interviewing people is not the
amount of money given to them in compensation
but the data organization and interpretation.
• It is not easy to have somebody with proper or
formal education in conducting interviews. Not
everyone has the skills.

Case Study: Rainbow Six friendly fire
1. Research Question: What are the effects of removing friendly fire in Rainbow Six Siege’s on
UX (at a player and at a team level)?
2. Concepts/Theoretical Framework: Social Psychology theories for community
3. Methodology: Interviews, Survey, Data Analysis, Playtesting
EXERCISE: Do you think the interview is going to be a good method to answer this research question?

Case Study: Rainbow Six friendly fire
Data results:
• Some say that it might make team-work more friendly and nice (no toxicity).
• Others say that they would not like it because it is good to ‘punish’ team members that are in
the wrong place.
• Others think that they would not like it, but the ‘shot’ member should be able to punish the
shooter back instead.
With an interview we can extract the possible effects of friendly fire and removing it but we
cannot say the impact of removing it on the playing community.
1. We would need a representative sample of the different players and that is costful.
2. It is better to identify different possible aspects in an interview and ask them in a
questionnaire. If this is possible, this is recommendable for several reasons (no moderator
biases being one of them).
3. However, in order to take the decision about removing it or not, we should know that: “what
players say are not what they really think, not because they are lying but because human
psychology is complex”. Then it would be better to playtest and analyse the interactions.
Players may complaint but they may still secretly prefer it. Ideas vs. Actions.
What a player says about a recalled experience has a propagating value. This is what they say and
this is what sells and creates the community.
But what a player does has a real value because this is what matters (if they play less, they play
less, and this will end up killing the entire community user experience).
[http://www.levelup.com/noticias/447556/Ubisoft-podria-remover-fuego-amigo-en-Rainbow-Six-Siege]

5.3 Focus Groups
Focus group (also known as group interviews) is a specific technique
originating from the work of American sociologist Robert Merton.
Focus groups are widely used in marketing, where the aim often is to
assess people’s reactions to certain products or brands, as well as in
many areas of research such as phenomenological sociology,
audience reception and media studies, and evaluation research, to
mention a few examples (Stewart, Shamdasani and Rook, 2007).
While focus groups have been used extensively in media research and
other sciences the method has not yet found its place in game
research, although there are a few examples of focus group studies.
In focus groups, a moderator asks participants to discuss their
experiences and their understanding of them; focus groups thus bring
the social construction of meaning into the data gathering process.
Most of the aspects mentioned for interviews are valid for focus
groups too.
Focus group interviews as a way to evaluate and understand game play experiences. Lina Eklund. Game
Research Methods: An Overview. Patri Lankoski et al. 2015 (p. 93)

Focus groups are based on the idea that group dynamics can provide much more data.
However, group dynamics are also its biggest advantage and its biggest flaw.
a) What are we studying with focus groups?
In an interview, we main study the recalled
Experience – What does she think it happened?
Focus groups are good at finding desires, motivations,
values and first-hand experiences (attitudes and
perceptions of game mechanics and goals); they are
not good at people telling you usability issues.
b) What kind of method it is
They answer well some ‘why’ questions.
Qualitative and attitudinal
UX
Recalled UX (later)
Now

Focus groups are based on the idea that group dynamics can provide much more data.
However, group dynamics are also its biggest advantage and its biggest flaw.
a) What are we studying with focus groups?
In an interview, we main study the recalled
Experience – What does she think it happened?
Focus groups are good at finding desires, motivations,
values and first-hand experiences (attitudes and
perceptions of game mechanics and goals); they are
not good at people telling you usability issues.
b) What kind of method it is
They answer well some ‘why’ questions.
Qualitative and attitudinal
UX
Now
Recalled UX (later)
Prior opinions, beliefs, habits…

Before alpha: we are obtaining general ideas. We
are not polishing the game.
Focus groups are good for:
• Exploratory. General attitudes about a given
topic (e.g. best game of a saga?).
• Feature prioritization. For instance, adding a
new mode in multiplayer.
• Competitive analysis. Comparisons with other
experiences from the competitor.
• Multiplayer/social experiences. They may
remember better the playtesting experience.
Setting
• Make sure mobile phones are switched off
• Seating is important as the participants must be
able to face each other properly.
• Test your recording device and place it in a
central location.
• Before the interview, make sure that a topic
guide is prepared.
5.3.2 Environment / Production Pipeline (Where and When)
It is like a group therapy but
people talk about the ‘games
problems’ instead of theirs.

Be careful not to construct groups with playtesters you found just to fill the gap, unless they
match your research purpose. The same rules of “quality playtesters” apply to focus groups.
If one of the focus group participants is very different and outforward, he may break the group
dynamics completely.
In a focus group, in contrast to an individual interview, the moderator plays a more passive role.
The aim, as described above, is for the participants to discuss the focus among themselves in
their own words.
To further the aim of group discussion, open-ended questions are necessary. The more open the
better. Participants will be in charge of posing questions among themselves.
The moderator role foremost places great demands on interpersonal skills as well as being a
good listener who does not judge or play favourites.
5.3.4 Conducting a Session (How)
The questions are planned in the exact same way as if they were prepared for an interview.

5.3.5 Data Gathering and Interpretation
Analyzing focus group data is in many ways similar to the analysis of any interview data.
Depending on the research focus, different transcribing techniques can be used but there are
some guidelines that should be adhered to:
1) be rigorous, shortcuts gain you nothing and reduce quality.
2) use the same transcription technique all the way through.
3) ask yourself what information you need to answer the research question and adapt
your transcription technique to that.
• It is fundamental to identify the different themes and search for data that could fit in that
theme even though it is ”out of its time” in the session.

5.3.6 Pros / Cons
Pros
• The basic principle here is that several minds are better than one. Focus groups allow for a
sharing of ideas amongst participants. New ideas can also be generated from this sharing of
ideas and opinions, as can design strategies leading to more efficient and targeted gameplay,
mechanics, and story.
• All the feedback from participants is collected all at once over a couple of hours. In an
iterative design or decision making process this means you can collect a lot of data very fast.
A strength of focus groups is their flexibility, that they can be carried out in many ways and
accommodate varying numbers of participants, different topics, sampling strategies, budgets
and so on.
• Designers, developers and producers can observe the focus group directly or through
recorded video and audio feeds.
• Focus groups can, due to the active role of the participants, be very good at exploring topics
about which little is known or understood; the participants become experts that the
researcher can learn from. It allows follow-up questions and going into detail when
discussing concepts.
[https://www.nngroup.com/articles/focus-groups/]

Cons
• This technique works best with an experienced moderator. In a pinch, a team member with
good listening skills will work (producers are often good at this) but they must be very careful
not to push the discussion toward ideas or answers that they prefer, or the focus group is
useless. Leading questions from the researcher can manipulate the participants into
answering questions a certain way, which reduces the usefulness of your findings.
• Several minds can easily be lead by one ego. If one participant has strong opinions and
expresses them loudly and confidently, other participants in the focus group may feel
pressured to agree with the dominant member. Then, much of the value of the group is lost.
Remember that the loudest or most talkative participant does not always have the best ideas.
• It's important that all the participants feel that their opinions are equally valued. Very
unequal levels of skill or experience can lead to participants deferring to the member who as
logged the most hours in similar games. Often it is helpful to have groups made up entirely of
beginners or experts. You may want to screen participants ahead of time to help with this.
• Focus groups are a subjective method, and all you have to go on is what people say -- and as
much as we like to judge people on the "attitudes" they hold, and think that they predict
behaviour (they usually don’t), what people say is not always what they actually do.
Focus groups are not a good method to understand how usability affects the players.
But we can find out about aspects of balance, challenge, narrative, minigames, etc.

Key Questions and Concepts (TakeAways)
• Interviews, focus groups and questionnaires are methods to understand attitudes.
They are the closest approach to understanding the recalled user experience. The
first two can provide ‘bright insights’ but also bias.
• Surveys are the best approach to characterize a population – as long as the
questions are well-written and the sample is large enough.
• These methods are used in prior phases of development, but never to polish the
gameplay. Important to remember: they are not good to find out usability issues.
They can complement the results from a playtesting session.
• The most difficult aspect in an interview is conducting it properly with the right
tone and analysing the answers with no misinterpretation. In regards of
questionnaires, the most difficult part is to generate the questions and to have a
sample. Generally, we are always aware not to ‘dirty’ the data.

References and Bibliography
• All the references provided in the Powerpoint are valuable.
• “Beyond Thunderdome: Debating the effectiveness of different user-research techniques”
[https://vimeo.com/26733185]
• Game Usability: Advancing the player experience. Isbister, Katherine, and Noah Schaffer. CRC
Press. 2015.
• Game Research methods: An overview. Lankoski, P., & Björk, S. 2015.
• Games User Research: A Case Study Approach. Miguel Angel Garcia-Ruiz. AK Peters/CRC
Press. 2016.
• Playful Design. John Ferrara. Rosenfeld Media, 2012.
• The Art of Game Design: A Book Of Lenses. Jesse Schell. Carnegie Mellon University. 2008.
• Kuniavsky, M. (2003). Observing the user experience: a practitioner's guide to user research.
Elsevier.
• Articles available in the site Gamasutra [gamasutra.com]
All images used in these slides belong to the cited sources.

EXERCISE: INTERVIEW MISTAKES
• What are the mistakes the interviewer makes in this video?
[https://www.youtube.com/watch?v=U4UKwd0KExc]

EXERCISE: FACILITATOR IN FOCUS GROUPS
• Organize groups of 6 people. The debate is on ‘Which one is best, Windows or
OS X?’ or ‘What game deserves more popularity, Battlefield or Call of Duty?’.
One becomes the facilitator and gathers opinions from each member of the
group. Another is the observant and remains silent. The four other members
participate.
• You have 7 minutes.

FACILITATION TIPS
As a focus group facilitator, it’s your job to create an environment where everyone is comfortable
enough to take risks and learn from what they experience. Demonstrate the attitude you want to see
in them: you should laugh, so participants can laugh – you should be passionate, so participants can
be passionate.
Your job as a facilitator has four important tasks:
• Suggest. Suggest and explain ways to apply principles throughout all stages of a project.
• Focus. Pull participants back into focus once straying away becomes unproductive.
• Remind. Remind workshop participants of various approaches.
• Restrain. Restrain yourself from being too dominant. Only interfere when it’s absolutely needed.
With those tasks in mind, your job is to steer the activities and reflections of your partipants by
following these principles:
• “Yes and…”. Defer judgement and build on each other.
• New territory. Use analogous situations to take you to new territory.
• Use constraints. Constrain thinking to generate volume in ideas.
• Selection criteria. Define selection criteria up to choose the ideas you will take forward.
[http://ui-patterns.com/blog/how-to-facilitate-and-plan-a-ux-workshop#a-good-facilitation]
[https://www.smashingmagazine.com/2017/01/becoming-better-facilitator/]

User Experience 6: Qualitative Methods, Playtesting and Interviews

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to User Experience 6: Qualitative Methods, Playtesting and Interviews

Similar to User Experience 6: Qualitative Methods, Playtesting and Interviews (20)

More from Marc Miquel

More from Marc Miquel (16)

Recently uploaded

Recently uploaded (20)

User Experience 6: Qualitative Methods, Playtesting and Interviews