Traning workshop on ‘Designing an conducting user studies” Module 4 - Thinking Aloud (Corné van Elzakker) @ ICC&GIS June 15th, 2016 Albena, Bulgaria

1. APPLICATION
0F THE
THINK ALOUD METHOD
IN
USE, USER AND USABILITY RESEARCH
IN
CARTOGRAPHY & GEOGRAPHIC
INFORMATION SCIENCE
CORNÉ VAN ELZAKKER
WORKSHOP DESIGNING & CONDUCTING USER STUDIES
6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016

2. 2
THINKING ALOUD
ONE OF THE MANY METHODS OF USER RESEARCH
Method Similar or Related Methods
Expert-
based
heuristic evaluation rulesof thumb
conformity inspection feature inspection, consistency inspection, standards
inspection, guideline checklist
cognitive walkthroughs pluralistic walkthroughs, prototyping, storyboarding, Wizard of
Oz
Theory-
based
scenario-based design personas, scenarios of use, use case, context of use, theatre
secondary sources contentanalysis, competitive analysis
automated evaluation automated interaction logs, unmoderated user-based
methods
User-based
participant observation ethnographies, field observation, MILCs, journal/diary sessions,
screenshot captures
surveys questionnaires, entry/exit surveys, blind voting, cognitive
workload assessment
interviews structured interviews, semi-structured interviews,
unstructured interviews, contextual inquiry
focus groups supportive evaluation
Delphi
card sorting Q methodology, conceptmapping, aflnity diagramming,
brainstorming
talk/think aloud verbal protocol analysis, co-discovery study
interaction study performance measurement,controlled experiments
Source: Robert Roth

3. 3
Important aspects of usability research
Requirements Analysis
Evaluate early design
Evaluate elaborate design
User feedback
Focus groups
Brainstorm sessions
Interviews
Surveys & questionnaires
Stakeholder meetings
Literature review
Card sorting
...
Paper prototype
Storyboarding
Heuristic evaluation
Automated evaluation
Wizard of Oz
Thinking aloud
Eye tracking
Card sorting
Cognitive walkthrough
....
Performance testing
User satisfaction
Heuristic evaluation
Cognitive walkthrough
Functional prototyping
Critical incidents
Logging user interactions
....
Questionnaires
User satisfaction
Logging user interactions
Observations
Diary keeping
...

4. Analyse
Requirements
Produce Design
Solutions
Conceptual
Design
& Prototype
Final Design
& Prototype
Interim
Designs &
Prototypes
Identify the
need for
user-centred
design
Design
satisfies
requirements
Evaluate Designs
Testing with
representative
users
input
Business
User
StakeholderOrganisational
Characteristics Preferences
Use contexts Tasks
4
(OF GEOSPATIAL DATA PROCESSING AND DISSEMINATION SYSTEMS)
Method = User-Centred Design
Prototyping
Requirement
analysis
Usability
evaluation
Techniques e.g.:
Qualitative
Focus groups
Interviews
Observation
Thinking aloud
Questionnaires
Screen logging
Eye tracking
Performance an.
Quantitative

5.  Distinction between qualitative and quantitative research
 Working with experts or with (representatives of) “real” end-users
 Doing research in the proper context (field-based vs. laboratory based
research)
 Distinction between methods & techniques of data collection and data
analysis
5
IMPORTANT ASPECTS OF UUU RESEARCH

6.  Interviews / Focus Groups
 Surveys / Questionnaires
 Product analysis
 Card sorting
 Interaction study
 Observation …
COLLECTING USE, USER & USABILITY DATA
ANOTHER LOOK AT CATEGORIES OF TOOLS

7. asking subjects to execute a particular problem-solving
task and to speak out their thoughts while solving that
problem / performing that task
participants are asked to say out loud whatever they are
looking at, thinking, doing and feeling at each moment
thereafter:
analyzing the resulting verbal and action protocols
7
THE THINK ALOUD METHOD

8. Some investigations into knowledge acquisition from existing (static) maps:
 Thorndyke & Stasz (1980)
 Kinnear & Wood a.o. (1987/1988)
 Kulhavy, Pridemore & Stock (1992)
 Crampton (1992)
 Heidmann & Johann (1997)
 Board (1997)
 Ungar, Blades & Spencer (1997)
EARLIER APPLICATIONS OF THE THINK ALOUD METHOD
IN MAP USE RESEARCH

9. Investigations into the use of cartographic visualization systems for
geographical data exploration:
 McGuinness, van Wersch, Stringer (1993)
 MacEachren, Boscoe, Haug, Pickle (1998)
 See online bibliography ICA Commission on UUUI for more examples of
case studies!
EARLIER APPLICATIONS OF THE THINK ALOUD METHOD
IN MAP USE RESEARCH

10.  Form pairs
 Execute map reading activity:
Locate the Google StreetView image on the map fragment provided while
thinking aloud
 Observer takes notes or records the thinking aloud
 Switch roles with Map 2
 After this, we will discuss your experiences
10
HANDS-ON EXPERIENCE
MAP READING ACTIVITY IN PAIRS

11. 11
HANDS-ON EXPERIENCE
MAP 1: BAILEUX

12. 12
HANDS-ON EXPERIENCE
MAP 2: FORGES

13.  What are your experiences with applying the think aloud method?
 What are the potential advantages?
 What are the disadvantages? What could be the problems?
 Questions?
HANDS-ON EXPERIENCE WITH THINKING ALOUD
MAP READING ACTIVITY IN PAIRS
13

14.  Leads to valid, unique, direct and the most complete data on cognitive
processes
 Least disturbance of the thought process
 No memory errors
 No interpretation, rationalization or explanation of thoughts by subjects
 No need to bring thoughts in predefined form
ADVANTAGES
THINK ALOUD METHOD

15.  Very time-consuming and tedious
 Protocol analysis difficult and subject to interpretation by researcher
 Protocols may be incomplete
 Verbalization problems
 Situation may be unnatural for test participant
DISADVANTAGES
THINK ALOUD METHOD

16.  Verbatim transcription of recordings: verbal protocols.
 Tools for qualitative data analysis
e.g. Atlas.ti or MacSHAPA
PROCESSING & ANALYSIS OF THINK ALOUD DATA
TOOLS

17. Keep your research questions in mind!
(but leave room to discover the unexpected ….)
PROCESSING & ANALYSIS OF THINK ALOUD DATA
ANALYSIS

18. 18
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
problem
solving
Predicted
coded
protocols
Raw
protocols
Coded
protocols
Segmented
protocols

19. 19
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
problem
solving
Predicted
coded
protocols
Raw
protocols
Coded
protocols
Segmented
protocols

20. 20
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
problem
solving
Predicted
coded
protocols
Raw
protocols
Coded
protocols
Segmented
protocols

21. 21
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
problem
solving
Predicted
coded
protocols
Raw
protocols
Coded
protocols
Segmented
protocols

22. 22
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
problem
solving
Predicted
coded
protocols
Raw
protocols
Coded
protocols
Segmented
protocols

23. INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION
Corné van Elzakker
The Use of Maps
in the
Exploration of Geographic
Data

24. 24
Exploratory cartography
maps as tools
for discovery

25. 25
Exploration of geographic data
This research project >
investigation into the use of maps:
the actual use of an existing
cartographic display
plus
the cognitive process of finding,
selecting, retrieving, adjusting or
generating 'private' map displays

26. 26
User task: regional exploratory studies
gaining insight into the geography of an
unknown region

27. 27
Example of a graphical model
constructed by a test person

28. 28
Database of the Overijssel region
Available to the test persons:
 digital geographic (attribute and base map)
data, to visualize within the ArcGIS
environment
 existing paper maps and atlases
 existing digital maps (static, interactive or
dynamic maps in different formats, ArcGIS
‘map documents’, maps on CD-ROM or on
diskette and Web maps

29. 29
Searching the Overijssel database

30. 30
Model of regional exploratory problem-
solving: map selection matrix

31. 31
Part C

32. 32
Model of regional
exploratory
problem-solving:
map use matrix

33. 33
Map use matrix (upper part enlarged)

34. 34
Model of regional exploratory problem-
solving

35. 35
Protocol analysis

36. 36
Map use for model construction

37. 37
Research methodology applied
 questionnaire
 thinking aloud
 video recording
 screen transaction logging
 thinking aloud in retrospect
 analysis of verbal protocols
 analysis of action protocols
 product analysis (graphical model)

38. Thinking aloud is a very useful qualitative method to
discover use, user and usability issues ánd
requirements.
Most profitable use in a mixed methods approach (e.g.
including video observation, survey to establish
participants’ backgrounds, screen / action logging, etc.)
Combining Eye-Tracking and Thinking Aloud: demonstration Kristien Ooms
38
CONCLUSION
INCORPORATE THINK ALOUD IN MIXED METHODS APPROACH

39. 39
ADDED VALUE OF THINKING ALOUD IN EYE-TRACKING
FIND STREETVIEW IMAGE ON MAP CHIMAY

40. 40
EFFECTS OF THINKING ALOUD ON ET RESULTS?

41.  Synchronization of recordings to facilitate analysis ...
Further practical hints will be provided by Kristien Ooms.
For now...
41
IMPLEMENTATION ISSUES
THINKING ALOUD IN MIXED METHODS APPROACHES

42. Thinking aloud is a very useful qualitative method to
discover use, user and usability issues ánd
requirements.
Most profitable use in a mixed methods approach.
42
CONCLUSION
THINK ALOUD METHOD

43. APPLICATION
0F THE
THINK ALOUD METHOD
IN
USE, USER AND USABILITY RESEARCH
IN
CARTOGRAPHY & GEOGRAPHIC
INFORMATION SCIENCE
c.vanelzakker@utwente.nl
WORKSHOP DESIGNING & CONDUCTING USER STUDIES
6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016