Where researcher sets up different conditions of action to see how it affects humans’ behaviour: Experimental (lab) research, Quasi-experimentation, Wizard-of-Oz simulations, Usability evaluations, Field trials, Research through design approach

1. Antti Salovaara
Aalto University, School of Business
22 January 2015
Methods in User–Technology Studies
Interventionist studies
16.15 – 17.00
Methods in User–Technology Studies / Interventionist
studies by Antti Salovaara is licensed under a
Creative Commons Attribution-ShareAlike 4.0
International License.

2. Interventionist studies
= where researcher sets up different conditions of action
to see how it affects humans’ behaviour
Experimental (lab) research
Quasi-experimentation
Wizard-of-Oz simulations
Usability evaluations
Field trials
Research through design approach

3. Experimental (lab) research
= studies where nuisance factors are eliminated or controlled
Owes a lot to research methods in psychology and social psychology
Needs:
Predefined quantitiable research questions
Precise measurements
Sometimes (not always) special hardware (eye tracking, EEG, ..)
Comparison between “treatment” and “control” (i.e., baseline)
groups

4. Example: Experimental (lab) research
Source: Lehtinen et al. (2012)
Vibrotactile
actuators
(bzzz)
© Lehtinen et al.
Image credits: See last slide

5. Quasi-experiments and analogue experiments
Quasi-
experimentation:
experimental
research without
the control group
Shadish et al. (2002)
Analogue experiments:
Controlled studies that
replicate simulate real-life
situations
Source: Oulasvirta et al. (2005)
Image credit: See last slide

6. Wizard-of-Oz simulations
Used in studies on futuristic
technologies
e.g., artificial intelligence
Applies deception:
Participant believes that the
system is real while a part of its
operation is controlled by the
experimenter
Setup is revealed after the study
E.g., speech recognition studies
Chess-playing automaton constructed
by Wolfgang von Kempelen in 1770
Imagecredit:Seelastslide

7. Example: a Wizard-of-Oz study
Nokia 9500
communicator
HP
PDA
Suunto
sports
wrist-
watch
Plastic
tray
attached
to pram’s
handlebar
©Author
© Author

8. Image credit: See last slide

9. Wizard-of-Oz setup
Wizard Facilitator User Recorder
Image credits: See last slide

10. Usability evaluations
Typical method:
1. List the representative tasks that
technology is used for
2. Write a realistic scenario around
these tasks
3. Create materials for these tasks
4. Present the scenario for the
participant and ask him/her carry
out the tasks.
5. Record with video
6. Repeat with more participants
until findings saturate (i.e., when
new usability problems are not
found)
= studies on the quality of
technology design with users
Exist in many variations
An effective method for
identifying problems
Rarely used in academic
research
Imagecredit:Seelastslide

11. Field trials
= open-ended studies or quasi-experiments on research
prototypes “in the wild”
e.g., Muller et al’s study on a new cooperation technology
Open-ended field trials’ differences to observational studies:
Technology is provided to users by researchers
Technology may also have been developed by the researchers
RQ focuses on the change caused by the new technology
The prototype’s design often embodies a research hypothesis
e.g., Muller et al: “Users do not need to use so many systems if a
system would support object-centric sharing, object-level awareness,
user-created structured collections, and dynamic membership”

12. Research through design
Is a conceptualization of the reason for organizing field
trials in academic research
Basic principles:
“Consider information systems as research instruments that are
realizations of your research hypotheses”
“Research prototypes should be created and studied for the
purposes of studying phenomena in IS use, not for finding out if
they make a positive difference”
⇒ Research prototypes do not need to be “good” design
Instead, they must help you answer to your research question

13. Example of a field trial: Comeks
Comeks – An MMS-based mobile comic strip creator
How the idea for Comeks originated:
Discovery of live action role-players’ need for light-weight story-like
game documentation
Image credit: See last slide.
Source: Salovaara (2007)

14. © Author

15. Image credit: See last slide.

16. How Comeks worked
Phone
with
Comeks
Message
creation
MMS Any
phone
Message
reception
© Author

17. Study on appropriation
First encounter with
a new technology
Appropriation
Integration into
daily activities
RQ: How do users appropriate Comeks and use its features?

18. Methods
U U
R
1.
2. 2.
R
3.
Phone
with
Comeks
MMS
Any
phone
Context
Logger
U U
Cued recall interview
every 2-3 weeks
R

19. Using lots of methods together
An observation-based field trial with
logging,
cued recall interviews,
questionnaire,
Wizard-of-Oz
That followed the research
through design approach
Source: Jacucci et al., CHI 2007

20. Mobile multimedia group communication
chat with a support for collocated interaction
Motivation: mobile
messages are often
shared with collocated
others
RQ: How will a system with dedicated support for
collocated interaction be used by groups?
Rally visitors in
Neste Rally 2005
Video >
©Theauthor
© Author
Source: Salovaara et al. (2006)

21. CoMedia user interface presence indicators
Blue man icon: a nearby
CoMedia user who is using
CoMedia and is in Bluetooth
range
Green man icon: a remote
CoMedia user who is using
CoMedia but is not in Bluetooth
range
Gray man: an off-line CoMedia
user who is not using CoMedia
Violet man: the user him/herself
Phone icon: a user has used
the phone recently – be it
CoMedia or any other
application.
!
!
!
!
!
People that are present in different chat spaces
© Author

22. P
“The Panopticon”
Observation, logging, “The Panopticon”, and
Wizard of Oz
Phone
with
CoMedia
Phone
with
CoMedia
U U R
SMS
R “Rally news”
&
Wizard of Oz
“The Panopticon”: Source: Salovaara et al. (2006)

23. Other methods
Cued recall interviews:
After the rally weekend, interviews about each message sent and
received
Questionnaire:
Sense of presence scale

24. References
Jacucci, G., Oulasvirta, A., Ilmonen, T., Evans, J., & Salovaara, A. (2007). CoMedia: mobile group
media for active spectatorship. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI 2007) (pp. 1273--1282). New York, NY: ACM Press.
Lehtinen, V., Oulasvirta, A., Salovaara, A., & Nurmi, P. (2012). Dynamic tactile guidance for visual
search tasks. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and
Technology (UIST 2012) (pp. 445--452). New York, NY: ACM Press.
Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005). Interaction in 4-second bursts: The
fragmented nature of attentional resources in mobile HCI. In Proceedings of the SICCHI Conference
on Human Factors in Computing Systems (CHI 2005) (pp. 919--928). New York, NY: ACM Press.
Salovaara, A., Oulasvirta, A., & Jacucci, G. (2006). ``The panopticon'': a method for observing inter-
group interactions. In Workshop on Reality Testing at CHI 2006. Montreal, Canada:
Salovaara, A., Jacucci, G., Oulasvirta, A., Saari, T., Kanerva, P., Kurvinen, E., & Tiitta, S. (2006).
Collective creation and sense-making of mobile media. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI 2006) (pp. 1211-1220). New York, NY: ACM Press.
Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for
Generalized Causal Inference. Boston, MA: Houghton Mifflin.

25. Image credits
Figure 1 from Lehtinen et al.
(2012). © Paper authors.
Figure 3 from Lehtinen et al.
(2012). © Paper authors.
Figure 3 from Oulasvirta et al.
(2005). Used with permission.
Figure 1 from Oulasvirta et al.
(2005). Used with permission.
Copper engraving from the book:
Karl Gottlieb von Windisch, Briefe
über den Schachspieler des Hrn.
von Kempelen, nebst drei
Kupferstichen die diese berühmte
Maschine vorstellen. 1783. Public
domain: Copyright expired.
The Ruoholahti canal by Oghmoir.
http://commons.wikimedia.org/wiki/
File:Ruoholahden_kanava.jpg.
Licensed under Creative Commons
Attribution-Share Alike 3.0 Unported
Images © Author and HIIT Uix
research team 2004
CodeSyntax usability lab by
garaolaza,
http://www.argazkiak.org/photo/
codesyntax-usability-lab/size/l/.
Licensed under Creative Commons
Attribution-Share Alike 3.0 Unported
By unnamed study participant 2005.
Used with permission.