Research question, Research methods as a toolbox, Reliability and validity, Open vs. closed research designs, Grounded theory, Interventionist vs. observational research, Having multiple research questions and methods, Triangulation and methodological overlap

1. Antti Salovaara
Aalto University, School of Business
22 January 2015
Methods in User–Technology Studies
Broad concepts
13.45 – 14.30

2. Broad concepts
Research question
Methods as a toolbox
Reliability and validity
Research designs (open vs closed)
Grounded theory
Interventionist vs observational research
Multiple research questions and methods
Triangulation & methodological overlap

3. Deciding the research question
RQ
Methods
Research
setting
Practical
realizability
Literature
and trends RQ

4. Every study has a research question
Grounded theory:
“Does setting X have something interesting?”
Ethnographic field study:
“How will the participants collaborate in context X?”
Field trial:
“Will a system with a feature X lead to some interesting user
behaviours?”
Controlled experiment:
“Does X lead to a higher effect than Y”?”

5. Research question determines the methods
RQ
Methods
Research
setting
Practical
realizability
Literature
and trends
Method 2Method 1
RQ Methods as a “toolbox”:
Always choose the best method(s)
for your research question!

6. Evaluating which methods to choose
Validity:
Does this method really
measure what I’m
interested about?
Reliability:
How accurate is the
method?
Do the participants
act naturally if I’m
present?
Observation:
Will I observe
enough
representative
situations with good
quality?
Interviewing:
Are the participants
willing and able to
tell the truth?
How can I make the
informants tell rich
stories?

7. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
1999)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

8. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
2000)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

9. Grounded theory
Best suited for open, observational studies
No predefined research focus or question
Heavily data-driven analysis
Open, axial, selective coding
Ideally: Alteration of analysis and data
collection
Constant comparative method:
Formulation of hypotheses and testing them with
theoretical sampling of more data
Glaser &
Strauss (1967)
Strauss &
Corbin (1990)

10. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
2000)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

11. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
2000)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

12. Controlled experiments
“Does X increase Y?”
“Does X increase Y more than Z does?”
“Controlled” =
All factors other than X (i.e., “nuisance variables”) are removed
from the research setting
Requires usually a lab setting
Typical in psychology and natural sciences

13. How each statistical method has a purpose
of answering a particular kind of quwstion:
https://www.uvm.edu/~dhowell/methods8/
Errata/DecisionTree.jpg

14. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
2000)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

15. Research designs (open vs closed)
Open Closed
Grounded
theory
(research
focus and
question
develops
iteratively)
Controlled
experiments
(research
question
determined
in advance
e.g., as a
testable
hypothesis)
Kramer et al
(PNAS 2014)
“naturalistic
field
experiment”
Mackay
(ACM TOCHI
2000)
“open-ended
ethnographic
field study”
Muller et al
(CHI2004)
“open-ended
field trial”
Observational Interventionist

16. Observational vs. interventionist studies
Observational: studying natural action
Researcher affects the research setting as little as possible
Benefit: High ecological validity
Risk: Collecting a messy set of individual observations without any
overall pattern
Interventionist: studying specific phenomena
Researcher carefully introduces changes in the setting (i.e.,
“intervenes”), to foreground interesting user behaviours
Benefit: Focusedness increases the amount of analysable data
Risks: Limited ecological validity, intervention does does not
foreground interesting behaviour

17. Examples of intervention techniques
Trialing
Asking the participants use a system that researchers have (at
least partly) designed and where certain features have been
foregrounded
Setting selection
E.g., arranging the study in a setting where the actions of interest
are frequent
Repetition
E.g., asking the participant perform the same task several times
Stabilizing
Removing nuisance variables that would threaten firm conclusions

18. Having multiple RQs and methods
…is a good idea.
Humans and research contexts are unpredictable
=> All user–technology studies have a high likelihood of
failure:
Participants don’t do anything interesting
(wrong RQ, i.e., a literature–RQ error)
Participants do something interesting, and you have data about it,
but your data is not convincing (wrong method, i.e., RQ–method
error)
You don’t get enough data (wrong method)
Etc

19. Triangulation and redundancy
Data from
method 2
Data from
method 1
RQ
Methodological triangulation:
Studying the same question with
two independent methods
1. Interviews and system logs
2. Your and somebody else’s data
Redundancy:
Data from
method 2
Data from
method 1
RQ
Studying the same
question with
overlapping methods
Interviews and a diary

20. Other forms of triangulation
Method 2Method 1
RQ
Fielding &
Fielding (1986)
Methodological Theoretical
Theory 2Theory 1
RQ
Data
Context 2Context 1
RQ
Denzin (1978)
Investigator
Researcher
2
Researcher
1
RQ

21. The final result
RQ1
RQ2
RQ3
Method 1
Method 2
Method 3
Method 4
“answers”
Literature
Trends
“motivates”

22. A “war story” before coffee
(14.30-14.45)?
“War stories”:
see Julian Orr (1996):
Talking about machines:
An ethnography of a modern job

23. Study on programmers’ instant messaging
RQ
Methods
Research
setting
Practical
realizability
Literature
and trends RQ
Starting point: 16,000 messages and 4 visits to Friday meetings
Content analysis
What is being
communicated?
Practical
realizability
Knowledge
management
Developers helping
each other
Knowledge
management
with IM
Developers helping
each other in a rapidly
changing environment
1. What is being
communicated?
2. Does IM support
knowledge sharing?
Developers sharing
ephemeral knowledge
(ephK)
Knowledge
sharing with
IM
Knowledge
sharing
mechanisms
Content classification
Outomes: 1. identification of new knowledge category (ephK); 2. sharing of
ephK through IM is poorly captured in existing KSM frameworks
Content classification
Mapping to KSM frameworks
Sharing direction analysis
Salovaara & Tuunainen (ICIS2013)

24. Coffee
14.30-14.45

25. References
Denzin, N. K. (1978). Sociological Methods: A
Sourcebook. New York, NY: McGraw-Hill.
Fielding, N. G. & Fielding, J. L. (1986). Linking
Data. Beverly Hills, CA: Sage Publications.
Glaser, B. G. & Strauss, A. L. (1967). The
Discovery of Grounded Theory: Strategies for
Qualitative Research. New Brunswick, NJ:
Transaction Publishers.
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T.
(2014). Experimental evidence of massive-scale
emotional contagion through social networks.
Proceedings of the National Academy of
Sciences, 111(24), 8788–8790.
Mackay, W. E. (1999). Is paper safer? The role of
paper flight strips in air traffic control. ACM
Transactions on Computer–Human Interaction,
6(4), 311--340.
Muller, M. J., Geyer, W., Brownholtz, B., Wilcox,
E., & Millen, D. R. (2004). One-hundred days in
an activity-centric collaboration environment
based on shared objects. In Proceedings of the
SIGCHI Conference on Human Factors in
Computing Systems (CHI 2004) (pp. 375–382).
New York, NY: ACM Press.
Orr, J. E. (1996). Talking about Machines: An
Ethnography of a Modern Job. Ithaca, NY: ILR
Press.
Salovaara, A. & Tuunainen, V. K. (2013).
Software developers' online chat as an intra-firm
mechanism for sharing ephemeral knowledge. In
Proceedings of the Thirty Fourth International
Conference on Information Systems (ICIS 2013).
Strauss, A. L. & Corbin, J. (1990). Basics of
Qualitative Research: Grounded Theory
Procedures and Techniques. Thousand Oaks,
CA: Sage Publications.