A tutorial on qualitative methods in CSCW presented at the CSCW Asia Winter School 2019 (Kyoto, Japan - 14 December 2019)

1. Qualitative Methods in CSCW
Research
Luigina Ciolfi
Professor of Human Centred Computing
Sheffield Hallam University, UK
http://luiginaciolfi.net
https://blogs.shu.ac.uk/c3riimpact/
CSCW Asia Winter School 2019, Kyoto, Dec 2019

2. Who’s Talking?
Background in Communication Studies and HCI…My work moved to CSCW
during my PhD
Qualitative researcher. Domains: cultural heritage and mobile/nomadic work
Learning from service: I read a lot of CSCW papers!
Also experience in supporting regional CSCW communities (EUSSET – European
Society for Socially Embedded Technologies; SIGCHI Ireland)
Teaching to CS students

3. Some questions I get asked all the time
● Why do we need to know about qualitative research?
● Is qualitative research objective?
● Why should I spend so much time studying people when they could just be
saying one thing and doing another?
● How can qualitative results be generalised?
● How do I know my research findings are valid?
● Is qualitative better than quantitative? Or the other way around?

4. “The two hardest problems in computer science are: (i) people,
and (ii) convincing computer scientists that the hardest
problem in computer science is people.”
Jeff Bigham, Carnegie Mellon University

5. They change their mind
They are complicated
They are part of complicated organisations and groups
They are busy and might not want to talk to you
They often say one thing but mean, or think, another
They often want what experts know will not work

6. They change their mind
They are complicated
They are part of complicated organisations and groups
They are busy and might not want to talk to you
They often say one thing but mean, or think, another
They often want what experts know will not work
They are smart and adaptable
They can empathise
They make very complicated systems work, using very complex tools
They like to be and work with other humans
They can figure things out
They find ingenious workarounds, solutions, or alternatives

7. What This Tutorial is NOT About
A step-by-step guide to ethnography…
…or interview design in a nutshell…
…or a pocket guide to qualitative data analysis

8. What This Tutorial IS About
Some history of qualitative enquiry in CSCW: why do we do it? And why do we do
it in this way?
Some key approaches and milestones shaping qualitative CSCW research
Challenges and opportunities
Some experiences in doing qualitative work in CSCW… Let’s share some food for
thought!

9. Qualitative Research in CSCW
Qualitative methods are mainly concerned with the ‘how’ and ‘why’ of research on
collaborative computing and computer-supported collaboration
Examples going back to the inception of the discipline, applied to several domains
But, why is so much CSCW work based on qualitative methods?

10. CSCW and Qualitative Methods: A bit of history
From Human Factors to Human Actors
“Understanding the user's needs, and the tasks performed by the user is basic to
the system development process. However, it is a mistake to think that simply
having a human factors person on the design team is by itself sufficient to ensure
that the "human factor" has been adequately taken into account” (Bannon, 1991)

11. CSCW and Qualitative Methods: A bit of history
CSCW researchers were advocating moving away from lab-centred, one-user
focused studies, which “have deliberately stripped away the supporting context of
the everyday world, in an effort to study 'pure' internal processes" (G. Olson 1990)

12. CSCW and Qualitative Methods
Systems were certainly providing users with collaborative functionalities, but what
about the “articulation work” of coordination and cooperation?
Studies of cooperation in actual work settings were required – “second order work”
“Situated” character of cooperative work (Suchman, 1983; Gershon & Star, 1986)
Looking at the complexity of people, artefacts, spaces, contextual and cultural
settings: “assumption that the social and material are ‘constitutively entangled’ “
(Orlikowski, 2007)

13. CSCW and Qualitative Methods: A bit of history
“The indigenous CSCW research program implies an approach to technology
development radically different from that of the previous development of
interactive and collaborative computing, namely, an approach to technology
development in which ethnographic and other forms of in-depth workplace studies
play an essential and proactive role.” (Schmidt & Bannon, 2013)

14. CSCW and Qualitative Methods: A bit of history
Early work on Control Rooms and ‘centres of coordination’
The “turn to ethnography” in human-centred computing
A “practice-focused” approach: “CSCW was the first research community in
applied computer science which stressed the importance of an in-depth
understanding of practices when designing ICT artefacts” (Wulf et al, 2011)
Schmidt, K (2014), “The Concept of ‘Practice’: What’s the point?”

15. Data Presentation and Analysis
From Heath & Luff, 1992

16. CSCW and Qualitative Methods: A bit of history
‘Studies of the social organisation of work will need to proceed in a manner which
recognises this heterogeneity of domains and develops analytic tools which are
capable of exhibiting the relevant scope of this variety’ (J. Hughes et al 1994)

17. Ethnography in CSCW
Fieldwork in HCI draws heavily from the ethnographic tradition, which “seeks to
present a portrait of life as seen and understood by those who live and work
within the domain concerned” (D. Randall, R. Harper and M. Rouncefield,
Fieldwork for Design, emphasis added)
Ethnography in CSCW (and HCI) is now widespread, including variations such as
“quick and dirty ethnography” and “design ethnography” (about which we
could debate!)
Recommended reading: “Reflections on 25 Years of Ethnography in CSCW”, by
Jeanette Blomberg and Helena Karasti, JCSCW 22: 4-6 (2013)

18. Ethnography in CSCW
We “borrowed” ethnography and other qualitative methods from social science…
…but we do things quite differently:
- Not so interested in creating theory or expanding existing theory out of
qualitative work
- Not so interested in generalising at the macro scale
- We are quite much more fond of mixed methods than social scientists
- “Implications for design”

19. Domains of Ethnographic study in CSCW
“Centres of Coordination”
Workplace studies
“Diagnostic Work”
“Mobility Work”

22. “Centres of Coordination”
Workplace studies
“Diagnostic Work”
“Mobility Work”
Disaster response
Health and clinical settings (many aspects of it)
Collaborative software development
Collaborative writing
Museum visiting
Domains of Ethnographic study in CSCW

23. Key outlooks provided to the broader field of HCI/HCC:
- Unanticipated use
- Appropriation practices
- Concerns for social context/social dimension of technology use (well before
social became “paradigm”)
- Domain-specific knowledge and understanding
- Ethnographic methods now belong in HCI
Domains of Ethnographic study in CSCW

24. Moving away from “work”
Leisure domains
Education (CSCL)
“Work” as the serious business of getting things done
“Work” (intended as paid labour is also changing…organizational and culture
changes, including micro-work)
Domains of Ethnographic study in CSCW

25. Doing qualitative research when you cannot be in the field…or has the field
moved?
“it has become increasingly problematic in CSCW to conceptualize the field site
as single-sited. The field site has become a multifaceted and intricate
constellation of people, technologies, activities, entities, and relations; and the
boundaries of the field site are less clear, even unbounded, involving
extended spatial and temporal scope” (Blomberg and Karasti, 2013)
Other methods for qualitative inquiry

26. Other methods for qualitative inquiry
Not just ethnography, other qualitative methodologies trying to capture the
complexity of “situated collaboration”
Well-known examples:
Interviews
Action research
Questionnaires
…often in combination (triangulation)

27. Social media and online collaboration
● Online ethnography (“digital ethnography”)
● Trace ethnography (Geiger and Ribes, 2011; Østerlund, Crowston and
Jackson, 2019)
Other methods for qualitative inquiry

28. Design/Creative methodologies in CSCW
● Another important aspect of qualitative research in CSCW
● Not only for designing or for generating design requirements
● E.g. “workshops” as ways to construct field sites (Rosner, Kawas, Li, Tilly and
Sung, “Out of Time, Out of Place: Reflections on Design Workshops as a
Research Method”, CSCW 2016)
● Influence by (and often convergence with) Participatory Design (Greenbaum
and Kyng (1991) “Design at Work”)
● The “Interaction Design” turn, early 2000s
● The “Fiction Turn” (late 2010s)

29. CSCW and Qualitative Methods
“The steps through which CSCW became institutionalized were rather practical
steps in a process of interlinking a range of research activities and communities
that participants began to see as converging or in some other sense related.
CSCW emerged more as a bazaar than as a cathedral…there is a sense in which
CSCW is continually being formed” (Schmidt & Bannon, 2013)

30. Doing qualitative research in CSCW: Challenges?
● Access to field
● …linked to research design
● Collecting data
● Analysis
● Writing up (it takes a lot of space and time)

31. Some Challenges from My Research

32. Role of digital interpretation aids at heritage sites

33. Figure : The complete set of smart replicas.
Role of digital interpretation aids at heritage sites

34. Figure : The complete set of smart replicas.

35. Figure : The complete set of smart replicas.

37. Beyond “running a study”: Action Research
PhD projects:
Dr Laura Maye at the
Hunt Museum (Limerick,
Ireland)
Dr Caroline Claisse at
Bishop’s House (Sheffield,
UK)

38. Researching Nomadic Work
● Researching workers who are not usually based at company premises (incl
freelancers)
● Operating from home or from “temporary workplaces” (cafes, airports, hotels,
etc.)
● Difficult to follow, difficult to observe how cooperation with colleagues and
clients happens
● Extensive interview studies, complemented by focused shadowing sessions
and other techniques such as diaries

40. Some books I found useful
● “Fieldwork for Design”, by Richard Harper, David Randall and Mark
Rouncefield
● “Digital Ethnography”, by Sarah Pink
● “My Life as a Night Elf Priest” by Bonnie Nardi
● “Workplace Studies”, “Video in Qualitative Research”, J Hindmarsh, P Luff
and C. Heath
● “Convivial Toolbox”, by L. Sanders and P. Stappers
● “Critical Fabulations” by Daniela Rosner
● …many “role model” papers using qualitative research and data presentation
in CSCW

41. Questions or Comments?