Diary Studies in HCI & Psychology

Diary Studies in HCI & Psychology:
Why They’re Useful and How to
Conduct Them
Demetrios Karis
demetrios.karis@gmail.com

UPA Boston’s Tenth Annual
Mini UPA Conference
May 25, 2011

0

What is a Diary Study?

In a diary study, participants go about their
normal lives except that they report what they’ve
done or experienced (via questionnaire or freeform
using an electronic or paper diary) at some interval,
signal, or event, and they do this multiple times over
days or weeks.

• © 2011 Karis • Diary Studies •
1

I’ll Go Over

• How people report
• When they report
• What they report

• Example of what to report
– Study on information needs while mobile: ―What time does the post
office close?‖
– Questions:
1. Where were you?
2. What were you doing?
3. What was your information need?
4. I addressed the need (At the time, Later, Not at all)
5. If you attempted to address the need, how did you do so? If
you didn’t make an attempt, why didn’t you?
6. Could you have addressed your need by looking at your
personal data (e.g., email, calendar, web browsing history, chat
history, or other)

2

Outline
• Why diary studies?
• History
• Example 1: Understanding the mobile phone music experience
• Methodology
– Three types of diary designs
– Paper and pencil versus electronic diaries
– The role of media, and the ―snippet‖ technique
– Practical considerations: number of participants, duration, data
collection, and analysis
• Example 2: Information needs while mobile
• Example 3: Electronic chronic pain diary
• Example 4: A diary study of work-related reading
• Related methods
– Experience sampling method (ESM)
– Day-in-the-life observation
• References

3

Why Diary Studies?
• Diary studies (in HCI):
– Provide information on how people are using a device or
application in natural everyday situations – and provide information
on how this use unfolds over time
• When, how, and where they use it
• Problems, similar products they’re using, unmet needs
– Provide information on user needs in order to provide
requirements & design guidance for new devices or applications
– Eliminate biases introduced by retrospection
– Often more accurate than retrospective techniques
• Limitations
– Participants’ commitment and dedication required
– Training sessions often required

4

History
• Paper and pencil diary studies began appearing in the psychological
literature 70 years ago in the early 1940s
• In psychology, used to study
– Personality processes
– Marital and family interaction
– Physical symptoms
– Mental health
• Example: An investigation of women’s perception of intimacy in their
romantic relationships in everyday life, by assessing
– The degree of intimacy the typical woman in a committed relationship
feels on average
– The extent to which the typical woman’s feelings of intimacy vary over
time
– Whether women differ from one another in their average feelings of
intimacy and in the variability of their feelings of intimacy over time
This example, and much of this presentation, comes from Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life as
it is Lived. Annual Review of Psychology, 2003, Volume 54, 579-616.

For a review of diary studies in HCI and CSCW prior to 2002, see Palen, L., & Salzman, M. Voice-Mail Diary Studies for
Naturalistic Data Capture under Mobile Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACM
Press, 87 – 95.
5

Example 1: A Diary Study to Help
Understand the Cellphone Music Experience
Goals
Understand how using a phone to listen to music fits into a person’s
―musical ecology‖ -- i.e., how do people use their phone and other
music devices in their daily life?

– More Specific Goals:
• Understand the user experience on a particular phone – with
respect to music – and whether it’s an improvement over the
last version
• Recommend changes to improve future music phones
• Five Studies
1. Expert review
2. Usability study
3. Diary Study
4. Interviews with iPod users
5. User group study of Rhapsody and online music

6

Methodology

• Week 1: use your phone as you normally would
– Email sent every day asking if they used their phone, and how

• Week 2: a task given each day (e.g., rewind song a few seconds)
– We receive information on usability and utility, and also whether they
had performed the task before

• Interviews at the beginning and end of the study

Kate Dobroth ran this study.

7

Methodology: More Details
• Nineteen participants between 18-44 (most between 18 and 24) who
used their phone for listening to music (recruited from a list provided
by a wireless carrier)
• Initial interview
– Impressions of the phone, music consumption, and experiences with
loading music onto the phone
• Daily emails about how phone was used
– Did you use your phone today? What did you use it for?
– Did you use any of the music features? If so, which ones?

• Second week of emails included music-related tasks
– Allowed us to explore whether participants had discovered various
features, and whether they were useful
• Final wrap-up interview
– Explore issues and questions from emails
8

Lots of Questions
• Where are you when you listen to music?
• How do you listen to music? (devices)
• Why do you listen using each type of device?
• How many songs do you have on your phone?
• Getting music onto the phone
– Where do people get music?
– How do they get it onto the phone?
• Usage
– Which phone features are used most often?
– How often do people use the music feature?
• Task performance
– Performance and impressions of specific tasks completed during
the second week
• Overall impressions of the phone
– Like most and least

9

Questions to Consider
• Are these results representative of all people who might like to listen
to music on their phones?
• Would you make design decisions or change requirements based on
these results?
• What other studies might be worth doing?
• Could there be memory issues with a diary study like this? That is, a
person may tell you what they did at the end of the day, many hours
after they actually did it.
• How can diary and quantitative techniques be combined?
• Other questions?

10

Log Analysis Plus a Diary Study

Adding quantitative data from log analyses helped
tremendously to answer basic questions.
• In the days before Smartphone apps, users could download a version of
SuperPages onto their feature phones (a BREW application)
• Several usability studies — valuable, but little information on utility
• Combining a diary study and log analysis helped to answer a number of
questions:
– What prevented people from using the service, or finding what they wanted?
D
– Where were they when they used the service, and what was the context?
Both – What are the opportunities for improvement, to make the service fit in with
how people actually use the application?
– When did people use the service?
– What types of searches were most popular?
L – How many searches did people conduct in each session?
– Which location options were used most often?
– Which businesses did people look for?

11

Three Types of Diary Designs
• Participants can be asked to report on their actions or experiences in
three ways
1. At specific intervals
• Once a day, or specific times of day (after breakfast, after school,
before bed) or specific time intervals (every three waking hours)
2. At a signal from the researcher
• Participants prompted to provide diary reports at fixed, random, or
a combination of fixed and random intervals
3. When an event occurs
• Participants provide reports whenever an event occurs that meets
some pre-established criteria
• How often should participants report?
– Need to consider the temporal patterns of the phenomenon being
studied; what’s the expected frequency of the events of interest?
– Also, need to consider the type of event or phenomena
• Pain: need to report immediate experience, not past
• Major events can be reported daily with much less error or bias (did
you take mass transit today?)

12

Paper and Pencil Diaries
• Easy to implement, but several problems
– Participants may fail to remember the scheduled response times
– Or may fail to have diaries at hand
– Potential for retrospection errors: participants rely on reconstruction to
complete missed entries
– For researcher, the burden of data entry and handling
• Suggestions
– Make diaries easily portable
– Reduce possibility of participant errors by preprinting the dates and
times of expected responses onto diary sheets
– Reduce demand characteristics by emphasizing the importance of
accuracy over the number of responses
– Pilot test your diary material on a representative sample of people
• Augmented paper diaries
– Responses still collected via paper, but these are augmented by
signaling devices such as pagers, preprogrammed wristwatches, phone
calls, or mobile phones (alarms, email, text messages, etc.)

13

Advantages of Electronic Devices
• Studies started appearing about 2000
• Many advantages
– Allow for signaling; can remind you to complete diary entry
– Can date and time-stamp entries
• Prevents ―back-filling‖ diaries to appear compliant
– Can provide location information if device has GPS sensor
– Flexibility in presentation of questions, plus the ability to force
responses before proceeding
– Improved data quality and management, & eliminates data entry
• Data doesn’t need to be transcribed and checked
• Out-of-range responses prevented
• Fewer missing responses
– Eliminates risk of skipped questions
– Much higher compliance rates using electronic diaries compared to
paper (in both children and adults)
• More diary entries completed (6.89 days out of 7 for electronic
diaries versus 4.97 using paper in Lewandowski et al., 2009)
• Early studies: participants provided with PDAs; now, many people
carry smartphones with them at all times, and these can be used
14

Diary Studies: The Role of Media
• Up until now we’ve been assuming that people are just responding to
questions by writing their answers or selecting from multiple choices
• Photos, video, audio recordings, & objects can help in diary studies
– Participants capture events (e.g., via a photo); these can stand alone or
can be used to elicit more detail during a later interview
• Annotation produced at the same time can be very beneficial (but
can add significantly to the effort involved)
– Audio can be used in some situations when taking a picture isn’t
appropriate (recording an audio memo or calling a number and
responding to questions)
– In some situations, it may make sense to have participants collect
objects

Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary
Studies. Proceedings of the CHI ‘05 (Portland, OR, April, 2005), ACM Press, 899-908.

15

The Snippet Technique
• While out and mobile there may not be time to complete a thorough
diary entry
• The snippet technique: ―The in situ capture of snippets: bits of text,
audio, or pictures captured in a matter of seconds…then, at a
convenient time, participants access a website to review their
snippets and complete thorough, structured diary entries.‖
– Use existing mobile device and send text (SMS), pictures (MMS),
voicemail, or a combination
• Preliminary results
– Participants wrote more using snippet technique
– Snippet technique doesn’t seem to decrease accuracy or completeness
– Participants typically had a strong preference for one type of media;
some used two, no one used all three
• Text used most, then audio, then pictures (but some had
inadequate cameras)

Brandt, J., Weiss, N., & Klemmer, S. R. (2007). Txt4l8r: Lowering the Burden for Diary Studies Under
Mobile Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308.

16

Practical Considerations
Exact values depend on the situation and study goals.
• Number of participants
– 10 to 20 is typical within CHI
– But wide range: 5 to > 50
• BBC diary study on audio listening: 2,000 respondents
• Duration
– One or two weeks typical
– A day to a month or more
• Reducing drop-out rates and improving data quality
– Recruit ―appropriate‖ participants, interview them if possible, explain
requirements
– Withhold all or most pay until the study is completed
– Keep the time requirements for each diary entry short
– Check to make sure responses written when requested, not later
– Provide examples so participants know what’s expected
– Communicate with participants during study
• Personal contact is often more important than monetary incentives in
motivating participants
• Provide encouragement and feedback on responses (is it detailed enough?)
17

Techniques for Collecting Data
• Paper diaries
– Still a good idea in some situations (e.g., seniors without computers)
• Email
– Have participants email you their diary entries every day
– We’ve found it effective to email a reminder, which can also include questions or
rating scales
• Mobile phone-based applications/text messages
– Participants text their diary entry, or use a smartphone application
– ―Snippets‖ (SMS, MMS, voicemail) during day as aid for later recall (see Brandt
et al. 2007)
• Some now use Twitter as a snippet technique
• Voicemail
– Participant calls dedicated voicemail line rather than recording event
electronically or on paper
– Free-form or structured (can be IVR system)
– See Palen & Salzman (2002)

18

Techniques for Collecting Data (2)

• Excel spreadsheet (e.g., worksheets for each day of the week)
• Web-based techniques
– Google docs or spreadsheets & Google forms (https://docs.google.com)
• Free text or preset forms
– Blogs (e.g., http://wordpress.org/)
• One for each participant; can be password protected
– Design a diary as a forum (http://www.phpbb.com/)
• Used by Lichtner et al. (2009)
– Online surveys

19

So you want to run a diary study…

• What are you studying, and what’s the expected frequency?
– Do you need to collect information multiple times a day, or is once
sufficient?
– How long should you continue the study?
• How many reports of experiences or device use do you need from
each participant?
• Is reporting event driven?
– If not, is reporting at a set interval sufficient, or should you signal the
participant to respond?
• How will people report on their activities or experiences?
– Will they be at a desk or mobile?
– Will computers or smartphones be available?
– Will they have web access?
– Will a snippet technique be useful? (e.g., photo or text message during
day, details filled in at night)

20

Analysis

• In the applied CHI world, the studies are usually qualitative, and
simple descriptive statistics are typically sufficient
– Exceptions, of course: e.g., Czerwinski et al. (2004)
• 10 participants, analysis of task switching, difficulty, length, etc.
• Multivariate logistical regression with each user’s task switch entry
as input
• In academic psychology, multilevel models modified to handle
repeated-measures data are recommended (these are also called
hierarchical linear models)
– Lots of data are often collected; one study on emotion included 2,300
person-days of data

21

Example 2: Typical Data in HCI
• Keep a diary for two weeks of all your information needs when you’re
mobile. Snippet technique used: Text messages sent to email
address, which were then posted automatically to the web
Later, six questions about each snippet
answered on the web
1. Where were you?
2. What were you doing?
3. What was your information need?
4. I addressed the need (At the time,
Later, Not at all)
5. If you attempted to address the need,
how did you do so? If you didn’t
make an attempt, why didn’t you?
6. Could you have addressed your need
by looking at your personal data (e.g.,
email, calendar, web browsing history,
chat history, or other)

Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. A
Diary Study of Mobile Information Needs. In
Proceedings of CHI ’08 (Florence, Italy, April 2008),
ACM Press, 433-442.
22

Breakdown of Information Needs
Need Example % of Total # of # of Part.
Category Diary Category Reporting
Entries Entries Cat.
Trivia ―What did Bob Marley die of, and 18.5% 78 17
when?‖

Directions ―Directions to Sammy’s Pizza‖ 13.3% 56 17

Point of Interest ―Where is the nearest library or 12.4% 52 17
bookstore?‖

Friend Info ―Where are Sam and Trevor?‖ 7.6% 32 8

Shopping ―How much does the Pantech phone 7.1% 30 16
cost on the AT&T website?‖

Business Hours ―What time does the post office 6.9% 29 15
close?‖

Personal Item ―What is my insurance coverage for 6.4% 27 12
cat scans?‖

First 7 of 16 categories from Table 1 of Sohn et al., 2008.
23

Example 3: Electronic Chronic Pain Diary
Study to record pain by adolescents with juvenile idiopathic arthritis using a
Palm Tungsten W PDA (2004)

Stinson et al. e-Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents with
Arthritis. The Clinical Journal of Pain, 2006,• Volume 22 Diary Studies •
© 2011 Karis •
(3).
24

Example 4: Requirements for a New Device

It’s 1997, and you want to gather information so you can create a device for
work-related reading (remember: Sony Reader, 2006; Kindle, 2007)
– What kinds of reading do you need to support?
– How do people read as part of their jobs?
– What are the implications for design?

• There’s not much literature on this…what should you do?
• One solution: a diary study
– Pilot study, to create initial taxonomy of reading activities
– 15 participants, diverse occupations (pilot, surgeon, architect, optician)
– 5 days
– Logged daily document activities during work day
• Including: books, journals, electronic documents, PDAs, post-it
notes
• Used paper log forms, included estimates of duration
– Daily interviews with each participant (30 minutes to 3 hours)
• Interviewer expanded on description: type of document,
collaborative or not, where occurred, etc.
25

Work-Related Reading: Results
• Taxonomy of reading activities, 10 categories, top three:
– Skimming (subcategories: sorting, proofing, browsing)
– Reading to search/answer questions
– Reading for cross-referencing (to integrate information)
• Writing categories
– Creation
– Note-taking
– Annotation
– Form-filling
– Updating (calendars/schedules)
• Reading was accompanied by writing most of the time
• Great deal of reading and writing using multiple documents in
parallel
• About half the time participants used at least two display surfaces
concurrently
• Shared document viewing common
– Different clusters, with high sharers viewing documents with others over
50% of the time
26

Design of Digital Reading Devices
―The linear, continuous reading of single documents by people on their
own is an unrealistic characterisation of how people read in the course
of their daily work.‖

• ―Reading occurs more frequently in conjunction with writing than it
does in isolation….[reading devices should] support the marking or
writing of documents during the reading process.‖
• There is a ―need to consider how single display devices can support
the range of cross-document activities people carry out.‖
• Flexible search and navigation is important for some common
activities.
• Collaborative viewing needs to be supported.

Do the Kindle and Nook meet work-related needs?

Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading:
Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April
1998), ACM Press, 241-248.
27

Four Examples

• Example 1: Understanding the cellphone music experience
– The use of a device (mobile phone) for a specific purpose (listening to
music)
• Example 2: Information needs while mobile
– How existing devices and strategies are used to address a need
• Example 3: Electronic chronic pain diary
– Recording an experience (pain) over time
• Example 4: A diary study of work-related reading
– Understanding the requirements for a new device

28

Two Related Methods

• Experience Sampling Method (ESM)
– Used to obtain a random selection of everything a person does and
experiences in everyday life
– Can be considered a type of diary study
• Day-in-the-Life Observation
– Observation can provide a more accurate record than diary studies (and
diary studies are often more accurate than surveys)
– But very labor intensive

29

Experience Sampling Method (ESM)
• Used to obtain a random selection of everything a person does and experiences in
everyday life
– Participant signaled 8 or more times a day for 1-2 weeks; then questions/ratings presented
• Used in psychology for almost 40 years to understand the dynamics of mental health,
what and how often people think about things, happiness, moods, etc.

―ESM is a means for collecting information about both the context and
content of the daily life of individuals…the unique advantage of ESM
is its ability to capture daily life as it is directly perceived from one
moment to the next, affording an opportunity to examine fluctuations
in the stream of consciousness and the links between the external
context and the contents of the mind.‖

• An excellent method for determining
1. ―What people do all day, where, and with whom‖
2. ―How people report experiences, different moments in their lives, along a great
variety of dimensions‖

Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method:
Measuring the Quality of Everyday Life. Sage Publications, Inc, 2006
30

Day-in-the-Life Observation
• Observation can provide a more accurate record than surveys or
diary studies
– One study found that ―next day telephone interviews understate TV use
by 62% and magazine use by 43% when compared with observation;
diaries also understate these media consumption (13% and 29%,
respectively)‖ (Middletown Media Study 1, Ball State University)
• A Day in the Life: An Ethnographic Study of Media Consumption,
Middletown Media Study II, Online Publishers Association, July
2006. (Ball State University, Indiana)
– 350 adults in Muncie and Indianapolis IN were observed in Spring 2005
– Participants were observed on average nearly 13 hours – about 80% of
the waking day (over 5,000 hours in total)
– Every 15 seconds, their media consumption and life activities were
recorded (15 media and 17 life activities tracked; e.g., eating, child care)
– Observations were made at home, work, auto and other locations
• Example on next page

31

(He works from home.)

(No print usage all day.)

32

What is a Diary Study?
In a diary study, participants go about their
normal lives except that they report what they’ve
done or experienced (via questionnaire or freeform
using an electronic or paper diary) at some interval,
signal, or event, and they do this multiple times over
days or weeks.
• Participants go about their normal lives • A qualitative technique
• They report on some activity or experience – Can be combined with surveys, log
– Electronic analyses or other quantitative
– Paper techniques
• They report according to some schedule • Media can be used
– Interval – Photos, video, audio recordings
– Signal • Snippets can be recorded and
– Event expanded upon later
• They report repeatedly over days or weeks • Practical considerations
– Number, duration, reducing drop-out
rates
– Ten techniques for collecting data

33

References

• Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading:
Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April 1998),
ACM Press, 241-248.

• Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life as it is Lived. Annual Review of
Psychology, 2003, Volume 54, 579-616.

• Brandt, J., Weiss, N., & Klemmer, S. R. Txt4l8r: Lowering the Burden for Diary Studies Under Mobile
Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308.

• Brown, B. A. T., Sellen, A. J., & O’Hara, K. P. A Diary Study of Information Capture in Working Life.
Proceedings of CHI ‘2000 (The Hague, Amsterdam, April 2000). ACM Press, 438 – 445.

• Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary Studies.
Proceedings of CHI ‘05 (Portland, OR, April 2005), ACM Press, 899-908.

• Conducting Commercial Ethnography, GreenBook White Paper: http://www.greenbook.org/marketing-
research.cfm/conducting-commercial-ethnography [This paper includes much practical information].

• Czerwinski, M., Horvitz, E. & White, S. A Diary Study of Task Switching and Interruptions. Proceedings
of CHI ‘04 (Vienna, Austria, April 2004), ACM Press, 175-182.

• Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method: Measuring
the Quality of Everyday Life. Sage Publications, Inc, 2006.

34

References (2)
• Lewandowski, A.S., Palermo, T.M., Kirchner, H.L, & Drotar, D. Comparing Diary and Retrospective
Reports of Pain and Activity Restriction in Children and Adolescents with Chronic Pain Conditions. The
Clinical Journal of Pain, 2009, Volume 25 (4), 299 – 306.

• Lichtner, V., Kounkou, A. P., Dotan, A., Kooken, J. P., Maiden, N. A. M. An Online Forum as a User
Diary for Remote Workplace Evaluation of a Work-Integrated Learning System. Proceedings of CHI
’09 (Boston, MA, April 2009), ACM Press, 2955-2969.

• Palen, L., & Salzman, M. Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile
Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACM Press, 87 – 95.

• Papper, R., Holmes, M., & Popovich, M. (2004). Middletown Media Studies 1. Center for Media
Design, Ball State University. Available free at
http://cms.bsu.edu/Academics/CentersandInstitutes/CMD/InsightandResearch/Capabilities/ProjectGall
ery/MiddletownMediaStudies/MiddletownMediaStudiesI.aspx

• Papper, R., Holmes, M., Popovich, M., Bloxham, M. (2005). Middletown Media Studies 2 Report
Package: Media Day and Concurrent Media Exposure. Center for Media Design, Ball State University.
Available for $750 at http://www.bsu.edu/webapps2/cmdreports/product_select.asp?product_id=10

• Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. A Diary Study of Mobile Information Needs.
Proceedings of CHI ’08 (Florence, Italy, April 2008), ACM Press, 433-442.

• Stinson, J. N., Petroz, G. C., Tait, G., Feldman, B. M., Streiner, D., McGrath, P.J., & Stevens, B.J. e-
Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents with Arthritis. The Clinical
Journal of Pain, 2006, Volume 22 (3).

35

Thank you! Questions?
Comments?
Suggestions?

36

Diary Studies in HCI & Psychology

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Diary Studies in HCI & Psychology

Similar to Diary Studies in HCI & Psychology (20)

More from UXPA Boston

More from UXPA Boston (20)

Recently uploaded

Recently uploaded (20)

Diary Studies in HCI & Psychology