Talk at the International Symposium on Companion Technology 2015, Ulm University, on creating digital companions

1. ICST 2015
What Companions
Know and Remember
Maria Wolters
@mariawolters
School of Informatics
School of Philosophy, Psychology, and
Language Science
University of Edinburgh
ForgetIT Team: Robert Logie, Elaine Niven (Edinburgh), Heiko Maus, Sven Schwarz (DFKI)
CADENCE: Johanna Moore, Myroslava Dzikovska, Jonathan Kilgour, Sarah MacPherson (Edinburgh), Marek Grzesz, Jesse Hoey (Waterloo)

2. Companions need to focus on what matters,
not on what they can store.
If companions are to learn what matters,
they need to be good students.

3. Example Case Study
Leafy Green General Practice
gives all their patients a health care
companion app
that provides electronic record access, and
facilitates communication with the practice.

4. Old Style Knowing and Remembering
❖ Dawn Jones is Patient ID SCO4711. She has Type II
diabetes.
❖ Dawn last saw Dr Smith three months ago, on July 23 at
2.15pm; as a result, her medications were changed

5. Is That Enough?
❖ The caller is Dawn Jones, Patient ID SCO4711. She has Type II diabetes.
❖ Dawn is quite chatty. It’s hard to get a straight Yes or
No out of her.
❖ Dawn last saw Dr Smith three months ago, on July 23 at 2.15pm; as a result, her
medications were changed
❖ Dawn normally goes every quarter for a check up.
Almost all of her appointments are with Dr Miller,
whom she trusts and likes.

6. The Additional Edge
❖ Know:
❖ how people talk, not just what they are likely to talk
about
❖ to what extent people can and will adapt to computer
voice interfaces
❖ Remember: remember the gist, forget the details

7. Companions and Knowing
❖ Interaction style (MATCH project)
❖ How to Help (CADENCE project)
1. Wolters MK, Georgila K, MacPherson S, Moore J. Being Old Doesn’t Mean Acting Old: Older Users' Interaction with Spoken Dialogue Systems. ACM Trans Access Comput. 2009;2(1):1–39.
2. Maria K. Wolters, Jonathan Kilgour, Sarah E. MacPherson, Myroslava Dzikovska, and Johanna D. Moore. 2015. The CADENCE Corpus: A New Resource for Inclusive Voice Interface Design.
In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3963-3966. http://doi.acm.org/10.1145/2702123.2702372

8. MATCH: Appointment Scheduling
❖ Older (50+) and younger (18-30) participants schedule
one appointment each using 9 different voice interfaces,
fill in long usability questionnaire, and then recall
agreed health care professional, date, time, and location
❖ Voice interfaces vary by number of options presented (1,
2, 4) and confirmation (no, implicit, explicit)
❖ Wizard-of-Oz with human speech recognition and
natural language understanding

9. Interaction Analysis
❖ All interactions were transcribed orthographically
❖ Semi-automatic dialogue act annotation
❖ dialogue act = speech act (request, confirm, …) +
modifier (half day, time, person)
❖ Clustering of behaviour based on statistics (frequency of
dialogue acts, turn length, type/token ratio …)

10. Differences in Interaction Style
❖ Cluster 1 - Factual:
very few social dialogue acts, short utterances, to the point.
❖ Cluster 2 - Social:
more polite, fairly chatty, give reasons why they can’t make
certain dates, …
❖ Most younger people, and 1/3 of the older people were factual,
the remaining older people were more social.
❖ Users with a social interaction style did not change it over the
course of 9 dialogues

11. Open Questions from MATCH
❖ Why do Social people fail to adapt?
❖ no adverse effects (perfect speech recognition)
❖ social cognition (can’t pick up cues)
❖ don’t want to
❖ What happens if we force Social people to become
Factual?

12. CADENCE: Intelligent Cognitive Assistant
❖ Idea: The ICA watches people perform tasks and
provides help if needed
❖ requires a model of users’ abilities, a task model,
plans for achieving goals
❖ Origin: COACH Handwashing assistant (Mihailidis /
Hoey and collaborators)

13. CADENCE Task
❖ Participants with (n=10) and without (n=44) cognitive
impairment use a very simple tablet app to look up
details of appointments
❖ A simulated ICA (Wizard of Oz) is available to help
❖ system versus user initiative
❖ neutral versus encouraging

14. Wizard of Oz Dialogue Corpus
System: Have you found what you are looking for?
DF09: Yes, thank you.
System: You have finished the task.
DF09: (hey)
System: Please write down the answer.
DF09: Well I am thank you, just be quiet.

15. Key Findings
❖ Deciding when to intervene, how long to leave it, is
tricky
❖ A human assistant would generally be preferred, but …
❖ … one doesn’t have to be polite to a computer

16. Open Questions from CADENCE
❖ How does a dialogue system cope with people thinking
aloud?
❖ When should the system intervene?
❖ How can we make sure we „teach people to fish“?

17. Companions and Remembering
❖ We can’t preserve everything - humans cope because
they abstract at each level of neural and cognitive
information processing
❖ But computers / companions remember everything?
http://www.redorbit.com/reference/floppy_disk/

18. Preservation is hard.
http://www.reed.edu/reed_magazine/june2013/articles/eliot_circular/gilgamesh.html
http://www.redorbit.com/reference/floppy_disk/

19. The ForgetIT Approach
❖ Preservation Value:
how much effort should we put into archiving?
❖ Memory Buoyancy:
what do we need right now? what happened recently?
❖ EU FP7 IP, led by L3S (Hanover)

20. Case Study: Photo Preservation
❖ DFKI Semantic Desktop technology (via PIMO5) system
allows rich contextualization and rich set of evidences

21. Open Questions
❖ How do we find out what people value
and how much they want to invest, when
they still insist on managing their photos
using the file manager?
(Source: ForgetIT Photo Preservation
Survey of 1300+ people world wide)
❖ How can we support Curators? How
those who prefer to „file and forget“?
❖ What is the difference between useful
and creepy?
http://forums.androidcentral.com/ambassador-guides-tips-how-s/300182-guide-google-now.html
1. Wolters MK, Niven E, Runardotter M, Gallo F, Maus H, Logie RH. Personal Photo Preservation for the Smartphone Generation. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in
Computing Systems - CHI EA ’15 [Internet]. New York, New York, USA: ACM Press; 2015 [cited 2015 May 6]. p. 1549–54. Available from: http://dl.acm.org/citation.cfm?id=2702613.2732793

22. Conclusion
Companions need to focus on what matters,
not on what they can store.
If companions are to learn what matters,
they need to be good students.
maria.wolters@ed.ac.uk @mariawolters