This document discusses understanding user satisfaction with intelligent assistants. It proposes three main types of usage scenarios: controlling devices, web search, and structured dialogue search. It describes a user study designed to measure user satisfaction across these scenarios. The study examines key factors like effort that determine satisfaction for different scenarios. It also aims to characterize "good abandonment" in web search and examine how query-level satisfaction relates to overall satisfaction in dialogue search. The document concludes that effort is key to satisfaction across scenarios and that abandonment and session context are important to consider.

1. Understanding User Satisfaction
with Intelligent Assistants
Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah,
Aidan C. Crook, Imed Zitouni, Tasos Anastasakos
Eindhoven University of Technology
Pennsylvania State University
University of Massachusetts Amherst
Microsoft
CHIIR’16, Chapel Hill, NC, USA

2. Q1: how is the weather in Chicago
Q2: how is it this weekend
Q3: find me hotels
Q4: which one of these is the cheapest
Q5: which one of these has at least 4 stars
Q6: find me directions from the Chicago airport to
number one
User’s dialogue
with Cortana:
Task is “Finding
a hotel in
Chicago”

3. Q1: find me a pharmacy nearby
Q2: which of these is highly rated
Q3: show more information about number 2
Q4: how long will it take me to get there
Q5: Thanks
User’s dialogue
with Cortana:
Task is “Finding
a pharmacy”

4. Research Questions
• RQ1: What are characteristic types of scenarios of use?

5. Controlling Device
• Call a person
• Send a text message
• Check on-device calendar
• Open an application
• Turn on/off wi-fi
• Play music

7. Knowledge Pane
Image Answer

8. Knowledge Pane
Image Answer Image Answer
Organic Results

9. Knowledge Pane
Image Answer Image Answer
Location Answer
Organic Results

10. User:
“Do I need
to have a
jacket
tomorrow?”
Search Dialogue

11. User:
“Do I need
to have a
jacket
tomorrow?”
Cortana: “You
could probably
go without one.
The forecast
shows …”
Search Dialogue

12. Cortana:
“Here are ten
restaurants
near you”
User:
“show
restaurant
s near me”
Search Dialogue

13. Cortana:
“Here are ten
restaurants
near you”
Cortana:
“Here are ten
restaurants near
you that have
good reviews”
User:
“show
restaurant
s near me”
User:
“show the
best
restaurants
near me ”
Search Dialogue

14. Cortana:
“Here are ten
restaurants
near you”
Cortana:
“Here are ten
restaurants near
you that have
good reviews”
Cortana:
“Getting you
direction to the
Mayuri Indian
Cuisine”
User:
“show
restaurant
s near me”
User:
“show the
best
restaurants
near me ”
User:
“show
directions to
the second
one”
Search Dialogue

15. Research Questions
• RQ1: What are characteristic types of scenarios of use?
• RQ2: How can we measure different aspects of user satisfaction?
• RQ3: What are key factors determining user satisfaction for the
different scenarios?
• RQ4: How to characterize abandonment in the web search
scenario?
• RQ5: How does query-level satisfaction relate to overall user
satisfaction for the search dialogue scenario?

16. Research Questions
• RQ1: What are characteristic types of scenarios of use?
• RQ2: How can we measure different aspects of user satisfaction?
• RQ3: What are key factors determining user satisfaction for the
different scenarios?
• RQ4: How to characterize abandonment in the web search
scenario?
• RQ5: How does query-level satisfaction relate to overall user
satisfaction for the search dialogue scenario?
USERSTUDY

17. User Study Participants
55%
45%
LANGUAGE
English Other
• 60 Participants
• 25.53 +/- 5.42 years

18. User Study Participants
75%
25%
GENDER
Male Female
55%
45%
LANGUAGE
English Other
• 60 Participants
• 25.53 +/- 5.42 years

19. User Study Participants
75%
25%
GENDER
Male Female
55%
45%
LANGUAGE
English Other
82%
8%
2%
8%
EDUCATION
Computer Science Electrical Engineering
Mathematics Other
• 60 Participants
• 25.53 +/- 5.42 years

20. User Study Design
• Video Instructions (same for all participants)
• Tasks are realistic – mined from Cortana logs:
o Control type of tasks
o Queries where users don’t click
o Search dialogue tasks – mostly localization type of queries

21. Find out what
is the hair
color of your
favorite
celebrity.

22. You are planning a
vacation. Pick a place.
Check if the weather is
good enough for the
period you are planning
the vacation. Find a hotel
that suits you. Find the
driving directions to this
place.

23. You are planning a
vacation. Pick a place.
Check if the weather is
good enough for the
period you are planning
the vacation. Find a hotel
that suits you. Find the
driving directions to this
place.

24. Questionnaire: Controlling Device
• Were you able to complete the task?
o Yes/No
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• How well did Cortana recognize what you said?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale

25. Questionnaire: Controlling Device
• Were you able to complete the task?
o Yes/No
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• How well did Cortana recognize what you said?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale
5 Tasks
20 Minutes

26. Questionnaire: Good Abandonment
• Were you able to complete the task?
o Yes/No
• Where did you find the answer?
o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?
o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale

27. Questionnaire: Good Abandonment
• Were you able to complete the task?
o Yes/No
• Where did you find the answer?
o Answer Box, Image, SERP, Visited Website
• Which query led you to finding the answer?
o First, Second, Third, >= Fourth
• How satisfied are you with your experience in this task?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale
5 Tasks
20 Minutes

28. Questionnaire: Search Dialogue
• Were you able to complete the task?
o Yes/No
• How satisfied are you with your experience in this task?
o If the task has sub-tasks participants indicate their graded
satisfaction e.g.
o a. How satisfied are you with your experience in finding a hotel?
o b. How satisfied are you with your experience in finding directions?
• How well did Cortana recognize what you said?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale

29. Questionnaire: Search Dialogue
• Were you able to complete the task?
o Yes/No
• How satisfied are you with your experience in this task?
o If the task has sub-tasks participants indicate their graded
satisfaction e.g.
o a. How satisfied are you with your experience in finding a hotel?
o b. How satisfied are you with your experience in finding directions?
• How well did Cortana recognize what you said?
o 5-point Likert scale
• Did you put in a lot of effort to complete the task?
o 5-point Likert scale
8 Tasks: 1 simple,
4 with 2 subtasks,
3 with 3 subtasks
30 Minutes

30. Search Dialog Dataset
• 540 tasks that incorporated
• 2, 040 queries, of which 1, 969 were unique
• the average query-length is 7.07
• The simple task generated 130 queries in total
• Tasks with 2 context switches generated 685 queries
• Tasks with 3 context switches generated 1, 355 queries

31. Factors Determining Satisfaction
RQ3: What are key factors determining user satisfaction
for the different scenarios?

32. 0
1
2
3
4
5
6
Across
Scenarious
Device
Control
Web
Search
Structured
Dialog
5
0
1
2
3
4
5
6
Across
Scenarious
Device
Control
Web
Search
Structured
Dialog
5
SatisfactionLevel
Efforts
Results Over Scenarios
Mean of Satisfaction

33. Results `Good Abandonment’
RQ4: How to characterize abandonment in the web
search scenario?

34. 0
1
2
3
4
5
6
First Query Second
Query
Third
Query
>= Fourth
Quey
0
1
2
3
4
5
6
Answer
Box
Image SERP Visited
WebSite
5
SatisfactionLevel
Results `Good Abandonment’
Mean of Satisfaction

35. Search Dialogue Satisfaction
RQ5: How does query-level satisfaction relate to overall
user satisfaction for the structured search dialogue
scenario?

36. Cortana:
“Here are ten
restaurants
near you”
Cortana:
“Here are ten
restaurants near
you that have
good reviews”
Cortana:
“Getting you
direction to the
Mayuri Indian
Cuisine”
User:
“show
restaurant
s near me”
User:
“show the
best
restaurants
near me ”
User:
“show
directions to
the second
one”
SAT? SAT? SAT?
SAT? SAT? SAT?
Overall
SAT?
?

37. Search Dialogue Satisfaction
RQ5: How does query-level satisfaction relate to overall
user satisfaction for the structured search dialogue
scenario?

38. Satisfaction Over Different Tasks
Satisfaction Level
Weather Task
NumberofAnswers
1 2 3 4 5

39. Satisfaction Over Different Tasks
Satisfaction Level
Weather Task Mission Task (2 sub-tasks)
NumberofAnswers
1 2 3 4 5

40. Satisfaction Over Different Tasks
Satisfaction Level
Weather Task Mission Task (2 sub-tasks)
Mission Task (3 sub-tasks)
NumberofAnswers
1 2 3 4 5

41. Q1: what do you have medicine for the stomach ache
Q2: stomach ache medicine over the counter
Q3: show me the nearest pharmacy
Q4: more information on the second one
Q5: do they have a stool softener
Q6: does Fred Meyer have stool softeners
General Search
Search Dialog
Combination
of scenarios
User’s dialogue with Cortana related to the ‘stomach ache’ problem

42. Conclusions (1)
• RQ1: What are characteristic types of scenarios of use?
• We proposed three main types of scenarios
• RQ2: How can we measure different aspects of user
satisfaction?
• We designed a series of user studies tailored to the three
scenarios
• RQ3: What are key factors determining user satisfaction for
the different scenarios?
• Effort is a key component of user satisfaction across the
different intelligent assistants scenarios

43. Conclusions (2)
• RQ4: How to characterize abandonment in the web search
scenario?
• We concluded that to measure good abandonment we need
to investigate the other forms of interaction signals that are
not based on clicks or reformulation
• RQ5: How does query-level satisfaction relate to overall user
satisfaction for the search dialogue scenario?
• We looked at user satisfaction as ‘a user journey towards an
information goal where each step is important,’ and showed
the importance of session context

44. Questions?
• We proposed three main types of scenarios of use
• We designed a series of user studies tailored to the three scenarios
• Effort is a key component of user satisfaction across the different
intelligent assistants scenarios
• We concluded that to measure good abandonment we need to investigate
the other forms of interaction signals that are not based on clicks or
reformulation
• We looked at user satisfaction as ‘a user journey towards an information
goal where each step is important,’ and showed the importance of session
context on user satisfaction
Questions?