Presented by Jon Temple and Dabby Phipps. Chatbots have emerged as a powerful new technology in our daily lives. Sometimes they attempt to answer our questions or provide advice, while other times they ask screening questions before handing off to another human. Despite their ubiquity, the capabilities of chatbots are often misunderstood with many people believing the chatbot can generate unique answers or solve problems on its own. In reality, the answers chatbots provide are only as good as the human thought and writing that goes into creating the cognitive intents, which form the corpus of a chatbot’s knowledge base. In the following, we will describe the complex process of authoring cognitive intents, such as: what is an intent; how to select intents based on user feedback and metrics; how to improve confidence matching; and how UX research can iteratively improve intent performance. These concepts will be tied together in a chatbot demonstration.

1. CIO Design, IBM
Putting words in the
mouths of chatbots:
Designing cognitive
intents
Dabby Phipps & Jon G. Temple, Ph.D.
| IBM, CIO Design

2. UXPA 2021
Meet the Presenters
Dabby Phipps
Senior UX Designer/
Researcher
Jon G. Temple
IBM Design Principal

3. UXPA 2021
Meet ELIZA:
The first chatterbot
• ELIZA - natural language conversational
program (1966)
• Simulates Rogerian psychotherapist
• ELIZA takes the user input, looks for
keywords and synonyms, creates a
response - often a transformation of the
user’s input2
• ELIZA is not intelligent and can’t learn.
• May not make sense or be
grammatically correct
1https://www.masswerk.at/elizabot/

4. UXPA 2021
Chatbots Today
• Software programs - leverage artificial
narrow intelligence (ANI)
• Answer user questions or perform
certain tasks within a narrow scope
• Natural language processing (NLP) -
utterance
• For chatbots, the key is natural
language understanding (NLU) - to
understand the user’s “intent” or what
they’re trying to achieve3
https://www.bold360.com/features/chat-and-messaging

5. UXPA 2021
The Pros and Cons of Chatbots
Pros
• Automate simple conversations
• Available 24x7
• Multiple customers simultaneously
• Great for small businesses
• User in control
• NLP conversational style of interaction
Cons
• Hard to support broad domains
• Poor performance leads to decreased user
confidence
• Takes time and effort to build the knowledge
corpus
• May require natural language when not needed

6. UXPA 2021
What Users Really Want
• Businesses may want an AI with
personality or conversational
• But users just want answers or
assistance - efficiently as possible
• Unrelated to chatbots: people using an
IT help system rated the same system
different based solely on outcome
(NPS: 31 v. -32)
• If chatbots can provide the appropriate
answers faster, they will be preferred

7. UXPA 2021
Chatbots: They are
what you make them

8. UXPA 2021
Chatbot Myth
Watson on Jeopardy (2011)5

9. UXPA 2021
Where does the chatbot's knowledge come from?
• Cognitive theories of mind -
“homunculus problem" as a straw man
• Cognitive processes - little people
inside your head
• Recursion - “what’s inside the head of
the homunculus?”
• Yet - this is true for chatbots:
knowledge comes from conversation
designers (aka people)
• Chatbots aren’t reading textbooks,
forums and writing own answers
• Machine learning - (if used) to improve
the query

10. UXPA 2021
Chatbot knowledge
It comes from people!*
*Soylent Green (1973)

11. UXPA 2021
Role of Conversation Designer in Training Chatbots
• Conversation designers need to
determine:
• Chatbot’s purpose
• Type of interaction (i.e., playful,
neutral, professional)
• The user intents within the domain
• Craft concise responses - engage the
user and address their need

12. UXPA 2021
What type of chatbot
do you want?

13. UXPA 2021
Questions to ask yourself before you build a chatbot
Moorjani (2017)6:
• What is the user outcome we want to drive?
• What is the business outcome we want to drive?
• Is the conversational UI the best medium for your
outcome?
• Is the “cost” for using the chatbot less than the
perceived value?
“This is not the chatbot you are looking for”

14. UXPA 2021
Determine the Chatbot’s Purpose
• End users
• Solve user problems
• Business goals
• Orgs’s vision
• Specific job or jobs for the chatbot

15. UXPA 2021
Tone of Voice
• Dimension of formality:
• Informal & friendly: "Hi there, how
can I help you?”
• More formal, still friendly: "Good
morning. How may i help you today?
Type your question in the box below
to get started."
• Formal, not very friendly: "This
service is designed to answer your
question in the box below.”
• Consider: Your brand, target audience,
tone of other content sources used by
the solution, nature of the specific
questions and responses

16. UXPA 2021
Designing the
cognitive intents

17. UXPA 2021
Chatbot Components
• Why are users
interacting with your
chatbot? What tasks are
they trying to complete?
What goals are they
trying to achieve?
• Example:
• The user wants to
reset their email
password.
Intents Utterances
Entities Dialog
• What are the nouns the
user might want to take
action on? What are
synonyms for those
nouns?
• A minimum of five user
examples should be
created for each intent,
but 10-15 is better.
• Examples:
• Entity : synonyms
• Email : mail, outlook,
notes, verse, e-mail,
apple mail
• For each intent, create a
list of user examples or
how your users might
state their intentions.
• Examples:
• I need to reset my
@email password.
• I want to change my
@email password.
• How do I change my
@email password?
• Error says to reset my
@email password.
• Matches the user’s
utterance (user’s stated
goal) with a dialog
response (chatbot’s
response).
• Example:
• [user] How do I reset
my email password?
• [chatbot] Outlook
now uses your w3id
and password to sign
in. To change your w3
password ….

18. UXPA 2021
Chatbot Flow Overview
• Intents - user’s intention or what they
want to do (verb)
• Entities - keywords modify the intent
(noun) - the what
• Dialog - response generated by the
chatbot based on its understanding of
the user’s input using intents and
entities
User input
(utterance)
Bot assesses
user intent
Bot identifies
best solution
Bot provides
response (dialog)

19. UXPA 2021
Intent Creation Process
Prioritize backlog
Identify end user
phrases for the issue
Determine entities
Talk to SME’s &
review support materials
Generate utterances
Create flow charts
Write dialog
Work with developers to
map out API integrations
Add intents, entities, and
dialog to chatbot system
Manual and automated
testing

20. UXPA 2021
Example: Researching a new intent
Identify top call drivers
from support tickets
“We get a lot of calls about
resetting two types of
passwords. I have a script that I
read to callers”.
Interview human support
agents
Input from SME’s Search data
User feedback New content/products
Data science analysis
“We have an API that can be
used to reset the intranet
password easily; but to reset your
email password, you need the
vendor to provide us new
services”.
“I have too many different kinds
of passwords to keep track of
and I often forget to change them
on time”. — Ima Uzer
Top call drivers for August:
1. Email
2.Reset intranet password
3. VPN…
Top search queries:
1. Outlook, HCL Notes, Apple mail
2. Checkpoint
3.Intranet password change,
reset…
Added 200K Macs to user base.
New support content written for:
Mac account access, Filevault,
Screentime, Apple ID…
Data science report:
Patterns found in multiple sources -
surveys, searches: password reset
complexity in multiple contexts,
including…
Reset password

21. UXPA 2021
Exercise

22. UXPA 2021
Instructions: Design an intent
• Use the index cards for everyone (use your own pen)
• User’s intent: Ordering a pizza with toppings (#orderPizza)
• Generate up to 3 utterances for that intent
• Define toppings as an entity (@toppings)
• Create some synonyms for that entity

23. UXPA 2021
Utterances for #orderPizza
• i want a pizza
• i'd like a pizza
• give me a pie
• please give me a pizza
• order a pizza pie

24. UXPA 2021
Entity
@toppings:
[value: synonyms]
meat: pepperoni, sausage, ham, anchovies, chicken
veggies: black olives, green olives, onions, bell peppers, jalapenos, artichoke
cheese: mozzarella, white cheddar, asiago, three cheese blend

25. UXPA 2021
Dialog
- Excellent choice! Pizza is our specialty. Do you want toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Sounds like you want a [variable] pizza. Do you want any other toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Let's make sure I have this right. You want a pizza with [selected options]. Is that correct?
Yes: Great! What size do you want? [@size]
No: Oops! Let's go over the toppings again. What toppings do you want on your pizza?
- Ok, so you want a large. Is that correct? Yes | No
Yes: Would you like anything else? We also have appetizers, desserts, and drinks.
No: Sorry, I must have mozarella in my ears. What size pizza do you want?

26. UXPA 2021
Dialog Flow

27. UXPA 2021
Best practices for
authoring

28. UXPA 2021
What Makes a Chatbot Good?
• Objectives as a conversation designer:
• Easy
• Intuitive
• Intentional language choices
• Feel more like a conversation with
another human
• User walks away feeling satisfied

29. UXPA 2021
Erika Hall's Eight Principles for Conversation(al) Design7
Cooperative
Turn-based
Goal-oriented Context-aware Quick & clear
Truthful Polite Error-tolerant

30. UXPA 2021
Writing Tips
• Active voice
• Use contractions
• Keep sentences short
• Use formatting to improve readability
(bulleted lists, paragraphs, commas,
dashes)
• Check for misspelled words and other
grammar issues
• Avoid “Yes" and "No" answers
• Incorporate part of the user's question
in response
• Be succinct and accurate answers (link
out to website for longer responses)

31. UXPA 2021
Chit Chat & Small Talk
• Exclamations (yes, no, okay, sure, oh)
• Greetings
• Personal
• Topic-based (e.g. Weather, geography)
• Threats, insults, and pleas for help
• Explicit and offensive language
• Compliments
• Complaints
• Repair (I don't understand, what do you
mean, why)

32. UXPA 2021
Response Types
• Substantive - answers the question (q:
what's your favorite movie? a: My
favorite movie is Bladerunner b/c …)
• Deflecting - responds to the question
without actually answering (q: what's
the weather like in london? a: humans
must be fascinated by the weather b/c
they're always asking me about it!)
• Redirecting - redirects the user back to
the core areas (q: what do you think of
religion? a: religion is not my area of
expertise, but I do know about ABC
banking cards!)

33. UXPA 2021
Demo:
PoohBot

34. UXPA 2021
Intent

35. UXPA 2021
Entities

36. UXPA 2021
Synonyms

37. UXPA 2021
Dialog

38. UXPA 2021
Summary

39. UXPA 2021
Take aways
• Chatbots provide a unique conversational style
• Successful in many contexts and industries
• Most chatbot knowledge is created by people
• Role of machine learning is more nuanced
• Conversational designers must:
• Understand the purpose, target audience, branding,
tone, users’ intents, and more
• Conduct research, meet with SMEs and architects
• Write cognitive intents and associated components
(entities, utterances, dialogs)
Build an IT support chatbot by
using IBM Watson Assistant
Building a Chatbot to Order a Pizza
Getting started

40. UXPA 2021
Parting words:
Meeting expectations

41. UXPA 2021
Failure to meet user
expectations
• New chatbots lack a full blown knowledge
corpus on launch
• Build out the knowledge corpus to answer direct
questions (e.g., point to a long video for the
answer)
• In 2018, Gartner expected that 40% of first-
generation chatbot/virtual assistant applications
launched in 2018 would be abandoned by 20208
• A chatbot that fails to live up to user
expectations will quickly lose adoption
(abandonment)

42. UXPA 2021
Cognitive Intercept: Overview
• With cognitive intercept9, embed the
chatbot technology into an existing user
experience (single point of entry)
• Monitors in the background, assesses the
user’s input
• Intercepts if it has high confidence
• Deploy a chatbot that covers a narrow
scope of topics without frustrating the
user
• Continue to gather user input to increase
its breadth and depth of coverage, as well
as improving accuracy
User initiates
action
High
confidence
match?
Continue with
standard
experience
Intercept
Cognitive layer
screens input
in background
Yes No

43. UXPA 2021
Cognitive Intercept: Real World Example - No Match
• Where cognitive intercept has been used:
• Web search (first integration point)
• Voice Response Unit (VRU)
• Online chat
• Users enter their input in a standard
search field
• System evaluates the input as both
traditional search terms and using NLP
• If a high-confidence match is not found,
the “chatbot” remains silent
• Running in the background effectively
manages expectations around what
chatbots can or cannot deliver

44. UXPA 2021
Cognitive Intercept: Real World Example - Match
• If the chatbot is trained on a particular
user intent and finds a high-confidence
match, then the chatbot “intercepts” and
displays associated dialog
• Full conversations using NLP are avoided
by using close-ended prompts to
represent user options
Cognitive intercept benefits:
• Manages user expectations
• Provides a single point of entry
• Reduces errors associated with NLP
• Permits a graceful ramp up of the
chatbot’s knowledge corpus

45. UXPA 2021
References
1 https://www.masswerk.at/elizabot/
2 https://www.chatbotpack.com/how-eliza-chatbot-works/
3 https://www.bold360.com/learn/what-is-natural-language-processing
4 https://www.smallbizgenius.net/by-the-numbers/chatbot-statistics/#gref
5 https://www.csmonitor.com/Science/2011/0214/IBM-s-Watson-Can-a-computer-outsmart-a-Jeopardy!-braniac
6 https://chatbotsmagazine.com/to-build-a-successful-chatbot-ask-these-5-questions-b7fe3776c74c
7 https://marvelapp.com/blog/principles-of-conversational-design/
8 https://www.gartner.com/smarterwithgartner/2-megatrends-dominate-the-gartner-hype-cycle-for-artificial-
intelligence-2020/
9 Temple, J.G., Elie, C., Schenkewitz, L., Nezbedova, P., Phipps, D., Roberson, K., Scherpa, J., & Malatesta, M. (June, 2019.
Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery. Paper presented at the UXPA
International Conference, Scottsdale, AZ.

47. UXPA 2021
Principle 1: Cooperative
• Back and forth flow
• Mimic how people have conversations -
pattern understood by the user

48. UXPA 2021
Principle 2: Goal-oriented
• Main purpose - help users reach their
goals
• Chatbot should reflect this the user’s
intentions
• Help your customers achieve their goals
helps your business achieve its goals

49. UXPA 2021
Principle 3: Context-aware
• Don’t rely on overly automated
messages
• Can frustrate users
• Does not feel like a natural conversation
• Builds confidence in the user
User: What are your holiday hours?
Chatbot: Our normal hours of operation
are …..

50. UXPA 2021
Principle 4: Quick and clear
• Conversational, but as efficient as
possible
• Use plain language and keep it simple
• Offer logical next steps
• Keep the conversation moving
• Stay on point
User: How many flavors of ice cream do
you have?
Chatbot: We have a huge selection of ice
cream flavors so we’re sure we have
something for everyone.

51. UXPA 2021
Principle 5: Turn-based
• Replies - succinct and to the point. No
monologues!
• Do not send a string of responses
• Best conversations are back and forth
dialogs
• Be clear if response from the user is
needed
• Validate user input to avoid errors

52. UXPA 2021
Principle 6: Truthful
• Validate are accurate and truthful
• Plan for dialog maintenance. Answers
change over time - may need to be
updated
• Verify it is behaving as expected
• Responses should align with user
expectations

53. UXPA 2021
Principle 7: Polite
• Don’t be rude
• Be respectful of your user’s time
• Users come to a chatbot with a goal.
Stay on topic
• Don’t try to push your own agenda if not
aligned
• Once the chatbot has solved the user’s
issue, offer to help with other issues,
next steps

54. UXPA 2021
Principle 8: Error-tolerant
• To errr is human, to forgive divine.
• Teach chatbot common spelling
alternatives or errors
• Disambiguate the request as needed
User: How do I reset my pwd?
Chatbot: To reset your password…

55. UXPA 2021
Quick Facts4
• 1.4 billion people use chatbots on a
fairly regular basis (Acquire)
• $5 billion would be invested in chatbots
by 2021. (Chatbots Magazine)
• 64% users cite round-the-clock support
biggest benefit (Drift)
• If chatbot can't solve a problem, 79%
want a real person

56. UXPA 2021
Planning Dialog Flow
• Ok! Now we’re ready to start thinking
about the chatbot’s dialog!
• Before you start writing the exact
words, think about how the
conversation with your users will flow
for each intent you’ve identified.
• The mapping will help you understand if
there are redundant dialog nodes across
scenarios and how your scenarios might
interact.
• <-disambiguation->

57. UXPA 2021
Why the Explosive Growth?
• Primarily the emergence of two
technologies:
• Messenger applications
• Artificial Intelligence / Machine
Learning