World IA Day Bristol 2017 Linguistic User Interfaces like Alexa and Google Assistant look set to become widespread in homes and perhaps other environments too. What does this mean for Information Architects and User Experience professionals used to working with Graphic User Interfaces? What new skills will we need to develop to make conversational interactions both usable and pleasurable?

1. Talking to machines, listening to humans
Gordon Plant, WIAD 2017
@gordonplant

2. Why talk to machines?
https://www.theguardian.com/culture/gallery/2015/jan/08/the-top-20-artificial-intelligence-films-in-pictures#img-7

3. If I could talk to the animals machines…
“if I could talk to the animals, just imagine it
Chatting to a chimp in chimpanzee
Imagine talking to a tiger, chatting to a cheetah
What a neat achievement that would be”
Dr Doolittle (1967)

4. Humans anthropomorphise everything
• “54 per cent of people have verbally assaulted their computers,
while 40 per cent have resorted to physical violence”
http://www.telegraph.co.uk/technology/5086091/Computer-rage-affects-more-than-half-of-Britons.html

5. Why talk to machines?
• Talking doesn’t interrupt other tasks
• Attention can remain focussed elsewhere
• Machines are ‘effort multipliers’
• Effort / Reward ratio is improved as effort is reduced
• Some intentions are hard to express via a GUI

6. Who’s talking already?
Survey of 1250 people by Creative Strategies, October 2016
• 22% use a voice assistant four to six times a week
• 33% think it is more convenient to talk than type
• 27% would prefer to act with bots in the car
• 26% would prefer to act with bots in the home
http://creativestrategies.com/no-bots-please-europeans/

7. How does a LUI work?

8. Utterance, intent, invocation
• Utterance
• The spoken words
• Intent
• A recognisable intent extracted from parsing the utterance
• Invocation phrase
• The phrase that launches the relevant ‘skill’
A ‘skill’ is a
bit like an
App on your
phone

9. Utterance strcuture
Alexa, tell HAL to open the pod bay doors
Utterance
IntentRequestWake
word
Connecting
words
Invocation
name

10. Let’s turn on
the heating

11. command boiler
Voice input
Let’s turn on the heating
Skill
Touch input App

12. Let’s turn on the heating
Alexa, tell Hive to turn the heating on
Utterance
IntentConnecting
words
Wake
word
Invocation
name
Request

13. Alternative request words
• Talk to
• Open
• Launch
• Start
• Resume
• Run
• Load
• Begin

14. Let’s turn on the heating
Alexa, tell Hive to turn the heating on
Alexa, tell Hive to turn the heating on to 20 degrees
Alexa, tell Hive to put the heating on to 20 degrees
Alexa, tell Hive to boost my heating
Tell <invocation name> <connecting word> <some action>

15. Let’s turn on the heating
Alexa, tell Hive to turn the heating on
Alexa, turn the heating on
Alexa, tell Hive to turn the heating on to 20 degrees
Alexa, Hive to 20 degrees
Alexa, tell Hive to put the heating on to 20 degrees
Alexa, put the heating on for 20 degrees
Alexa, tell Hive to boost my heating
Alexa, tell Hive to put the heating on for 1 hour
Tell <invocation name> <connecting word> <some action>
For every phrase
that works, there
are many similar
ones that don’t

16. GUI vs LUI
Wake Invoke App Navigate app Tap button Confirmation
Alexa tell Hive to turn the heating on “ok”
<invocation name> <connecting word> <some action>

17. Model / Modes / Actions
are discoverable at launch
Model / Modes / Actions
only discoverable by trial and error
User only needs to remember
the name or location of the app
User needs to remember
complete, structured sentences
GUI LUI

18. Heating on
0 0.5 1.0
1.0
0.5
Alexa, tell Hive to turn
the heating on
Alexa, ask Hive heating on
Alexa, tell Hive to turn
the heating off
Alexa, tell Hive heating off
Alexa, turn the heating on
We don’t have to think about
resolving the click – it just
happens
GUI LUI
Many inputs may resolve to the same ‘click’,
and others may not resolve at all
Heating off
Tell me a joke
After Matthew Honnibal

19. Heating on
After Matthew Honnibal
0
Heating off
Tell me a joke What’s the weather?
Play some jazz
What’s the time in Seattle?
Play some rock
How’s my diary looking?
Add beer to my shopping list
Did Arsenal win last night?
Get the Batmobile ready
Buy more dishwasher tabs
Lock the back door
Where’s that Beer I ordered?
https://medium.com/swlh/a-natural-language-user-interface-is-just-a-user-interface-4a6d898e9721#.bogmc1aru

20. The Invisible Canvas
“[With a LUI] you have a vastly bigger canvas on which the user
can “click”…But you still have to paint buttons, forms, navigation
menus etc. onto this canvas. You’re still wiring a UI to some fixed
underlying set of capabilities.”
Matthew Honnibal
https://medium.com/swlh/a-natural-language-user-interface-is-just-a-user-interface-4a6d898e9721#.bogmc1aru

21. Listening to people https://www.youtube.com/watch?v=RFqe8U8qw-M

22. All conversation has a shared context
• When two people talk, their context will modify tone and content…
• We have social rules around phone calls, texts, IM etc.
• These are modifications to the rules of face to face conversation
• We use different language for work / home / social contexts
• …but machines have no context to share
• We have to explicitly model the context for the machine
• Alexa does not have ‘common sense’

23. Authentication
• In human-to-human conversation we authenticate on sound of
voice almost instantly
• To Alexa, all voices are equal
• Without authentication, many potential use for LUIs are unsecure
https://uxdesign.cc/what-we-can-learn-from-alexas-mistakes-a4670a9e6c3e#.3l540jsf4

24. You talkin’ to me?
• We rely on tone of voice to provide meta-data about the
message content
• “It’s not what you said, it’s the way you said it.”
• Without the meta data, the communications capacity
(bandwidth) is greatly reduced

25. GSOH essential
• “Amazon, Google, Apple and Facebook have been recruiting a diverse
cast of script writers, audio specialists and comedians. It is part of a much
wider drive in digital industry to hire those with an understanding of
how etiquette, creativity, dramatic timing and humour can elevate a digital
experience. Google, for instance, is reportedly working with joke writers
from Pixar and The Onion to imbue its new Assistant with some
personality.”
• http://www.mobileuserexperience.com/?cat=79

26. Tone of voice

27. When will it be like the
movies?
https://www.theguardian.com/culture/gallery/2015/jan/08/the-top-20-artificial-intelligence-films-in-pictures#img-17

29. Conclusion
https://goo.gl/images/lHH00z

30. Consider the context
• Spatial context
• Is the user in a space where speech can be recognised?
• Social context
• Is it socially acceptable to talk to a machine?
• Task context
• Is the user engaged in some other task ?

31. The Hype Cycle & Amara’s Law
“We tend to overestimate
the effect of a technology in
the short run and
underestimate the effect in
the long run”
Roy Amara
Source https://en.wikipedia.org/wiki/Hype_cycle
We are
here

32. Thankyou
@gordonplant
gordon@basekit.com
https://goo.gl/images/K9wtTK