Building a voice application for Amazon Alexa requires the Voice First approach. But with the growing device family with displays like the Echo Spot, the Echo Show, or the Fire TV, you are able to support your voice experience with photos, illustrations, or videos. This session concentrates on how to build a Multi-Modal application with Amazon Alexa. We will have a closer look on the best-practices as well as some tools and techniques to help you to create richer voice applications.

1. 1 / 78
Multimodal
Voice Development
with Amazon Alexa

2. 2 / 78
Ralf Eggert
CEO Travello GmbH
Alexa Skill Developer
Advisory Board Voice Con
Alexa Champion

3. 3 / 78
Portfolio (excerpt)

4. 4 / 78
Three questions

5. 5 / 78
Question 1:
Who owns an Echo
device with a display?

6. 6 / 78
Question 2:
Who has developed a
Multimodal Skill for Alexa?

7. 7 / 78
Question 3:
Who has an APL based
Multimodal Skill for Alexa
live?

8. 8 / 78
A definition of multimodal

9. 9 / 78
What is this
multimodal stuff
anyway?

10. 10 / 78
Multimodal
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

11. 11 / 78
Textual
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

12. 12 / 78
Aural
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

13. 13 / 78
Linguistic
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

14. 14 / 78
Spatial
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

15. 15 / 78
Visual
Multimodality describes
communication practices in
terms of the textual, aural,
linguistic, spatial, and visual
resources - or modes - used to
compose messages.
Murray, Joddy (2013) / Wikipedia

16. 16 / 78
Textual Aural Linguistic Visual
Let's focus on

17. 17 / 78
Interlude

18. 18 / 78
OTIS REM
Who knows this smart
voice assistant?
Is it multimodal?

19. 19 / 78
Alexa & Multimodal

20. 20 / 78
OK!
How about Alexa?

21. 21 / 78
AWS Lambda /
HTTPS Endpoint
Server
Alexa Voice Service


Headless

22. 22 / 78
Cards
Support headless Echo
devices with visuals.
Are they multimodal?

23. 23 / 78
Echo Buttons
Another input media.
Are they multimodal?
What do you think?

24. 24 / 78
AWS Lambda /
HTTPS Endpoint
Server
Alexa Voice Service

Multimodal

25. 25 / 78
Echo Dot Echo Echo Show Echo Spot
Headless vs. Multimodal

26. 26 / 78
Display devices
Only 5.9 % of Alexa users in
the US own an Echo device
with a display
Voicebot.ai, June 24, 2018, see https://goo.gl/7WSkjD

27. 27 / 78
More numbers
56.2 % own Echo Spot
25.0 % own Echo Show
18.7 % own both devices
Voicebot.ai, June 24, 2018, see https://goo.gl/7WSkjD

28. 28 / 78
Mind the trap!
Many Alexa Skills rather
focus on Echo Show than on
Echo Spot.
More than twice as much
Echo Spot than Echo Show
devices sold.

29. 29 / 78
History: display interface

30. 30 / 78
Echo Show
1st generation Echo Show
was introduced in May 2017.

31. 31 / 78
Echo Spot
1st generation Echo Spot
was introduced in
September 2017.

32. 32 / 78
Display Interface
Support for displays added to
Alexa developer console

33. 33 / 78
Reference
Learn how to build
multimodal Alexa Skills
with the Display
Interface Reference
https://goo.gl/eeSQah

34. 34 / 78
Display Interface Templates

35. 35 / 78
Body Template 1
Echo Show Echo Spot

36. 36 / 78
Body Template 2
Echo Show Echo Spot

37. 37 / 78
Body Template 3
Echo Show Echo Spot

38. 38 / 78
Body Template 6
Echo Show Echo Spot

39. 39 / 78
Body Template 7
Echo Show Echo Spot

40. 40 / 78
List Template 1
Echo Show Echo Spot

41. 41 / 78
List Template 2
Echo Show Echo Spot

42. 42 / 78
Wait!
What happened to
Body Templates
4 and 5?
Does anyone know?

43. 43 / 78
Template Docs
Learn to choose the best
template for your needs.
https://goo.gl/pp4STX

44. 44 / 78
{
"version": "1.0",
"session": {},
"context": {
"Display": {},
"System": {
"device": {
"deviceId": "...",
"supportedInterfaces": {
"Display": {
"templateVersion": "1.0",
"markupVersion": "1.0"
}
}
}
}
},
"request": {}
}
Alexa Request
Very simplified

45. 45 / 78
{
"body": {
"version": "1.0",
"response": {
"outputSpeech": {},
"card": {},
"directives": [
{
"type": "Display.RenderTemplate",
"template": {}
},
{
"type": "Hint",
"hint": {}
}
],
"reprompt": {},
"shouldEndSession": false
}
}
}
Alexa Response
Also very simplified

46. 46 / 78
{
"type": "BodyTemplate6",
"token": "launch",
"backgroundImage": {
"contentDescription": "Hello",
"sources": [
{
"url": "https://host.com/image.jpg",
"widthPixels": 0,
"heightPixels": 0
}
]
},
"textContent": {
"primaryText": {
"type": "RichText",
"text": "<font size="7">Hello</font>"
}
},
"backButton": "HIDDEN"
}
RenderTemplate
As well simplified

47. 47 / 78
Cons of Display
Interface
Only 7 templates
Distinction Show vs. Spot
Templates on Spot suck
Static layouts
Hard to test

48. 48 / 78
Alexa Presentation Language

49. 49 / 78
Alexa
Presentation
Language
For the win

50. 50 / 78
APL schedule
Private beta June 2018
Public beta October 2018
General availability soon

51. 51 / 78
Echo Show
2nd generation Echo Show
was presented in September
2018.

52. 52 / 78
Alexa
Presentation
Language
Support for APL added to
Alexa developer console

53. 53 / 78
Reference
Learn how to get started with
the Alexa Presentation
Language
https://goo.gl/Zw9m62

54. 54 / 78
Authoring Tool
Design your own templates
https://goo.gl/jbC535

55. 55 / 78
Authoring Tool
Pick from the predefined
samples, start from scratch
oder upload your code

56. 56 / 78
APL terminology
Document, package,
layout, resources, styles,
and components
https://goo.gl/ufq6SH

57. 57 / 78
APL document
JSON file containing list of
packages, resources, layouts,
and styles.
Works like a container and is
send to the device.

58. 58 / 78
APL package
Packages contain APL
documents and images.
Can easily be reused and are
cached on the device.

59. 59 / 78
APL layout
Hierarchy set of components
for rendering one the display.
Can contain text, images,
scrolling regions and even
other layouts.
Can be used to build libraries.

60. 60 / 78
APL resources
Defined constants to be used
for drawing text or images on
the screen.
For example font sizes,
colours or spacing.

61. 61 / 78
APL styles
Collection of grouped
resources to build a style.
Defines size, background
colour, text colour, borders,
etc.

62. 62 / 78
APL components
Components are primitive
types to be added to a layout.
Examples are containers,
text, images, sequences,
scroll views, or touch
wrappers

63. 63 / 78
{
"version": "1.0",
"session": {},
"context": {
"Display": {},
"System": {
"device": {
"deviceId": "...",
"supportedInterfaces": {
"Alexa.Presentation.APL": {
"runtime": {
"maxVersion": "1.0"
}
}
}
}
}
},
"request": {}
}
Alexa Request
With active APL support

64. 64 / 78
{
"body": {
"version": "1.0",
"response": {
"outputSpeech": {},
"card": {},
"directives": [
{
"type": "Alexa.Presentation.
APL.RenderDocument",
"version": "1.0",
"document": {},
"datasource": {}
}
],
"reprompt": {},
"shouldEndSession": false
}
}
}
Alexa Response
Also very simplified

65. 65 / 78
{
"type": "APL",
"version": "1.0",
"import": [],
"resources": [],
"styles": {},
"layouts": {},
"mainTemplate": {}
}
APL Document
Just the structure

66. 66 / 78
Authoring tool hands-on

67. 67 / 78
Pros of APL
Start with samples
Build from scratch
Build reusable packages
Authoring tool
Easy to test
Split display design from code

68. 68 / 78
Cons of APL
Response is more complex
No saving option in the
authoring tool
Not fully rolled out yet
No graphical editor yet

69. 69 / 78
SDKs / tools with APL support
Third party toolsOffical SDKs
December 2018
January 2019
Alexa.NET

70. 70 / 78
Compare
display interface and APL

71. 71 / 78
Display interface start screen
Thanks toDanielMittendorf

72. 72 / 78
APL start screen
Thanks toDanielMittendorf

73. 73 / 78
Display interface list screen
Thanks toDanielMittendorf

74. 74 / 78
APL list screen
Thanks toDanielMittendorf

75. 75 / 78
Conclusion

76. 76 / 78
Start with APL
Public beta phase
You could start today
Consider the numbers of sold
display devices!

77. 77 / 78
Need more
motivation?
Alexa Skills Challenge
$150K in total prizes
Bonus prize for Germany
Enter til 22th of January 2019
https://goo.gl/EETRu5

78. 78 / 78
Any questions?
ralf@travello.audio
https://www.travello.audio