Technology advances have enabled people with disabilities to communicate more meaningfully and participate more fully in their daily lives. In this session, we discuss the many challenges for those with special needs and how AWS voice technologies empower this population. In this workshop, participants learn how to build Pollexy (Polly + Lex), a Raspberry Pi, and a mobile-based special needs verbal assistant that lets caretakers schedule audio task prompts and messages both on a recurring schedule and on-demand.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Building a Virtual Assistant with
Amazon Polly and Amazon Lex: Pollexy
T r o y L a r s o n , A W S S e n i o r D e v O p s C l o u d A r c h i t e c t
R u c h i k a A b b i , A W S S o l u t i o n s A r c h i t e c t
J e r e m y W a l k e r , A W S I n f r a s t r u c t u r e A r c h i t e c t
M C L 3 3 1
N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workshop goals
• Understand the challenges of building a multiroom, multi-person virtual assistant
• Experience the simplicity of connecting a Raspberry Pi to AWS services
• Be inspired to use AWS technologies to build impactful solutions

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Pollexy Project
https://www.youtube.com/watch?v=BUewiOZTNzM

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AI solutions for every developer
Pollexy uses AI-
enabled managed
API services
UsabilityandSimplicity
ControlandAdvancedDeepLearningCapabilities

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Conversational interfaces for applications
Fast, flexible, and scalable NoSQL
database
Event-driven, serverless compute Managed message queuing service
AWS services used: F u lly m a nage d, s e r ve rle ss
Amazon DynamoDB AWS Lambda Amazon Simple Queue Service (SQS)
Amazon Polly
Lifelike text-to-speech service
Amazon Lex

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hardware components
Raspberry Pi 3 Headset Pluggable USB audio adapter

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Amazon Polly?
• A service that converts text into lifelike speech
• Offers 50 lifelike voices and 24 languages
• Low latency responses enable developers to build real-time systems
• Developers can store, replay, and distribute generated speech
• Cost-effective, pay-as-you-go, $4 for 1M characters, free tier of 5M
characters/month – first year

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Polly – Lifelike text-to-speech
Natural sounding speech
A subjective measure of how close TTS output is to human speech.
Accurate text processing
Ability of the system to interpret common text formats such as abbreviations,
numerical sequences, homographs, and so on
Today in Las Vegas, NV, it's 90°F.
"We live for the music," live from the Madison Square Garden.
Highly intelligible
A measure of how comprehensible speech is.
“Peter Piper picked a peck of pickled peppers.”

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Lex?
• A service for building conversational interfaces for your applications using voice
and text, powered by the same deep learning technologies as Amazon Alexa.

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Speech
recognition
Natural language
understanding
Powered by the same deep learning technology as Alexa
Amazon Lex – T e x t a n d s p e e c h l a n g u a g e u n d e r s t a n d i n g

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Lex – E x a m p l e
Utterances
Spoken or typed phrases that invoke
your intent
BookHotel
Intents
An intent performs an action in
response to natural language user
input
Slots
Slots are input data required to fulfill
the intent
Fulfillment
Fulfillment mechanism for your intent

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pollexy basic concepts

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Pollexy?
• AWS open-source Python project
• Multiroom messaging system
• Multi-person messaging system
• Built-in scheduling
• Initiates messages to each location

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overall flow

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Person
• Individual who can receive a message
• Each person can have custom attributes
• Require physical confirmation before hearing message
• Mute windows

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Location
• Anywhere a device is located
• Each location can have custom attributes
• Person can be scheduled at each location
• Mute windows
• Device capabilities (examples: buttons,
cameras)

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scheduling engine
• Simple to use built-in scheduler
• Uses the common iCalendar format
• What can be scheduled?
• People at locations
• Mute windows
• Individual notifications

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scheduled message
• Recurring message/notification
• Messages are scheduled for a person, not location
• Lambda reads the scheduled message determines
where the person located by the person schedule

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hands-on
GitHub repo:
https://github.com/awslabs/Amazon-Pollexy
Twitter:
@TheTroyLarson