A talk I gave at Conversations AI 2020 Winter

1. FROM ALEXA PRIZE SOCIALBOT TO
MULTISKILL AI ASSISTANT PLATFORM
Daniel Kornev | CPO @ DeepPavlov.ai

2. DeepPavlov.ai
Zero Interaction Cost
Wish → Action

3. DeepPavlov.ai
Little Problem:
Poor Usability*
* Intelligent Assistants Have Poor Usability, Nielsen Norman Group © 2018

4. DeepPavlov.ai
Good*
Bad
Bad
Bad
Terrible
Soon to be great
Hard to do
Limited by design
Super hard to do
Needs lot of work
UI Technique Status Potential
* For Non-native English speakers it’s still a pain
Voice Input
Natural Language Understanding
Voice Output
Intelligent Interpretation
Agency

5. DeepPavlov.ai
Holy Grail
of
NLP
human-like
comprehension
of texts &
languages

6. DeepPavlov.ai
Bad
Bad
Hard to do
Super hard to do
UI Technique Status Potential
Natural Language Understanding
Intelligent Interpretation

7. DeepPavlov.ai
Grand challenge: create a socialbot that can engage in a fun, high
quality conversation on popular societal topics for 20 minutes and
achieve an average rating of at least 4.0/5.0.
Alexa Prize 3 Winners:
1.Emora - $500K, 3.8/5.0, 7 min, 32 sec, Emora University
2.Chirpy Cardinal -- $100K, 3.17/5.0, Stanford University
3.Alquist -- 50k$, (2nd in ‘17, 3rd in ‘19, ‘20), Czech Technical University

8. 1000+ Commits
800+ PRs
~30 skills
10+ annotators
1 PhD!
DeepPavlov.ai

9. DeepPavlov.ai

10. DeepPavlov.ai
THE JOURNEY
Turning Alexa Prize Socialbot into a Multiskill AI Platform
500+ Commits
100+ PRs
+own KBQA
+own classifiers for
Topics & Dialog Acts
+tutorials

11. DeepPavlov.ai

12. DeepPavlov.ai

13. DeepPavlov.ai
?

14. DeepPavlov.ai

15. DeepPavlov.ai
© Copyright PresentationGO.com
Pre-Purchase
Post-Purchase
Surveys
Promotions
Campaigns
Customer Service
Technical Support
Product Usage
Billing & Payment
Account Management
Logistics

16. DeepPavlov.ai
▪ Open repository of NLP models
and pipelines
• easy to find and reuse NLP
components for development of
new skills or extension of existing
▪ Open repository of
conversational skills
• alternative implementations of
the most popular skills
▪ Open hub for AI Assistant
distributions
• general and domainindustry
specific distributions of skill sets

17. DeepPavlov.ai
Multiskill
orchestration
Conversa-
tionalskills
NLP
frameworks
ML platforms
Proprietary Open Source
▪ Multiskill orchestration
• DeepPavlov Agent is an engine for
conversational skill deployment and
orchestration
▪ Conversational skills
• DeepPavlov Dream is a collection of pre-
build conversational skills and a default
distribution package for Dream AI
Assistant
▪ NLP frameworks
• DeepPavlov Library provides pretrained
models and simple declarative approach
to build NLP processing pipelines
▪ ML platforms
• TensorFlow and PyTorch as backends

18. DeepPavlov.ai
LIVING IN PANDEMICS
MAKES US LONELY
HOW TO BUILD
MULTISKILL AI ASSISTANT
WITH DEEPPAVLOV
Wouldn’t it be nice to have a friend to care about us?

19. DeepPavlov.ai
MEET GERTY 3000
Can help Sam with problems on
the Moon Base Sarang?
Can entertain Sam? Yes ✔
Yes ✔
Can we emulate it with
DeepPavlov DREAM?
Yes ✔
Main Question
Functionality Analysis

20. DeepPavlov.ai
Text OR
Voice
Input
TTS
(NeMo)
Spell
Checking
NeMo ASR Harvesters
Status
Chit-Chat
(AIML)
Emotion
BUILT-IN
SKILL
SELECTOR
RULE-BASED
RESPONSE
SELECTOR

21. DeepPavlov.ai
services:
agent:
[..]
depends_on:
- mongo
harvesters_maintenance_skill:
[..]
mongo:
[..]
rule_based_response_selector:
[..]
nemo:
[..]
depends_on:
- agent
emotion_classification:
[..]
program_y:
[..]
spell_checking:
[..]
Spell Checking
Annotators
Emotion Classification
Harvesters Status Skill
Skills
Chit-Chat Skill
NeMo ASR & TTS
Other Services
Rule-Based Response Selector

22. DeepPavlov.ai
services:
agent:
[..]
depends_on:
- mongo
harvesters_maintenance_skill:
[..]
mongo:
[..]
rule_based_response_selector:
[..]
nemo:
[..]
depends_on:
- agent
emotion_classification:
[..]
program_y:
[..]
clone_tts:
[..]
Annotators
Services Groups
Skills
Response Annotators
Depend on groups (e.g., “skills”)
Services Are Isolated
Limited in what they see in dialog
Invoke Agent’s State Manager
Can run via HTTP or be Python-based
"skills": {
"harvesters_maintenance_skill": {
"connector": {
"protocol": "http",
"url": "http://harvesters_maintenance
_skill:3002/respond"
},
"dialog_formatter": "dp_formatters:ful
l_dialog",
"response_formatter": "dp_formatters:b
ase_skill_formatter",
"state_manager_method": "add_hypothesi
s",
"previous_services": ["annotators"]
},
Response Selectors

23. DeepPavlov.ai
What is (all) harvesters’ status?
Intents
What is harvester status?
Prepare rover for a trip
domain.yml
intents:
- all_statuses_request
- status_request
[..]
- trip_request
responses:
utter_status_request:
- text: "The harvester {harv_id} is {harv_status}.“
[..]
nlu.md
## intent:all_statuses_request
- What is the harvesters status?
- What is the combines status?
[..]
stories.md
## harv_status + prepare_trip
* status_request
- utter_status_request
stories.md – training for dialogs
RASA Configs
nlu.md – training for intents & slots
domain.yml – basic ontology for skill
Simple and easy to use

24. DeepPavlov.ai
Works with GoBot
GoBotWrapper
Obtains data from DB
Generates NLG
stories.md – training for dialogs
Tutorial in Google Colab
nlu.md – training for intents & slots
domain.yml – basic ontology for skill
Full sample: use it to train your GoBot
and save its output to your Skill
GoBotWrapper
[..]
@app.route("/respond", methods=["POST"])
def respond():
[..]
dialogs = request.json["dialogs"]
for dialog in dialogs:
sentence = dialog['human_utterances'][-1]
['annotations'].get("spelling_preprocessing")
[..]
uttr_resp, conf = gobot(sentence)
response = gobot.getNlg(uttr_resp)
responses.append(response)
confidences.append(conf)
return jsonify(list(zip(responses, confidences)))

25. DeepPavlov.ai
<?xml version="1.0" encoding="UTF-8"?>
<aiml version="2.0">
<category>
<pattern>I AM ^ TIRED</pattern>
<template>
🙁
<random>
<li>Get some sleep<get name="name"/>.
You're very tired.</li>
<li>Have a rest and be happy! How can
I help you?</li>
</random>
</template>
</category>
[..]
</aiml>
</xml>
Assistant Profile (Name, Place, etc.)
Patterns
Greeting scenario
Topics
Looks up for patterns
Dialog Processing
Picks random pre-defined response
If not sure, confidence is low (0.2)
Returns response + confidence

26. DeepPavlov.ai
DEEPY 3000: DEMO
A prototype of a fictional Moonbase A.I. Assistant, inspired by the Moon Movie
(2009) made by Duncan Jones
> docker-compose up --build
> curl --location --request POST 'localhost:4242' --header 'Content-
Type: application/json' --data-raw '{"user_id": "name", "payload":
“what do I do here?"}'

27. DeepPavlov.ai
demo.deeppavlov.ai
Web Demo
@deeppavlov_dream_ai_bot
TG Bot
github.com/deepmipt/dp-dream-demos
Play with demo:
Clone and build your own!
medium.com/deeppavlov
Read us:
forum.deeppavlov.ai
Talk to us: @DeepPavlovDreamDiscussions
TG:
@DeepPavlov
Twitter/TG: