A talk I gave at NVIDIA GTC Spring 2022.

1. GOD MODE FOR BUILDING OPEN-
DOMAIN BOTS WITH DISCOURSE-
DRIVEN INTEGRATED DIALOG
DEVELOPMENT ENVIRONMENT
Daniel Kornev
CPO @ DeepPavlov

2. Grand Challenge: create a socialbot that can engage in a fun, high
quality conversation on popular societal topics for 20 minutes and
achieve an average rating of at least 4.0/5.0.
Alexa Prize 3 vs 4 Winners:
1. $500K, 3.8 vs 3.28, 7 min 32 sec vs 14 min 14 sec
2. $100K 3.17 vs 3.25
Alexa Prize 4 Winner:
Alquist (2nd in ‘17, 3rd in `18,`20) – Czech Technical University 2

3. DeepPavlov.ai
“The aim is to identify a way of looking at and manipulating
commonsense knowledge that is consistent with and can support
what we consider to be the two most fundamental aspects of
intelligent cognitive behaviour: the ability to learn from experience
and the ability to reason from what has been learned. We are
therefore seeking a semantics of knowledge that can
computationally support the basic phenomena of intelligent
behaviour.”
Leslie Valiant

4. 1000+ Commits
800+ PRs
~30 skills
10+ annotators
1 PhD!
DeepPavlov.ai

5. DeepPavlov.ai +

6. DeepPavlov.ai
pip install
Python-Native
Lots of extensions
Battle-tested
Entity Detection
Extensions
Intent Classifier
Dialog Acts & Speech Functions
Dialog Analysis Dashboard

7. DeepPavlov.ai
Scenario-Driven approach enables sequential experience and is strategic, yet it takes a long time and still misses
on what user can say.
User
•Let’s chat
Bot
•(greeting)
(question)
User
•[utterance]
Bot
•[utterance]
User
•[utterance]
Bot
•[utterance]
User
•[utterance]
Bot
•[utterance]
Small Talk Movies Skill
Pros: Scenario-Driven skills enabled us to give users several conversation turns with
the same context, and limited user understanding within human-curated scenarios
Cons: given that we can’t predict what user might say, lots of iterations needed.

8. DeepPavlov.ai
Problem
Designing Open-Domain Chatbots is a complex process.
User can say anything
Interpretation How to recognize pattern?
Response How to react?
Bot: Hi, how are you doing?
User: Hi!
Regular Expressions: Require detailed scenarios and long iterative development process
Intents: Work only within the given domain (task-oriented chatbots) or at high-level (greeting, etc.)
Dialog Acts: Work in limited cases (yes/no, question/statement)
User: Hi! User: What’s up? User: I’m sad. User: Had an awful convo. User: Switch off lights

9. DeepPavlov.ai
There are at least 2 ways* look at a Conversation:
* S. Eggins & D. Slade, Analysing Casual Conversation.London: Cassell, 1997
Pragmatic Conversation
• Motivated by clear pragmatic purpose. Aka task-
oriented. Usually very short. Formal.
Casual Conversation
• NOT motivated by clear pragmatic purpose. Can
and often are lengthy. Informal, can have humor.
Aka chit-chat.

10. DeepPavlov.ai
There are several ways* to analyze a Casual Conversation:
* S. Eggins & D. Slade, Analysing Casual Conversation. London: Cassell, 1997

11. There are at least 4 different approaches to classify utterances & sentences:
Speech Acts*
• Work at utterance level. Hearer interprets
speaker’s intentions and tries to interpret
desired actions from hearer.
Dialog Acts
• Work at sentence level. Ascribe each sentence’s
dialog function to the entire utterance.
Speech Functions
• Work at sentence level. Similar to Speech Acts
but they produce utterance’s through its role in
Discourse.
Utterance Acts**
• Work at utterance level but
include body movements.
**Not applicable for us as we can’t see the person
*Original authors were not concerned with
Discourse

12. DeepPavlov.ai
There are many taxonomies for each of the classification types:
Speech Acts
• Austin (1962)
• Searle (1975)
• Bach and Harnish (1979)
Dialog Acts
• Jurafsky (1997) – SWBD-DAMSL
• MIDAS (2019)
Speech Functions
• Halliday (1984, 1994)
• Eggins and Slade (1997)
Speech Acts & Dialog Acts:
Speech Acts == Dialog Acts (Jurafsky, 2019)
Speech Acts != Dialog Acts (Webb, 2010)
vs.
Speech Acts & Speech Functions:
Speech Acts = interpretation (logico-phil.)
Speech Functions = production of utterances
(discourse)

13. MIDAS has two groups of Dialog Acts: Semantic and Functional.
Gunrock’s Report for
Alexa Prize 2 (2018)

14. MIDAS has two groups of Dialog Acts: Semantic and Functional.
Gunrock’s Report for
Alexa Prize 2 (2018)

15. DeepPavlov.ai
Eggins and Martin
(1997)
Discourse structure patterns operate across turns: thus overtly interactional & sequential
Discourse Management is a basis for acting across turns, thus becoming strategic

16. Eggins and Slade
(1997)
Speech Functions control Discourse:
Give
information
Demand
information
Speech Acts
Discourse Moves
Speech Function Example:
open:initiate:give_opinion

17. Eggins and Slade
(1997)
Speech Functions have hierarchy based on the role in Discourse:
move
open
attend command Initiate
Give
Fact
opinion
Demand
Open
Fact
Opinion
Closed
Fact
Opinion
sustain
Continue
Monitor Prolong
Elaborate
Extend
Enhance
Append
Elaborate
Extend
Enhance
React
Respond
Support
Develop
Elaborate
Extend
Enhance
Engage Register Reply
Accept
Comply
Agree
Answer
Acknowledge
Affirm
Confront
Disengage Reply
Decline
Non-comply
Disagree
Withold
Disawow
Contradict
Rejoinder
Support
Track
Check
Confirm
Clarify
Probe
Response
Resolve
Repair
Acquiesce
Confront
Challenge
Detach
Rebound
Counter
Response
Unresolve
Refute
Re-challenge

18. Eggins and Slade
(1997)
Removing SFs we don’t have to classify from user’s utterances in Alexa Prize
move
open
attend Initiate
Give
Fact
opinion
Demand
Open
Fact
Opinion
Closed
Fact
Opinion
sustain
Continue
Monitor Prolong
Elaborate
Extend
Enhance
Append
Elaborate
Extend
Enhance
React
Respond
Support
Develop
Elaborate
Extend
Enhance
Engage Register Reply
Accept
Comply
Agree
Answer
Acknowledge
Affirm
Confront
Disengage Reply
Decline
Non-comply
Disagree
Withold
Disawow
Contradict
Rejoinder
Support
Track
Check
Confirm
Clarify
Probe
Response
Resolve
Repair
Acquiesce
Confront
Challenge
Detach
Rebound
Counter
Response
Unresolve
Refute
Re-challenge

19. DeepPavlov (2021)
(removed SFs for commands, discussing physical goods, non-verbal, multiple people)
move
open
Attend Command Initiate
Give
Fact
opinion
Demand
Open
Fact
Opinion
Closed
Fact
Opinion
sustain
Continue
Monitor Prolong
Elaborate
Extend
Enhance
React
Respond
Support
Develop
Elaborate
Extend
Enhance
Engage Register Reply
Agree
Acknowledge
Affirm
Confront
Reply
Disagree
Disawow
Contradict
Rejoinder
Support
Track
Check
Confirm
Clarify
Probe
Response
Resolve
Confront
Challenge
Detach
Rebound
Counter
Response
Unresolve
Refute
Re-challenge

20. DeepPavlov.ai
We used our Speech Functions Classifier as well as Dialog Acts Classifier
(MIDAS) made by Gunrock (Alexa Prize 3 team) to measure the effect

21. Original Book Skill was designed during Alexa Prize 3 and rewritten in DFF by
one of our interns.
?
?

22. DeepPavlov.ai
Step 1
The question: could use of Speech Functions and/or MIDAS make dialog more predictable for dialog designer?
Let people talk to the bot with only this skill enabled
Step 2 Classify utterances w/ Speech Functions & MIDAS
Step 3 Run Speech Functions & MIDAS predictors
Step 4 Compute # of times predictors predicted user utterance classes correctly & not
Step 5 Compare results with vanilla data

23. DeepPavlov.ai
Primitive but working approach: ignorance, yes/no, custom conditions.

24. DeepPavlov.ai
Comparing original approach to MIDAS:

25. DeepPavlov.ai
Manual Labeling vs. Classifier:

27. DeepPavlov.ai
Comparing original approach to SFs:

28. DeepPavlov.ai
Manual Labeling vs. Classifier:

29. Speech Functions Predictor shows the potential of the Speech Functions-based scenario-driven skill

30. DeepPavlov.ai
For Dialogue Designers for Doom Players
God Mode for

31. DeepPavlov.ai
Extension for VS Code
Open-Source Dev Tool
Draw.io-based
Draw.io <> Python DSL of DFF
With Discourse Recommendations…
Aids in Rapid Dialog Design
Statistics…
Plugins!

33. DeepPavlov.ai +

34. DeepPavlov.ai
Custom spatial UI, re-built suggestions, auto response generation and… a
new name!

54. DeepPavlov.ai
Multiskill
orchestration
Conversa-
tionalskills
NLP
frameworks
ML platforms
Proprietary Open Source
▪ Multiskill Orchestration
• DeepPavlov Agent is an engine for
conversational skill deployment and
orchestration
▪ Conversational Skills
• DeepPavlov Dream - collection of pre-build
conversational skills and a default AI
Assistant distribution package
• DeepPavlov DFF - framework for building
scenario-driven chatbots & Dream skills
▪ NLP Frameworks
• DeepPavlov Library - pretrained models
and both Python & declarative approaches
for building NLP pipelines
▪ ML Platforms
• TensorFlow and PyTorch as backends

55. DeepPavlov.ai
DeepPavlov
DF Designer
DeepPavlov
DF SDK
DeepPavlov
DF Engine
DeepPavlov
Annotators
Library, Agent, Annotators, DF Engine, DF SDK, DF Designer, Dream
Builder, Dream + Dashboards (DF + Dream)
DeepPavlov
Dream Builder
DeepPavlov
DF Dashboard
DeepPavlov
Dream Dashboard
DeepPavlov
DFF

56. DeepPavlov
DF Designer
DeepPavlov
DFF
DeepPavlov
DF Engine
DeepPavlov
Annotators
DeepPavlov
Dream Builder
Dev Tools
Chatbot &
AI Assistant
Platforms
Engines
Models
DeepPavlov
DF Dashboard
DeepPavlov
Dream Dashboard
DeepPavlov
DF SDK

57. SPECIAL
THANKS!
Denis Kuznetsov
DFF Team Lead
Dmitry Evseev
Wiki for DFF SDK
Lida Ostyakova
Speech Functions
Bálint Magyar
VS Code Extension
Dmitry Babadeev
VS Code Extension
Ksenia Petyxova
Skill Writer
Nika Smilga
Skill Writer
Oleg Serikov
VS Code Extension
Maria Molchanova
Product Manager
Daniel Kornev
Deputy Head of Lab
Mikhail Burtsev
Head of Lab
Yuri Kuratov
Senior Researcher
Maxim Talimanchuk
Dev Ops Engineer
Fedor Ignatov
Senior SW Engineer
Dilyara Baymurzina
Dream Team Lead

58. bit.ly/df-designer or DF Designer in VS Code
Marketplace
DF Designer:
bit.ly/dff-sdk
DF SDK:
bit.ly/df-designer-qa or use QR code:
Talk to us about DF
Designer:
Come join us!
medium.com/deeppavlov
Read us:
forum.deeppavlov.ai
Talk to us: @DeepPavlovDreamDiscussions
TG:
@DeepPavlov
Twitter/TG: