When we talk about (chat) bots, we think of a bot with which to chat through texts entered with a keyboard. Between typos, buttons and AdaptiveCards, we tend to lose sight of the true essence of a Conversational Agent: conversing through a dialogue, perhaps through the voice. The maximum value of a bot is expressed when using the Azure Cognitive services for Speech to Text and Text to Speech. We will understand how to develop a bot, the use of predefined entries and multilingual support. And what are the consequences in defining a conversation. And then we'll see how to give the bot "physicality". Through the use of Unity, we will create a scene with an Avatar modeled in 3D that synchronizes with the conversation, giving us the feeling of talking to us like a real person. The whole in the context of an enterprise scenario.

1. Realizzare
un Virtual Assistant
con Bot Framework,
Azure e Unity
Marco Parenzan
Solution Sales Specialist @ Insight
Microsoft Azure MVP
Community Lead @ 1nn0va
www.wpc2019.it 1

2. www.wpc2019.it 2
The classic way to the chatbot

3.  Comprehensive framework for the lifecycle
of your project
Create a
Conversation
with Bot
Service
Author Dialogs
Design Cards
Visualize
Bot Framework
SDK
Cognitive
Services
Getting Started
Templates
Emulator​
Application
Insights
DevOps Tools
Azure
BYO Web Host
WebChat
Cortana
Skype, FB
+
Custom
(DirectLine)
Application
Insights
LUIS Suggestions

4. Conversation
al AI: Azure
Bot Service +
Cognitive
Services
User Input Channels Business ProcessBot Intelligence

5.  WebAppBot
 Bot Framework Emulator
 Bot Channel Registration
 Ngrok
 App Credentials
 Storage State
 GitHub samples
 https://github.com/microsoft/botframework
 https://github.com/microsoft/BotBuilder-Samples
 https://github.com/microsoft/botframework-solutions
 https://github.com/microsoft/BotFramework-Samples
www.wpc2019.it 5
My starting
point for bot

6.  Portable - To any app, device, and UI framework
 Open - Libraries and schema are open source and shared
 Low cost - Easy to define, easy to consume
 Expressive - Targeted at the long tail of content that
developers want to produce
 Purely declarative - No code is needed or allowed
 Automatically styled - To the Host application UX and brand
guidelines
Adaptive
Cards
(https://adapti
vecards.io/)
Open Framework Multiple Canvases
Notification Microsoft Teams Skype Android iOS
Bot Service Lifecycle: Plan | Build | Test | Publish | Connect | Evaluate

7. DEMO
www.wpc2019.it 7

8. What I don’t
like of Bot
Framework
 Developer Centered
 Dialog methaphor
 Form based

9. www.wpc2019.it 9
New pillars

10. Rationalise
management to
avoid abandonment
Governance -
Highlight and
optimize the asset
value of a CA
Conversational
Agents are another
resource that IT
needs to manage
So you need to
identify where the
costs and points of
attention are
Apply some
"devops" style
discipline
Need for
governance

11. Autonomy
Not having
skill IT
No channel
dependency
More text,
less forms
Encouraging
the role of
conversationa
l designer

12. Different
"channels"
Text
Voce
Virtual
Reality
 We are meeting different
needs
 Text/form
 Desktop/Mobile
experience
 Voice
 Mobility/Hand-free
experience
 Virtual reality
 Human simulation
experience
 Integration with non-chat-
based facilities

13. www.wpc2019.it 13
…a long journey not yet
finished…

14. www.wpc2019.it 14
The activity architecture

15. Intro on how
the bot
framework
works
 Every interaction creates
an activity
 Bot Framework interacts
with the user via channel
 ConversationalUpdate and
Message
 Requests arrive to the bot
via HTTP POST
 Status responses are an
HTTP 200
 Conversation in managed
via TurnContext

16. Intro on how
the bot
framework
works
 Activity is executed via
JSON
 JSON is deserialized and
sent to the Adapter, using
the ProcessActivity method
 Adapter creates the
TurnContext and calls the
middleware
 Activities are managed by
the ActivityHandler

17. DEMO
www.wpc2019.it 17

18. www.wpc2019.it 18
Where is intelligence in a bot?

19.  “Conversational AI refers to the use of messaging apps,
speech-based assistants and chatbots to automate
communication and create personalized customer
experiences at scale”
 “A chatbot is a computer software program designed to
simulate human conversation via text, audio messages, or
images”
Conversation
al AI

20. Adding
Intelligence
Bing Speech API
Convertspeech to text and back
again, and understandits intent
Speaker Recognition API
Give your app the ability
to know who's talking
Custom Speech Service
Fine-tune speech recognition for
anyone, anywhere
QnA Maker
Distill information intoconversational,
easy-to-navigate answers
Language Understanding
Intelligent Service
Teach your apps to understand
commands from your users
Easilyperform speech
and text translation
Translator
Azure Search
Integrate search into a
conversational experience

21.  You will not lower people in your company
 You will use them to more valuable tasks
TRAIN YOUR
INTELLIGEN
CE

22. UNAUTHENTICATED
EXPERIENCE
AUTHENTICATED
EXPERIENCE
THE TWO
STEPS OF A
COMPANY
BOT
STRATEGY

23.  Answer to simple questions
 «How to do this...»
 «How to do that...»
 Objective of the company
 Engaging experience
 Lower pressure on customer care
UNAUTHENT
ICATED
EXPERIENC
E

24. QnA Maker
 Based on Azure. The data
and runtime hosted in your
Azure subscription.
 Data privacy and
compliance. QnA data is
hosted within your Azure
compliance boundary.
 Azure Application Insights
integration. Helpful to get
insights on how the
services is being utilized.

25. DEMO
www.wpc2019.it 25

26. www.wpc2019.it 26
Integrating an API…

27.  The GW integration implements the Business Logic for
intents
 1 intent1 API
 Enables microservices governance to be implemented
 Direct call to internal business applications
 Or indirect if you implement MS on corporate data lakes
 Infrastructure setup
 Integration Gateway API Setup
 Azure VNET Configuration
 On-premises infrastructure configuration support (Firewall, Host
involved
Integration
GW
(Gateway)
Result
Intent
you CA
Utterance Intent
Intent Utterance
Channel
Entities
Result
IntentAPI

28. HTTPS
HTTPS
App Service
Help Desk
Commerciale
API
Doc Commerciale
API
Doc interna API
Technical
architecture
of a
COMPANY
INTEGRATIO
N
BOT FW
Channel
Teams
Helpdesk API
HTTPS
Skype
for
Business
Channel
Azure
AD
Active
Directory
GWIntegrazione
AD Connect
SAP
Sharepoint 2010
FirewallDMZ/PrivateLAN
Win10
Mac
iOS
Android
LUIS NLP
1 x Contesto x Lingua
Contesto
Risposte
1 x Contesto x Lingua
Sharepoint 2019

29. www.wpc2019.it 29
Let’s build a Conversation DSL

30. The user's intention
The idea behind the
user's action
Every time you
interact with an
application, you
express an intention
The intent is an
NLP concept...
... but intent is
everywhere
What is an
intent

31.  Intent – the action/task the user wants to perform
 Utterance – the sentence where the user expresses the goal
 Entity – words or phrases inside an entity
 Domain –Intents related to a specific type of actions (e.g.
Calendar intents) – prebuilt domains
Let’s talk
about intent

32. ... a universally
recognized text key
Probably a key
classified in a
domain
.. characterized by a
set of parameters
(called entities) that
argue the intention
Technically
speaking, an
intent is...

33.  Each Intent is a user story in its own right
 Separation of contexts promotes maintainability
 For each Intent
 Defining sprints, design thinking, and iterations with business
users
 Defining Utterances
 Test
 Implementation in BOT
 Api implementation (independent call grouped inconversation
contexts by GW API)
 You don't always need
How to
Implement
Using User
Story - Agile
Methodology

34. DEMO
www.wpc2019.it 34

35. www.wpc2019.it 35
From intent to text

36.  Machine learning-based service to build natural language into
apps, bots, and IoT devices.
Quickly create enterprise-ready, custom models that
continuously improve.
Language
Understandin
g (LUIS)
Quickly build a custom
language solution
Alays learning &
improving
Enterprise-ready,
available worldwide
Add natural language
to your apps

37.  Intent – the action/task the user wants to perform
 Utterance – the sentence where the user expresses the goal
 Entity – words or phrases inside an entity
 Domain –Intents related to a specific type of actions (e.g.
Calendar intents) – prebuilt domains
LUIS –
terminology

38.  It means adding examples (utterance) so that statistically the
tool recognizes and classifies the similarities, meaningful
words, excludes the least important ones
 It is not always true (e.g. personalization, sentiment, ...)
 Beware of ambiguity
What it
means
“training”

39. The intent is
an NLP
concept...
... but
intents are
everywhere
What is intent
The user's intention
The idea behind the
user's action
Every time you
interact with an
application, you
express an intention

40. ... a universally
recognized text key
• Probably a key
classified in a domain
.. characterized by a set
of parameters (called
entities) that argue the
intention
Technically
speaking, an
intent is...

41.  Cost Effective Based on statistical algorithms (pattern
matching on utterance)
 Identifying the correct set of utterance is the work of the
conversational designer
 Pervasive (coverage)
 Unambiguous (to avoid false positive)
 Anyway, a big piece of engineering
 Out-of-the-box solutions for major players
 Offline support
 Adapt Set expectations correctly
Natural
Language
Processing
(NLP)
FUTURESReti Neurali (verticali, taylor made,
efficaci, ma costosi)

42.  LUIS
 Request: utteranceIntent
 Multiple LUIS instances for multiple languages…
 …but one conversation
 There are the answers
 There is the conversation
 www.luis.ai: one app at a time
Luis
Authoring

43. DEMO
www.wpc2019.it 43

44. www.wpc2019.it 44
Using voice

45. Cognitive
Speech
Services
overview
Real-time & Batch Mode
single voice, 2-waydialogues or multi-person conversations
Enhancedadaptation
New Call Center specific features
REST for all operations on new Portal!
Over 80 standard voices in 49 locales
Customizable for 9 locales
Neural TTS GA for 4 locales
Available via REST and SDK!
Speech-to-speech translationbased on neural machine translation
(NMT) technologies
Available in 10+ locales
Customizable!
Speech SDK 1.5 (out now)
Easy to integrate with the full range of
cloud-basedMicrosoftSpeech services,
creatingan end-to-endexperience for
customers.
Multi-Platform
Multi-language support
Speech devices SDK
Optimized audio capturingsoftware
stack seamlesslyintegrated with our
Speech SDK that connects to our
Speech Services
Enable your applications to talk
Swiftly convert audio to text for natural responsiveness
Speech-to-Text
Text-to-Speech
Speech translation
Easily integrate real-time speech translation to your app

46.  In NLP terms, it is used to transcribe a "multimedia" utterance
 We're talking about S2T...
 ... but even an OCR could have its role...
 .. or a custom vision using the camera...
 Cost Effective
 Pre-established entries
 Success will involve the availability of variants...
 ... and the increase in quality
 You can do the custom training
 For noise reduction, domain verticalization
 Expensive (hours of recordings and pages of transcripts)
Speech To
Text

47.  Full cloud solution. Twilio
 Integration with existing Call Center
 Feasibility verification
 Human Hand-off
Fonia

48.  interactive voice response (IVR)
 First level
 Massive call schedule for data retrieval
 Compliance compliance
 Artifact Development Appliances
 Hands-free interaction
Esempi di
Scenari

49. DEMO
www.wpc2019.it 49

50. www.wpc2019.it 50
Virtual Assistant

51. Virtual
Assistant
architecture
on Bot
Framework
BotChannel
Registration
App
Service WPF+Unity
Player
Microsoft Azure Virtual Assistant (on premises)
Management
Web App
AI
[Bot Framew ork
V4]
Config
Cognitive
Services
Text-To-
Speech
Speech-To-
Text
NLP (LUIS)
NLP
Design
Text/Audi
o
Camera
Micropho
ne
Speakers
Video
Audio
Audio
Testo
Testo
Intent
Audio
input
output
Avatar
Channel
Touch(x,y)
Proiezion
e
3D
Audio
DirectLine
Windows PC

52. DEMO
www.wpc2019.it 52

53. www.wpc2019.it 53
Interesting Projects

54.  https://github.com/microsoft/BotFramework-Composer
 Bot Framework Composer is an integrated development tool for developers
and multi-disciplinary teams to build bots and conversational experiences
with the Microsoft Bot Framework. Within this tool, you'll find everything you
need to build a sophisticated conversational experience.
 https://github.com/microsoft/BotFramework-Emulator
 The Bot Framework Emulator is a desktop application that allows bot
developers to test and debug bots built using the Bot Framework SDK. You
can use the Bot Framework Emulator to test bots running either locally on
your machine or connect to bots running remotely through a tunnel.
 https://github.com/microsoft/BotFramework-WebChat
 This repository contains code for the Bot Framework Web Chat component.
The Bot Framework Web Chat component is a highly-customizable web-
based client for the Bot Framework V4 SDK. The Bot Framework SDK v4
enables developers to model conversation and build sophisticated bot
applications.
 https://github.com/microsoft/botframework-cli
 The new BF Command Line Interface (CLI) tool replaces the collection of
standalone tools used to manage Bot Framework bots and related services.
We have ported most tools and are in process of porting the rest. The new
BF CLI aggregates the collection of cross-platform tools into one cohesive
and consistent interface.
GitHub

55. www.wpc2019.it 55
Q&A

56.  OverNet Education
 Info@OverNetEducation.it
 www.OverNetEducation.it
 Rozzano (MI) +39 02 365738
 Bologna +39 051 269911
 www.wpc-overneteducation.it
 #wpc19it
www.wpc2019.it 56
Contatti