Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Developed Google Home app
(Actions on Google)
using Microsoft Azure Functions
 Everyone calls me “Chomado”.
 Based in Tokyo, Japan. Love
anime/manga
 Before joining Microsoft, I’d been a
mobile app...
My manga work:
“Run! Coding Academy”
37.4K
Twitter : @chomado
I love my Grandma
She has presbyopia
(aged eyes) and
difficult to read.
So I bought a smart
speaker for her.
I think VUI (...
What is “Smart
Speaker”?
Wireless speaker with AI assistant!
Virtual
Assistant
(logo)
Virtual Assistant Owned by Devices Languages/Markets
Microsoft Cortana
Harman Kardon
INVOKE
Engli...
Virtual
Assistant
(logo)
Virtual Assistant Owned by Devices Languages/Markets
Microsoft Cortana
Harman Kardon
INVOKE
Engli...
How Microsoft Cortana skill works?
How's the weather like in
Tokyo today? voice
How's
the weather like in Tokyo
today? text
Weather
Service
When you say a
co...
Connect to
HelloAzureFunctio
ns service voice
HelloAzureFunctions
I don’t know such service.
I don’t know what to do
When ...
When you say a command
which Google Assistant
does NOT know
Speech to Text,
NLP,
Knowledge Graph,
ML Ranking,
User Profile...
You need to implement
this function (Web API)
called by Webhook
Speech to Text,
NLP,
Knowledge Graph,
ML Ranking,
User Pro...
Hi, I’m Madoka
(voice)
Hi, I’m Madoka
(in cool voice)
I made a voice
changer app (action)
working like this!
I post its demo video to Twitter
two weeks ago (16 March)
I made this app
using Microsoft Azure
Functions (C#) and it
work...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
Microsoft Azure
Dialogflow
(Natural language user int...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
Azure Functions
(where my code running)
“I’m Madoka”
...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
Azure Functions
(where my code running)
“I’m Madoka”
...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
Azure Functions
(where my code running)
“I’m Madoka”
...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
Azure Functions
(where my code running)
“I’m Madoka”
...
Actions on Google
“I’m Madoka”
She says “I’m Madoka”.
Invoke Webhook
I’m Madoka
(in cool voice)
Azure Functions
(where my ...
Trigger: HttpTrigger
Output: CloudBlockBlob (I chose this because it
can directly access the stored mp3 file’s URL)
Azure ...
In the Function
In speech execution, it’s better to use
SSML format. (Speech Synthesis Markup
Language: an XML-based marku...
It got 58 stars
and forked 10 times
Google Assistant app (Actions on Google) with Microsoft Azure Functions in C# (Serverless service by Microsoft)
Nächste SlideShare
Wird geladen in …5
×

Google Assistant app (Actions on Google) with Microsoft Azure Functions in C# (Serverless service by Microsoft)

3.061 Aufrufe

Veröffentlicht am

https://github.com/chomado/GoogleHomeHack
Please use "Webhook" in FulFillment page in Dialogflow to use Azure

Veröffentlicht in: Technologie
  • Login to see the comments

Google Assistant app (Actions on Google) with Microsoft Azure Functions in C# (Serverless service by Microsoft)

  1. 1. Developed Google Home app (Actions on Google) using Microsoft Azure Functions
  2. 2.  Everyone calls me “Chomado”.  Based in Tokyo, Japan. Love anime/manga  Before joining Microsoft, I’d been a mobile app developer using Xamarin + Microsoft Azure  I’m not only an engineer but also a manga artist:
  3. 3. My manga work: “Run! Coding Academy”
  4. 4. 37.4K Twitter : @chomado
  5. 5. I love my Grandma She has presbyopia (aged eyes) and difficult to read. So I bought a smart speaker for her. I think VUI (Voice UI) is good for old-aged people.
  6. 6. What is “Smart Speaker”? Wireless speaker with AI assistant!
  7. 7. Virtual Assistant (logo) Virtual Assistant Owned by Devices Languages/Markets Microsoft Cortana Harman Kardon INVOKE English for US, Canada, Australia, India Amazon Alexa Amazon Echo English for US, UK, Ireland, Canada, India, Australia, New Zealand; German (Germany, Austria); Japanese (Japan); Google Assistant Google Home English (U.S., U.K., Canada, Australia), French (France, Canada), German, Japanese Siri Apple HomePod English for US, UK, Australia Clova Clova Friends, Clova WAVE Japanese and Korean for Japan and South Korea markets
  8. 8. Virtual Assistant (logo) Virtual Assistant Owned by Devices Languages/Markets Microsoft Cortana Harman Kardon INVOKE English for US, Canada, Australia, India Amazon Alexa Amazon Echo English for US, UK, Ireland, Canada, India, Australia, New Zealand; German (Germany, Austria); Japanese (Japan); Google Assistant Google Home English (U.S., U.K., Canada, Australia), French (France, Canada), German, Japanese Siri Apple HomePod English for US, UK, Australia Clova Clova Friends, Clova WAVE Japanese and Korean for Japan and South Korea markets (ToT) Google Home
  9. 9. How Microsoft Cortana skill works?
  10. 10. How's the weather like in Tokyo today? voice How's the weather like in Tokyo today? text Weather Service When you say a command which Google Assistant knows How Google Assistant works Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Weather Service
  11. 11. Connect to HelloAzureFunctio ns service voice HelloAzureFunctions I don’t know such service. I don’t know what to do When you say a command which Google Assistant does NOT know Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Connect to HelloAzureFunction s service text How Google Assistant works
  12. 12. When you say a command which Google Assistant does NOT know Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant How Google Assistant works So, to make Google Assistant do your own command, you need to create an Google Assistant app called “Actions on Google” (same as Cortana/Alexa skill development motivation)
  13. 13. You need to implement this function (Web API) called by Webhook Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech Devices Google Assistant Create Actions on Google to run your own command on your Google Assistant
  14. 14. Hi, I’m Madoka (voice) Hi, I’m Madoka (in cool voice) I made a voice changer app (action) working like this!
  15. 15. I post its demo video to Twitter two weeks ago (16 March) I made this app using Microsoft Azure Functions (C#) and it works like this … 47.5 K views in 2 weeks! It means many Google developers got interested in Microsoft technology. https://twitter.com/chomado/status/974534165419892736
  16. 16. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  17. 17. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  18. 18. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  19. 19. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  20. 20. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) mp3 data’s URL + Request to play the mp3 file (written in SSML) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  21. 21. Actions on Google “I’m Madoka” She says “I’m Madoka”. Invoke Webhook I’m Madoka (in cool voice) Azure Functions (where my code running) “I’m Madoka” (json data) VoiceText Web API (text to speech service by HOYA, Inc) “I’m Madoka” (string) Azure Storage (Blob) mp3 data’s URL + Request to play the mp3 file (written in SSML) Microsoft Azure Dialogflow (Natural language user interface like MS LUIS)
  22. 22. Trigger: HttpTrigger Output: CloudBlockBlob (I chose this because it can directly access the stored mp3 file’s URL) Azure Function (v1) which is called by Google’s Webhook Google Assistant users’ input text (“I’m Madoka!”) is stored in this JSON property named “ResolvedQuery”, so we use it.
  23. 23. In the Function In speech execution, it’s better to use SSML format. (Speech Synthesis Markup Language: an XML-based markup language for speech synthesis applications) Not only Google Assistant but also our Cortana and Amazon Alexa support it. You can access the URL which stores the mp3 file in Azure Blob Storage As you know, Azure Functions allows us to connect & post data to Azure Storage sooooo easily! Azure Functions is awesome. Everyone knows it.
  24. 24. It got 58 stars and forked 10 times

×