Using voice commands has been ubiquitous nowadays, as more mobile phone users use voice assistants like Siri, as well as devices like Amazon Echo and Google Home have been invading our living rooms. You can interact with the intelligent assistant without leaving your couch. At the same time, chatbots have gained popularity, and services like Slack and Facebook Messenger let you achieve tasks without leaving the client- you can schedule a meeting, order some pizza, call a taxi, etc. Historically in web development, we have been relying on various UI elements to interact with users. Now with the new technologies, you can develop rich applications with the natural user interactions with a minimal visual interface. This enables countless use cases for richer and more accessible web applications. In this talk, Tomomi Imura talks about the trend of the conversational interface, and what and how you can build with JavaScript. The talk is given at: - TokyoJS (shorter ver) July 28, 2017 - ModernWeb Taiwan Aug 10, 2017

1. Future of the Web with
Conversational Interface
BOTS, AI, AND JAVASCRIPT
Tomomi Imura (@girlie_mac)
#ModernWeb2017

2. @ girlie_mac
● advocate open web & technology
● code JavaScript & Node.js
● write & speak about tech
● hack useless stuff
● dev relations at Slack
● live in foggy San Francisco
tomomi imura

3. "Bots are like new
applications that
you can converse
with. "
-- Satya Nadella, Microsoft

4. @ girlie_mac
“We will evolve
in computing
from a mobile
first to an AI first
world.”
-- Sundar Pichai, Google

5. @ girlie_mac CC BY-SA: nextdayblinds.com

6. @ girlie_mac
Traditional Web & App Interactions

7. @ girlie_mac
Modern Web & Apps with Social Interactions

8. @ girlie_mac
Conversational User Interactions: Siri and Alexa (Voice Assistants)
Alexa, how is the
weather?
Hey Siri
In various form-factors

9. @ girlie_mac
Conversational User Interactions for Kids - with Voice
CogniToys Dino - https://cognitoys.com

10. @ girlie_mac
Conversational User Interactions in a robot shape - with Voice

11. @ girlie_mac

12. @ girlie_mac
Conversational User Interactions: Goole Assistant (Voice & Text)

13. @ girlie_mac
Conversational User Interactions: SlackBots (Text)
!

14. Slack Integrations & Bots for Better Productivity
+
+
+

15. Graphic
Interface to
Conversational
Interface

16. @ girlie_mac
Deliver me a large margherita pizza!
What kind of crust do you
want?
1. Regular crust
2. Thin crust
3. Gluten free crust
Thin crust

17. Natural user
interactions
with a
minimal visual
interface

18. No UI Clutter
Less Time Spent

19. @ girlie_mac
Where is the address
of the place?
It’s 325 Pearl Street.
Come on now!
Request a ride

20. @ girlie_mac
Yes, get me a ride now
Your driver, Sam will
pick you up in 6 minutes.
Look for the red Toyota
Prius!

21. Old concepts.
Technology
caught up
with the ideas.

22. @ girlie_mac
“I'm sorry Dave, I'm afraid I can't do that”

23. @ girlie_mac
“I'm sorry Dave, I'm afraid I can't do that”

24. Conversational
Interface is:

25. ● Intuitive
● Accessible
● Productive

26. @ girlie_mac
Messaging Platforms
● Slack
● Facebook Messenger
● Telegram
● WeChat
● Kik
● Viver
● LINE etc.

27. Messaging + Bots for More Interactive Communications

28. Slackbots at

29. Slackbots at
basuke@sony.com

30. TacoBot by Taco Bell
https://www.tacobell.com/feed/tacobot

31. @ girlie_mac
Chatbot Building APIs
● API.ai (Google)
● Wit.ai (Facebook)
● Microsoft Bot
Framework
● Motion.ai
● Converse AI
● Chatbots.io
● Recast.ai
● Flow XO
● ManyChat
● Fluent.ai (Voice UI)
etc.

32. AIaaS
Motion.ai
API.ai
Artificial
Intelligence as
a Service

33. @ girlie_mac
Hi,
I want to book a flight!
Yes, from SFO.
Where are you flying to?
Taipei
Hi Linda! Are you flying from
San Francisco, as usual?
An airline customer
service bot
Linda may not
know she is talking
to a bot!

34. @ girlie_mac
NLP & Conversational Platforms / APIs
Natural Language Processing & Cognitive platforms:
● IBM Watson
● Google Cloud Natural Language API
● Microsoft LUIS
● Amazon Lex
● Baidu UNIT

35. Build Your Own
Conversational
Interface

36. ...with
JavaScript

37. @ girlie_mac
{ REST }
JSSDK
The APIs are
mostly
accessible with
JS

38. @ girlie_mac
Example: API.ai + FB Messenger
const app = require('apiai')(CLIENT_ACCESS_TOKEN);
function sendMessage(event) {
let sender = event.sender.id;
let text = event.message.text;
let ai = app.textRequest(text, {
sessionId: SESSION_STRING });
ai.on('response', (response) => {
// Got a response. Let's POST to Facebook Messenger
});
ai.end();
}
API.ai Node.js SDK

39. @ girlie_mac
Example: IBM Watson + Slack

40. @ girlie_mac

41. @ girlie_mac
Example: IBM Watson + Slack
The bot workflow:
1. Read each message on a Slack channel
2. Send the message to IBM Watson for examination
3. If the likelihood of an emotion is above the given
confidence threshold post the most prominent
emotion

42. @ girlie_mac
Example: IBM Watson + Slack
Slackbot Config:
1. Set up a bot user
2. Set OAuth permission
scopes
3. Subscribe a bot event, so
the bot can post messages
on Slack channels
IBM Bluemix Config for
Watson

43. @ girlie_mac
Example: IBM Watson + Slack
app.post('/events', (req, res) => {
let q = req.body;
if (q.type === 'event_callback') {
if(!q.event.text) return;
analyzeTone(q.event);
}
});
Use Slack Events API to grab the
text when a user post a message
Pass the text data to Watson to analyze
HTTP POST using ExpressJS

44. @ girlie_mac
Example: IBM Watson + Slack
const watson = require('watson-developer-cloud');
let tone_analyzer = watson.tone_analyzer({
username: process.env.WATSON_USERNAME,
password: process.env.WATSON_PASSWORD,
});
const confidencethreshold = 0.55;
tone_analyzer.tone({text: text}, (err, tone) => {
tone.document_tone.tone_categories.forEach((tonecategory) => {
if(tonecategory.category_id === 'emotion_tone'){
tonecategory.tones.forEach((emotion) => {
if(emotion.score >= confidencethreshold) {
postEmotionOnSlackChannel(emotion);
}});
}});
});
Returns
emotions score in
0 to 1
Just initializing it w/
your API credentials
Post the result on Slack

45. @ girlie_mac
Example: IBM Watson + Slack + Raspberry Pi (for fun)
function colorEmotion(emotion) {
if (emotion.tone_id === 'anger') {
setLED(red);
} else if(emotion.tone_id === 'joy') {
setLED(yellow);
} else if(emotion.tone_id === 'fear') {
setLED(purple);
} else if(emotion.tone_id === 'disgust') {
setLED(green);
} else if(emotion.tone_id === 'sadness') {
setLED(blue);
}
} Change the LED color to
match the emotion

46. @ girlie_mac
Ack, this sucks! I want my
money back!
An angry customer detected. Connect
the customer with a human!

47. Conversational
Interface with Voice
in Browser?

48. @ girlie_mac
Hello!
1
2
Howdy
3rd Party NLP API for
artificial conversations
Voice command:
Voice -> Text 3
Text -> Synthetic Voice
Text

49. @ girlie_mac
Web Speech API
Speech Recognition & Speech Synthesis
http://caniuse.com/#feat=speech-recognition
http://caniuse.com/#feat=speech-synthesis
1 3

50. @ girlie_mac
Web Speech API: Speech Recognition
const SpeechRecognition =
window.SpeechRecognition || window.webkitSpeechRecognition;
const recognition = new SpeechRecognition();
Get an instance of the SpeechRecognition,
the controller interface
In the current Chromium, it is still
prefixed

51. @ girlie_mac
Web Speech API: Speech Recognition (Cont’d)
recognition.lang = 'en-US';
recognition.interimResults = false;
recognition.start();
recognition.addEventListener('result', (e) => {
let last = e.results.length - 1;
let text = e.results[last][0].transcript;
});
Some properties
Methods: start(), stop(), abort()
Events: onresult,
onerror, onaudiostarted,
onaudioend, etc.

52. @ girlie_mac
Web Speech API: Speech Synthesis
const synth = window.speechSynthesis;
const utterance = new SpeechSynthesisUtterance();
utterance.text = 'I am a robot';
utterance.pitch = 1.5;
utterance.voice = 'Alex';
synth.speak(utterance);
No vendor prefix
Properties of the
SpeechSynthesisUtterance interface
Get available voices with synth.getVoices()
method

53. @ girlie_mac

54. Demo on Chrome
https://webspeech.herokuapp.com/

55. @ girlie_mac
https://www.smashingmagazine.com/2017/08/ai-chatbot-web-speech-api-node-js/

56. @ girlie_mac

57. The future was here already.
Long Live Clippy!
1997 - 2007

58. @ girlie_mac
謝謝 !
@girlie_mac
girliemac.com
github.com/girliemac
slideshare.net/tomomi

59. @ girlie_mac
Attribution:
Open Emoji by Emoji-One (CC-BY 4.0)