Master thesis presentation, Graz University of Technology, May 2020

1. Chatbots for Brand Representation in
Comparison with Traditional Websites
Johannes Kühnel, BSc
May 28, 2020
Institute of Interactive Systems and Data Science
Graz University of Technology, Austria

2. Table of contents
1. Introduction
2. Proof of Concept
3. User Study
4. Conclusion
1

3. Intro

4. What are Chatbots?
Chatbots are computer programs that interact with users us-
ing natural languages. (Shawar and Atwell 2007)
Usually chatbots ...
• provide a service (e.g. bookings)
• use conversational interfaces
• feature simple or more complex AI
2

5. “Bots are the new apps”
Satya Nadella, Microso t CEO
2

6. Motivation
Messaging Platform Monthly Active Users
WhatsApp 1,500
Facebook Messenger 1,300
Weixin / Wechat 1,083
Table 1: Monthly active users of the top 3 messaging platforms in millions.
Sources: Hootsuite and We Are Social (2019) and Nadella (2016) and https://telegram.org/blog/200-million (visited on
2019-02-13).
3

7. Related Work
Exploratory study by Beriault-Poirier, Tep, and Sénécal (2018)
• comparison of websites and chatbots of 3 brands
• participants perform 1 task per brand and platform
• Keypoints: websites offered better user experience, positive
emotions with chatbot
4

8. Related Work
Exploratory study by Beriault-Poirier, Tep, and Sénécal (2018)
• comparison of websites and chatbots of 3 brands
• participants perform 1 task per brand and platform
• Keypoints: websites offered better user experience, positive
emotions with chatbot
Case study by Shawar and Atwell (2015)
• Frequently Asked Questions (FAQ)
• chatbot vs search engine
• Keypoints: more relevant answers, higher preference
4

9. Proof of Concept

10. Theodore, a Company Chatbot
A chatbot — named Theodore — to represent Graz based so tware
developer CodeFlügel was created.
5

11. Theodore, a Company Chatbot
A chatbot — named Theodore — to represent Graz based so tware
developer CodeFlügel was created.
Goal: chatbot capable of representing and informing about the
company (like the existing website)
5

12. Theodore, a Company Chatbot
A chatbot — named Theodore — to represent Graz based so tware
developer CodeFlügel was created.
Goal: chatbot capable of representing and informing about the
company (like the existing website)
Theodore should
• reproduce most of the website’s features
• run on Facebook Messenger and a custom webchat
• re-use existing Application Programming Interfaces (APIs)
5

13. Dialog Design
Chatbot has to provide information about
• the company itself
• products & services
• contact information
• vacancies
• newsletter subscription
• social media & blog posts
6

14. Dialog Design
Chatbot has to provide information about
• the company itself
• products & services
• contact information
• vacancies
• newsletter subscription
• social media & blog posts
Additional requirements include
• help function & fallback mechanism
• informal language
• aware of being a bot
6

15. Technologies
Backend
Languages & Frameworks: Node.js with Express.js and Socket.IO
APIs: Facebook, Dialogflow, Mailchimp, Wordpress 1
Webchat
Languages & Frameworks: Angular (using TypeScript) with Socket.IO
APIs: Backend 2
1connection via REST API / HTTP requests
2connection via WebSockets
7

16. Message Format
Facebook Messenger’s message format (JSON) used for both
Messenger and webchat.
Support for various components:
• Generic Template
• List
• Button
• Media
• Quick Replies
8

17. Natural Language Understanding
Natural Language Understanding (NLU): process of “understanding”
natural language in computer science
Intents: the meaning or purpose of the user input
Parameters (entities): terms tied to the intents (e.g. context, amount,
time etc.)
9

18. Natural Language Understanding
Natural Language Understanding (NLU): process of “understanding”
natural language in computer science
Intents: the meaning or purpose of the user input
Parameters (entities): terms tied to the intents (e.g. context, amount,
time etc.)
Figure 1: Dialogflow Basics — Intents
Source: https://cloud.google.com/dialogflow/docs/basics
9

19. Natural Language Understanding cont.
• initial design was framework agnostic
• looked at several services/frameworks for NLU
• Dialogflow (Google), Wit.ai (Facebook), LUIS (Microso t), Watson
Assistant (IBM), Amazon Lex
10

20. Natural Language Understanding cont.
• initial design was framework agnostic
• looked at several services/frameworks for NLU
• Dialogflow (Google), Wit.ai (Facebook), LUIS (Microso t), Watson
Assistant (IBM), Amazon Lex
• final implementation only for Dialogflow
• scope of thesis / PoC
• differences in framework/service concepts
• pricing
10

21. Bot Frameworks
Evaluated bot development frameworks:
• Microso t Bot Framework
• Botkit
• Botmaster.ai
• Botpress
• BotMan
11

22. Bot Frameworks
Evaluated bot development frameworks:
• Microso t Bot Framework
• Botkit
• Botmaster.ai
• Botpress
• BotMan
Disadvantages include:
• reliance on third party services, higher latencies
• only one platform per instance
• slow development
• limited content handling
• poor platform support
11

23. Bot Frameworks
Evaluated bot development frameworks:
• Microso t Bot Framework
• Botkit
• Botmaster.ai
• Botpress
• BotMan
Disadvantages include:
• reliance on third party services, higher latencies
• only one platform per instance
• slow development
• limited content handling
• poor platform support
Due to these shortcomings, a custom implementation was chosen.
11

24. Architecture
Figure 2: Request – Response Architecture
12

25. UI Components
Figure 3: Blog post components in the webchat and Facebook Messenger.
13

26. UI Components cont.
Figure 4: Quick Replies in the webchat and Facebook Messenger.
14

27. UI Components cont.
Figure 5: Generic Template components showing products in the webchat
and Facebook Messenger.
15

28. User Study

29. User Study
• comparison of chatbot and website
• identify advantages and weaknesses
• selection based on relevance to the company
• 3 groups
• (potential) clients
• potential employees
• blog readers and everyone else
• 20 participants (at least 5 per group)
16

30. User Study cont.
• pre- & post-test questionnaires
• users perform 9 typical tasks
• e.g. “Find out where CodeFlügel’s office is located and how to call
them.”
• each with the chatbot and website
• interview a terwards
Figure 6: Test Setup
17

31. Results cont.
Figure 7: Average time per task
18

32. Results
Figure 8: Average total time and average time for tasks
19

33. Results cont.
Figure 9: What users say about the company’s chatbot and website
20

34. Results cont.
Figure 10: Which platform was more appealing to the users
21

35. Results cont.
Figure 11: 95% would use more chatbots
22

36. Conclusion

37. Conclusion
• positive feedback about the chatbot
• train for whole sentences and keywords
• chatbot more entertaining
• high acceptance rate
• chatbot was faster (specific information)
23

38. Conclusion
• positive feedback about the chatbot
• train for whole sentences and keywords
• chatbot more entertaining
• high acceptance rate
• chatbot was faster (specific information)
• menus offer quick navigation
• minor issues with intent matching
• exploration better with website
23

39. Future Work
• improve intent matching
• broader audience / more participants
• different chatbots and personas
• brand perception
• conversion rates and other Key Performance Indicators (KPIs)
24

40. Live Demo
URL: https://theodore.kuehnel.co.at
24

41. Questions?
24

42. Website
Figure 12: Screenshot of the company homepage, taken on 2018-11-04.

43. Webchat cont.
Figure 13: List, contact and typing indicator components in the webchat.

44. Facebook Messenger cont.
Figure 14: List, contact and typing indicator components in Facebook
Messenger.

45. Message Format - Details
{
"message" : {
"attachment" : {
"type" : "template" ,
"payload" : {
"template_type" : "generic" ,
"elements" : [
{
"title" : "<TITLE_TEXT>" ,
"image_url" : "<IMAGE_URL_TO_DISPLAY>" ,
"subtitle" : "<SUBTITLE_TEXT>" ,
"default_action" : {
"type" : "web_url" ,
"url" : "<DEFAULT_URL_TO_OPEN>" ,
"messenger_extensions" : <TRUE | FALSE > ,
"webview_height_ratio" : "<COMPACT | TALL | FULL>"
} ,
"buttons" : [ < BUTTON_OBJECT > , . . . ]
} ,
. . .
]
}
}
}
}
Listing 1: A generic template JSON message in Facebook Messenger’s
format.

46. Task List
1. Find out what CodeFlügel does or which services they provide.
2. Find out where CodeFlügel’s office is located and their phone
number.
3. Find out which companies CodeFlügel has already implemented
projects for.
4. Find and open the latest blog entry.
5. Sign up for the newsletter with the e-mail address
<firstname>.<surname>@codefluegel.com.
6. Find out if and which jobs are currently available.
7. Find a way to try Augmented Reality (AR) for yourself.
8. Find at least one Augmented Reality (AR) project created by
CodeFlügel.
9. Find out what Augmented Reality (AR) actually is.

47. Participant Details
• age from 20–33, averaging 27.7
• 60% with bachelor degree
• 45% studying (bachelor or master’s program)
• 70% with technical background (mostly IT)
• mostly male

48. Results cont.
Figure 15: What users expect of a company website and chatbot

49. Results cont.
Figure 16: The chart shows how the users communicated with the chatbot

50. Results cont.
Figure 17: Preferred type of speech of the chatbot

51. Results cont.
Figure 18: Total time needed to complete the tasks (per User)

52. References i
Beriault-Poirier, Amélie, Sandrine Prom Tep, and Sylvain Sénécal (Oct.
2018). “Putting Chatbots to the Test: Does the User Experience
Score Higher with Chatbots Than Websites?” In: Human Systems
Engineering and Design. Springer International Publishing,
pp. 204–212. isbn: 978-3-030-02053-8. doi:
10.1007/978-3-030-02053-8_32.
Hootsuite and We Are Social (2019). Digital 2019. Global Digital
Overview. Tech. rep. Mindbowser. url:
https://datareportal.com/reports/digital-2019-
global-digital-overview (visited on 02/04/2019).
Nadella, Satya (Mar. 25, 2016). Build 2016. Keynote Presentation.
Microso t. url: https://channel9.msdn.com/Events/
Build/2016/KEY01#time=1h41m11s (visited on 02/13/2019).

53. References ii
Shawar, Bayan Abu and Eric Atwell (2007). “Chatbots: Are They Really
Useful?” In: LDV-Forum 22.1, pp. 29–49. url:
https://jlcl.org/content/2-allissues/20-Heft1-
2007/Bayan_Abu-Shawar_and_Eric_Atwell.pdf (visited on
01/24/2019).
– (Sept. 2015). “A chatbot as a Question Answering Tool”. In: 2015
International Conference on Advances in So tware, Control and
Mechanical Engineering. 2015 International Conference on
Advances in So tware, Control and Mechanical Engineering, pp. 1–6.
isbn: 978-93-84422-37-0. doi: 10.17758/UR.U0915120.