These slides were presented at the CONVERSATIONS 2019 - 3rd International Workshop on Chatbot Research. The preprint of the corresponding paper can be found here: https://conversations2019.files.wordpress.com/2019/11/conversations_2019_paper_22_preprint.pdf

1. An Approach for Ex-Post-Facto Analysis of Knowledge
Graph-Driven Chatbots – the DBpedia Chatbot
Rricha Jalota, Priyansh Trivedi, Gaurav Maheshwari, Axel-Cyrille Ngonga Ngomo, Ricardo
Usbeck
November 20, 2019
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 1 / 17

2. Introduction
Figure: U.S. Chatbot Market by Vertical, 2014 - 2025 (USD Million) 1
1
Source: https://www.grandviewresearch.com/industry-analysis/chatbot-market
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 2 / 17

3. Introduction
Knowledge Graphs and Knowledge Graph-Driven Systems
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 3 / 17

4. Background
Knowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions
- Answer questions related to DBpedia
- Expose the research work being done in DBpedia as product features
- Casual conversation/banter
2
Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbot
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17

5. Background
Knowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions
- Answer questions related to DBpedia
- Expose the research work being done in DBpedia as product features
- Casual conversation/banter
Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) +
domain-agnostic factual questions (using DBpedia KG)
2
Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbot
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17

6. Background
Case Study: The DBpedia Chatbot
Total: 9084 users, 90,800 interactions
Table: Feedback Statistics
Feedback-asked 28953
Feedback-received 7561
Negative-feedback 4155
Figure: Architecture of the DBpedia Chatbot
Check http://chat.dbpedia.org
https://github.com/dbpedia/chatbot
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 5 / 17

7. Objective of the Ex-Post-Facto Analysis
Understand the nature of user-requests
- query-patterns
- user-intentions
Examine whether the chatbot can serve its purpose – satisfy user-requests
Get insights about the conversation flow to improve the chatbot’s architecture
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 6 / 17

8. Approach Overview
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 7 / 17

9. Approach
Request Analysis - Intent Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 8 / 17

10. Approach
Request Analysis - Intent Analysis
Figure: Visualization of clusters obtained via HDBSCAN on sentence embeddings. Each cluster consists
of at least 25 samples. The top 10 clusters out of a total of 33 have been labeled with their top terms.
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 9 / 17

11. Approach
Request Analysis - Complexity of utterances
Complex Query
Example: Can you give me the names of women born in the Country during the 19th century?
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 10 / 17

12. Approach
Request Analysis - Miscellaneous Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 11 / 17

13. Response Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 12 / 17

14. Response Analysis
Entity Types in Utterances prior to Negative Feedback
Figure: Entity type distribution from 1000
manually annotated failed utterances.
Table: spaCy-NER and DBpedia Spotlight
accuracy for detecting person and location
mentions.
System Person Location
spaCy-NER 41.3% 42.2%
DBpedia Spotlight 69.2% 46.1%
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 13 / 17

15. Conversation Analysis
Figure: Topics as identified by DBpedia Spotlight
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 14 / 17

16. Implications for DBpedia Chatbot
Adding support for multilingualism
Smart Suggestions
Detecting implicit feedback and
out-of-scope queries
Knowledge-based QA
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 15 / 17

17. Implications for Knowledge-driven Chatbots
Multilingual Support
Guide User Input
Guiding User Expectations
Adding explainability
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 16 / 17

18. That’s all Folks!
Get in touch:
Rricha Jalota
Data Science Group, Paderborn University
rricha.jalota@uni-paderborn.de
github.com/dice-group/DBpedia-Chatlog-Analysis
Follow us on Twitter: @DiceUPB, @FraunhoferIAIS,
@RrichaJalota
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 17 / 17