1. The document presents an educational question routing framework for online student communities that recommends questions to users suitable to answer them based on their expertise, willingness, and workload. 2. An experiment applying this framework to a quantum cryptography MOOC with over 4,600 students showed it increased accurate recommendations, community participation in answering questions, and average user contributions compared to a baseline method. 3. The framework incorporated additional data from the MOOC beyond community question answering activity and its effectiveness was demonstrated in a real online experiment over 7 weeks with the MOOC students.

1. Educational Question Routing in
Online Student Communities
Jakub Macina Slovak University of Technology
Ivan Srba Slovak University of Technology
Joseph Jay Williams Harvard University / National University of Singapore
Maria Bielikova Slovak University of Technology
11th ACM Conference on Recommender Systems, Como, Italy, 27th-31st August 2017

2. 2 / 31
Online Student Communities
• Massive Open Online Courses (MOOCs)
Dropout rate up to 94%
Community Question Answering
(CQA)
Discussion forum

3. 3 / 31
Challenge for MOOC Discussions
• Up to 50% of unanswered questions
• Course instructors are overloaded with many
students to serve
• Low participation of students in question answering
1. Lurkers who are not contributing
2. Willing to participate but overloaded with many
questions

4. 4 / 31
Our Idea: Question Routing
• Recommendation of new questions to users who
are suitable to answer them
• Well-known research task from CQA systems
What is capital
city of Italy?

5. 5 / 31
Related Work
• Question routing in standard CQA
• Asker-oriented approaches
• Overloading small group of experts
• Based mainly on QA data

6. 6 / 31
Related Work
• Question routing in standard CQA
• Asker-oriented approaches
• Overloading small group of experts
• Based mainly on QA data
• Question recommendation in MOOCs
• Constraints optimization framework (Yang et al 2014)
• Any question beneficial to user
• With significant time-delay
Not appropriate
for MOOCs

7. 7 / 31
Educational Question Routing

8. 8 / 31
Educational Question Routing Task
Given new question 𝑞
find an ordered list of users 𝑢1, … , 𝑢 𝑛
who are most suitable to answer question 𝑞
Opportunities
Data from MOOC course
(grades, accomplished exercises)
Constraints
Appropriate knowledge
Willingness to answer
Working capacity

9. 9 / 31
Goals of Educational Question
Routing
• G1: Decrease information load of users by accurate
recommendations
• G2: Engage a greater part of the community in the
question answering
• G3: Increase an average number of contributions

10. 10 / 31
Educational Question Routing
Framework

11. 11 / 31
Educational Question Routing
Framework
User modeling phase

12. 12 / 31
Educational Question Routing
Framework
Routing phase

13. 13 / 31
1. Construction of Question Profile
• Question text profile 𝜃 𝑞
• Captures question’s content
• Text pre-processing
• Bag-of-words model (tf-idf weights)
• Metadata
• Asker, category, etc.

14. 14 / 31
2. Construction of User Profile
• User text profile 𝜃 𝑢
• Captures topics of question user previously answered
(user’s interests)
𝜃 𝑢 = ෍
𝑞𝜖𝑄 𝑢
(𝜃 𝑞 + 𝜃 𝑎,𝑞)
• Metadata about previous user activities
• Quantity, quality and time distribution
• In CQA and MOOC

15. 15 / 31
3. Matching of Questions and Users
• Ranking of users given new question
• Ensemble of two classification tasks:
• Appropriate expertise to answer a new question
• Willingness to answer a new question
• Combination:
𝑃(𝑦 = 1) = 𝑃(𝑒𝑥𝑝𝑒𝑟𝑡𝑖𝑠𝑒 = 1) ∗ 𝑃(𝑤𝑖𝑙𝑙𝑖𝑛𝑔𝑛𝑒𝑠𝑠 = 1)

16. 16 / 31
3. Matching of Questions and Users
• Features derived from text and metadata comparison between
question and user profile
• Features for expertise classification (# of features = 11)
• Level of difficulty for a user to answer a new question - knowledge
gap
• Portion of related lectures watched
• Grades
• Features for willingness classification (# of features = 14)
• Overall count of answers, questions and comments
• Amount of latest activity
• Response time on rec.

17. 17 / 31
4. Optimization
• Balancing routed questions by considering current
student’s workload 𝐿 𝑢

18. 18 / 31
Experiments – CQA system
• Educational and organizational CQA system Askalot
• Open source, developed at Slovak University of Technology
• Builds on diversity in students’ knowledge and
educational/organizational specifics
• University/MOOC variant
github.com/AskalotCQA/askalot askalot@fiit.stuba.sk

19. 19 / 31
Experiments - MOOC
• QuCryptox Quantum Cryptography at edX
• Offered by Caltech and TU Delft
• 10 weeks (Sept. 2016 – Dec. 2016)
https://courses.edx.org/courses/course-v1:CaltechDelftX+QuCryptox+3T2016

20. 20 / 31
Course Statistics
Metric Quantity
Students enrolled in the course 8115
Students started the course 4618
Users participating in CQA (contributors + lurkers) 1098 (24%)
Users contributing in CQA 377 (8%)
Questions 361
Answers 386
Comments 476

21. 21 / 31
Evaluation Methodology
• Offline experiment
• Online experiment
• Very rare in context of CQA systems
• Ecologically valid
• Total impact on student community
• Baseline: non-educational asker-oriented question
routing method with optimization

22. 22 / 31
Offline Experiment Setup
• Standard ML pipeline including:
• Feature transformation
• Feature selection
• Chi square selection
• Model selection
• SVM, Random forest, Logistic regression
• Hyper-parameter tuning

23. 23 / 31
Offline Experiment Results
• Comparison with actual answerers of a question

24. 24 / 31
Online Experiment Setup
• A/B test during 7 weeks
• Stratified random assignment to three groups:
1. Educational (n=1306)
2. Baseline (n=1306)
3. Control (n=1306)
• Recommendation to top 10 users
• Constraint for workload 𝐿 𝑢
• maximum 4 recommendations per 7 days
• Real-time profile updates, re-training each day

25. 25 / 31
Online Experiment Setup
Notification
Dashboard

26. 26 / 31
Online Experiment Results
• 132 new questions were routed to potential
answerers
• Resulting in 2640 recommendations

27. 27 / 31
G1: Accurate Recommendations
Decreased Information Load
Metric Our method Baseline Statistical significance
CTR 23.25% 18.29% 𝜒2 1, 𝑁 = 2640 = 10.03, 𝑝 < 0.01
Success@10 15.91% 10.61% 𝜒2
1, 𝑁 = 264 = 1.61, 𝑝 = 0.20

28. 28 / 31
G2: Greater Part of the Community
Got Involved
Period Our method Baseline Control
Before 7.60% (62/816) 8.99% (73/812) 9.12% (74/811)
During 13.16% (40/304) 9.35% (26/278) 8.72% (28/321)
Active CQA users / active MOOC users

29. 29 / 31
G3: Average Number of
Contributions Increased
Before experiment During experiment
(with recommendation)

30. 30 / 31
Possible Improvements
• Duplicate questions identification
• Question retrieval (another well-known task in CQA)
• Question type identification
• Some questions can be answered only by instructors
• Scalability

31. 31 / 31
Educational Question Routing in
Online Student Communities
1. Answerer-oriented question routing framework
considering not only expertise, but also
willingness and workload of answerers
2. Incorporating additional MOOC data beyond CQA
activity
3. Effectiveness in real world is demonstrated by
online experiment with more than 4600 MOOC
students. Code available at:
https://github.com/dmacjam/dp-analysis-evaluation