The goal of the current study is to develop an algorithm for generating pedagogically valuable questions. We focus on verbatim questions whose an- swer, by definition, can be literally identified in a source text. We assume that an important keyphrase relative to a specific learning objective can be identified in a given source text. We then further hypothesize that a pedagogically valuable verbatim question can be generated by converting the source text into a question for which the keyphrase becomes an answer. We therefore propose a model that identifies a keyphrase in a given source text with a linked learning objective. The tagged source text is then converted into a question using an existing model of question generation, QG-Net. An evaluation study was conducted with existing authentic online course materials. Corresponding course instructors judged 66% of the predicted keyphrases were suitable for the given learning objective. The results also showed that 82% of the questions generated by pre-trained QG-Net were judged as pedagogically valuable.

1. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Pedagogical Question Generation towards
Automatic Online Courseware Engineering
Machi Shimmei, Noboru Matsuda
Dept of Computer Science
North Carolina State University

2. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Introduction
• Learning by doing on MOOC
– Answering questions facilitates learning when
combined with watching videos
• Test-enhanced learning
– Answering questions (any kind) facilitate learning
• Spaced repetition
– Repetitive exposure to answering questions
strengthen learning
2
7/27/22

3. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Gap and Need
• Creating questions is challenging
• Creating effective questions is even more
challenging
• A pragmatic technology for automated
question generation is desired
3
7/27/22

4. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Solution: QUADL
• QUestion generation with an Application of
Deep Learning
• Goal
– Given a didactic text and a learning objective,
generate a question that is suitable to achieve
the learning objective.
• Type of question: Verbatim question
– The answer can be found in the didactic text
4
7/27/22

5. INNOVATIVE EDUCATIONAL
COMPUTING LAB
QUADL: Example
• Learning objective (LO): Describe metabolic pathways as
stepwise chemical transformations either requiring or releasing
energy; and recognize conserved themes in these pathways.
• Sentence (S): Among the main pathways of the cell are
photosynthesis and cellular respiration, although there are a
variety of alternative pathways such as fermentation.
• Question (Q): Along with photosynthesis , what are the main
pathways of the cell ?
– Answer: cellular respiration
7/27/22 5

6. INNOVATIVE EDUCATIONAL
COMPUTING LAB
QUADL: Architecture
6
Sentence (S)
Learning
Objective (LO)
Answer
Prediction
Verbatim
Question (Q)
S <Index_start, Index_end> (S is suitable with a predicted answer)
S <0, 0> (S is not suitable for LO)
Question
Conversion
QG-net
BERT
= Non-Target Sentence
= Target sentence
7/27/22

7. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Generating Training Data from OLI
• Made exhaustive combination of LO and
corresponding S
• Instructors tagged answer index for randomly
sampled 395: <LO, S<Is, Ie>>
– 30% of the pairs are not-target (i.e.,<Is=0, Ie=0>)
• 345 pairs for training; 50 pairs for testing
7
7/27/22

8. INNOVATIVE EDUCATIONAL
COMPUTING LAB
QUADL: Preliminary Evaluation
• Trained Answer Prediction model on data
from Open Learning Initiative
• Used pre-trained QG-Net for Question
Conversion
• Had Amazon Mechanical Turkers (AMT) /
instructors forecast the usefulness of
generated questions
8
7/27/22

9. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Measure:
• Participants judged <LO, S<Is, Ie>, Q>
(1) Is it adequate to convert sentence S into a
question whose answer is token <Is, Ie>
to attain the learning objective LO?
(2) Is the question Q suitable for attaining the
learning objectives LO?
9
7/27/22

10. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Results: Answer Prediction (by AMT)
• Is it adequate (not) to convert S<Is, Ie> into Q?
10
7/27/22

11. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Results: Question Conversion
11
All S with <Is, Ie> output from the
Answer Prediction model
• Does Q help students attain the learning goal?
7/27/22

12. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Results: Question Conversion
12
All S with <Is, Ie> output from the
Answer Prediction model
Only S with <Is, Ie> judged as
adequate by participants
• Does Q help students attain the learning goal?
7/27/22

13. INNOVATIVE EDUCATIONAL
COMPUTING LAB
Examples
7/27/22 13
A participant judged both a target term and a question were adequate :
A participant judged a target term was adequate, but the question was not adequate:

14. INNOVATIVE EDUCATIONAL
COMPUTING LAB
QUADL: Future Work
• We proposed QUADL for generating questions that are aligned with the
given learning objective. As far as we are aware, there have been no
studies that aim to generate questions that are suitable for attaining the
learning objectives.
• Improve model performance
– Alternative Question Conversion model
– Extremely low data regime
• Efficacy study
– Measure learning outcome with authentic students @ OLI
14
7/27/22