This presentation introduces Content Wizard, a concept-based recommender system to help instructors of programming courses build intelligent tutoring systems. Content Wizard analyzes code examples to extract concepts and generates a course model. It then recommends annotated examples and parameterized problems for each course unit based on whether concepts are past, current or future. An evaluation compared Content Wizard's recommendations to a baseline method and found it performed better, especially for annotated examples. Future work includes online experiments and improving transparency of recommendations.

1. PRESENTATION
Content Wizard: Concept-Based Recommender System for
Instructors of Programming Courses
UMAP’17, July, 09, 2017
Hung Chau, Jordan Barria-Pineda and Peter Brusilovsky
School of Information Sciences
University of Pittsburgh
Pittsburgh, PA, USA

2. Introduction
1
• Intelligent tutoring Systems (ITS) help students to find an “optimal
path” to achieve their learning goals
• The current generation of adaptive learning systems rarely support
an adaptation to a teacher’s preferences
• Course authoring tools are needed to support non-computer
instructors in creating an effective IST

3. Context
2
Introduction to Java
Programming Course
Mastery Grids [1]
[1] Julio Guerra, Roya Hosseini, Sibel Somyurek, and Peter Brusilovsky. 2016. An Intelligent Interface for Learning Content: Combining an Open Learner
Model and Social Comparison to Support Self-Regulated Learning and Engagement

4. Context
3
• The quality of recommendation and navigation support depends on the
quality of the course structure
• We already developed an authoring tool that provides some level of
support in defining the course structure and assigning learning content
• However, the basic level of support of the tool was not sufficient.
• We propose Content Wizard which augments the course authoring tool
with a concept-based recommender system
• decrease the effort required to build an ITS
• facilitate the task of maintaining a coherent sequential structure

5. Course generation
4
Content Wizard workflow diagram

6. 5
Course generation
Concepts extracted by the Java parser [2]: ArrayCreationStatement,
ArrayDataType, ArrayInitializer, DoubleDataType, IntDataType, MethodDefinition, etc.
[2] Roya Hosseini and Peter Brusilovsky. 2013. JavaParser: A Fine-Grain Concept Indexing Tool for Java Problems
Course example

7. 6http://adapt2.sis.pitt.edu/quizjet/
Course generation
Parameterized problem

8. 7
Course generation
Annotated example

9. Course generation
8
Kc_n_1 Kc_n_2
Kc_1_1 Kc_1_2 Kc_1_m1
CourseUnit1CourseUnit2CourseUnitn
Kc_n_mn
…
…
Kc_2_1 Kc_2_2 Kc_2_m2…
. . .
Course model

10. Course generation
9
Content Wizard workflow diagram

11. Course generation
10
Mastery Grids student interface
(a) Concept visualization of a specific course unit (b) Concept visualization of a specific learning activity

12. Course generation
11
Content Wizard workflow diagram

13. 12
Recommendation algorithm
• Content Wizard adaptively provides two valuable sources of information: a ranking
list and warning flags
• Based on the course model and the concepts extracted from the code examples of
the current unit, all the concepts in the ontology are classified into three categories:
• Past concepts (P)
• Current concepts (C)
• Future concepts (F)
• The ranking score of each learning activity ai
α, β, and γ are three parameters controlling the importance of each category
(2)
(1)

14. 13
Course Authoring tool Interface

15. 14
Evaluation
• We use data from two Java classes taught in SIS, University of
Pittsburgh in Fall 2016 (referred as IS17F16)
• a lecture-based format
• a course structure includes 18 units
• To run the study:
1. collected the code examples provided by the teachers
2. created a new course with the same structure
3. provided the corresponding code examples to get the recommended
contents for each unit
4. compare them to the ones the instructors selected for the IS17F16
course

16. 15
Evaluation
• Baseline: adopted TF-IDF method
• Run the same process to get a set of recommended items
• Treat each activity as a list of concepts extracted by the same parser
• Calculate the TF-IDF weighting for each concept of the activity
• Metrics: precision, recall, and F1 score (at top 3, top 5, top 10, and
top 15)

17. 16
Results
Performance comparison of the Content Wizard and the baseline
(a) Annotated Examples
(b) Parameterized Problems

18. 17
Discussion & Future work
vLimitations
• Some concepts are more important than others in the same learning item
• The lack of knowledge about the relationship between similar concepts clearly
affects system performance
For example, PostIncrementExpression (+=) and PostDecrementExpression (-=))
v Future work
• Run an online experiment
• Using Programming courses on other systems to test our approach
• Add features that contribute to the transparency of the recommendation
process and perform a more extensive evaluation of the new tool

19. THANK YOU!!!
Hung Kim Chau
University of Pittsburgh
School of Information Science
135 N Bellefield Ave, Pittsburgh, PA, 15260
Phone: (412) 624-5230