ITS 2015 (YRT)

1. Example-Based Problem Solving Support Using
Concept Analysis of Programming Content
Roya Hosseini1 and Peter Brusilovsky1,2
1 Intelligent Systems Program (ISP)
2 School of Information Sciences
University of Pittsburgh
12th International Conference on Intelligent Tutoring Systems
Young Researchers Track
June 8, 2014
University of Pittsburgh Intelligent Systems Program

2. Purpose
 Select relevant examples for:
 Supporting problem solving in Java Programming
 Introduce two concept-based approach for:
 Finding similar examples to a question
2

3. Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab Study
3

4. Example-based Problem Solving
 Helps students in solving problems
 Finds most relevant examples for problems
4

5. ELM-ART
 ELM-ART: an ITS for LISP programming (Weber 1996)
 Required advance analysis for:
 Task Description
 Domain Knowledge
 Learner Model
5

6. Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab study
6

7. Concept-based Similarity
 Uses a Standard Ontology
7
 Extracts concepts in
programming content
(Hosseini & Brusilovsky 2013)
 Measures similarity of
concepts in contents
is-a
Abstraction Inheritance Encapsulation
Overriding
Method
Inheritance
Field
Inheritance
Overriding
Equals
Overriding
Hash Code
is-a is-a
is-a
is-a
is-a is-a

8. Example
8
public class Tester {
public static void main (String[] args)
{
int x = 0;
for (int i = 0; i < 10; i++){
for (int j = 0; j < 10; j++){
…
}
}
}
}
for (int i = 0; i < 10; i++){
…
}
for (int j = 0; j < 10; j++){
…
}
Global
for (int i = 0; i < 10; i++){
for (int j = 0; j < 10; j++){
…
}
}
Local

9. Cosine Similarity
Question vector
Example vector
9
Global Similarity
Wn-1 WnW2 ...W1
Cn-1 CnC2 ...C1
Wn-1 WnW2 ...W1
Cn-1 CnC2 ...C1

10. Local Similarity
 #1: Forms a subtree from concepts
in the same block
10
if
++<
If ( x < 2 ) {
x++;
}

11. Local Similarity
 #2: Compares subtrees of question and example
 Tree Edit Distance (TED)
11
Example
a
cb
e
if
Question
e
gf
a
cb d
a
cb d
a
cb
e
if
1 2
TED: 3
e
gf

12. Local Similarity
 #3: Measures similarity based on TED
12
Example
a
cb
e
if
a
hd
e
gh i
i
lk m n
Example
Question
e
gf
a
cb d
TED: 3
TED: 6

13. 13
Global vs. Local
We Are Globally
Similar!

14. Outline
 Literature Review
 Concept-based Similarity Approach
 Evaluation: Lab study
14

15. Study Design
Time Rating #Example Approach
Failing in question Optional 5 Random
End of question Mandatory 4 Both
15

16.  Date: January, 2014
 12 students
 Java Contents:
 6 topics
 83 annotated examples
 24 parametric questions
16

17. Task
 Pretest
 Solving 4 questions in 3 Java topics
 Rating helpfulness of examples
 ‘Not helpful at all’(0) - ‘Very helpful’(3)
 Post-test
17

18. Solving question
18

19. Examples & Rating
19

20. Results
20
1.95
1.49
0
1
2
3
GLOBAL LOCAL
Average Users Ratings

21. Rating Evaluation
21
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.29 0.32
0
0.5
1
Global Local
Average RMSE

22. Rating Evaluation
22
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.69 0.62
0
0.5
1
Global Local
Average Precision 2+

23. Rating Evaluation
23
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.28 0.23
0
0.5
1
Global Local
Average Precision 3

24. Rating Evaluation
24
Work in
progress
 RMSE
 Precision 2+
 Precision 3
 MRR
0.7
0.61
0
0.5
1
Global Local
Average MRR

25. 25
Work in
progress
0
0.2
0.4
0.6
0.8
1
RMSE Precision 2+ Precision 3 MRR
Ratings over Difficulty Level of Question
Global Easy Local Easy
Global Moderate Local Moderate

26. HOW-TOs
26
Work in
progress
 #1: Define structure of the content?
 #2: Do personalized example selection?

27. 27
Work in
progress
 Concepts that appear together in a:
 Block/line
#1: Define structure of the content?
if
++<
If ( x < 2 )
{
x++;
}

28. 28
Work in
progress
 Consider user knowledge information
 Select examples with:
 least unknown parts
 enough new parts
#2: Do personalized example selection?

29. 29
Work in
progress
 Lab study : 30 subjects
 Personalized & non-personalized Global-Local approach
 Personalized example selection did not work out!
 Hard question: Local approach has the least RMSE
 Easy-Medium question: Global approach has the least
RMSE
Concept-based
Similarity
User Knowledge
Level
Personalized
Example-Selection

30. Discussion
 Global & Local Concept-based approach:
 Generalizability across other programming domains
 Limitations:
 Few contents
 Few subjects
30

31. Next Steps
 Investigating:
 learning gains of students in the study
 other approaches for capturing content structure
 personalized example selection
○ user knowledge, …
 Adaptive visualization of problem-example
space
31

32. Thank You!
Personalized Adaptive Web Systems
School of Information Sciences
University of Pittsburgh
Roya Hosseini (roh38@pitt.edu)
http://people.cs.pitt.edu/~hosseini/