Cross-System Personalization for College Students

Cross-System
Personalization for
College Students
Peter Brusilovsky with
Sergey Sosnovsky
Michael Yudelson
Shaghayegh Sahebi
Chirayu Wongchokprasitti
Sharon Hsiao

User-Adaptive Systems Then…

Collects information
about individual user User Modeling side

Adaptive
System User Model

Adaptation side
Provides
adaptation effect

Classic loop user modeling - adaptation in adaptive systems

University of Pittsburgh - PAWS Lab

… and now


Personalization Challenges in the New
Context
•  How an adaptive system can benefit from
information about users collected by other
systems?
–  What is the framework for UM integration?
–  Will it improve cold-start situation?
–  Will it improve parallel use of multiple systems
•  Can we do it for different types of user models?
–  Knowledge model
–  Interest model
University of Pittsburgh - PAWS Lab 4

Concept-Level Knowledge Model

Concept 4

Concept 1

3

10

Concept N

Concept 2

0

7

2

4

Concept 5

Concept 3


Cross-System Knowledge Modeling
Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

http://adapt2.sis.pitt.edu/kt/

The Approach: Ontology-Based Cross-
System Personalization

Connect DM
(ontologies)

Missing links


Main Stages of Our Work

•  Centralized user modeling (1990-1998)
•  Multi-system personalization based on single
domain model: ADAPT2 (2003-2007)
•  Cross-domain mapping for cold start (2007)
–  C to Java
•  Single domain guided evidence mapping
(2008-2010)
–  Topic to concept mapping for Java
–  Constraints to concepts mapping for SQL
•  Single domain automatic mapping (2010-2012)

How we started – from C to Java
•  Manual vs. Automatic
ontology mapping
•  Knowledge mapping using
ontology mapping Java
•  Compare predicted and C
demonstrated knowledge
•  Automatic mapping is
comparable with manual
•  Overall gain for translated
knowledge is not high UM of C UM of
•  We got concerned about knowledge Java
model to model mapping knowledge
•  Started exploring evidence
mapping

SEDONA: UM exchange with ontology
servers Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 3

Concept 5

Ontology A

Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

Ontology B
Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

University of Pittsburgh - PAWS Lab http://adapt2.sis.pitt.edu/kt/

SEDONA: UM Exchange

•  Ontology server is an exchange point for concept-
level overlay student models that are based on the
stored ontology
•  Each UM server or adaptive system that can deduce
student knowledge in terms of this ontology reports
it to the server
•  Each adaptive system that need to know the level of
student knowledge for concepts of this ontology can
query the ontology server


Lightweight event-based centralized
user modeling Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

Concept 4

Concept 1

no

yes

Concept N

Concept 4

no

Central UM
Concept 1

no

Concept 2

yes

Concept N

yes

no

no

Concept 2

yes

yes

Concept 5

no

Concept 3

yes

Concept 5

Concept 3

Concept 4

Concept 1

no

yes

Concept N

no

Concept 2

yes

no

yes

Concept 5

Concept 3

University of Pittsburgh - PAWS Lab http://adapt2.sis.pitt.edu/kt/

Goal: True Integration
•  Student side:
– Use systems in parallel (any order, any
combination)
– No extra overhead (single sign-on, single
place to access)
•  System side:
– Integrated environment > (system1 +
system2)
– Each system should try to increase the
quality of user modeling and adaptation

Java Problets: The Interface

Help
Question
text

Sample
program
System s
feedback
Student s
answer

Java Problets: Domain Model
•  Problets implement traditional overlay user modeling to
adapt to student s performance
l  The domain model
of a problet is a
concept map
enhanced with
learning objectives,
that combine
pedagogical and
domain knowledge

QuizJET (1):
System Description
•  QuizJet (Java Evaluation Toolkit) is a system for authoring
and delivery of online self-assessment quizzes for Java
programming language
•  A typical QuizJET problem is a sample program (consisting
of one or several classes), that a student needs to evaluate
and provide an answer a follow-up question
•  QuizJET generates problems by substituting a numerical
value in the program template with a randomized
parameter
•  Upon receiving a student s answer QuizJET provides a
feedback indicating the correctness of the answer and the
right answer (if the student s attempt was not successful)

QuizJET (2):
Student Interface
•  Students can access QuizJET problems through the
KnowledgeTree portal

Problem's
Topics in the classes
course

Problem
Activities text
available for the
current topic
QuizJET s
feedback

QuizJET (3): Domain Model
•  Java Ontology
specifies about 500
classes connected
with 3 types of
relations: subClassOf,
partOf/hasPart, and
related
•  About 300 classes are
available for indexing
•  A class can play one of
two roles in the problem
index: prerequisite or
outcome

Domain Model Integration
•  Main problem: different modeling paradigms
–  A learning objective models application of a concepts in the certain
context
–  Extra classes from the Java ontology have been used for context
modeling
–  Weights are assigned to prevent too aggressive propagation of
classes responsible for context modeling
•  Example:
–  This learning objective models a situation when the conditional part
of the if-else statement is a relational expression evaluated into true
value

Evidence-based UM integration in
CUMULATE


SQL-Exploratorium


SQL Explorer: SQL Ontology

http://www.sis.pitt.edu/~paws/ont/SQL.owl

SQL-Tutor: Constraints


Domain Model Mapping
•  Constraints and Concepts are too difficult
to map them automatically
•  A typical constraint models syntactic or
semantic relation between several concepts
•  Manual connect constraint to concepts
with some
degree (small-1,
medium-2, or large-3)

Evidence-Based Modeling
•  Solution to SQL-Tutor problem, triggers a
number of constraints satisfied and or
violated
•  Mapping model calculates knowledge
update for every concepts related to every
triggered constrained:

•  The updates are reported to SQL-
Exploratorium s user modeling server

Evaluation
•  University of Pittsburgh,
2 courses: undergraduate and graduate
•  ½ of semester
•  42 students tried SQL-KnoT, 18 – SQL-
Tutor
•  Out of 103 sessions of using SQL-KnoT
66 co-located with SQL-Tutor usage


Results

•  Questionnaire (21 students)
– I1 / I2: Overall, I like the interface of SQL-
KnoT/SQL-Tutor.
– U1 / U2: SQL-KnoT/SQL-Tutor is a useful
learning tool.
– C1 / C2: SQL-KnoT/SQL-Tutor problems
challenged me intellectually.

What was presented in the past UbiqUM
•  An example of semantic integration of two working
adaptive systems relaying on very different domain
models
•  Students used the systems together during single
sessions and liked the opportunity
•  More evaluation is needed to verify the effect of
integration of user modeling accuracy and adaptation
•  It is interesting to evaluate the combined adaption
(adaptive navigation from SQL-Exploratorium
followed by intelligent coaching from SQL-Tutor)

Discussion

+ Experts only need to produce relations b/w KIs
– the rest is automatic
+ Relations can be removed (strength=0)
-  Cannot add relations


References: Past UbiqUM papers
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and Sharma, D.
(2008) Towards integration of adaptive educational systems: mapping domain models to
ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-
Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based
integration of adaptive educational systems. Proceedings of 16th International Conference on
Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.
Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model
Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User
Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive
Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.
Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V.
(2009) Database exploratorium: a semantically integrated adaptive educational system. In:
Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User
Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009
Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009)
Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F.
Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture
Notes in Computer Science, Vol. 5830, pp. 134-158

Evaluating and improving mapping:
SQL Exploratorium and SQL Tutor
•  Authoring constraint mapping is time consuming
•  How we can evaluate weights?
•  How we can improve mapping?
Constraints Concepts
w=1 Join
207 "You need to specify the join w=2/3
condition in FROM!" FROM Clause
w=2/3 Attribute
147 "You have used some names
w=2/3
in the WHERE clause that are not Table
w=1/3
from this database." Database


SQL KnoT and SQL-Tutor (2)
•  6 experts (2 teachers, 2 GSA, 2 practitioners)
•  1012 constraint-concept relations: strong (1/1),
medium (2/3), weak (1/3)
•  Usage log of 3544 SQL-Tutor problem-solving
attempts of 38 users
•  Dataset specific subset
–  282 constraints, 576 relations, 61 concepts


Fitting The Source
(Constraint) Model

•  Experts only need to produce relations b/w KIs
– the rest is automatic


What was not presented at UbiqUM
•  Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl,
W. (2007) Translation of overlay models of student knowledge
for relative domains based on domain ontology mapping. 13th
International Conference on Artificial Intelligent in Education,
AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp.
289-296
•  Yudelson, M., Brusilovsky, P., Mitrovic, A., and Mathews, M.
(2010) Using Numeric Optimization To Refine Semantic User
Model Integration Of Adaptive Educational Systems.
Proceedings of the Third International Conference on
Educational Data Mining (EDM 2010), Pittsburgh, PA, June
11-13, 2010, pp. 221-230.
•  Sosnovsky, S. (2011) Ontology-based Open-Corpus
Personalization for e-Learning. PhD Thesis, University of
Pittsburgh

What Happened with auto-mapping?

University of Pittsburgh - PAWS Lab 38 Sergey Sosnovsky PhD Thesis

OOPS Interface: Reading Phase
Feedback/exit
buttons

Navigation links to
the next and the
previous topics

content of the
chosen topic

University of Pittsburgh - PAWS Lab 39 Sergey Sosnovsky PhD Thesis

Cross System Interest Modeling

•  CoMeT: a social system for sharing information
about research colloquia in Pittsburgh
•  Models user research interests by observing
bookmarking and sharing behavior
•  Cold start problem – can’t recommend with no
bookmarks
•  Can we seed user profiles using other systems
that represent user research interests?
–  Paper bookmarking systems – CuteULike

CoMeT: Collaborative Management of
Talks (try pittcomet.info)


Traditional Interest Modeling with
Keywords
•  Document model
–  a bag of words represented as a vector in keywords
vector space with TF.IDF weighting scheme

Keywords

W W W W W W
1 2 3 4 5 6
D1 0 1 0 0 0 0
D2 .5 0 0 .5 0 0
Talks/Papers
D3 .12 .13 0 .25 .5 0
D4 .25 0 .25 0 .25 .25


Recommending Talks to Users
•  User model
–  A combination of vectors of “interesting documents”
–  Possibly weighted by the rare of interest
•  K-nearest neighbor method
–  recommend top K closest documents to user profile
UP: User Profiles
U: User Profiles in D: Documents in in Keywords Space
Talks/Papers Space Keywords Space
w1 w w3
W W w3
D1 D2 D3 D4 2
1 2
U1 1 0 1
U1 1 0 0 0 D1 0 1 0

D2 0 0 .5 user U2 .25 0. .37
user U2 .25 0 .5 .25 s 5
s D3 0 1 0 U3 0 . .37
U3 0 .5 .25 .25 25
D4 0 0 .5

Keywords
Documents Keywords


Recommendation with Additional
Sources of Information
•  Sources of information about user interests:
–  Standard information: Keywords of bookmarked talks
in CoMeT
–  Tags of talks in CoMeT
–  Keywords of bookmarked papers from CiteULike
–  Tags of papers in CiteULike (CUL)
•  Explore the impact of additional sources
•  Also explore different models for fusion of tags
and keywords

Document Representation Models

•  Control condition: Keywords Only (KO)
–  Keywords extracted from documents’ titles and abstracts

•  Keywords+n*Tags (KnT)
–  Keywords extracted from documents’ titles and abstracts +
tags assigned to documents

•  Keywords Concatenated by Tags (KCT)
–  Keywords extracted from documents’ titles and abstracts +
tags assigned to documents


Keywords+n*Tags (KnT) Model
•  Each document: a bag of words containing :
–  document’s abstract, title and tags
•  Tags: regular keywords
–  Each tag appears n times
•  Merge CUL and CoMeT data in this model: same as KO
Common
Tag
Keywords Keywords & Tags
s
D3
W3 W4
W3=T1 W1 W2 T3 T4
/T1 /T2
W4=T2
Keywords:
w1, w2, w3, w2 n=2 D1 0 1 1 0 0 0
D2 1 0 3 5 0 0
Tags: Talks/Papers
T1, T3 D3 1 2 3 0 1 0
D4 2 0 5 0 2 1


Keywords Concatenated by Tags (KCT)
Model
•  Tags: a separated source of information
•  Each document: a bag of keywords and a bag of
tags
–  Concatenating keywords and tags vectors
–  TF.IDF weightening scheme Keywords
Tags

D3

W1 W2 W3 W4 T1 T2 T3 T4

Keywords: W3=T1
w1, w2, w3, w2 W4=T2 D1 0 1 1 0 0 0 0 0

Talks/Papers D2 1 0 3 1 0 2 0 0
Tags:
T1, T3 D3 1 2 1 0 1 0 1 0
D4 2 3 3 0 1 0 2 1


Merging CUL and CoMeT Data
D: Merged Documents’ Matrix

Dc: CUL Papers’ Matrix Dt: CoMeT Talks’ Matrix W1 w2 W3 T1 T2
w1 w T1 T2
C1 0 0 1 0 0
2
W W T1
P1 1 0 0 0 2 3 K C2 0 0 0 .5 0

k P2 .25 0 .5 .25
e C1 0 1 0 + P1 1 0 0 0 0
C2 0 0 .5 e P2 .25 0 0 .5 .25
P3 0 .5 .25 .25
P3 0 .5 0 .25 .25

m+i l+j
l+m+i+j-o-p
k- the number of CiteULike papers
m- the number of keywords used in CiteULike papers
i- the number of tags used in CiteULike papers
e- total number of talks in CoMeT
l- total number of keywords in CoMeT
j- total number of tags in CoMeT
o- the number of common keywords between two CoMeT and CiteULike systems
P- the number of common tags between two CoMeT and CiteULike systems

Experimental Results
•  User study:
–  8 real users of both CoMeT and CiteULike systems
•  Questionnaire for each recommended talk:
–  Is this talk related to your interest? (yes/no question)
–  How interesting this talk to you? (in 5-point scale)
–  If the talk is related to your interests, how novel is this talk
to you? (in 5-step scale)
•  Measures:
–  Relevance: precision by yes/no answers
–  Interest: nDCG by 5-point scale
–  Novelty: averaged the novelty ratings (Non-relevant = zero
novelty)

Precision and Novelty for different
number of recommendations


Conclusion
•  Including another reliable user profile →
increase precision of
recommendations;
•  Using CiteULike data for all models
–  Increased Relevance of recommended documents
–  Decreased novelty for KO model
•  CiteULike: adding, reviewing and rating related papers to their research field
•  CoMeT: information about talks happening within a specific time given on a particular
date users bookmark a more novel, less relevant talk
•  Adding tags
–  Increased novelty of recommendations (both using CoMeT and CUL data)
–  Increased relatedness in larger number of recommendations
•  Injection of keywords from another source of data: more reliable
than including tags for relevance
•  Including tags from various sources of information: more reliable
for interestingness or novelty

Back to the start

One user, many models of the same user

Let’s look from the other side

User Model

User Model
Adaptive
System
User Model

User Model

Many users, many models of different users


Treemap Group UM for Java
Programming


The more the students compared to their peers, the higher post-quiz
scores they received (r= 0.34 p=0.004)

Progressor for Java Programming


Progressor+ for rich content


And some more references
•  Sahebi, S., Wongchokprasitti, C., and Brusilovsky, P. (2010) Recommending research
colloquia: a study of several sources for user profiling. In: Proceedings of the
1st International Workshop on Information Heterogeneity and Fusion in
Recommender Systems (HetRec 2010) at the 2010 ACM conference on
Recommender systems, RecSys '10, Barcelona, Spain, ACM, pp. 32-38
•  Brusilovsky, P., Hsiao, I.-H., and Folajimi, Y. (2011) QuizMap: Open Social Student
Modeling and Adaptive Navigation Support with TreeMaps. Proceedings of 6th
European Conference on Technology Enhanced Learning (ECTEL 2011), Palermo,
Italy, Sptember 20-23, 2011, Springer-Verlag, pp. 71-82.
•  Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B. (2011) Open Social
Student Modeling: Visualizing Student Models with Parallel Introspective Views. In:
Proceedings of 19th International Conference on User Modeling, Adaptation, and
Personalization, UMAP 2011, Girona, Spain, Springer-Verlag, pp. 171-182.
•  Hsiao, I.-H., Guerra, J., Parra, D., Bakalov, F., König-Ries, B., and Brusilovsky, P.
(2012) Comparative Social Visualization for Personalized E-learning. Proceedings of
the Working Conference on Advanced Visual Interfaces, AVI 2012, Capri, Italy, ACM
Press, pp. 303-307. 58

Cross-System Personalization for College Students

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von Peter Brusilovsky

Mehr von Peter Brusilovsky (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Cross-System Personalization for College Students