Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL

Ratings, tags, bookmarks and
other species: some examples of
quantitative research on
information filtering in TEL
Salvador Sánchez-Alonso

About this talk
 Some context
 about me, my group and my research

 Research coordinates

 Revision of successful cases

 Conclussions and open research directions

 Practical exercise

JTEL Summer school 2011 - Ratings, tags, bookmarks and other species

About me
 Remember Pecha-kucha? 


About IE
 Research lines

 Projects

 Doctorate studies

 Publications

 Journals and conferences


About our PhD program
 Not officially online but definitely not face-to-face!

 High performance-oriented
 No more “read this” or “have a look at…”
 Lots of autonomous work but with REAL help/guidance

 Procedure:
 Presentation (including CV)
 Finding a few ideas PhD candidate likes
 Writing objectives
 Usually avoiding conferences unless veeeeeery junior
 Paper accepted in an impact factor journal  PhD finished.
 2-3 years usually enough


Objectives of this talk
 Find research opportunities in quantitative TEL
research

 Learn from our experience

 See how TEL research can target high impact factor
journals


Assumptions
 You are familiar with
 The concept of Learning object
 The concept of metadata
 Learning objects repositories

 … and of course with IEEE LOM standard

 You have (ideally) visited one or more learning object
repositories (e.g. MERLOT, CNX, Organic.Edunet…)


Research coordinates


The process
 Develop a research question related to some
functionality and state a hypothesis (not formally yet)

 Identify the data source

 Build a software engine to collect the data

 Find the more apropriate technique(s) to analyse the
data and apply it on the dataset

 Use statistics to assess if the hypothesis holds


Data


Techniques


Social Network Analysis (SNA)
 Social network: a graph made up of nodes (individuals,
organizations...) and edges representing relationships
between nodes (friendship, work, membership...)

 Social Network Analysis: a set of techniques to discover
features of a network by means of its numerical or visual
representation.
 Find network measures such as betweenness and centrality
 Most SNA software uses a plain text ASCII data format to
represent the network


Collaborative filtering
 Informally: a form of automating the process of "word-of-mouth"
 But you would rather like to hear the opinions of those who have
interests similar to your own!

 Basic mechanism:
 A large group of people's preferences are registered
 Using a similarity metric, a subgroup of people is selected whose
preferences are similar to the preferences of the person who seeks
advice;
 A (possibly weighted) average of the preferences for that subgroup is
calculated;
 The resulting preference function is used to recommend options on
which the advice-seeker has expressed no personal opinion as yet.


Ontologies
 Formal representation of knowledge

 Concepts, relations and properties are represented in
an ontology language (eg OWL)

 Ontologies can be used to
 Enhance information retrieval
 Power advanced services such as more accurate web
search
 Communicate between systems
 Evaluate the correctness of a conceptual model
 …


Statistical profiling
 A set of techniques that allow to discover patterns or
correlations in large quantities of data

 Helps in dealing with the increasing data overload,
helping to discriminate information from noise

 Metrics:
 Precision: the fraction of correct instances among those
that the algorithm believes to belong to the relevant
subset
 Recall: the fraction of correct instances among all
instances that actually belong to the relevant subset


Cases


Assesing LO
reusability from
their metadata
PhD. Javier Sanz
Timeline: August 2008 -April 2010
Status: Final


Hypothesis
• It is possible to find an aprioristic reusability
evaluation based on LO metadata
• This metric would span all the factors affecting the
reusability of a learning object

• It is possible to compute reusability in an
automated way


Functionalities
• Estimating reusability would provide useful information
when it comes to selecting reusable objects
• This measure of reusability might constitute an indicator of
quality which would allow for search results to be ordered,
with those which have greater possibilities of being reused
taking priority.


The process in a nutshell
 “Polish” the hypothesis

 Gather data from 2 repositories: MERLOT and eLERA

 Find the metadata elements having an impact on
reusability

 Create the metrics
 Adjust them with empirical data

 Assess their effectiveness


Method
Cohesion
Learning object

Reusability
Size
+
Expert
Metadata Technological Portability Agregation

Educational Portability
Repository

> Aggregation methods: Weighted mean, Choquet’s integral, Multiple linear regression
> Evaluation of the efficiency of the model: Average absolute error, Average relative
error, Correlation between the real and the estimated value, Quality of the prediction
> Expert questionnaire + LORI


Factors, metrics and metadata
elements


Results
– Statistically significant
correlation between
estimated reusability and
content quality evaluated
by the experts.
– Statistically significant
correlation between
estimated reusability and Correlation K endall’s Spearman’s Rho
Tau
effectiveness as a
Content quality 0.228 0.307
learning tool and ease of Effectiveness 0.153 0.217
use. Ease of use 0.190 0.250


Publications
 Sanz-Rodriguez, J., Dodero, J. and Sanchez-Alonso, S.
(2010). Metrics-based evaluation of learning object JCR
reusability. Software Quality Journal 19(1), pp. 121-140.

 Sanz-Rodríguez, Dodero, Sánchez-Alonso. (2010) Ranking
Learning Objects through Integration of Different Quality
Indicators, IEEE Transactions on Learning Technologies, 2008
3(4), pp. 358-363.


Expanding queries
in LO repositories
with ontologies
PhD. Alejandra Segura
Timeline: June 2009 - December 2010
Status: Final


Hypothesis
Evaluación y resultados

 Query expansion can help repository users to find
additional relevant resources not retrieved using the
regular built-in search
 Scenario of application: A teacher searches for
educational resources to design a new course or to
create a new resource


Procedure
Query Query Search

other
Synonym
Original

Part of

Is a
extraction

Expanded concepts
Query

is a
Expansion
part of
• Exists
• Aproximate Remove
• Doesnt exist other duplicates

synonyms

List of LO Filter results
A 
Contrast B +/- Relevance
results C  evaluation


Procedure
 24 Official courses in genetics
 In Universities and Higher education institutions
 Syllabi published in the web
 Academic period 2009

 711 different concepts identified (lists of contents)
 91 test queries (concept retrieval frequency >1)


Method
 Experts in the field evaluated the results (3
experts each query).
 Topical relevance
 Expert profile: medical practitioners and genetics
specialists, 5 years experience in teaching and practice
extrictly necessary.
 Expert correlation analysis using rater agreement
metrics
 Precission and recall where used to state relevance
and novelty


Results
 Rater agreement moderate according to
Cohen’s Kappa and Spearman correlation
 COVERAGE: More than half the results (54%)
retrieved from non-expanded queries are also
retrieved when expanding the query.
 NOVELTY: 53% of the relevant LOs retrieved by
the expanded queries are new (e.g. different
from those retrieved without expansion).


Results
0,90
0,80
0,70
0,60
0,50
0,40
0,30
0,20
0,10
0,00
isa hermanos isa hijos isa padres par todo par partes syn exacto

Novedad Cobertura


Conclusions
Conclusiones

Results of expanded queries are affected by:
 The quality of the ontology
 Built-in retrieval mechanism
 Intrinsic characteristics of the learning objects collection

Best novelty results when:
 Polysemic queries
 Results from all types of expansions are merged in a
unique list


Publications
 Segura, A., Sánchez-Alonso, S., Garcia-Barriocanal, E.
and Prieto, M. (2011). An empirical analysis of ontology-
JCR
based query expansion for learning resource searches
using MERLOT and the Gene ontology. Knowledge Based
Systems, 24(1), pp. 119-133.


Exploring affiliation
network models as a
collaborative filtering
mechanism in e-learning
Not linked to any PhD
Status: ready for anyone interested


Hypothesis
 Social network analysis of relations in learning
environments will make it possible to re-conﬁgure…

 A) the learning contents and/or activities
 E.g. including new activities, changing the future course
structure or taking other kind of actions.

 B) the learning environment
 E.g. group formation, rearranging groups once the course
is being delivered.


Method
 Modeling and analysing learners participation in
activities organized around communication forums,
(very common in e-learning environments)

 Forum participation as an affiliation network (a kind of
social network with different types of nodes)

 One of the possible applications: identifying common
interests of groups of learners.

 Technique: Blockmodeling (aimed at transforming an
apparently non-coherent network into a more easily
comprehensible arrangement)


Results
 We identified two
different groups, one
interested in the
learning tools used
during the course and
the other group more
interested in the
theoretical aspects of
the course.


Results
 33-slice cluster with
introductory topics (most
people are interested)

 As long as course
progresses interest is
less cohesive

 16-slice cluster for topics
on general LO definitions
and concepts

 8-slice for highly
technical issues
(SCORM + LD)


Publications
 Rodríguez, D., Sicilia, MA., Sánchez-Alonso, S., Lezcano,
L. and García-Barriocanal, E. (2009) Exploring affiliation
JCR
network models as a collaborative filtering mechanism in e-
learning, Interactive Learning Environments. DOI:
10.1080/10494820903148610


Automated quality
assessment of
Learning Objects
PhD. Cristian Cechinel
Timeline: from July 2009
Status: Almost final


Hypothesis

 Some intrinsic features of learning objects stored in
existing repositories may present significant
difference between highly rated (good) learning
objects and poorly rated (not-good) learning objects.
 These quantitative measures could serve as the
basis for the development of models for quality
prediction.

Functionality
Developing models for automatically assessing
quality of learning objects inside repositories
based on the intrinsic features of the resources

Method

 Identification of intrinsic metrics of learning resources
that could serve as potential indicators of quality
 Database gathered from the MERLOT repository by
using a web crawler.
 In total, 35 metrics were extracted from 6,740
learning objects. From these resources, only 1,765
(27.27%) had at least one peer review or one user
rating and were used in the analysis, the rest were
discarded

Metrics

Class of Measure Metric

Link Measures Number of Links, Number of Unique Links, Number of Internal Links,
Number of Unique Internal Links, Number of External Links, Number of
Unique External Links
Text Measures Number of Words, Number of words that are links

Graphic, Interactive and Number of Images, Total Size of the Images (in bytes), Number of
Multimedia Measures Scripts, Number of Applets, Number of Audio Files, Number of Video
Files, Number of Multimedia Files
Site Architecture Size of the Page (in bytes), Number of Files for downloading, Total
Measures Number of Pages
Evaluation Metadata Number of Personal Collections
(contrast metric)

Method
 Learning objects were classified into three groups
(good, average, poor) according to their ratings.
 A Mann-Whitney (Wilcoxon) test was performed to
evaluate whether the selected metrics presented
significant difference in their medians between the
groups of good and not-good (average + poor)
resources
 Kolmogorov-Smirnov test was performed to evaluate
differences regarding the distributions.

Preliminary Results
The two groups of ratings available on MERLOT
(i.e. peer-reviewers and user comments ratings)
differ substantially regarding the intrinsic
characteristics of the resources.
The tested metrics present different profiles and
tendencies between good and not-good materials
depending on the category of discipline and the
type of material to which the resource belong

Preliminary Results
 We developed a Linear Discriminant Analysis (LDA) to
build models in order to distinguish
1. good from not-good resources,
2. good from average resources, and
3. good from poor resources

 For the Science and Technology discipline intersected
with the Simulation material type in the context of peer-
reviews thresholds.
 The third model achieved 91.49% of overall accuracy, with
a squared canonical correlation of 0.81130 (significant at
the 99% level)

Preliminary Results
 The pursuit for an automated model for the quality
evaluation of learning objects must consider the
development of profiles taking into account the
intersection of the categories of disciplines and material
types, as well as the distinct groups of raters.


Publications
 Cechinel, C., Sánchez-Alonso, S. and García-Barriocanal,
E. (2011). Statistical profiles of highly-rated learning
JCR
objects. Computers & Education, 57(1), pp. 1255-1269.


Can 3D platforms
improve training of
trainers' programs?
PhD. Carlos M. Lorenzo
Timeline: from April 2010
Status: in progress


Hypothesis
 Using Massively Multiuser Online Learning
environments (MMOL) platforms in virtual courses can
improve training-of-trainers program

 The aim is to explore how a specific MMOL Platform
facilitates online tutor’s tasks in a rich virtual learning
environment with a pedagogical framework, and to
identify essential issues of interactivity in this context


Method
 MMOL session using the
collaborative LORI approach
 A group of users contribute their
individual evaluations on a learning
object and try to reach a consensus
after hearing everyone else’s opinion.

 2D session in LCMS

 Comparing results in terms of
satisfaction and efectiveness


The experiments
 A prototype of 3D MMOL platform was created in a
realXtend server with an interactive space called
“MadriPolis”.

Case A: 11 master
students Both cases consist in training-
of-trainers experiences about
• LCMS on-line tutor experiment
collaborative Learning Object
• MMOL on-line tutor experiment evaluation based on Learning
Object Review Instrument
Case B: 10 graduated (LORI) with the Convergent
students Participation Model (CPM)
(Vargo et al., 2003) to
• LCMS on-line tutor experiment determinate the quality of e-
• MMOL on-line tutor experiment learning resources


Results

•Case «A». Data
•LCMS Experim. • Data Analisys
Collection and SNA:
•MMOL Experim.
•Case «B» •Log events. • Density
•MMOL Experim. •On-line surveys • Centrality
•LCMS Experim. •Direct
Observations
•Triangulation

Case studies Evaluation


Publications
 Manuscript submitted to Computer & Education (April
2011). Still waiting… JCR


Recommenders
inside learning object
repositories:
requirements for
meaningful datasets
Not linked to any PhD
Status: ready for anyone interested


Hypothesis
Implicit communities found via SNA
blockmodeling & component analysis
have a potential for recommending
learning objects to repository users


Application
Implicit communities found via SNA
blockmodeling & component analysis
have a potential for recommending
learning objects to repository users


Method
 Evaluate parameters for Collaborative Filtering
Algorithms for two datasets from MERLOT
1. Resources including ratings given by users

2. Resources present in the users’ Personal
Collections
 Generating recommendations for the datasets
using optimized parameters


Method
 Compare the results generated by the different
algorithms for the two datasets
 Contrast the results of the recommendations
generated by the algorithms with existing
endorsement mechanisms of the repository


Results
 Very high Precision values (varying from 20% to
100%) and not so high Recall percentages (with a
maximum of 18%).
 Recommendations generated are related to other
endorsement mechanisms in MERLOT
 Big differences between the recommendations
generated using the two distinct datasets
 Reinforcement of the initial idea that these two datasets
represent very distinct information


Future work
 Involving users’ opinions in the process
 Contrasting if recommendations for a given user
fall in the disciplinary area of that user or are
crossing disciplines
 Evaluating if users are already familiar with the
recommended resources, and if they would
recommend such resources to their fellows


Publications
 Sánchez Alonso, S., Sicilia, MA., García, E., Pagés, C. and
Lezcano, L. (2011) Social models in open learning object
repositories: A simulation approach for sustainable JCR
collections. Simulation Modelling Practice and Theory
19(1): 110-120

 Sicilia, M.-Á., García-Barriocanal, E., Sánchez-Alonso, S.,
& Cechinel, C. (2010). Exploring user-based recommender
results in large learning object repositories: the case of
MERLOT. Procedia Computer Science, 1(2), 2859-2864.


Conclussions and
open reseach
directions


Lessons learned
 Quantitative research is usually well received by
impact factor journals editorial boards

 It is feasible to have a PhD ready in about 2 years

 Respect repositories’ policies on acceptable use

 Collecting data, either manually or through automated
processes may be not permitted


Lessons learned
 LOR data should be shared!

 Evangelize repository owners to share data for
research and to include that in their conditions for use.
 A common dataset sharing format for LOR needed


Open research opportunities
 More in-depth study of social interaction in LCMS
(software ready for SNA in Moodle)

 Open courseware research studies similar to those
presented (OCW Finder crawler ready)

 Several project-related research (e.g. Assessment
of automated translation mechanisms in
Organic.Edunet)

 Any other research extending previous cases...


Your turn!
 5 next minutes [individually]: think of an experiment
similar to those reported in my talk

 15 minutes [all]: share your ideas with us

 [In pairs] Write down a summary with at least:
 Hypothesis
 Functionality
 Techniques
 Assessment metod


To be written down
 Salvador: salvador.sanchez@uah.es

 Want to know more about our distance PhD program
@ IE-UAH?
 Talk to me today or email me at your wish

 Fancy to work with us in a EU project?
 Contact me or prof. Sicilia: msicilia@uah.es


Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL

Recommended

Recommended

More Related Content

Similar to Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL

Similar to Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL (20)

Recently uploaded

Recently uploaded (20)

Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL