The document summarizes Riina Vuorikari's presentation on using social information retrieval techniques to enhance the discovery and reuse of learning resources. The presentation outlines the context of Vuorikari's dissertation work, which examines how teachers tag and use social bookmarking tools in a multilingual environment. It then discusses the main research questions, experimental design, and some early analysis of user behavior based on data from pilot projects with teachers across Europe. The analysis shows that teachers tag resources in multiple languages and the tags have potential to connect teachers across borders if certain challenges are addressed.

1. Doctoral Consortium, RecSys 2007
Can Social Information Retrieval Enhance
the Discovery and Reuse
of
Learning Resources?
Riina Vuorikari
Katholieke Universiteit Leuven, Department of Computer Science
European Schoolnet, Belgium

2. Outline of the presentation
Context of the dissertation work
●
Main research questions
●
Experimental design
●
First evaluations so far:
●
Multi-lingual use of tags
–
Levels of user engagement
–

3. Context of the dissertation work

4. Context
European education, especially that of K-12
●
education, is inherently multilingual and
multicultural.
European teachers have access to multiple
●
repositories of digital learning resources by
Educational Authorities,
–
publishers,
–
other teachers,..
–

5. EUN partners..image

6. Context
Resources
●
In many different languages
–
For different national and regional curriculum
–
Contain metadata (e.g.title, keywords, language)
–
Of varying quality
–
Repositories have formed federations to
●
make resources available
Federated search based on metadata
–
Harvesting of metada
–

7. Challenge for users
End-users (e.g. teachers) have difficulties to
●
discover and find resources from educational
repositories
Metadata does not always match search terms
–
Locating content across linguistic and
●
national borders within Europe has proven
hard
Despite the use of a multilingual Thesaurus and
–
controlled vocabularies

8. Challenges for repositories
Users become more demanding and expect
●
services that are seen elsewhere (own
collections, pedagogical hints, ..)
European Schoolnet leading projects that
●
build services on top of federation of
European repositories
Social bookmarking tool
–
Tags
–
My networks
–

9. My Main Question
Can Social Information Retrieval
Enhance
the Discovery and Reuse
of
Learning Resources?

10. Social Information Retrieval
(SIR)
Refers to a family of techniques that assist
●
users in obtaining information to meet their
information needs by harnessing the
knowledge or experience of other users.
Examples of SIR techniques include:
●
sharing of queries,
–
collaborative filtering,
–
social network analysis,
–
social bookmarking,
–
subjective relevance judgements such as
–
tags, annotations, ratings and evaluations,
etc.

11. What is SIR for education?
Is education as a field of implementation that
●
different from other fields (e.g. music, movies)?
What are the domain specific requirements, where
●
does the data come from and what are its
semantics?
What are objects of recommendation?
●
SIR TEL http://ariadne.cs.kuleuven.be/sirtel/
●
My audience are teachers. Metaphor: it's like
●
recommending for DJs?

12. Context of this dissertation
Education
Social Information
Information seeking theories
Digital
Retrieval Digital
libraries content
(SIR)
methods
To empower the social and contextual aspects
of teachers' work

13. Main research questions

14. Main research questions 1
Teachers, tagging, languages:
How do teachers tag and use social
●
bookmarking in a multi-lingual environment?
Are those bookmarks and tags useful for
●
discovery of resources?
How about tags in multiple languages?
●

15. Main research questions 2
SIR aspect:
Can bookmarks and tags be used to connect
●
like-minded teachers cross country and
linguistic borders?
...and thus used for social information
●
retrieval?
What are the levels of user engagement
●
with the system?

16. Main research questions 3
Information Seeking aspect:
What are the main information seeking tasks
●
that teachers have?
What are the main SIR retrieval methods that
●
they use for them?
Can we match a task to a SIR method?
●

17. Experimental design

18. Data source 1
Calibrate project (http://calibrate.eun.org),
●
now to end of 2007
K-12 digital learning resources
●
Personal collections and tags (not shared)
●
78 pilot schools in Hungary, Austria, Estonia,
●
Czech Republic, Lithuania and Poland

20. Implementation area and data
source 2
MELT project (http://info.melt-project.eu),
●
from now to March 2009
K-12 digital learning resources from a
●
federation of about 10 repositories
Implementation of a social bookmarking tool,
●
annotations and my networks
About 70 teachers from Austria, Belgium,
●
Finland and Hungary

22. Data gathering
Diverse data collection methods to allow
●
triangulation of collected data.
log files from the portals to see the grand lines,
–
patterns, etc
complimented by some questionnaires to
–
understand groups or communities
possible interviews, thinking alouds, observation,
–
etc. on some few users to understand individual
behaviour.

23. Experimental Design
Independent Social Condition
Condition
Salganik, M., Dodds, P., & Watts, D.
●
Experimental Study of Inequality and
Unpredictability in an Artificial Cultural
Market. Science, 311(5762), (2006), 854-
856.

24. Experimental Design
Independent Social Condition
Condition
Tag input No tags shown Tags shown Tags shown
when tagging within users in all
spoken language languages
Social Ranking of Social navigation based on
Information resources bookmarks, tags, annotations
Retrieval and my networks

25. Some early analysis

28. Analysis of User Behavior on Multi-
lingual Tagging of Learning Objects
January 24 to April 21 2007
●
77 teachers /173 total participating
●
459 bookmarks
●
417 multilingual tags
●
320 different learning resources
●

29. Cross-border and language use
5
Tag
Tag 1
fr
fi 3
Tag
de
4
Tag
Tag 1
fr
fi 2
Tag
en
LO 2
in Fr
LO 1
in Fi
fr
de
fi
fi fi

30. Language should not divide..
5
Tag
fr
2
Tag
2
Tag
en de
Tag 1 4
Tag
fi fr
Tag 1
fi
LO 2 LO 2 LO 2
LO 1 LO 1
in Fr in Fr in Fr
in Fi in Fi
fi
de
fr
fi fi

31. ..but bring like-minded people
together
2
Tag
en
Tag 1
Tag 5
fi
fr
Tag 1
Tag 2
fi
de
LO 2
LO 1 in Fr
in Fi
de
fr
fi
fi fi

32. Visualisation tool for cross-
country use of bookmarks
Prototype tool to visualise
●
Bookmarks (title, classification keyword, country)
–
Tags (language)
–
Users (name, country, language)
–
–
Wanna play around with it?
●
http://www.cs.kuleuven.ac.be/~hmdb/infovis/
calibrate/calibrate.html

34. Distribution of bookmarks
Average: 6 bookmarks
●
Wide distribution:
●
10% “Super users”
–
more than 20
15% 20-6 bookmarks
–
45% 6-2 bookmarks
–
About 30% only
–
experimented (1)

35. Language analysis
Out of 417 tags many were with multiple
●
terms, when separated we found 585 terms
1/3 in Hungarian
●
26% in English, even though none of the
●
users were native English speakers
1/3 in German and Polish
●

36. Language analysis
The language was right in about 70% of
●
cases (from the interface), and found out
that...
...users tag in many different languages:
●
at the same time (e.g. Baum, arbre, tree)
–
at different times (once in Pl, other times in En)
–
use the interface in different languages (seems
–
like not only to test)

37. Btw, what do others do?
del.icio.us, Yahoo.fr, MyWeb.Yahoo.uk,
●
blogmarks.net, MisterWong.de...
Two different ways to deal with multiple
●
languages can be observed;
ones taken care of by users (i.e. crowd-
–
sourcing”)
others where the system supports multiple
–
languages to certain extent

38. Does the language matter?
Need for better ways to identify the language
●
Give rules (if the user first preferred languages is.., then..)
–
Automate the recognition of languages
–
Out-source it to users
–

39. Semantic analysis
Factual tags 63%
●
(Golder: item topics, kinds of
item, category refinements)
Subjective tags 29%
●
( Golder: item qualities)
Personal tags 3%
●
(Golder: item ownership, self-
reference, tasks organisation)
5% other
●
Sen et al. (2006).
●

40. Why tag categories?
In Sen et al. (2006)
●
it was found that
tags of different
categories can be
useful for different
tasks
In our case it is too
●
early to say
anything, but ...we'll
have an eye on it!

41. “Travel well” tags
About 13% of tags contain a general term, a
●
name, place
e.g. EU, Euroopa, Europa, europe,
●
geograafia, Pythagoras, etc.

42. What's the point of travel well
tags?
If those tags need no translation or language
●
filtering to be understood, and ..
..if they can be identified
●
We can be sure to show at least some tags
●
to users
whose language preferences we don't know, and
–
in which language there are no tags or keywords
–
available.

43. Do users find tags useful?

44. Usefulness of tags..
Overall, the thesaurus terms performed
●
better than the tags,
However, it can be argued that tags, after all
●
being produced with no outlay, showed an
overall encouraging and potential gain in
overall usefulness!

45. So what is needed?
HIDE ALL BUT THE RIGHT STUFF!
●
In the tagging interface (guided tagging)
●
Show tags in all languages?
–
Show only travel well tags?
–
Show only tags in users' preferred languages
–
While viewing the tags
●
In a tag cloud
–
For social navigation (resource-user-tag)
–
Q: does the system translate tags or only when a
–
user-given translation exist?

46. Future studies
Similar language and semantic analysis are
●
planned for a more thorough data in 2008
Moreover, our goals are to find out:
●
How do users use the tags (e.g. language and
–
tag convergence) ?
How are tags and the relation resource-tag-user
–
used for discovery?
Identify teachers information seeking tasks and a
–
best fit for a retrieval system.

47. User engagement
Inspired by Yahoo!'s START
●
rating shows the first level of engagement;
–
then tags it;
–
user views a page;
–
forwards it to friends,
–
and finally writing a review
–
How can this be used for recommending
●
purpose?

48. User engagement
In our case these
●
look very different:
views the page
–
views metadata
–
bookmarks and tags
–
rates
–
actual use?
–

49. That's it for now!
http://www.cs.kuleuven.be/~riina
riina.vuorikari@eun.org
riina.vuorikari@cs.kuleuven.be