1. Knowledge Management Institute
Pragmatic Evaluation of Folksonomies
20th International World Wide Web Conference (WWW2011)
Hyderabad, India
D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman
Markus Strohmaier
Assistant Professor, Graz University of Technology, Austria
Visiting Scientist, (XEROX) PARC, USA
Markus Strohmaier 2011
1

2. Knowledge Management Institute
Taxonomies: Categorization by Experts
Taxonomy: Usually produced and maintained by
few (e g dozens of) domain experts
(e.g. experts.
Alternative: Folk-generated taxonomies
(„Folksonomies“)
( F lk i “)
But how useful are such hierarchical
structures? How can they be evaluated?
Markus Strohmaier 2011
2

3. Knowledge Management Institute
Outline of this talk
1. Folksonomies
Construction and E l ti
C t ti d Evaluation
2.
2 Decentralized Search
J. Kleinberg‘s algorithm
3. Pragmatic Evaluation Framework
Presentation and discussion
4. Results & Findings
Markus Strohmaier 2011
3

4. Knowledge Management Institute
Outline of this talk
1. Folksonomies
Construction and E l ti
C t ti d Evaluation
2.
2 Decentralized Search
J. Kleinberg‘s algorithm
3. Pragmatic Evaluation Framework
Presentation and discussion
4. Results & Findings
Markus Strohmaier 2011
4

5. Knowledge Management Institute
Tagging: Social classification by users
Users label and categorize
Resources resources with concepts (tags)
Tags
Users
U
is a tuple V:= (U, T, R, Y) where
• th th
the three di j i t fi it sets U T R correspond t
disjoint, finite t U, T, d to user 1
– a set of persons or users u ∈ U
– a set of tags t ∈ T and
– a set of resources or objects r ∈ R tag 1 res. 1
• Y ⊆ U ×T ×R, called set of tag assignments
Tag similarity based on
users and resources
Markus Strohmaier 2011
5

6. Knowledge Management Institute
Construction of Folksonomies
From tag centrality to tag tag centrality:
F t t lit t high generality:
t lit
more abstract
low tag centrality:
more specific
Other existing folksonomy algorithms:
k-means, affinity propagation, …
[Heyman and Garcia-Molina 2006]
Markus Strohmaier 2011
6

7. Knowledge Management Institute
Semantic Evaluation of Folksonomies
Emerging Hierarchy
g g y Expert Hierarchy
p y
(Emergent) (Golden Standard)
via e.g. hierarchical clustering WordNet: a lexical DB for English
computers
Map- Synset Hierarchy
Programming ping
programming
distance d1 = 1 distance
d2 = 2
Python
Design
g languages
g g
patterns
abs. difference |d1 - d2| a Semantic
simple p y for the q
p proxy quality
y grounding j
java python
of emergent semantics
Markus Strohmaier 2011
8

8. Knowledge Management Institute
Outline of this talk
1. Folksonomies
Construction and E l ti
C t ti d Evaluation
2.
2 Decentralized Search
J. Kleinberg‘s algorithm
3. Pragmatic Evaluation Framework
Presentation and discussion
4. Results & Findings
Markus Strohmaier 2011
9

9. Knowledge Management Institute
Decentralized Search
Idea: use folksonomies as
Then, the performance of decentralized search
p background knowledge
g g
Background knowledge: Shortest path to target
depends on the suitability of folksonomies.
(a tag hierarchy)
In other words, we can evaluate the suitability of
folksonomies for decentralized search through
simulations. Folksonomy Folksonomy Folksonomy
1 ... n
shortest path found with
A (tag-tag) network: local k
l l knowledge pLK = 4
l d
Goal: Navigate from START to TARGET Δ = pLK-pGK
using local and background knowledge
only
candidates start target
shortest path with
p
global knowledge pGK = 3
Markus Strohmaier 2011
J. Kleinberg. The small-world phenomenon: An algorithmic perspective. Proc. 32nd ACM Symposium on Theory of Computing, 2000. Also appears as Cornell Computer Science Technical Report 99-1776 (October 1999) 10

10. Knowledge Management Institute
Outline of this talk
1. Folksonomies
Construction and E l ti
C t ti d Evaluation
2.
2 Decentralized Search
J. Kleinberg‘s algorithm
3. Pragmatic Evaluation Framework
Presentation and discussion
4. Results & Findings
Markus Strohmaier 2011
11

11. Knowledge Management Institute
Pragmatic Evaluation Framework
General idea:
• Use the OUTPUT produced by folksonomy algorithms
(hierachical structures) as INPUT (b k
(hi hi l t t ) (background
d
knowledge) for decentralized search.
Framework Instantiation
K-means, Aff.Prop.,
1. Generate n folksonomies DegCentrality, CloCentrality
exploratory navigation
2. Model navigational task
success rate, stretch
3. Select evaluation metrics
decentralized search
4. Simulate navigation
4 Sim late na igation
comparative evaluation
5. Evaluate performance
Markus Strohmaier 2011
12

12. Knowledge Management Institute
Simulating Exploratory Navigation
Topically
related
START TARGET tags
tags
resources
Topically
related
Random resources
Random
R d
start
resource
Usefulness of: page: e.g.
landing
page from
search
engine We generate 100.000 search pairs
(start, target) for each dataset, and
Folksonomy F lk
F lk Folksonomy Folksonomy
F lk run simulations
1 ... n
Markus Strohmaier 2011
13

13. Knowledge Management Institute
Outline of this talk
1. Folksonomies
Construction and E l ti
C t ti d Evaluation
2.
2 Decentralized Search
J. Kleinberg‘s algorithm
3. Pragmatic Evaluation Framework
Presentation and discussion
4. Results & Findings
Markus Strohmaier 2011
14

14. Knowledge Management Institute
Success Rates Across Different Folksonomies
flickr dataset
Tag generality
approaches
k-means /
affinity propagation
Random
folksonomy
Success rate:
The number of times an agent is successful
in finding a path using a particular
folksonomy as background knowledge All approaches outperform a
random folksonomy y
n
max hops n: the maximal number of steps an agent
Tag generality approaches
is allowed to perform before stopping (a tunable outperform k-means / Aff.
parameter e.g., an agent only f ll
t t l follows n li k )
links). Propagation
Markus Strohmaier 2011
16

15. Knowledge Management Institute
Success Rates Across Different Datasets
Holds for all But how
datasets efficient are
(to diff.
diff those
extents) folksonomies
during
search?
Markus Strohmaier 2011
17

16. Knowledge Management Institute
Stretch Δ = pLK-pGK
p
Shortest Paths found with Local Knowledge
Bibsonomy K M
Bib K-Means
Finds no path:
Δ = infinite
Finds paths that is +1 longer:
Δ=1
Holds for all
datasets
d t t Finds shortest possible path: Tag
T generality
lit
(to diff. Δ=0 approaches (d+e)
extents) find much shorter
paths!
Markus Strohmaier 2011
18

17. Knowledge Management Institute
Pragmatic Evaluation Framework
Framework Instantiation Alternatives
K-means, Aff.Prop., other folksonomy
1. Generate n folksonomies DegCentrality, algorithms or
CloCentrality expert taxonomies
exploratory other tasks
2. Model navigational task navigation
success rate, stretch other evaluation metrics
3.
3 Select evaluation metrics
decentralized search actual click data
4. Simulate navigation
comparative other evaluation
5. Evaluate performance evaluation approaches
Pragmatic evaluation produces different results for different
tasks and different assumed or observed navigation behavior.
The evaluation framework is completely general with regard to
the task, data and evaluation metrics adopted.
Markus Strohmaier 2011
19

18. Knowledge Management Institute
Results & Findings: Summary
1. Folksonomies are useful b k
1 F lk i f l background k
d knowledge f
l d for
navigation.
2. Existing folksonomy algorithms are more useful
than a random baseline.
baseline
3.
3 Tag generality approaches perform better than
existing hierarchical clustering approaches.
4. Pragmatic results support theoretical analysis
(not presented in talk – included in paper).
Markus Strohmaier 2011
20

19. Knowledge Management Institute
Thank You.
Th k Y
Markus Strohmaier
markus.strohmaier@tugraz.at
D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K. Lerman
Pragmatic Evaluation of Folksonomies
20th International World Wide Web Conference (WWW2011)
Hyderabad, India, March 28 - April 1, ACM, 2011.
http://kmi.tugraz.at/staff/markus/documents/2011_WWW2011_Folksonomies.pdf
Markus Strohmaier 2011
21