Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Web Science & Technologies                          University of Koblenz ▪ Landau, GermanyMeasuring the Influence of Tag ...
Collaborative Tagging Systems Objectives of tag recommenders:   Improve indexing quality ⇒ retrieval results   Reduce t...
Outline Measures of indexing quality   What to understand under “indexing quality”?   Inter-resource consistency ⇔ inte...
Measures of                                      Indexing QualityMeasuring the Influence of Tag Recommenders              ...
What does “indexing quality” mean?                                              user perceived similarity                 ...
Measures of indexing quality Inter-resource consistency    Compare resource similarity to the tag vector distance    Re...
Research Hypotheses Hypothesis: Inter-indexer consistency does not measure the  influence of tag recommenders on the inde...
Measuring Inter-Resource Consistency  Idea: Compare resource similarity and tag vector distance      ai: Average distanc...
Measuring Inter-Indexer Consistency   Idea: Do users agree on common description for a resource?   Tag Reuse Rate      ...
EvaluationMeasuring the Influence of Tag Recommenders                   Slide 10 of 21Klaas Dellschaft (klaasd@uni-koblenz...
Experimental Setup Objective:    Are inter-resource and inter-indexer correlated if tag     recommendations are given? ...
Suggestion of Popular Tags – ScreenshotMeasuring the Influence of Tag Recommenders               Slide 12 of 21Klaas Dells...
Clustering of Similar Web Pages – ScreenshotMeasuring the Influence of Tag Recommenders               Slide 13 of 21Klaas ...
ResultsMeasuring the Influence of Tag Recommenders                   Slide 14 of 21Klaas Dellschaft (klaasd@uni-koblenz.de...
Sizes of the Tagging Data Set                German User Group:                                       #Users   #Tags    #T...
The Clustering Data Set  In average, each user identified 4.59 clusters  Overall, 146 distinct clusters have been identi...
Differences in the Topical Clusters               Cluster probabilities in English experiment                             ...
Measuring the Inter-Resource Consistency H1a: Popular Tags decrease the inter-resource consistency H2a: User Tags increa...
Measuring the Inter-Indexer Consistency H1b: Popular Tags increase the inter-indexer consistency H2b: User Tags lead to ...
Conclusions Measures of indexing quality   Inter-resource consistency   Inter-indexer consistency   Measures do not co...
Paper:K. Dellschaft & S. Staab. Measuring the Influence of Tag   Recommenders on the Indexing Quality in Tagging Systems. ...
Nächste SlideShare
Wird geladen in …5
×

Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems

377 Aufrufe

Veröffentlicht am

This presentation is about our paper which was presented at the Hypertext conference 2012. In this paper, we investigate a methodology for measuring the influence of tag recommenders on the indexing quality in collaborative tagging systems. We propose to use the inter-resource consistency as an indicator of indexing quality. The inter-resource consistency measures the degree to which the tag vectors of indexed resources reflect how the users understand the resources. We use this methodology for evaluating how tag recommendations coming from (1) the popular tags at a resource or from (2) the user's own vocabulary influence the indexing quality. We show that recommending popular tags decreases the indexing quality and that recommending the user's own vocabulary increases the indexing quality.

Links to the paper:
http://dx.doi.org/10.1145/2309996.2310009
http://www.west.uni-koblenz.de/files/publications/dellschaft2012mti.pdf

Veröffentlicht in: Bildung, Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems

  1. 1. Web Science & Technologies University of Koblenz ▪ Landau, GermanyMeasuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems Klaas Dellschaft klaasd@uni-koblenz.de Steffen Staab staab@uni-koblenz.de
  2. 2. Collaborative Tagging Systems Objectives of tag recommenders:  Improve indexing quality ⇒ retrieval results  Reduce tagging effortMeasuring the Influence of Tag Recommenders Slide 2 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  3. 3. Outline Measures of indexing quality  What to understand under “indexing quality”?  Inter-resource consistency ⇔ inter-indexer consistency Evaluation of the measures  Are the measures correlated with each other?  User study: Apply measures for two recommenders Evaluation results ConclusionsMeasuring the Influence of Tag Recommenders Slide 3 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  4. 4. Measures of Indexing QualityMeasuring the Influence of Tag Recommenders Slide 4 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  5. 5. What does “indexing quality” mean? user perceived similarity Resources r1 r2 r3 describe  science  4 0 0 Tag Vectors          news  10  8 0  humor  v1 =   v2 =   v3 =   0 6 9          patents  0 0 5         sim(v1, v2) sim(v2, v3)Measuring the Influence of Tag Recommenders Slide 5 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  6. 6. Measures of indexing quality Inter-resource consistency  Compare resource similarity to the tag vector distance  Requires external knowledge about similarity of resources  Direct but sophisticated measure of indexing quality Inter-indexer consistency  Do users agree on common description for a resource?  Assumption: Users select tags independent of each other  Indirect but easy measure of indexing quality Which measure to use for evaluating tag recommenders?Measuring the Influence of Tag Recommenders Slide 6 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  7. 7. Research Hypotheses Hypothesis: Inter-indexer consistency does not measure the influence of tag recommenders on the indexing quality! Popular Tags: Suggest most popular tags of a resource  H1a: Popular Tags increase the inter-indexer consistency  H1b: Popular Tags decrease the inter-resource consistency User Tags: Suggest all tags previously applied by the user  H2a: User Tags lead to a decreased or unchanged inter-indexer consistency  H2b: User Tags increase the inter-resource consistency The measures do not correlate when evaluating tag recommendersMeasuring the Influence of Tag Recommenders Slide 7 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  8. 8. Measuring Inter-Resource Consistency  Idea: Compare resource similarity and tag vector distance  ai: Average distance to resources in the same cluster  bi: Average distance to resources in the closest other cluster bi − ai si = max(ai , bi ) resource cluster of similar resources inconsistent consistent even more-1 0 consistent +1Measuring the Influence of Tag Recommenders Slide 8 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  9. 9. Measuring Inter-Indexer Consistency Idea: Do users agree on common description for a resource? Tag Reuse Rate  Average number of users who apply a tag  Used in the related work  news  8 8 8          humor   4 6  6  fun   2  2  0          patents  0 0  0         Tag Reuse Rate: 4.7 5.3 7Measuring the Influence of Tag Recommenders Slide 9 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  10. 10. EvaluationMeasuring the Influence of Tag Recommenders Slide 10 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  11. 11. Experimental Setup Objective:  Are inter-resource and inter-indexer correlated if tag recommendations are given? Task given to users:  Assign keywords to 10 web pages.  After tagging, cluster web pages according to their similarity (⇒ inter-resource consistency). Three different experimental conditions:  No Suggestions  User Tags  Popular Tags Further divided into an English and German user groupMeasuring the Influence of Tag Recommenders Slide 11 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  12. 12. Suggestion of Popular Tags – ScreenshotMeasuring the Influence of Tag Recommenders Slide 12 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  13. 13. Clustering of Similar Web Pages – ScreenshotMeasuring the Influence of Tag Recommenders Slide 13 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  14. 14. ResultsMeasuring the Influence of Tag Recommenders Slide 14 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  15. 15. Sizes of the Tagging Data Set German User Group: #Users #Tags #TAS #TAS / #User No Suggestions 74 706 2134 28.84 Popular Tags 78 531 2228 28.56 User Tags 79 466 1507 19.08 English User Group: #Users #Tags #TAS #TAS / #User No Suggestions 115 973 3150 27.39 Popular Tags 118 550 3003 25.45 User Tags 118 819 2919 24.74Measuring the Influence of Tag Recommenders Slide 15 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  16. 16. The Clustering Data Set  In average, each user identified 4.59 clusters  Overall, 146 distinct clusters have been identified  11 most frequent clusters ⇒ 70% of the data  The web pages cover ~7 topics  3 web pages are on the border between two topicsMeasuring the Influence of Tag Recommenders Slide 16 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  17. 17. Differences in the Topical Clusters Cluster probabilities in English experiment No Suggestions Popular Tags User Tags The Onion + BBC The Onion + Patents ⇒ News ⇒ Humor  English Popular Tags condition has to be excludedMeasuring the Influence of Tag Recommenders Slide 17 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  18. 18. Measuring the Inter-Resource Consistency H1a: Popular Tags decrease the inter-resource consistency H2a: User Tags increase the inter-resource consistency Expectation: E(spt,i) < E(sns,i) < E(sut,i)  E(spt,i) E(sns,i) E(sut,i) German Users 0.1474 0.1847 0.2367 English Users N/A 0.1713 0.1915 (All differences are significant!)Measuring the Influence of Tag Recommenders Slide 18 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  19. 19. Measuring the Inter-Indexer Consistency H1b: Popular Tags increase the inter-indexer consistency H2b: User Tags lead to a decreased or unchanged inter-indexer consistency Expectation: E(trpt,i) > E(trns,i) ≥ E(trut,i)  E(trpt,i) E(trns,i) E(trut,i) German Users 3.60 2.44 2.39* English Users 4.67 2.76 2.68* * Differences between E(trns,i) and E(trut,i) not significantMeasuring the Influence of Tag Recommenders Slide 19 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  20. 20. Conclusions Measures of indexing quality  Inter-resource consistency  Inter-indexer consistency  Measures do not correlate if recommendations are given  Only inter-resource consistency can be used Popular Tags  Do not lead to consistent descriptions across resources  Are rather counterproductive for indexing resources User Tags  Lead to consistent descriptions across resource  Consolidate the personomy of usersMeasuring the Influence of Tag Recommenders Slide 20 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de
  21. 21. Paper:K. Dellschaft & S. Staab. Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems. Proceedings of the Hypertext Conference, 2012 http://dl.acm.org/citation.cfm?id=2310009Experimental Interface:http://userpages.uni-koblenz.de/~klaasd/experiment/Data Set:http://west.uni-koblenz.de/Research/DataSets/tagging-experiment/Measuring the Influence of Tag Recommenders Slide 21 of 21Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de

×