Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Human and Machine Judgements
about Russian Semantic Relatedness
• A. Panchenko, D. Ustalov, D. Paperno, C. Meyer, N.
Konst...
Motivation
• A semantic similarity measure is a specific kind of
• similarity measure for nouns or multiword expressions.
...
Russian Datasets for Measuring Word
Semantic Similarity
• Human Judgement dataset (HJ dataset)
– Word pairs with human jud...
Human judgements about semantic
similarity (HJ)
• This is the standard way to assess a semantic similarity
measure.
• The ...
Human judgements: Crowdsourcing
Example of human judgements about
semantic similarity (HJ)
RuThes Lingustic Ontology
http://www.labinform.ru/pub/ruthes/index.htm
• 96 thousand unique words and expressions
– Synony...
Thesaurus Sociation.org
•Non-commercial
Internet-project
• contains 325,863
associations for 37,463
words
Structure of the semantic relation
classification (RT, AE) benchmarks
Russe: Best models according to the
HJ benchmark
MJ: Machine Judgements of Word Pairs
from the RUSSE Shared Task
• This dataset contains 12 886 word pairs coming
from HJ, ...
Gathering Machine Judgements
• Select one best submission for each of 19
participating teams for HJ, RT and AE datasets
• ...
Machine Judgements: Example
• word1,word2,sim,wmean
• препарат,вещество, 1.0,0.484418
• препарат,лекарство, 1.0,0.634770
•...
DT: Open Russian Distributional Thesaurus
• skip-gram model (Mikolov et al., 2013)
• trained on a 12.9 billion word collec...
Conclusion
• We presented new Russian resources for evaluating of
semantic relatedness measures
• Russian HJ datasets: Mil...
Alexander Panchenko - Human and Machine Judgements about Russian  Semantic Relatedness
Nächste SlideShare
Wird geladen in …5
×

Alexander Panchenko - Human and Machine Judgements about Russian Semantic Relatedness

186 Aufrufe

Veröffentlicht am

Human and Machine Judgements about Russian Semantic Relatedness

Veröffentlicht in: Daten & Analysen
  • Login to see the comments

  • Gehören Sie zu den Ersten, denen das gefällt!

Alexander Panchenko - Human and Machine Judgements about Russian Semantic Relatedness

  1. 1. Human and Machine Judgements about Russian Semantic Relatedness • A. Panchenko, D. Ustalov, D. Paperno, C. Meyer, N. Konstantinova, N. Loukachevitch, Ch. Bieman
  2. 2. Motivation • A semantic similarity measure is a specific kind of • similarity measure for nouns or multiword expressions. • … high values for synonyms, hyponyms, free associations, etc. • … low values for unrelated pairs • Applications: • information retrieval, document clustering, topic detection, question answering, word sense disambiguation, text summarization… • Most datasets, approaches were proposed for English • 2015 Russe • The First International Workshop on Russian Semantic Similarity Evaluation (RUSSE) • 19 participants, 105 runs, special session at the Dialog-2015 conference.
  3. 3. Russian Datasets for Measuring Word Semantic Similarity • Human Judgement dataset (HJ dataset) – Word pairs with human judgements • Russian Thesaurus dataset (RT dataset) – synonyms and hypernyms from RuThes thesaurus • Associative Thesaurus dataset (AE dataset) – cognitive associations between words • Machine Judgements – combination of submissions from a shared task on Russian semantic similarity • Russian Distributional Thesaurus
  4. 4. Human judgements about semantic similarity (HJ) • This is the standard way to assess a semantic similarity measure. • The HJ dataset contains word pairs translated from the widely used benchmarks for English: • Miller-Charles set – 30 word pairs • Rubenstein, H., Goodenough – 65 word pairs • WordSim – 353 word pairs: • Additionally subdivided into similarity set and relatedness set • Evaluation: Correlations with human judgments in terms of Spearman’s rank correlation • Agreement in ordering
  5. 5. Human judgements: Crowdsourcing
  6. 6. Example of human judgements about semantic similarity (HJ)
  7. 7. RuThes Lingustic Ontology http://www.labinform.ru/pub/ruthes/index.htm • 96 thousand unique words and expressions – Synonyms – Conceptual relations: class-subclass, part-whole, conceptual dependence •The dataset contains 114 066 relations for 6 832 nouns. •Half of these relations are synonyms and hypernyms from the RuThes-lite thesaurus •half of them are unrelated words.
  8. 8. Thesaurus Sociation.org •Non-commercial Internet-project • contains 325,863 associations for 37,463 words
  9. 9. Structure of the semantic relation classification (RT, AE) benchmarks
  10. 10. Russe: Best models according to the HJ benchmark
  11. 11. MJ: Machine Judgements of Word Pairs from the RUSSE Shared Task • This dataset contains 12 886 word pairs coming from HJ, RT, and AE datasets • The pairs have continuous relatedness scores • To estimate these scores we averaged 105 submissions of the shared task on Russian semantic similarity, RUSSE. • Each run consisted of 12 886 word pairs along with their similarity scores.
  12. 12. Gathering Machine Judgements • Select one best submission for each of 19 participating teams for HJ, RT and AE datasets • Rank the 19 best submissions. The best one has rank r1 = 19; the worst has rank r19 = 1 • Combine scores of these 19 best submissions – The score of a pair is equal to sum of run scores multiplied by run weight – Run weight: rank, exponent of rank, or square root of rank • Combined approach is better than single submission
  13. 13. Machine Judgements: Example • word1,word2,sim,wmean • препарат,вещество, 1.0,0.484418 • препарат,лекарство, 1.0,0.634770 • препарат,перестройка, 0.0,0.157699 • препарат,барселона, 0.0,0.105411 • инспекция,проверка, 1.0,0.532748 • инспекция,гол, 0.0,0.107823 • латы,меч, 1.0,0.428076 • латы,щит, 1.0,0.441120 • латы,рыцарь, 1.0,0.453718 • латы,броня, 1.0,0.414047 • латы,доспехи, 1.0,0.543852
  14. 14. DT: Open Russian Distributional Thesaurus • skip-gram model (Mikolov et al., 2013) • trained on a 12.9 billion word collection of books in Russian – minimal word frequency -- 5, – number of dimensions in a word vector -- 500, – Context window size: 10 words – For the most frequent 932,000 words, 250 nearest neighbours with the cosine similarity between word vectors are calculated. – These related words were lemmatized using PyMorphy2.
  15. 15. Conclusion • We presented new Russian resources for evaluating of semantic relatedness measures • Russian HJ datasets: Miller-Charles, Rubenstein, Goodenough; WordSim-353 • RuThes dataset and Human associations dataset • Machine Judgements Dataset and Distributional Thesaurus • The resources can be obtained from • http://panchenko.me/rsr/ • The semantic similarity and relatedness are useful in many NLP and information retrieval applications

×