The document describes experiments conducted to evaluate measures of association for identifying the compositionality of word pairs. It discusses two hypotheses: 1) word pairs with higher association scores are less compositional, and 2) more frequent word pairs are more compositional. Three systems are described that use different measures of association (t-score, PMI, PMI) to classify word pair compositionality in a shared task. While the t-score performed best at identifying compositionality, PMI and frequency-based measures showed less success.

1. Identifying Collocations to Measure Compositionality : Shared Task System Description Ted Pedersen Department of Computer Science University of Minnesota, Duluth http://www.d.umn.edu/~tpederse

5. compositional

8. http://ngram.sourceforge.net

10. Well suited for shared task

12. Mutual Information (tmi)

13. Pearson's chi-squared test (x2)

14. Pointwise Mutual Information (pmi)

16. Jaccard Coefficient (jaccard)

17. Odds Ratio (odds)

18. Dice Coefficient (dice)

19. T-score (tscore)

22. p(w1) = n_1+ / n_++

23. p(w2) = n_+1 / n_++

29. Duluth-2 (aka “Coward's Comfort”) : use the measure most distinct from Duluth-1

30. Duluth-3 (aka “Why not?”) : three submissions allowed, so why not...

31. Rank correlation with fine grained gold standard 2 4 10 tscore 0.1481 0.2114 0.2674 tmi 0.1335 0.1908 0.2361 ll 0.1336 0.1913 0.2358 frequency 0.1865 0.2100 0.2126 ps 0.0992 0.1554 0.1874 x2 0.1157 0.1172 0.1654 phi 0.1253 0.1167 0.1646 jaccard 0.1253 0.1255 0.1602 dice 0.1253 0.1255 0.1602 odds 0.0216 0.0060 0.0257 pmi -0.0241 -0.0145 0.0143 rightFisher -0.1768 -0.0817 0.0740 leftFisher 0.1316 0.0686 -0.0870 twotailed -0.1445 -0.0651 -0.1064

32. “So many tests?? I suggested the t-score in 1991...”

38. RightFisher and two tailed Fisher had lower correlation, but aren't really suitable for collocation discovery

39. PMI has long history of use in collocation discovery …

40. “I'm no coward, and I like PMI!”

43. Narrow window in Duluth-3 might tend to miss other occurrences of words (outside window)

45. In measures of association higher scores mean less literal (at least according to hypothesis 1)

47. Binned 0-33 low, 34-66 medium, 67-100 high

49. Duluth-2 and Duluth-3 generally less effective for both coarse and numerical scoring

52. Scoring of compositionality is less successful

54. more frequent word pairs are likely to be compositional

56. http://ngram.sourceforge.net