The Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an observer with respect to a true and objective ranking, and measure the agreement between two observers in the absence of a true ranking. These two variants prove useful in cases where ties are possible in either ranking, and may indeed result in very different scores. AP correlation was devised to incorporate a top-heaviness component into Kendall tau, penalizing more heavily if differences occur between items at the top of the rankings, making it a very compelling coefficient in Information Retrieval settings. However, the treatment of ties in AP correlation remains an open problem. In this paper we fill this gap, providing closed analytical formulations of AP correlation under the two scenarios of ties contemplated in Kendall tau. In addition, we developed an R package that implements these coefficients.

1. Amsterdam
October 3rd
Future Work
• A �e means that items really are indis�nguishable: a �ed pair in X is expected to be �ed in Y too
• A �e means that items are within some threshold like ΔnDCG<0.05: �es aren't transi�ve anymore
Open source package
ircor
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τ � 0.891
τap � 0.883
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τa � 0.874
τap,a � 0.873
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τa � 0.866
τap,a � 0.788
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τb � 0.885
τap,b � 0.831
Examples
Some extreme cases:
• If there are no �es, τ=τa=τb and τap=τap,a=τap,�es
• If all items are �ed, τa=τap,a=0
and both τb and τap,�es are undefined
Yilmaz τap Correla�on With Ties
Defini�ons: := number of items �ed with (inclusive), := rank of the first item in 's group (typical sports ranking)
Scenario B: agreement between two observersScenario A: accuracy of observer w.r.t. integral truth
• Tied pairs don't contribute
• Tied pairs are not expected
• Ignore all top items (they always contribute 0)
contribution of items above 's group
contribution of items within 's group
(in all permutations in which they're ranked above)
• A concordant's contribution depends on 's rank across permutations
• Ignore all top items (they always contribute 0)
• Each item is concordant in half the permutations
• And it's contribution depends on 's rank across permutations
• τap assumes is the true ranking, so let's symmetrize:
and adapt τap to ignore ties:
Kendall τ Correla�on With Ties
"this effect [of ties] may arise either because the objects really are indistinguishable,
[...] or because the observer is unable to discern such differences as exist"
Scenario A (Woodbury, 1940):
accuracy of observer w.r.t. integral truth
Scenario B (Student, 1921):
agreement between two observers
• Tied pairs don't contribute anything
• All pairs are expected to contribute #ties in X and Y
"average value of τ over all the t! possible ways
of assigning integral ranks to the tied members"
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
• Tied pairs don't contribute anything
• Tied pairs are not expected to contribute
Rank Correla�on Without Ties
Kendall (1938): Yilmaz et al. (2008):
• To compare two rankings and over the same set of items
• Very common in IR: rank pages by crawling policy, terms for relevance feedback, systems by mean score...
• Specially in Evalua�on: performance across topic sets, effec�veness measures, assessors, pooling methods...
• Based on the concordance between pairs of items:
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
T�� T�������� �� T��� �� AP C����������
Julián Urbano and Mónica Marrero