Some popular evaluation metrics for machine learning with examples for what are they used.

1. Evaluation Metrics for
Classification and
Information Retrieval

2. Who am I
● Katya
● Natural Language Processing
● CTO at Majio
● Sloth.Works - matches Candidates to Jobs
● Twitter - @kitkate87
● Medium - @ekaterinamihailova

3. Content
● Classification metrics
● Information Retrieval metrics
● Majio’s evaluation metrics
● Design your own metric

4. General ML flow
Define goals
and metrics
Gather and
clean data
Build ML
model
Evaluate
results
Analyze
results

5. Classification Metrics

6. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures

7. Confusion matrix
True
Negative
False
Positive
FNFP
TNTP
Algorithm
Truth

8. Confusion matrix
True
Negative
False
Positive
Algorithm
Truth

9. Accuracy
acc = T / (T + F)
= (TP + TN) / ALL
acc = ( + ) / ALL

10. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth- 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures

11. Accuracy with 1% sloth pictures
Algorithm - always says it is not a sloth
acc = 99%

12. Accuracy per class
accP = TP / (TP + FN)
accP = /( + )
accN = TN / (TN + FP)
accN = /( + )
acc = (accP + accN)/2

13. Accuracy per class with 1% sloth pictures
Algorithm - always says it is not a sloth
acc = 50%

14. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure

15. Logarithmic Loss

16. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure - log loss
○ Camera in the forest - 1% sloth pictures

17. Precision
p = TP / (TP + FP)
p = / ( + )

18. Precision with 1% sloth pictures
Algorithm – guesses right exactly one monkey and for
everything else says it is not a monkey
p = 100%

19. Recall (True positive rate)
r = TP / (TP + FN)
r = / ( + )

20. Recall with 1% sloth pictures
Algorithm - always says it is a sloth
r = 100%

21. f1-measure
f = 2 p*r / (p+r)

22. f1-measure with 1% sloth pictures
Algorithm - always says it is a sloth
f = 0%
Algorithm - always says it is NOT a sloth except for 1
f ~ 0%
Algorithm has 30% precision and 70% recall
f = 42%
Algorithm has 50% precision and 50% recall
f = 50%

23. Parametrized f-measure
f(b) = (1+b) p*r / ((b*p)+r)

24. Parametrized f-measure with 1% sloth pictures
b = 3; f = 4*p*r/(3*p + r)
Algorithm has 30% precision and 70% recall
f = 52.5%
Algorithm has 50% precision and 50% recall
f = 50%

25. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure - log loss
○ Camera in the forest - 1% sloth pictures - f-measure
○ Search results for sloth and non sloth - 50% sloth pictures

26. False positive rate
fpr = FP / (FP + TN)
fpr = / ( + )

27. False Positive Rate with 1% sloth pictures
Algorithm - always says it is not a sloth
fpr = 0%

28. Information Retrieval metrics

29. Search Results
1 2 3 4 5 6 7 8

30. Search Results
1 0 0 1 1 1 1 1

31. TPR and FPR for different points
● At point 1- TPR = 2%, FPR = 0%
● At point 25 - TPR = 40%, FPR = 10%
● At point 50 - TPR = 74%, FPR = 26%
● At point 75 - TPR = 96%, FPR = 54%
● At point 100 - TPR = 100%, FPR = 100%

32. ROC Curve and AUC (Area Under the Curve)
True Positive Rate
False Positive Rate

33. ROC Curve and AUC (Area Under the Curve)
True Positive Rate
False Positive Rate

34. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure - log loss
○ Camera in the forest - 1% sloth pictures - f-measure
○ Search results for sloth and non sloth - 50% sloth pictures - AUC
○ Search results for sloth and non sloth - 1% sloth pictures

35. Search Results
1 0 0 1 1 1 1 1

36. Precision and Recall at different points
● At point 1 - Recall = 2%, Precision = 100%
● At point 25 - Recall = 40%, Precision = 80%
● At point 50 - Recall = 74%, Precision = 74%
● At point 75 - Recall = 96%, Precision = 64%
● At point 100 - Recall = 100%, Precision = 50%

37. Precision - Recall curve
Precision
Recall

38. Average Precision
1 0 0 1 1 1 1 1

39. Average Precision
(1/1 + 0 + 0 + 2/4 + 3/5 + 4/6 + 5/7 + 6/8)
/6
70.5%

40. Average Precision
/6
(0 + 0 + 1/3 + 2/4 + 3/5 + 4/6 + 5/7 + 6/8)
59.4%

41. Average Precision
/6
( 1/1 + 2/2 + 3/3 + 4/4 + 5/5 + 6/6 + 0 + 0)
100%

42. Mean Average Precision
MAP = (70.5% + 59.4% + 100%) / 3 = 76.64%

43. Geometric Mean Average Precision
MAP = (70.5% * 59.4% * 100%) = 74.81%∛

44. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure - log loss
○ Camera in the forest - 1% sloth pictures - f-measure
○ Search results for sloth and non sloth - 50% sloth pictures - AUC
○ Search results for sloth and non sloth - 1% sloth pictures - MAP, GMAP
○ Create image search for sloths with different relevance

45. Cumulative Gain
2 0 1 2 2 1 1 2
11
CG =∑rel(i)

46. Discounted Cumulative Gain
2 0 1 2 2 1 1 2
DCG =∑rel(i)/log2(i+1)

47. Discounted Cumulative Gain
DCG =∑rel(i)/log2(i+1)
5.12
2 0 1/2 0.86 0.77 0.35 1/3 0.31

48. Ideal Discounted Cumulative Gain
2 1.24 1 0.86 0.39 0.35 1/3 0
6.17
DCG =∑rel(i)/log2(i+1)

49. Normalized Discounted Cumulative Gain
NDCG = DCG / IDCG = 5.12 / 6.17 = 0.83

50. Evaluating Image recognition algorithms
● Setup
○ Images with sloths and images without sloths
● Goals
○ Distinguish between a sloth and non sloth - 50% sloth pictures - accuracy
○ Distinguish between a sloth and non sloth - 1% sloth pictures - accuracy per class
○ Distinguish between a sloth and non sloth and ask a person if not sure - log loss
○ Camera in the forest - 1% sloth pictures - f-measure
○ Search results for sloth and non sloth - 50% sloth pictures - AUC
○ Search results for sloth and non sloth - 1% sloth pictures - MAP, GMAP
○ Create image search for sloths with different relevance - NDCG

57. Majio Usecase
Matching Candidates to Job
1 2 3

58. Evaluating Matching Candidates to Job - 1
1 3 2 1 1 2 2 1 2 2
( TP/T - FP/T + 1 ) / 2

59. Evaluating Matching Candidates to Job - 1
1 3 2 1 1 2 2 1 2 2
( 2/4 - 1/4 + 1 ) / 2
62.5%

60. Evaluating Matching Candidates to Job - 1
3 1 2 2 2 2 2 2 2 2

61. Evaluating Matching Candidates to Job - 1
3 1 2 2 2 2 2 2 2 2
( 0/1 - 1/1 + 1 ) / 2
0%

62. Evaluating Matching Candidates to Job - 2
1 3 2 1 1 2 2 1 2 2
Normalized MAP at points 5, 10, 15

63. Evaluating Matching Candidates to Job - 2
1 3 2 1 1 2 2 1 2 2
MAP = (3.3/5 + 5.7/10) / 2
31.8%

64. Evaluating Matching Candidates to Job - 2
1 1 1 1 2 2 2 2 2 3
best MAP=(4.3/5 + 5.7/10) / 2
32.8%

65. Evaluating Matching Candidates to Job - 2
3 2 2 2 2 2 1 1 1 1
worst MAP=(1.3/5 + 5.7/10) / 2
29.8%

66. Evaluating Matching Candidates to Job - 2
1 3 2 1 1 2 2 1 2 2
normalized MAP = (MAP - wMAP) / (bMAP - wMAP)
33.3%

67. Evaluating Matching Candidates to Job - 2
normalized MAP = (MAP - wMAP) / (bMAP - wMAP)
33.3%
40% 30% 20% 10% 9% 8% 7% 6% 5% 4%

68. Evaluating Matching Candidates to Job - 2
40% 30% 20% 10% 9% 8% 7% 6% 5% 4%
0.8 * f1 + 0.2 * AP

69. Inter-annotator agreement
How much the annotators make the same decision for the
same search result.

70. The experiment
● 4 Annotators
● 60 randomly generated search results (by order, percentage and cut off line)
● The search results were equally distributed with majio scores between 1 and
100
● Annotators had to give score to the search between 1 (perfect) and 4 (horrible)
● 2 of the search results were there twice but in different context
● At least 3 out 4 annotators have to agree on ranking in order to be accepted

72. The results
● Inter-annotator agreement on 32 out of 60 rankings
● Two groups of annotators - strict (no 1s fallen behind) and useful (can you do
you with the amount of good candidates we have sent you)
● 2 out of 4 annotators gave different score to the trap rankings
● On the rankings in the inter-annotator agreement the scoring was consistent.
Limits for good and bad ranking acquired values.

73. Conclusions
● There are a lot of Information Retrieval metrics in the world (only a chosen few
were shown here)
● None is perfect but some are useful
● You can craft a metric yourself but then you have to check how good of a metric
it is
● People don’t generally agree on things in the beginning. Experiment until there is
good enough agreement.