The document discusses different techniques for filtering human interactions in crowdsourced object segmentation tasks. It examines removing users based on error rates or Jaccard index, filtering clicks in the same superpixel, and using a foreground map algorithm to take advantage of all user interactions. The document also explores automatically categorizing users into types like painters, experts, or spammers based on analyzed click patterns. Key findings include Jaccard index being a better measure than error rate to filter users, and that partial click filtering outperforms total click filtering when combined with user filtering.

1. Co-filtering human interaction
and object segmentation
Ferran Cabezas
Supervised by:
Vincent Charvillat
Axel Carlier
Xavier Giró-i-Nieto
Amaia Salvador
1

2. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
2

3. Crowdsourcing object segmentation
3

4. Filtering out bad human interactions
Correct human interaction
GoalResult of a correct human interaction Result of an incorrect human interaction
Incorrect human interaction
4

5. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
5

6. Click’n’Cut
• Web tool for interactive object segmentation designed for crowdsourcing
tasks.
A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut: Crowdsourced Interactive Segmentation with Object
Candidates. In CrowdMM’14, 2014
DEMO
6

7. Data
20 users that have
fully realized the
Click’n’Cut experiment
100 objects with
associated ground
truth from the
Berkeley-DCU dataset.
Testing set
5 images from Pascal VOC
2012 to perform gold
standard techniques.
Training set
Training set
7

8. How are obtained the masks from the clicks?
• Combination of different precomputed
binary object candidates .
• Foreground map algorithm
?
A.Carlier, Combining Content Analysis with Usage Analysis to better understand visual
contents, PHD Thesis, 2014.
A. Carlier, V. Charvillat, A.Salvador, X.Giró-i-Nieto, O. Marques, Click’n’Cut:
Crowdsourced Interactive Segmentation with Object Candidates. In
CrowdMM’14, 2014
8

9. Information of users are not always reliable
Bad user interaction Good user interaction
9

10. First approach - How are separated good from bad
user interactions?
4th GS1st GS
Error rate Error rate Error rate Error rate Error rate
2nd GS 3rd GS 5th GS
Mean error rate
• Removing users based on their error rate on the Gold standard images (training set)
10

11. Removing users based on their error rate
Remove users based on an error rate threshold
5GS
User20
5GS
User18
5GS
User19
. . .
5GS
User3
5GS
User1
5GS
User2
Error rate Error rate Error rate Error rate Error rate Error rate
11

12. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
12

13. How are evaluated the obtained masks?
clicks
Object
candidate
technique
Ground truth mask
?
?
Foreground
map algorithm
13

14. Jaccard index
A ∪ B
A ∩ B
Measure of similarity between the mask obtained from the Click’n’Cut experiment and the ground
truth mask
14

15. 3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
15

16. Impact of good and bad users in the resulting mask
Image
1 user (good user)
Image
12 users (Good users)
• A lot of errors can be removed just by discarding bad users
Image
20 users
16

17. Jaccard index= 0.0214
Error rate = 0
Jaccard index= 0.9402
Error rate = 0
Users filtering
NO OBVIOUS CORRELATION
17

18. Jaccard index for each user
4th GS1st GS
Jaccard
index
Jaccard
index
Jaccard
index
Jaccard
index
Jaccard
index
2nd GS 3rd GS 5th GS
Mean Jaccard index
• Better idea of how it is the contribution of the user in the final result
18

19. Jaccard index for each user
5GS
User20
5GS
User18
5GS
User19
. . .
5GS
User3
5GS
User1
5GS
User2
Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index Jaccard index
Remove users based on a Jaccard index threshold
19

20. Image 100
Jaccard index 100
Image 1
Jaccard index 1
Image 2
Jaccard index 2
Image 3
Jaccard index 3
Image 98
Jaccard index 98
Image 99
Jaccard index 99
MEAN
Jaccard index for the test set
. . .
Maintained users
Removed users
20

21. Results for the test set
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of users
Jaccard index by taking different number of users
JaccardIndex
Users sorted by its ascendent Jaccard index
Users sorted by its descendent error rate
descendent
ascendant
21

22. 3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
22

23. Schematic
Combination of
Object
Candidates
Image with filtered clicks
Obtaining mask
Slic
Felzenszwalb
N-cuts
nothin
g
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Image with non filtered clicks
23

24. Schematic
Combination of
Object
Candidates
Image with filtered clicks
Obtaining mask
Slic
Felzenszwalb
N-cuts
nothing
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Image with non filtered clicks
24

25. Superpixel techniques
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Combination of
Object
Candidates
Slic
Felzenszwalb
N-cuts
nothing
Image with filtered clicks
Obtaining mask
25

26. Superpixel techniques
• Felzenszwalb
• K = 20
• σ = 0,5
• m = 20
• SLIC
• Region size = 10
• Regularizer = 0.1• N-cuts
26

27. Filtering Clicks in a same superpixel
Three different
techniques for over-
segment an image
Two techniques for discarding
the clicks in a same superpixel
Combination of
Object
Candidates
Slic
Felzenszwalb
N-cuts
nothing
Image with filtered clicks
Obtaining mask
27

28. Filtering Clicks in a same superpixel
1) Total removal of conflict clicks :
Discarding all clicks in conflicting
superpixels
2) Partial removal of conflict clicks :
Discarding the clicks in minority
/equality inside conflicting
superpixels
nothingnothing
28

29. Results
Without applying any
technique of filtering
clicks
0.14
Techniques of
filtering clicks in a
same sppxl.
Partial removal of
conflict clicks
Total removal of
conflict clicks
SLIC 0.2109 0.2412
N-CUTS 0.2735 0.3330
FELZ 0.2104 0.2240
• Jaccard index for all users in the test set
29

30. 3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
• Removing users
• Removing clicks
• Removing clicks and users
Outline
30

31. Results
• Users sorted by its descendent Jaccard index
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with partial filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard indexJaccardIndex
Comparing results with total filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
Partial filtering Total filtering
31

32. 3. Treatment of human interaction
b) Taking advantage of all human interaction - Foreground map algorithm
Outline
32

33. Foreground map algorithm
Set of clicks
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
Felzenzwalb
Superpixel
segmentation
with k=100
Felzenzwalb
Superpixel
segmentation
with k=300
• Each click have a measure of confidence
based on the user error on the 5GS.
• Weight superpixel based on clicks
33

34. Foreground map algorithm
• Superpixel combination
• Slic: 6 levels
• Felzenzwalb: 8 levels
. . . . . .
R.Vieux, J.Benois, J.Domenger, A.Braquelaire,
Segmentation-based multi-class semantic object
detection, Multimedia Tools and Applications, 2010 34

35. Parameters to adjust after the combination
• Threshold
• Structure element for hole filling
?
?
35

36. Combining all Felz. and Slic levels
Threshold 0.56  Jaccard index = 0.8603
• Felz: k: 10,20,50,100,200,300,400,500
• SLIC: Regions side: 5,10,20,30,40,50
• SE =7
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1 X: 0.56
Y: 0.8891
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the train set
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
X: 0.56
Y: 0.8603
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the test set
36

37. Results combining all Felz. and Slic levels
Threshold = 0.56
SE = 7
37

38. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
38

39. Type of users and their particularities
• Painter: Lot of foreground clicks inside the object to segment
39

40. Type of users and their particularities
• Tired: Few clicks per image
40

41. Type of users and their particularities
• Border guards: Most of the bg clicks are in the contour of the image.
41

42. Type of users and their particularities
• Surrounders: Most of the fg clicks are in the contour of the image
42

43. Type of users and their particularities
• Mirrors: Have understood the experiment upside-down
43

44. Type of users and their particularities
• Spammers: Randomly placed foreground clicks over the image.
44

45. Type of users and their particularities
• Experts: Have well-understood the experiment and just made few
mistakes
45

46. Type of users and their particularities
• Different pattern: Does not follow the same pattern of clicks in all images
46

47. Manually categorization
• It is done a manually
categorization by considering just
the 5 gold standard images
Users Manually categorization
1 Painter
2 Expert
3 Mirror
4 Expert
5 Border guard
6 Expert
7 Tired
8 Border guard
9 Expert
10 Different pattern
11 Different pattern
12 Expert
13 Expert
14 Expert
15 Expert
16 Expert
17 Tired
18 Surrounder
19 Spammer
20 Expert
47

48. Manual rules for automatic user categorization
Features Painter The
mirror
The border
guard
The
surrounder
The
spammer
The tired The expert
# clicks >150/image - - - - <5/image -
fg clicks(%) >95% - <20% >95% >90% - -
errors(%)
<3% >90% - - >40% <20% -
Jaccard index (%) - <10% - - - <80% >80%
Contour fg(%)
(fg contour clicks/total fg
clicks)
- - - >80% <80% - -
Contour bg(%)
(bg contour clicks/total bg
clicks)
- - >70% - - - -
• According to the particularities of each type of user, a set of features and its rules are created:
48

49. Automatic categorization evaluation for the test set
Prediction
Painter Mirror Expert Spammer Surrounder Border Guard Tired Diff. Pattern
Ground Truth
Painter 1 0 0 0 0 0 0 0
Mirror 0 1 0 0 0 0 0 0
Expert 0 0 9 0 0 0 0 1
Spammer 0 0 0 1 0 0 0 0
Surrounder 0 0 0 0 1 0 0 0
Border guard 0 0 0 0 0 1 0 1
Tired 0 0 0 0 0 0 1 1
Diff. pattern 0 0 0 0 0 0 0 2
49

50. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
50

51. Conclusions
• Jaccard index is a better measure compared to error rate to separate bad
users from good ones
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of users
Jaccard index by taking different number of users
JaccardIndex
Users sorted by its ascendent Jaccard index
Users sorted by its descendent error rate
51

52. Conclusions
• Better results with partial than with total filtering
• Filtering clicks only makes sense when treating with bad users
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with partial filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
0 2 4 6 8 10 12 14 16 18 20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Number of Users sorted by its descended Jaccard index
JaccardIndex
Comparing results with total filtering and without filtering
Felz. sppxl. technique
Ncuts spxxl. technique
SLIC spxxl. technique
With no filtering clicks
Partial filtering
Total filtering
52

53. Conclusions
• In the foreground map algorithm it is reached the best result by
combining Felzenzwalb and Slic superpixel techniques with different levels
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1 X: 0.56
Y: 0.8891
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the train set
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
X: 0.56
Y: 0.8603
Threshold
JaccardIndex
Combining Slic and Felzenzwalb superpixels techniques in the test set
53

54. Conclusions
Images from User 11
• It is not possible to automatically categorize users that does not
follow the same pattern of clicks in all images
54

55. 1. Motivation
2. Related Work
3. Treatment of human interaction
a) Removing human interaction - Combination of object candidates
b) Taking advantage of all human interaction - Foreground map algorithm
4. Automatic categorization of the users
5. Conclusions
6. Future work
Outline
55

56. Future work
• Study different techniques for filtering clicks in a same superpixel.
• Take advantage of the clicks of some users to create a better mask
(e.g. Border guard and Surrounder users)
• Train classifier for automatic user categorization
56

57. Questions & Answers
57