https://imatge-upc.github.io/rvos-mots/ Video object segmentation can be understood as a sequence-to-sequence task that can benefit from the curriculum learning strategies for better and faster training of deep neural networks. This work explores different schedule sampling and frame skipping variations to significantly improve the performance of a recurrent architecture. Our results on the car class of the KITTI-MOTS challenge indicate that, surprisingly, an inverse schedule sampling is a better option than a classic forward one. Also, that a progressive skipping of frames during training is beneficial, but only when training with the ground truth masks instead of the predicted ones.

1. CURRICULUM LEARNING FOR
RECURRENT VIDEO OBJECT SEGMENTATION
Co-directors: Xavier Giró Nieto and Carles Ventura Royo
Author: Maria Gonzàlez Calabuig

2. Introduction
Dataset
The model
Experiment sets
Techniques
Qualitative results
YouTube-VOS
Conclusions
CONTENTS

3. INTRODUCTION

4. INTRODUCTION
Curriculum Learning for Recurrent VOS - 4 of 144
Curriculum Learning:
Methodology inspired by the learning process of humans. The training data is presented in a
meaningful way, from simple to complex concepts.
Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.

5. INTRODUCTION
Curriculum Learning for Recurrent VOS - 5 of 144
Curriculum Learning:
Methodology inspired by the learning process of humans. The training data is presented in a
meaningful way, from simple to complex concepts.
4 curriculums
Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.

6. INTRODUCTION
Curriculum Learning for Recurrent VOS - 6 of 144
Curriculum Learning:
Methodology inspired by the learning process of humans. The training data is presented in a
meaningful way, from simple to complex concepts.
4 curriculums
THE DATASET
Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.

7. INTRODUCTION
Curriculum Learning for Recurrent VOS - 7 of 144
Curriculum Learning:
Methodology inspired by the learning process of humans. The training data is presented in a
meaningful way, from simple to complex concepts.
4 curriculums
THE DATASET THE MODEL
Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.

8. INTRODUCTION
Curriculum Learning for Recurrent VOS - 8 of 144
THE TASK
Semi-supervised or “one-shot” Video Object Segmentation

9. INTRODUCTION
Curriculum Learning for Recurrent VOS - 9 of 144
THE TASK
Semi-supervised or “one-shot” Video Object Segmentation

10. INTRODUCTION
Curriculum Learning for Recurrent VOS - 10 of 144
THE TASK
Semi-supervised or “one-shot” Video Object Segmentation

11. INTRODUCTION
Curriculum Learning for Recurrent VOS - 11 of 144
THE TASK
Estimated by the modelGiven to the model
Semi-supervised or “one-shot” Video Object Segmentation

12. DATASET

13. KITTI-MOTS
DATASET
Curriculum Learning for Recurrent VOS - 13 of 144
Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.

14. KITTI-MOTS
DATASET
Curriculum Learning for Recurrent VOS - 14 of 144
Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.
Its video sequences present challenges:

15. KITTI-MOTS
DATASET
Curriculum Learning for Recurrent VOS - 15 of 144
Its video sequences present challenges:
Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.

16. KITTI-MOTS
DATASET
Curriculum Learning for Recurrent VOS - 16 of 144
Its video sequences present challenges:
Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.

17. KITTI-MOTS
DATASET
Curriculum Learning for Recurrent VOS - 17 of 144
Its video sequences present challenges:
Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.

18. THE MODEL

19. THE MODEL
End-to-End Recurrent Network for video object segmentation: RVOS
Curriculum Learning for Recurrent VOS - 19 of 144
Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network
for Video Object Segmentation”, CVPR 2019.

20. THE MODEL
End-to-End Recurrent Network for video object segmentation: RVOS
Curriculum Learning for Recurrent VOS - 20 of 144
Athar, A., Mahadevan, S., Oˇsep, A., Leal-Taix´e, L., Leibe, B.: Stem-seg: Spatio-temporal embeddings for instance segmentation in videos., ECCV (2020)

21. EXPERIMENT SETS

22. SETS OF EXPERIMENTS
All techniques tested on two sets of experiments:
Resolution Batch Size Length clip
287x950 2 3
Resolution Batch Size Length clip
256x448 4 5
Curriculum Learning for Recurrent VOS - 22 of 144

23. METRICS
The results have been evaluated on the official metrics of the MOTS Challenge.
- sMOTSA has been defined as the reference metric:
Curriculum Learning for Recurrent VOS - 23 of 144
Paul Voigtlaender et al. “MOTS: Multi-Object Tracking and Segmentation”, CVPR 2019.

24. METRICS
The results have been evaluated on the official metrics of the MOTS Challenge.
- sMOTSA has been defined as the reference metric:
Curriculum Learning for Recurrent VOS - 25 of 144
Paul Voigtlaender et al. “MOTS: Multi-Object Tracking and Segmentation”, CVPR 2019.

25. EVALUATION METHOD
Proposal: Evaluation averaged per sequence
Curriculum Learning for Recurrent VOS - 25 of 144
Sequences

26. SCHEDULE
SAMPLING

27. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 27 of 144
VOS requires information about the previous step.

28. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 28 of 144
Train using the model’s outputs.

29. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 29 of 144
Train using the ground-truth annotations.

30. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 30 of 144
TEACHER FORCING

31. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 31 of 144
TEACHER FORCING
Fast and efficient

32. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 32 of 144
TEACHER FORCING
Fast and efficient Leads to exposure bias

33. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 33 of 144
time

34. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 34 of 144
time

35. Schedule Sampling
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 35 of 144

36. Schedule Sampling
Linear
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 36 of 144

37. Schedule Sampling
Linear Step
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 37 of 144

38. Schedule Sampling
Linear
Forward
Step
Forward
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 38 of 144

39. Schedule Sampling
Linear
Forward
Step
Forward
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 39 of 144

40. Schedule Sampling
Linear
Forward
Step
Forward
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 40 of 144

41. Schedule Sampling
Linear
Forward Inverse
Step
Forward Inverse
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 41 of 144

42. Schedule Sampling
Linear
Forward Inverse
Step
Forward Inverse
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 42 of 144

43. Schedule Sampling
Linear
Forward Inverse
Step
Forward Inverse
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 43 of 144

44. Schedule Sampling
Linear
Forward Inverse
Step
Forward Inverse
SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 44 of 144

45. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 45 of 144
RESULTS ON THE FORWARD STRATEGIES

46. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 46 of 144
RESULTS ON THE INVERSE STRATEGIES

47. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 47 of 144
OVERVIEW

48. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 48 of 144
OVERVIEW

49. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 49 of 144
OVERVIEW

50. SCHEDULE SAMPLING
Curriculum Learning for Recurrent VOS - 50 of 144
OVERVIEW

51. FRAME SKIPPING

52. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 52 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1

53. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 53 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1
frame #2

54. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 54 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1
frame #2
frame #3

55. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 55 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1
frame #2
frame #3
frame #4

56. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 56 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1
frame #2
frame #3
frame #4
frame #5

57. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 57 of 144
KITTI-MOTS has slow-motion video sequences.
frame #1
frame #2
frame #3
frame #4
frame #5
frame #6

58. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 58 of 144
Ideally:
…
...
.
…
…
..
N
fram
es of the sequence

59. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 59 of 144
Ideally:
But we have limitations (e.g. memory constraints)
…
…
..
N
fram
es of the sequence
…
...
.

60. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 60 of 144

61. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 61 of 144

62. FRAME SKIPPING
Frame Skipping
Curriculum Learning for Recurrent VOS - 62 of 144

63. FRAME SKIPPING
Frame Skipping
From 0 to 9
Curriculum Learning for Recurrent VOS - 63 of 144

64. FRAME SKIPPING
Frame Skipping
From 0 to 9
Curriculum Learning for Recurrent VOS - 64 of 144
time

65. FRAME SKIPPING
Frame Skipping
From 0 to 9
Curriculum Learning for Recurrent VOS - 65 of 144
time

66. FRAME SKIPPING
Frame Skipping
From 0 to 9
Curriculum Learning for Recurrent VOS - 66 of 144
time

67. FRAME SKIPPING
Frame Skipping
From 0 to 9
Curriculum Learning for Recurrent VOS - 67 of 144
time
...

68. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
Curriculum Learning for Recurrent VOS - 68 of 144
time

69. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
Curriculum Learning for Recurrent VOS - 69 of 144
time

70. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
Curriculum Learning for Recurrent VOS - 70 of 144
time

71. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
Curriculum Learning for Recurrent VOS - 71 of 144
time
...

72. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training All training
Curriculum Learning for Recurrent VOS - 72 of 144

73. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training All training
Curriculum Learning for Recurrent VOS - 73 of 144

74. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training All training
Curriculum Learning for Recurrent VOS - 74 of 144

75. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training
First half
training
All training
First half
training
Curriculum Learning for Recurrent VOS - 75 of 144

76. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training
First half
training
All training
First half
training
Curriculum Learning for Recurrent VOS - 76 of 144

77. FRAME SKIPPING
Frame Skipping
From 0 to 9 From 1 to 5
All training
First half
training
All training
First half
training
Curriculum Learning for Recurrent VOS - 77 of 144

78. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 78 of 144
RESULTS ON THE FRAME SKIPPING APPLIED DURING ALL TRAINING

79. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 79 of 144
RESULTS ON THE FRAME SKIPPING APPLIED ONLY WITH GROUND-TRUTH

80. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 80 of 144
OVERVIEW

81. FRAME SKIPPING
Curriculum Learning for Recurrent VOS - 81 of 144
OVERVIEW

82. TEMPORAL AND
SPATIAL
RECURRENCES

83. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 83 of 144
KITTI-MOTS is a crowded dataset:

84. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 84 of 144
time (frame sequence)
space(objectsequence)

85. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 85 of 144
time (frame sequence)
TEMPORAL RECURRENCE

86. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 86 of 144
space(objectsequence)
SPATIAL RECURRENCE

87. TEMPORAL AND SPATIAL RECURRENCES
Proposed curriculum:
Curriculum Learning for Recurrent VOS - 87 of 144

88. TEMPORAL AND SPATIAL RECURRENCES
Temporal and Spatial
Recurrence
Only temporal during
the first half of training
Curriculum Learning for Recurrent VOS - 88 of 144

89. TEMPORAL AND SPATIAL RECURRENCES
Temporal and Spatial
Recurrence
Spatio-temporal during
all training
Only temporal during
the first half of training
Curriculum Learning for Recurrent VOS - 89 of 144

90. TEMPORAL AND SPATIAL RECURRENCES
Temporal and Spatial
Recurrence
Spatio-temporal during
all training
Only temporal during
all training
Only temporal during
the first half of training
Curriculum Learning for Recurrent VOS - 90 of 144

91. TEMPORAL AND SPATIAL RECURRENCES
Temporal and Spatial
Recurrence
Spatio-temporal during
all training
Only temporal during
all training
Only temporal during
the first half of training
Only temporal during the
second half of training
Curriculum Learning for Recurrent VOS - 91 of 144

92. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 92 of 144

93. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 93 of 144

94. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 94 of 144
Ground-truth

95. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 95 of 144
Only Spatio-Temporal
Ground-truth

96. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 96 of 144
Only Spatio-Temporal Only Temporal
Ground-truth

97. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 97 of 144
Only Spatio-Temporal Only Temporal
Only Temporal first half
Ground-truth

98. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 98 of 144
Only Spatio-Temporal Only Temporal
Only Temporal first half Only Temporal second half
Ground-truth

99. TEMPORAL AND SPATIAL RECURRENCES
Curriculum Learning for Recurrent VOS - 99 of 144

100. LOSS
PENALIZATION
BY OBJECT AREA

101. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 101 of 144
KITTI-MOTS contains instances with different resolution:

102. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 102 of 144
KITTI-MOTS contains instances with different resolution:

103. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 103 of 144
KITTI-MOTS contains instances with different resolution:

104. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 104 of 144
An hypothesis is made:
DIFFICULT

105. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 105 of 144
An hypothesis is made:
DIFFICULT EASY

106. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 106 of 144
A curriculum is created:
time

107. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 107 of 144
A curriculum is created:
time

108. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 108 of 144

109. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 109 of 144
Resolution Batch Size Length clip
287x950 2 3
Resolution Batch Size Length clip
256x448 4 5

110. LOSS PENALIZATION BY OBJECT AREA
Curriculum Learning for Recurrent VOS - 110 of 144
Resolution Batch Size Length clip
287x950 2 3
Resolution Batch Size Length clip
256x448 4 5

111. LAST MINUTE
RESULTS

112. LAST MINUTE RESULTS
Curriculum Learning for Recurrent VOS - 112 of 144

113. QUALITATIVE
RESULTS

118. YouTube-VOS

119. YouTube-VOS
Curriculum Learning for Recurrent VOS - 119 of 144
Ning Xu et al. “YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark”, ECCV 2018

120. YouTube-VOS
Curriculum Learning for Recurrent VOS - 120 of 144
- Training parameters:
Resolution Batch Size Length clip
256x448 4 5

121. YouTube-VOS
Curriculum Learning for Recurrent VOS - 121 of 144
- Training parameters:
- Evaluated with the official metrics of the YouTube-VOS challenge.
Resolution Batch Size Length clip
256x448 4 5

122. YouTube-VOS
Curriculum Learning for Recurrent VOS - 122 of 144
- Training parameters:
- Evaluated with the official metrics of the YouTube-VOS challenge.
Resolution Batch Size Length clip
256x448 4 5

123. YouTube-VOS
Curriculum Learning for Recurrent VOS - 123 of 144
- Training parameters:
- Evaluated with the official metrics of the YouTube-VOS challenge.
Resolution Batch Size Length clip
256x448 4 5

124. YouTube-VOS
Curriculum Learning for Recurrent VOS - 124 of 144
Forward Linear Inverse Linear
Forward Step Inverse Linear

125. YouTube-VOS
Curriculum Learning for Recurrent VOS - 125 of 144
Results on KITTI-MOTS Results on YouTube-VOS

126. YouTube-VOS
Curriculum Learning for Recurrent VOS - 126 of 144
adapted
Frame skipping from 0 to 3From 0 to 9

127. YouTube-VOS
Curriculum Learning for Recurrent VOS - 127 of 144
Results on YouTube-VOSResults on KITTI-MOTS

128. CONCLUSIONS

129. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 129 of 144
SCHEDULE SAMPLING FRAME SKIPPING
LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES

130. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 130 of 144
SCHEDULE SAMPLING FRAME SKIPPING
LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES

131. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 131 of 144
SCHEDULE SAMPLING FRAME SKIPPING
LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES

132. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 132 of 144
SCHEDULE SAMPLING FRAME SKIPPING
LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES

133. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 133 of 144
SCHEDULE SAMPLING FRAME SKIPPING
LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES

134. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 134 of 144
Importance of knowing the dataset.
KITTI-MOTS YouTube-VOS

135. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 135 of 144
Importance of knowing the dataset.
KITTI-MOTS YouTube-VOS

136. CONCLUSIONS
Curriculum Learning for Recurrent VOS - 136 of 144
Importance of knowing the dataset.
KITTI-MOTS YouTube-VOS

137. FUTURE WORK
Curriculum Learning for Recurrent VOS - 137 of 144

138. FUTURE WORK
Curriculum Learning for Recurrent VOS - 138 of 144
Schedule Sampling

139. FUTURE WORK
Curriculum Learning for Recurrent VOS - 139 of 144
Schedule Sampling Frame Skipping

140. FUTURE WORK
Curriculum Learning for Recurrent VOS - 140 of 144
Schedule Sampling Frame Skipping
Loss penalization
by object area

141. FUTURE WORK
Curriculum Learning for Recurrent VOS - 141 of 144
Schedule Sampling Frame Skipping
Loss penalization
by object area
Other curriculums

142. FUTURE WORK
Curriculum Learning for Recurrent VOS - 142 of 144
Schedule Sampling Frame Skipping
Loss penalization
by object area
Other curriculums
Combination of the
best curriculums

143. WORKSHOP SUBMISSIONS
Curriculum Learning for Recurrent VOS - 143 of 144
Acceptance Notification: August 3, 2020
PAD2020

144. Curriculum Learning for Recurrent Video Object Segmentation
Maria Gonzalez Calabuig
Barcelona, 24th July 2020