2. MEAN TEACHERS ARE
BETTERROLE MODELS
Weight-averaged consistency targets improve
semi-supervised deep learning results
ANTTI TARVAINEN & HARRI VALPOLA
17. dog We need two predictions:
a student and a teacher.
Then, hopefully,
the student can learn
something from the
teacher.
???
cat horse
18. dog So how to do this?
Two ways.
???
cat horse
19. dog
We distort the student’s
input, making its task
more challenging.
1. MAKE THE
TASK HARDER.
cat horse
20. dog
1. MAKE THE
TASK HARDER.
cat horse
We distort the student’s
input, making its task
more challenging.
And then we train
the harder task to
predict the easier
tasks’ output.
21. dog
We maintain an exponential
moving average of weights
to create a better teacher.
2. MAKE THE
TEACHER BETTER.
???
cat horse
22. dog
We maintain an exponential
moving average of weights
to create a better teacher.
A mean teacher.
2. MAKE THE
TEACHER BETTER.
???
cat horse
23. dog
We maintain an exponential
moving average of weights
to create a better teacher.
A mean teacher.
Then we let the
student learn these
better predictions.
2. MAKE THE
TEACHER BETTER.
???
cat horse
47. 0
10
20
30
40
top-5validation
errorrate
using all the labels
35,2
9,1
Variational
Auto-Encoder
Mean Teacher
(ResNet-152)
(theseresults included in the
Arxiv version of the paper)
3,8
stateof the art
IMAGENET 2012 WITH10%OF THELABELS