https://imatge-upc.github.io/saliency-salgan-2017/ We introduce SalGAN, a deep convolutional neural network for visual saliency prediction trained with adversarial examples. The first stage of the network consists of a generator model whose weights are learned by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency maps. The resulting prediction is processed by a discriminator network trained to solve a binary classification task between the saliency maps generated by the generative stage and the ground truth ones. Our experiments show how adversarial training allows reaching state-of-the-art performance across different metrics when combined with a widely-used loss function like BCE.

1. 1
SalGAN: Visual Saliency Prediction with
Generative Adversarial Networks
Junting Pan Cristian Canton K.McGuinness Noel E. O’Connor Jordi Torres Elisa Sayrol Xavier Giró

2. 2
Saliency?

3. 3
Saliency Prediction

4. 4
MODEL
ARCHITECTURE

5. ARCHITECTURE OF GENERATOR
5
The encoder is initialized with
VGG-16, and we do fine tuning of
the last two groups of Conv Layers
The decoder is initialized randomly, the last
Conv Layer have tanh nonlinearities and the
output layer consist in a Conv Layer of kernel
size 1x1 with sigmoid activation.

6. Then according to the post about GAN model we applied the loss function
with smaller saliency maps
6
SALGAN-GAN: Downsample saliency map
[Inspiration from this blog post]
Compare
(BCE)
Downsampled
Generated
Saliency Map
Downsampled
Ground Truth
Saliency Map

7. SALICON VALIDATION
7
SALGAN: Downsample saliency map

8. 8
APPLYING GAN
GAN Training showing
saliency + image

9. 9
APPLYING GAN - Model Selection
SALICON validation set accuracy
metrics for GAN+BCE vs BCE on
varying numbers of epochs.

10. 10
APPLYING GAN - Model Selection
SALICON validation set
Information Gain for different
hyper parameter α on varying
numbers of epochs

11. 11
RESULTS

12. Qualitative Results
12
GroundTruth BCE SALGAN

13. Qualitative Results
13
GroundTruth BCE SALGAN

14. Qualitative Results- Failure case
14
GroundTruth BCE SALGAN

15. 15
Quantitative Results - SALICON TEST / MIT300