building ai to recreate our visual world

1. Learning to recreate our
visual world
Jun-Yan Zhu
UC Berkeley

2. 4.7 trillion
photographs
13 billion images
300 million images
uploaded daily
1.5 million images
uploaded daily
300 hours uploaded
per minute

3. Street
Visual
Understanding
Scene understanding
[Zhou et al. 2014]

4. people
bicycle
umbrella
Visual
Understanding
Object Detection
[Girshick et al. 2014]

5. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
[Karpathy and Fei-Fei 2015]
[Donahue et al. 2015]
[Misra et al. 2015]
Image Captioning

6. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis

7. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis
by Dom’s father

8. Man-Computer Symbiosis
Computers can help us draw, even if we can’t.
JCR Licklider, 1968

9. Sketchpad
Sutherland, 1960

10. Image warping
By Photoshop
Visual Synthesis is hard!
Why it does not work
- Is this a handbag?
- What makes a
handbag look real?

11. What makes an image look real?
Statistics of Natural Images and Models [Huang and Mumford. 2000]
Sample images from the dataset
(4000 images in total)
log histogram of pixels
log histogram of wavelet pairs

12. What makes an image look fake?
Mismatch color statistics
[Lalonde and Efros 2007] lack details [Johnson et al. 2011]
noticeable boundary
[Perez et al. 2003] “Bleeding” artifacts [Tao et al. 2010]

13. Deep Generative Models
• Generative Adversarial Network (GAN)
[Goodfellow et al. 2014]
• Variational Auto-Encoder (VAE) [Kingma and
Welling 2013]
• DRAW (Recurrent Neural Network) [Gregor et
al 2015]
• Pixel RNN and Pixel CNN ([Oord et al 2016])
• …

14. Generative Adversarial Networks (GANs)
Discriminato
r
Real vs. Fake
Generator
𝑥 ~ 𝐺(𝑧)
[Goodfellow et al. 2014]

15. Generator
[Goodfellow et al. 2014]
z G(z)
cat credit: aleju/cat-generator
G

16. real or fake?
[Goodfellow et al. 2014]
Discriminator
z G(z)
D
Generator
G
𝐺: generate fake samples that can fool 𝐷
𝐷: classify fake samples vs. real images

17. [Goodfellow et al. 2014]

18. [Goodfellow et al. 2014]
fake 0.1
z G(z)
DG

19. real 0.9
[Goodfellow et al. 2014]
z G(z)
DG
D
x
fake 0.1

20. fake 0.3
[Goodfellow et al. 2014]
z G(z)
DG
D
x
real 0.9

21. Images generated by GANs
Sample shoes images
from Zappos.com
[Yu and Grauman 2014]
Random image samples
from Generator G(z)
[Radford et al. 2015]

22. Explore the GANs latent space
[Radford et al. 2015]
𝐺(𝑧-) 𝐺(𝑧.)Linear Interpolation in z space: 𝐺(𝑧- + 𝑡 ⋅ (𝑧. − 𝑧-))
Handbag model trained on 137k Amazon handbags

23. Limitations of GANs
• produce images randomly; hard to control.
• not photo-realistic enough; low resolution.

24. Limitations of GANs
• produce images randomly; hard to control.
• not photo-realistic enough; low resolution.

25. Controllable Image Generation
[Zhu et al. ECCV 2016]
Trained on
Places Dataset

26. Generative Visual Manipulation on the Natural Image Manifold
Zhu, Krähenbühl, Shechtman, Efros 2016

27. Manipulating the Latent Vector
Objective:
user guidance imageconstraint violation loss 𝐿4
Generative model: 𝐺(𝑧)
z G(z)
G 𝐿4( )
,
𝑣4

28. Manipulating the Latent Vector
Objective:
user guidance imageconstraint violation loss 𝐿4
Generative model: 𝐺(𝑧)
z G(z)
G 𝐿4( )
,
𝑣4

29. Manipulating the Latent Vector
z G(z)
G 𝐿4( )
,
𝑣4
𝐺(𝑧)
Guidance
𝑣4

30. Product Design

31. Overview
original photo
projection on manifold
Project
Results generated by GANs
Editing UI

32. Overview
original photo
projection on manifold
Project Edit Transfer
Results generated by GANs
Final results
Editing UI

33. Edit Transfer via Generalized Optical Flow
𝐺(𝑧.)𝐺(𝑧-)
Input
Motion (u, v)+ Color (𝑨 𝟑×𝟒): estimate per-pixel geometric and color variation
Linear Interpolation in 𝑧 space
[Brox et al. 2004]
[Shih et al. 2013]

34. Edit Transfer via Generalized Optical Flow
𝐺(𝑧.)𝐺(𝑧-)
Input
Linear Interpolation in 𝑧 space
Motion (u, v)+ Color (𝑨 𝟑×𝟒): estimate per-pixel geometric and color variation Motion (u, v)+ Color (𝑨 𝟑×𝟒): estimate per-pixel geometric and color variation

35. Edit Transfer via Generalized Optical Flow
Result
𝐺(𝑧.)𝐺(𝑧-)
Input
Motion (u, v)+ Color (𝑨 𝟑×𝟒): estimate per-pixel geometric and color variation
Linear Interpolation in 𝑧 space

36. Editing handbags

37. Editing shoes

38. Generated
Image
User
Guidance
Scribble→ latent space → Image [Zhu et al. 2016]
G
Scribble ImageLatent space

39. Generated
Image
User
Guidance
Scribble→ latent space → Image [Zhu et al. 2016]
Image-to-Image Network
[Isola, Zhu, Zhou, Efros. 2017] [Zhu*, Park*, Isola, Efros. 2017]

40. Image colorization
Designing objective functions
L2 regression
[Johnson et al. 2016]
Super-resolution
[Zhang et al. 2016]
L2 regression
slides credit: Phillip Isola

41. Image colorization
Designing objective functions
Cross entropy objective,
with colorfulness term
Deep feature covariance
matching objective
[Johnson et al. 2016]
Super-resolution
[Zhang et al. 2016]

42. Universal loss?
…
…

43. …
Generated vs
Real
(classifier)
Real photos
Generated images
…
…
[Goodfellow et al. 2014]

44. Image-to-Image Translation
Image-to-image translation with conditional adversarial nets [Isola, Zhu, Zhou, Efros. CVPR 2017]

45. real or fake?
[Goodfellow et al. 2014]
Discriminator
z G(z)
D
Generator
G
𝐺: generate fake samples that can fool 𝐷
𝐷: classify fake samples vs. real images

46. real or fake?
[Goodfellow et al. 2014]
Discriminator
x G(x)
D
Generator
G
𝐺: generate fake samples that can fool 𝐷
𝐷: classify fake samples vs. real images

47. Real!
[Goodfellow et al. 2014]
Discriminator
x G(x)
D
Generator
G

48. [Goodfellow et al. 2014]
Discriminator
x G(x)
D
Generator
G Real too!

49. real or fake pair ?
slides credit: Many slides modified from Phillip Isola’s talk
x G(x)
G
D

50. fake pair: 0.1
G
D
x G(x)

51. real pair: 0.9
𝑦
D

52. Edges → Images
Input Output Input Output Input Output
Edges from [Xie & Tu, 2015]

53. Sketches → Images
Input Output Input Output Input Output
Trained on Edges → Images
Data from [Eitz, Hays, Alexa, 2012]

54. #edges2cats [Christopher Hesse]
Ivy Tasi @ivymyt
@gods_tail
@matthematician
https://affinelayer.com/pixsrv/
Vitaly Vidmirov @vvid

55. Input Output Groundtruth
Data from
[maps.google.com]

56. Input Output Groundtruth
Data from [maps.google.com]

57. Labels → Facades
Input Output Input Output
Data from [Tylecek, 2013]

58. BW → Color
Input Output Input Output Input Output
Data from [Russakovsky et al. 2015]

59. …
Paired

60. - Expensive to collect pairs.
- Impossible in many scenarios.
Label ↔ photo: per-pixel labeling
…
Paired
Horse ↔ zebra: how to get zebras?

61. …
…
…
Paired Unpaired

62. x G(x)
Generator
G
D
No input-output pairs!

63. Discriminator
x G(x)
D
Generator
G Real!

64. Discriminator
x G(x)
D
Generator
G Real too!
GANs doesn’t force output to
correspond to input

65. mode collapse!

66. …CycleGAN, or there and back aGAN
[Zhu*, Park*, Isola, Efros. ICCV 2017]
……
Discriminator D>: 𝐿?@A 𝐺 𝑥 , 𝑦
Real zebras vs. fake zebras

67. Discriminator DB: 𝐿?@A 𝐹 𝑦 , 𝑥
Real horses vs. fake horses
Discriminator D>: 𝐿?@A 𝐺 𝑥 , 𝑦
Real zebras vs. fake zebras
…
…CycleGAN, or there and back aGAN

68. Cycle-consistency Loss
Forward cycle loss: F G x − x .
G(x) F(G x )x

69. Cycle-consistency Loss
Large cycle loss
Forward cycle loss: F G x − x .
G(x) F(G x )x
Small cycle loss

70. Cycle-consistency Loss
Backward cycle loss: 𝐺 𝐹 𝑦 − 𝑦 .
Forward cycle loss: F G x − x .
G(x) F(G x )x F(y) G(F x )𝑦
See also [Yi et al. 2017] [Kim et al. 2017]

71. Results

72. Collection Style Transfer
Photograph
@ Alexei Efros
MonetVan Gogh
CezanneUkiyo-e

73. Cezanne Ukiyo-eMonetInput Van Gogh

74. Monet’s paintings → photos

75. Monet’s paintings → photos

80. Neural Style Transfer [Gatys et al. 2015]

81. Input Style Image I CycleGANStyle image II Entire collection
Photo → Van Gogh
horse → zebra
Input Style image I CycleGANStyle image II Entire collection

82. CG2Real: GTA5 → real streetview
Inspired by [Johnson et al. 2011]GTA5 CG Input Output

83. Real2CG: real streetview → GTA
Cityscape Input Output

85. [Richter*, Vineet* et al. 2016]
GTA5 images Segmentation labels

86. Use CG data to train recognition systems
Per-class accuracy Per-pixel accuracy
Oracle (Train and test on Real) 60.3 93.1
Train on ‘free’ synthetic data (GTA5) Test on real images
meanIOU (per-class) Per-pixel accuracy
Oracle (Train and test on Real) 60.3 93.1
Train on CG, test on Real 17.9 54.0

87. Domain Adaption with CycleGAN
[Tzeng et al. ] In submission
meanIOU (per-class) Per-pixel accuracy
Oracle (Train and test on Real) 60.3 93.1
Train on CG, test on Real 17.9 54.0
FCN in the wild [Hoffman et al.] 27.1 (+6.0) -
Train on CycleGAN, test on Real 34.8 (+16.9) 82.8
Train on CycleGAN data Test on real images

88. Failure cases

91. ImageNet
“Wild horse”

92. Conclusion

93. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis
by Dom’s father

94. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis
by iGAN

95. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis
by Pix2pix

96. A city street filled with
lots of people walking in
the rain.
Visual
Understanding
Visual
Synthesis
by CycleGAN

97. Visual
Synthesis
A few things I have learned
• Visual Synthesis is a learning problem.
• We can learn to do it with trillions of photos.
• Build general-purpose tools and find cool problems.
• Open-source the code and data.

98. Phillip Isola and Jun-Yan Zhu

99. Community-driven research: #pix2pix
Bertrand Gondouin @bgondouinBrannon Dorsey @brannondorsey
Kaihu Chen @kaihuchen Mario Klingemann @quasimondo

100. Jun-Yan Zhu and Taesung Park

101. Pix2pix: 144 lines
CycleGAN: 220 lines
Jun-Yan Zhu and Taesung Park

102. 20+ Implementations by others
• [Tensorflow] (by Harry Yang)
• [Tensorflow] (by Archit Rathore)
• [Tensorflow] (by Van Huy)
• [Tensorflow] (by Xiaowei Hu)
• [Tensorflow-simple] (by Zhenliang He)
• [Chainer] (by Yanghua Jin)
• [Minimal PyTorch] (by yunjey)
• [Mxnet] (by Ldpe2G)
• [lasagne/keras] (by tjwei)
⋯

103. Birds @Matt Powell
Bear → Panda @Matt Powell
#CycleGAN
Monet → Thomas Kinkade @David Fouhey
Resurrecting Ancient Cities @ Jack Clark

104. Portrait to Dollface
@Mario Klingemann
Colorizing legacy photographs
@Mario Klingemann
#CycleGAN

105. Face Swapping with #CycleGAN
Input

106. Face Swapping with #CycleGAN
Input
Generated

107. Face Swapping with #CycleGAN
Input
Generated
Reconstruction

108. #CycleGAN: Face ↔ Ramen
@ Takuya Kato

109. @itok_msi
+ Smaller cycle loss + Global image discriminator

110. Medical imaging appilcations
• Segmentation [Xue et al.], etc.
• CT denoising [Yi and Babyn]
• MR <-> CT [Wolterink et al. ]
• MRI Reconstruction [Quan et al. ]
⋯

111. MR <-> CT

112. MRI Reconstruction

113. Three views of GANs
• generative models
G 𝑧 unsupervised learning
• Trainable regression loss.
• Domain matching loss.
“What should I do”

114. Thank You!
Eli ShechtmanPhilipp Krähenbühl
Alyosha Efros
Yong Jae LeePhillip Isola
Taesung Park Richard Zhang Tinghui ZhouTing-Chun Wang Deepak Pathak
Trevor DarrellRavi
Ramamoorthi
Nima K. Kalantari
Jue WangOliver Wang Aseem Agarwala
Eric Tzeng
Ce Liu Miki Rubinstein
Judy Hoffman
Manmohan
Chandraker
Young GengAngela Lin
Aquarius

115. Questions?
@LynnHo

116. Domain adaption: train on source, adapt to target
• A recent survey: ~300 papers
• Minimizing distribution distance
– Borgwardt 06, Mansour 09, Pan 09, Fernando 13
• Deep model adaption
– Chopra 13, Tzeng 14, Long 15, Ganin 15, Hoffman 17.
Slides credit: Judy Hoffman

117. State-of-the-art domain adaption method
Per-class accuracy Per-pixel accuracy
Oracle (Train and test on Real) 60.3 93.1
Train on CG, test on Real 17.9 54.0
FCN in the wild [Hoffman et al.] 27.1 (+6.0) -
VGG-FCN: 17.9 [Long et al. 2015]; Dilated-VGG-FCN: 21.1 [Fisher and Koltun 2015’]
Train on ‘free’ synthetic data (GTA5) Test on real images
adapt

118. Cats are as popular as GANs
• GitHub: github.com/junyanz/CatPapers
• 90% data is visual; most of visual data are about Cats.
• 70+ vision, learning and graphics papers.

120. Most influential cat paper
• Fred Attneave. “Some informational aspects of visual
perception”. Psychological Review (1954).

121. Questions?

122. User-Guided Colorization
[Zhang*, Zhu* et al. SIGGRAPH 17]
Grayscale image
Output colorization
User colors, mask
Raw Data

123. Training Details: Objective function
• Conditional GAN

124. Training Details: Objective function
• Conditional GAN + L1
𝑦
,
{ }
• Stable training + fast convergence.

125. Training Details: Discriminator 𝐷
[Radford et al., 2015]
ImageGAN
Slide from Victor Garcia
• Faster, fewer parameters; Arbitrarily large images
• Equal or better results
Our discriminator
PatchGAN

126. Effects of discriminator 𝐷

127. Training Details: Generator 𝐺
Encoder-decoder
U-Net
[Ronneberger et al.]

128. Shallower depth of field

129. iPhone DSLR iPhone DSLR

130. Summer ↔ Winter

131. Ablation study on paired dataset
the same output
mode collapse!

132. Ablation study on paired dataset
the same output
mode collapse!

133. Ablation Study on Cityscapes dataset

134. Training Details: Objective
• GANs with cross-entropy loss
• Least square GANs [Mao et al. 2016]
Stable training + better results
Vanishing gradients

135. Training Details: Generator 𝐺
Encoder-decoderU-Net
[Ronneberger et al.]
ResNet [He et al.] [Johnson et al.]
• Both have skip connections
• ResNet: fewer parameters
better for ill-posed problems