Introduction (application) of generative models for general audiences. Many figures are borrowed from https://lilianweng.github.io.

1. Generation:
Deep Generative Models

2. What is Generative Model?
• Generative model learns the distribution of data without label
• Create new data & Modify existing data
• Image/video/language/speech generation
• Data augmentation & semi-supervised learning
• Data privacy (e.g., public release of medical dataset)

3. What is Generative Model?
• Generative model learns the distribution of data without label
• Unsupervised representation learning
• Learning “good” representation with unlabeled data
• Design an auxiliary task (hence often called self-supervised learning)
• Generative model is a popular approach for unsupervised learning

4. Major Breakthroughs in Deep Generative Models
1980 1985 1990 2000 2005 2010 2015
1985 2006
Boltzmann machine (1985)
• By G. Hinton et al.
• Undirected graphical model
• Computationally expensive
Helmholtz machine
(1986)
• Directed graphical
model
Contrastivedivergence(1989)
• G. Hinton et.al
• Easy method for training RBM
1986
Deep Boltzmann machine (2009)
• Undirected deep generative
model consists of stacks of RBM
• Layerwise training followed by
joint learning
Restricted Boltzmann
machine (1986)
• Bipartite version of BM
1995
Variational Autoencoder (2013)
• By Durk Kingma et al.
• Easy NN like back-propagation learning
in deep generative model
Greedilylayer-wisepre-training(2006)
• Deep Belief Networks
• Major breakthrough in learning
deep generative model Generative Adversarial Network
(2014)
• Large scale image generative model
G. Hinton, S. Ruslan D. Kingma, M. Welling I. GoodfellowG. Hinton, T. Sejnowski P. Smolensky G. Hinton, R. Neal
• Hierarchical feature learning• Restricted Boltzmann Machine • Contrastive Divergence • Variatianal Autoencoder
2002 2009 2013 2014 2015
Ladder Network (2015)
• Performance breakthrough in
Semi-supervised learning

5. Approaches for Generative Models
1. Flow-based (autoregressive) model
• Pros: exactly compute the probability of the data (many applications)
• Cons: slow inference (autoregressive) or low quality (non-autoregressive)
Autoregressive (e.g., PixelCNN)
Non-autoregressive (e.g., Normalizing Flow)

6. Approaches for Generative Models
2. Variational autoencoder (VAE)
• Pros: stable training & theoretical properties (lower bound of likelihood)
• Cons: known to produce blurry outputs1
1. Recent methods combine VAE and other methods, e.g., IAF-VAE (+ flow) or WAE (+ GAN) to improve the performance
Blurry!

7. Approaches for Generative Models
3. Generative adversarial network (GAN)
• Pros: good performance (most SOTA models are based on GAN)
• Cons: hard to train (alternating two networks leads instability)

8. Application: Image Generation
• BigGAN

9. Application: Image Generation
• StyleGAN (https://www.youtube.com/watch?v=kSLJriaOumA)

10. Application: Image-to-Image Translation
• pix2pix (paired)

11. Application: Image-to-Image Translation
• CycleGAN (unpaired)

12. Application: Image-to-Image Translation
• StarGAN (multi-domain)

13. Application: Image-to-Image Translation
• MUNIT (diverse output)

14. Application: Image-to-Image Translation
• InstaGAN (shape modification) (from our lab)

15. Application: Emoji Generation
• DTN (create personal avatar)

16. Application: Semantic Manipulation
• pix2pixHD (https://www.youtube.com/watch?v=3AIpPlzM_qs)

17. Application: Pose Guided Generation
• PG2 (change pose of person)

18. Application: Cloth Extraction
• PixelDTGAN (extract cloth from image)

19. Application: Text-to-Image Synthesis
• Reed et al.

20. Application: Text-to-Image Synthesis
• Hong et al. (control location with bounding box)

21. Application: Video-to-Video Translation
• vid2vid (paired) (https://www.youtube.com/watch?v=HCqXJth9t_k)

22. Application: Video-to-Video Translation
• everybody dance now (with pose) (https://www.youtube.com/watch?v=PCBTZh41Ris)

23. Application: Video-to-Video Translation
• Recycle-GAN (unpaired) (https://www.youtube.com/watch?v=F51RCdDIuUw)

24. Application: Data Augmentation
• DAGAN (augment data to improve neural network performance)

25. Application: Anomaly Detection
• AnoGAN (find anomaly from given data)