The deep learning AI revolution has been sweeping the world for a decade now. Deep neural nets are routinely used for tasks like translation, fraud detection, and image classification. PwC estimates that they will create $15.7 trillion/year of value by 2030. But most current networks are "discriminative" in that they directly map inputs to predictions. This type of model requires lots of training examples, doesn't generalize well outside of its training set, creates inscrutable representations, is subject to adversarial examples, and makes knowledge transfer difficult. People, in contrast, can learn from just a few examples, generalize far beyond their experience, and can easily transfer and reuse knowledge. In recent years, new kinds of "generative" AI models have begun to exhibit these desirable human characteristics. They represent the causal generative processes by which the data is created and can be compositional, compact, and directly interpretable. Generative AI systems that assist people can model their needs and desires and interact with empathy. Their adaptability to changing circumstances will likely be required by rapidly changing AI-driven business and social systems. Generative AI will be the engine of future AI innovation.

1. The Future of AI is Generative
(not Discriminative)
Steve Omohundro

2. z
Observed
Variables:
Hidden (Latent)
Variables:
Discriminative
Models
Generative
Models
z
x x
Bayesian: p(z|x) p(z) and p(x|z)

3. The Future of AI is Generative
1. The deep learning revolution has been discriminative
2. We now have brain-level hardware but not software
3. Generative 1.0: GANs, VAEs, GPT-3, etc.
4. Generative NG: Causality, Theory of Mind, Reasoning
5. AI Empathy and Human/AI co-creation
6. Will be the driver for innovative AI business and society

4. 2012 AlexNet: The Deep Learning Revolution!
https://paperswithcode.com/paper/imagenet-classification-with-deep
https://www.wired.com/2013/03/google-hinton/
Google purchased for $44 million!
https://www.wired.com/story/secret-auction-race-ai-supremacy-google-microsoft-baidu/

5. GPU Hardware and Big ImageNet Data
• NVIDIA GeForce GTX 580
• 1.6 TeraFLOPS, $499.99
https://www.techpowerup.com/gpu-specs/geforce-gtx-580.c270
ImageNet LSVRC-2010
1.2 million images
https://dl.acm.org/doi/10.1145/3065386

6. 2013 DeepMind DQN Plays 7 Atari Games
https://towardsdatascience.com/dqn-part-1-vanilla-deep-q-networks-6eb4a00febfb
https://arxiv.org/abs/1312.5602
Google purchased for $600 million!
https://www.cnbc.com/2020/12/17/deepmind-lost-649-million-and-alphabet-waived-a-1point5-billion-debt-.html

7. PwC: AI will create $15.7 trillion/year by 2030
https://www.pwc.com/gx/en/issues/data-and-analytics/publications/artificial-intelligence-study.html

8. NVIDIA DGX A100
5 PetaFLOPS
$199,000
https://www.nvidia.com/en-us/data-center/dgx-a100/

9. Human Brain: 1 PetaFLOP
https://www.openphilanthropy.org/brain-computation-report

10. Problems with Discriminative Models…
• Need lots of training data
• Have adversarial examples
• Don’t generalize beyond the
training set
• Inscrutable representations
• Knowledge is hard to transfer
• But can be applied by anyone,
anywhere, to anything!

11. Generative Models 1.0
• Bayesian Networks
• Generative Adversarial
Networks (GAN)
• Variational Autoencoders (VAE)
• Autoregressive Models
• Normalizing Flows
• SurVAE Networks
• …

13. NVIDIA Stylegan2
https://github.com/NVlabs/stylegan2

14. Transformer+VQGAN High-res Image Synthesis
https://compvis.github.io/taming-transformers/

15. GPT-3 Poetry
https://humanise.ai/blog/ai-writes-poetry/

16. DALL-E Generates Images from Text
https://openai.com/blog/dall-e/

17. More DALL-E:
https://openai.com/blog/dall-e/

18. Generative Models: The Next Generation
• Causality
• Simulation-based
• Generative process models reality
• Semantic and explainable
• Composable
• Transferable
• Metalearning
• Creative Synthesis

19. Will more red or yellow blocks fall if bumped?
https://www.pnas.org/content/110/45/18327

20. Human Cognitive Core
1. Objects and mechanical interactions
2. Agents and goal-directed actions
3. Sets and numbers
4. Geometric relationships
Infants, Adults, Cultures, Primates
- Spelke and Kinzler
https://www.today.com/shop/best-gifts-babies-our-
2017-holiday-gift-guide-t118481
https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-
7687.2007.00569.x

21. Google Dreamer V2
https://github.com/danijar/dreamerv2

22. Probabilistic Programming
• E.g. Pyro built on PyTorch
• Simulate probabilistic agents
• Probabilistic Inference for
modeling and reasoning
• Program synthesis (e.g.
DreamCoder) to generate new
programs
https://arxiv.org/abs/2006.08381v1
https://arxiv.org/abs/1604.00289

23. PPL Theory of Mind – AI Empathy
• Reasoning about reasoning
• Nested inference
• Inverse Reinforcement Learning
• AI Teachers – model student
• AI Counselors – model dysfunction
• AI Co-Creators – model intent
https://www.sciencedirect.com/science/article/abs/pii/S1389041713000387
https://www.psychalive.org/empathy-can-help-us-right-now/

24. Generative Models for AI Innovation
• Generative AI models will drive
rapid innovation
• Shift investment and business
• Shift social and entertainment
• Require generative models to
handle rapid change
• Generative models can
empathize and co-create
• A glorious human future much
bigger than current projections!
https://www.surfertoday.com/environment/sunrise-and-
sunset-interesting-facts-about-the-golden-hour