Philosophy of Deep Learning

1. Melanie Swan Philosophy & Economic Theory New School for Social Research, NY NY melanie@BlockchainStudies.org Pfizer, New York NY, March 30, 2017 Slides: http://slideshare.net/LaBlogga Philosophy of Deep Learning: Deep Qualia, Statistics, and Blockchain Image credit: Nvidia

2. 30 Mar 2017 Deep Learning ASA P value misuse statement 1 Source: http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503, http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108  ASA principles to guide P value use  The P value alone cannot determine whether a hypothesis is true or whether results are important

3. 30 Mar 2017 Deep Learning 2 Melanie Swan  Philosophy and Economic Theory, New School for Social Research, New York NY  Founder, Institute for Blockchain Studies  Singularity University Instructor; Institute for Ethics and Emerging Technology Affiliate Scholar; EDGE Essayist; FQXi Advisor Traditional Markets Background Economics and Financial Theory Leadership New Economies research group Source: http://www.melanieswan.com, http://blockchainstudies.org/NSNE.pdf, http://blockchainstudies.org/Metaphilosophy_CFP.pdf https://www.facebook.com/groups/NewEconomies

4. 30 Mar 2017 Deep Learning Deep Learning vocabulary What do these terms mean?  Deep Learning, Machine Learning, Artificial Intelligence  Deep Belief Net  Perceptron, Artificial Neuron  MLP/RELU: Multilayer Perceptron  Artificial Neural Net  TensorFlow, Caffe, Theano, Torch, DL4J  Recurrent Neural Nets  Boltzmann Machine, Feedforward Neural Net  Open Source Deep Learning Frameworks  Google DeepDream, Google Brain, Google DeepMind 3

5. 30 Mar 2017 Deep Learning Key take-aways 1. What is deep learning?  Advanced statistical method using logistic regression  Deep learning is a sub-field of machine learning and artificial intelligence 2. Why is deep learning important?  Crucial method of algorithmic data manipulation 3. What do I need to know (as a data scientist)?  Awareness of new methods like deep learning needed to keep pace with data growth 4

6. 30 Mar 2017 Deep Learning Deep Learning and Data Science 5  Not optional: older algorithms cannot perform to generate requisite insights Source: http://blog.algorithmia.com/introduction-to-deep-learning-2016

7. 30 Mar 2017 Deep Learning Agenda  Deep Learning Basics  Definition, operation, drawbacks  Implications of Deep Learning  Deep Learning and the Brain  Deep Learning Blockchain Networks  Philosophy of Deep Learning 6 Image Source: http://www.opennn.net

8. 30 Mar 2017 Deep Learning Deep Learning Context 7 Source: Machine Learning Guide, 9. Deep Learning

9. 30 Mar 2017 Deep Learning Deep Learning Definition “machines that learn to represent the world” – Yann LeCun  Deep learning is a class of machine learning algorithms that use a cascade of layers of processing units to extract features from data  Each layer uses the output from the previous layer as input  Two kinds of learning algorithms  Supervised (classify labeled data)  Unsupervised (find patterns in unlabeled data)  Two phases: training (existing data) and test (new data) 8 Source: Wikiepdia, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-on-deeplearning

10. 30 Mar 2017 Deep Learning What is Learning? When algorithms detect a system’s features or rules 9 Single-purpose AI: Deep Blue, 1997 Hard-coded rules Multi-purpose AI structure: AlphaGo, 2016 Algorithm-detected rules, reusable template Deep Learning machine General purpose AI: Deep Qualia, 2xxx? Novel situation problem-solving, Algorithm edits/writes rules Question-answering AI: Watson, 2011 Natural-language processing Deep Learning prototype

11. 30 Mar 2017 Deep Learning Deep Learning: what is the problem space? 10 Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" http://t.co/nPFlPZzMEJ  Level 1 – basic application areas  Image, text, speech recognition  Multi-factor recognition (label image with text)  Sentiment analysis  Level 2 – complex application areas  Autonomous driving  Disease diagnosis, tumor recognition, X-ray/MRI interpretation  Seismic analysis (earthquake, energy, oil and gas)

12. 30 Mar 2017 Deep Learning Deep Learning Taxonomy High-level fundamentals of machine learning 11 Source: Machine Learning Guide, 9. Deep Learning; AI (artificial intelligence) Machine learning Other methods Supervised learning (labeled data: classification) Unsupervised learning (unlabeled data: pattern recognition) Reinforcement learning Shallow learning (1-2 layers) Deep learning (5-20 layers (expensive)) Recurrent nets (text, speech) Convolutional nets (images) Neural Nets (NN) Other methods Bayesian inference Support Vector Machines Decision trees K-means clustering K-nearest neighbor

13. 30 Mar 2017 Deep Learning What is the problem? Computer Vision (and speech and text recognition) 12 Source: Quoc Le, https://arxiv.org/abs/1112.6209; Yann LeCun, NIPS 2016, https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view Marv Minsky, 1966 “summer project” Jeff Hawkins, 2004, Hierarchical Temporal Memory (HTM) Quoc Le, 2011, Google Brain cat recognition Yann LeCun, 2016, Predictive Learning, Convolutional net for driving

14. 30 Mar 2017 Deep Learning Image Recognition: Basic Concept 13 Source: https://developer.clarifai.com/modelshttps://developer.clarifai.com/models How many orange pixels? Apple or Orange? Melanoma risk or healthy skin? Degree of contrast in photo colors?

15. 30 Mar 2017 Deep Learning Regression (review)  Linear regression  Predict continuous set of values (house prices)  Logistic regression  Predict binary outcomes (0,1) 14 Logistic regression (sigmoid function) Linear regression

16. 30 Mar 2017 Deep Learning Deep Learning Architecture 15 Source: Michael A. Nielsen, Neural Networks and Deep Learning

17. 30 Mar 2017 Deep Learning Example: Image recognition 1. Obtain training data set 2. Digitize pixels (convert images to numbers)  Divide image into 28x28 grid, assign a value (0-255) to each square based on brightness 3. Read into vector (array) (28x28 = 784 elements per image) 16 Source: Quoc V. Le, A Tutorial on Deep Learning, Part 1: Nonlinear Classifiers and The Backpropagation Algorithm, 2015, Google Brain, https://cs.stanford.edu/~quocle/tutorial1.pdf

18. 30 Mar 2017 Deep Learning Deep Learning Architecture 4. Load spreadsheet of vectors into deep learning system  Each row of spreadsheet is an input 17 Source: http://deeplearning.stanford.edu/tutorial; MNIST dataset: http://yann.lecun.com/exdb/mnist 1. Input 2. Hidden layers 3. Output X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Vector data

19. 30 Mar 2017 Deep Learning What happens in the Hidden Layers? 18 Source: Michael A. Nielsen, Neural Networks and Deep Learning  First layer learns primitive features (line, edge, tiniest unit of sound) by finding combinations of the input vector data that occur more frequently than by chance  Logistic regression performed and encoded at each processing node (Y/N (0,1)), does this example have this feature?  Feeds these basic features to next layer, which trains itself to recognize slightly more complicated features (corner, combination of speech sounds)  Feeds features to new layers until recognizes full objects

20. 30 Mar 2017 Deep Learning Feature Recognition in the Hidden Layers 19 Source: Jann LeCun, http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

21. 30 Mar 2017 Deep Learning What happens in the Hidden Layers? 20 Source: Nvidia  First hidden layer extracts all possible low-level features from data (lines, edges, contours), next layers abstract into more complex features of possible relevance

22. 30 Mar 2017 Deep Learning Deep Learning  Core concept:  Deep Learning systems learn increasingly complex features 21 Source: Andrew Ng

23. 30 Mar 2017 Deep Learning Deep Learning  Google Deep Brain recognizes cats 22 Source: Quoc V. Le et al, Building high-level features using large scale unsupervised learning, 2011, https://arxiv.org/abs/1112.6209

24. 30 Mar 2017 Deep Learning Deep Learning Architecture 23 Source: Michael A. Nielsen, Neural Networks and Deep Learning 1. Input 2. Hidden layers 3. Output guess (0,1)

25. 30 Mar 2017 Deep Learning Deep Learning Math Test new data after system iterates 24 1. Input 2. Hidden layers 3. Output X X X X X X X X X X X X X X X Source: http://deeplearning.stanford.edu/tutorial; MNIST dataset: http://yann.lecun.com/exdb/mnist  Linear algebra: matrix multiplications of input vectors  Statistics: logistic regression units (Y/N (0,1)), probability weighting and updating, inference for outcome prediction  Calculus: optimization (minimization), gradient descent in back-propagation to avoid local minima with saddle points Feed-forward pass (0,1) 0.5 Back-propagation pass; update probabilities .5.5 .5.5.5 0 01 .75 .25 Inference Guess Actual

26. 30 Mar 2017 Deep Learning Hidden Layer Unit, Perceptron, Neuron 25 Source: http://deeplearning.stanford.edu/tutorial; MNIST dataset: http://yann.lecun.com/exdb/mnist 1. Input 2. Hidden layers 3. Output X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X  Unit (processing unit, logistic regression unit), perceptron (“multilayer perceptron”), artificial neuron

27. 30 Mar 2017 Deep Learning Kinds of Deep Learning Systems What Deep Learning net to choose? 26 Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" http://t.co/nPFlPZzMEJ  Supervised algorithms (classify labeled data)  Image (object) recognition  Convolutional net (image processing), deep belief network, recursive neural tensor network  Text analysis (name recognition, sentiment analysis)  Recurrent net (iteration; character level text), recursive neural tensor network  Speech recognition  Recurrent net  Unsupervised algorithms (find patterns in unlabeled data)  Boltzmann machine or autoencoder

28. 30 Mar 2017 Deep Learning Advanced Deep Learning Architectures 27 Source: http://prog3.com/sbdm/blog/zouxy09/article/details/8781396  Deep Belief Network  Connections between layers not units  Establish weighting guesses for processing units before run deep learning system  Used to pre-train systems to assign initial probability weights (more efficient)  Deep Boltzmann Machine  Stochastic recurrent neural network  Runs learning on internal representations  Represent and solve combinatoric problems Deep Boltzmann Machine Deep Belief Network

29. 30 Mar 2017 Deep Learning Convolutional net: Image Enhancement  Google DeepDream: Convolutional neural network enhances (potential) patterns in images; deliberately over-processing images 28 Source: Georges Seurat, Un dimanche après-midi à l'Île de la Grande Jatte, 1884-1886; http://web.cs.hacettepe.edu.tr/~aykut/classes/spring2016/bil722; Google DeepDream uses algorithmic pareidolia (seeing an image when none is present) to create a dream-like hallucinogenic appearance

30. 30 Mar 2017 Deep Learning How big are Deep Learning systems?  Google Deep Brain cat recognition, 2011  1 billion connections, 10 million images (200x200 pixel), 1,000 machines (16,000 cores), 3 days, each instantiation of the network spanned 170 servers, 20,000 object categories  State of the art, 2015-2016  Nvidia facial recognition example, 2016, 100 million images, 10 layers, 18 parameters, 30 exaflops, 30 GPU days  Google, 11.2-billion parameter system  Lawrence Livermore Lab, 15-billion parameter system  Digital Reasoning, 2015, cognitive computing (Nashville TN), 160 billion parameters, trained on three multi-core computers overnight 29 Source: https://futurism.com/biggest-neural-network-ever-pushes-ai-deep-learning, Digital Reasoning paper: https://arxiv.org/pdf/1506.02338v3.pdf

31. 30 Mar 2017 Deep Learning Deep Learning, Deep Flaws?  Even though now possible, still early days  Expensive and inefficient, big systems  Only available to massive data processing operations (Google, Facebook, Microsoft, Baidu)  Black box: we don’t know how it works  Reusable model but still can’t multi-task  Atari example: cannot learn multiple games  Drop Asteroids to learn Frogger  Add common sense to intelligence  Background information, reasoning, planning  Memory (update and remember states of the world)  …Deep Learning is still a Specialty System 30 AlphaGo applied to Atari games Source: http://www.theverge.com/2016/10/10/13224930/ai-deep-learning-limitations-drawbacks

32. 30 Mar 2017 Deep Learning We had the math, what took so long?  A) Hardware, software, processing advances; and B) more data  Key advances in hardware chips  GPU chips (graphics processing unit): 3D graphics cards designed to do fast matrix multiplication  Google TPU chip (tensor processing unit): custom ASICs for machine learning, used in AlphaGo  Training the amount of data required was too slow to be useful  Now can train neural nets quickly, still expensive 31 Tensor (Scalar (x,y,z), Vector (x,y,z)3, Tensor (x,y,z)9) Google TPU chip (Tensor Processing Unit), 2016 Source: http://www.techradar.com/news/computing-components/processors/google-s-tensor-processing-unit-explained-this-is-what- the-future-of-computing-looks-like-1326915

34. 30 Mar 2017 Deep Learning Deep Learning and the Brain 33

35. 30 Mar 2017 Deep Learning  Deep learning neural networks are inspired by the structure of the cerebral cortex  The processing unit, perceptron, artificial neuron is the mathematical representation of a biological neuron  In the cerebral cortex, there can be several layers of interconnected perceptrons 34 Deep Qualia machine? General purpose AI Mutual inspiration of neurological and computing research

36. 30 Mar 2017 Deep Learning Deep Qualia machine?  Visual cortex is hierarchical with intermediate layers  The ventral (recognition) pathway in the visual cortex has multiple stages: Retina - LGN - V1 - V2 - V4 - PIT – AIT  Human brain simulation projects  Swiss Blue Brain project, European Human Brain Project 35 Source: Jann LeCun, http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

38. 30 Mar 2017 Deep Learning Deep Learning Blockchain Networks 37

39. 30 Mar 2017 Deep Learning Blockchain Technology 38 Source: http://www.amazon.com/Bitcoin-Blueprint-New-World-Currency/dp/1491920491

40. 30 Mar 2017 Deep Learning What is Blockchain Technology?  Blockchain technology is an Internet- based ledger system for submitting, logging, and tracking transactions  Allows the secure transfer of assets (like money) and information, computationally, without a human intermediary  Secure asset transfer protocol, like email  First application is currency (Bitcoin) and FinTech re-engineering, subsequent applications in algorithmic data processing 39 Source: Blockchain Smartnetworks, https://www.slideshare.net/lablogga/blockchain-smartnetworks

41. 30 Mar 2017 Deep Learning Deep Learning Blockchain Networks Help resolve Deep Learning challenges 40 Source: http://www.melanieswan.com, http://blockchainstudies.org/NSNE.pdf, http://blockchainstudies.org/Metaphilosophy_CFP.pdf  Deep Learning systems need greater capacity  Put Deep Learning systems on the Internet in a secure- trackable, remunerable way; distributed not parallel systems  Deep Learning systems need more complexity and side modules  Instantiate common sense, memory, planning modules  Deep Learning systems do not reveal what happens in the hidden layers  Track arbitrarily-many transactions with smart contracts  Core blockchain functionality employed  Automated coordination of massive amounts of operations via smart contracts (automatically-executing Internet-based programs)

42. 30 Mar 2017 Deep Learning Deep Learning systems go online with Blockchain  Key point is to put Deep Learning systems on the Internet  Blockchain is perfect technology to control secure access, yet have all of the 24/7 availability, flexibility, scale, and side modules needed  Provide global infrastructure to work on current problems  Genomic disease, protein modeling, financial risk assessment, astronomical data analysis 41

43. 30 Mar 2017 Deep Learning  Combine Deep Learning and Blockchain Technology  Deep learning technology, particularly coupled with blockchain systems, might create a new kind of global computing platform  Deep Learning and Blockchains are similar  Indicative of a shift toward having increasingly sophisticated and automated computational tools  Mode of operation of both is making (statistically-supported) guesses about reality states of the world  Predictive inference (deep learning) and cryptographic nonce- guesses (blockchain)  Current sense-making model of the world, we are guessing at more complex forms of reality 42 Advanced Computational Infrastructure Deep Learning Blockchain Networks

45. 30 Mar 2017 Deep Learning Philosophy of Deep Learning 44

46. 30 Mar 2017 Deep Learning 45 Human’s Role in the World is Changing Sparse data we control Abundant data controls us?  Deep Learning is emphasizing the presence of Big Data

47. 30 Mar 2017 Deep Learning Philosophy of Deep Learning - Definition 46  The Philosophy of Deep Learning is the branch of philosophy concerned with the definition, methods, and implications of Deep Learning  Internal Industry Practice  Internal to the field as a generalized articulation of the concepts, theory, and systems that comprise the overall use of deep learning algorithms  External Social Impact  External to the field, considering the impact of deep learning more broadly on individuals, society, and the world

48. 30 Mar 2017 Deep Learning 3 Kinds of Philosophic Concerns  Ontology (existence, reality)  What is it? What is deep learning?  What does it mean?  Epistemology (knowledge)  What knowledge are we gaining from deep learning?  What is the proof standard?  Axiology or Valorization (ethics, aesthetics)  What is noticed, overlooked?  What is ethical practice?  What is beauty, elegance? 47 Sources: http://www.melanieswan.com/documents/Philosophy_of_Big_Data_SWAN.pdf

49. 30 Mar 2017 Deep Learning Explanation: does the map fit the territory? 48 1626 map of “the Island of California” Source: California Is An Island Off the Northerne Part of America; John Speed, "America," 1626, London  Explanandum  What is being explained  Explanans  The explanation

50. 30 Mar 2017 Deep Learning How do we understand reality?  Methods, models, and tools  Descartes, Optics, 1637  Deep Learning, 2017 49

52. 30 Mar 2017 Deep Learning Key take-aways  What is deep learning?  Advanced statistical method using logistic regression  Deep learning is a sub-field of machine learning and artificial intelligence  Why is deep learning important?  Crucial method of algorithmic data manipulation  What do I need to know (as a data scientist)?  Awareness of new methods like deep learning needed to keep pace with data growth 51

53. 30 Mar 2017 Deep Learning Conclusion  Deep learning systems are machine learning algorithms that learn increasingly complex feature sets from data via hidden layers  Deep qualia systems might be a step forward in brain simulation in computer networks and general intelligence  Next-generation global infrastructure: Deep Learning Blockchain Networks merging deep learning systems and blockchain technology 52

54. 30 Mar 2017 Deep Learning Resources 53 Distill, a visual, interactive journal for machine learning research http://distill.pub/

55. Melanie Swan Philosophy & Economic Theory New School for Social Research, NY NY melanie@BlockchainStudies.org Philosophy of Deep Learning: Deep Qualia, Statistics, and Blockchain Pfizer, New York NY, March 30, 2017 Slides: http://slideshare.net/LaBlogga Thank You! Questions? Image credit: Nvidia

Philosophy of Deep Learning

Du liest eine Vorschau.

Hier ansehen

Philosophy of Deep Learning

Weitere Verwandte Inhalte

Diashows für Sie (20)

Andere mochten auch (20)

Ähnlich wie Philosophy of Deep Learning (20)

Weitere von Melanie Swan (20)

Aktuellste (20)

Philosophy of Deep Learning