My slides from the presentation I gave

1. SOME EXPERIMENTS I HAVE DONE
WITH ART + DEEP LEARNING
Jason Toy

2. THE PRESENTATION
show some of my experiments, both testing the limits of other
people’s models and training my own models
overview of how the models work
for artists and machine learners in the audience, i tried to make it
so you will all learn a little
hopefully inspire some of you to go out and build something
awesome

3. MY BACKGROUND - JASONTOY
my main passion is general artificial intelligence
studied math and computer science
generalists, program a little of everything, master of nothing
founded a couple of companies: rubynow, socmetrics - using ML for
mining social media
CEO of filepicker,sold in beginning of 2016
exploring the intersection of machine learning,art, entrepreneurship

4. DEEP LEARNINGVSTRADITIONAL
MACHINE LEARNING
mostly automated feature extraction

5. DEEP LEARNINGVSTRADITIONAL
MACHINE LEARNING
much better at learning nonlinear relationships

6. WHAT IS GENERATIVE
MODELING
generative vs discriminative
architect of models
GM around for a long time - used in architect, design,games,etc
miniature systems that mimic something in real life,“artist in a
box”
more fun; I'm not as interested to increase ad clickthrough rates

7. DISCRIMINATIVEVS GENERATIVE
generative:
naive bayes
LDA
deep learning
discriminative:
SVM
random forest
linear/logistic
regression

8. GENERATIVE DEEP MODELS
tweak-able output w/ vectors

9. EXPERIMENTS

10. CHAR-RNN
“I'm not going anywhere. I will bring the poorly educated back
bigger and better. It's an incredible movement. ”
“We're losing companies, the economy.We are going to save it.
We're going to bring the party. Let's Make America Great Again”
“I want to thank the volunteers.They've been unbelievable, they
work like endlessly, you know, they don't want to die. My
leadership is good”

11. CHAR-RNN
1vanilla (image classification) 2 sequence output (image -> text)
3 sequence input ( sentiment analysis)
4 seq2seq (machine translation) 5 synced seq2seq (video
classification)

12. CHAR-RNN
RNN - recurrent because they perform the same task for every
element of a sequence; typically 2-3 layers
LSTM - long short term memory
similar, state is calculated differently

13. MY CHAR-RNN EXPERIMENTS
what does Hellen Keller think?
seeing is like or inspirents of a kiss licks, in child, for the last decting
of accomplish with me for the mistakes in silence is to keep the
moments filled whiter, the chaps of the house language was sends a
humanise.
i wish i could presepred its repepenting and the days like the poor
discuss of language of the poem in the letters, dotiment in the endless
good and eager and over the charicality of the hall of rubbings that I
hapmende the comprehend, the birds like your mind to perhaps the
not wind I should do?

14. MY CHAR-RNN EXPERIMENTS
“i love you. Now her before it just numberse idevening with the
press over. I was probably ever need to ever admit? Right” -
Trump
“life is an economy. I was in the LGBT communities can to the
worst of the gun not only the fight are of us safe and I start up
these are not grow…” - Hillary

15. FUTURE CHAR-RNN
EXPERIMENTS
train a model to talk like a person with little data? transfer
learning?
could we train a model off of a standard “human” model ?
could we train a model to talk in different emotions/styles?

16. DEEP DREAMING / INCEPTION

17. A MACHINE LEARNING IMAGE
CLASSIFIER

18. LAYERS LEARNED FEATURES
architects:
imagenet
googlenet
alexnet

20. GOOGLENET

21. LAYER AMPLIFICATION
objective function: activate as many neurons in a layer
key trick: push back to image
feedback loop
choose different layers for different effects:
conv2/3x3,inception_3a,etc

22. TEST IMAGE

24. –Johnny Appleseed
“Type a quote here.”

26. TRAINING MY OWN DREAMS

30. INCEPTION FUTURE
EXPERIMENTS
train with different image sets - sea life, reptiles?
different objective function - activate only 1 group of neurons?
selective regions of hallucinating?
testing different network architects

31. NEURALSTYLE
Paint images in the style of any painting

32. A NEURAL ALGORITHM OF
ARTISTIC STYLE
paper: http://arxiv.org/abs/1508.06576
The key finding of this paper is that the representations of
content and style in the CNNs are separable.
CNNs - convolutional Neural Network

33. high layers in the network act as the content of the image
style computed from multiple layers’ filter responses

35. ]
?{;.
/ΠK
;

36. NEURALSTYLE FUTURE
EXPERIMENTS
can we automatically find the “good” images from a combination?
can we know beforehand if a combo style/content will look
good?
currently trained on vggnet data, what happens if we train it on a
different data set, will the art look different?
will a different architect make better art?

37. MULTIMODAL! STORYTELLING

38. I ACCIDENTALLY GAVETHE ANIMAL
BACK OF MY HEAD , BREATHING
DEEPLY .THERE WAS NO DOUBT IN
HER EYES ,AND I COULDTELL BY
THE LOOK ON HIS FACETHAT HE
DID N'T APPROVE OF WHAT WAS
HAPPENINGTO ME . IN FACT , IT
MUST HAVE BEEN ONE OFTHOSE
RARE OCCASIONS ,AS WELL AS A
PET ANIMAL . HER SCENT FILLED
THE AIR .THAT 'S WHAT SHE WAS
LOOKING FOR ,AND NOW SHE
HADTO STAY AWAKE LONG
ENOUGHTO DIG UPTHE LEASH

39. SKIP-THOUGHTVECTORS
sentence -> vectors

41. TRUMP STORYTELLER

42. FUTURE NEURAL STORY
EXPERIMENTS
train with different text
a “seeing” Hellen Keller version
train on different visual features

43. AND MANY OTHER EXPERIMENTS…….
HOPEFULLY INSPIRING

44. DATA IS ESSENTIAL
many of these models are built on public datasets
always has been a problem; bigger problem for DL and general
models
very hard to get data; how can this be solved?
constantly on my mind ; lets connect me if interested

45. DL IS NOT ALL FUN AND
UNICORNS
data issue
specialized software/hardware pipelines; GPUs
be prepared to wait; think weeks, not hours
model tuning
architect tuning
techniques and architects changing everyday

46. WHY?
I dream of building larger models
AGI and multi modal models
larger experiments
want to collaborate with cool artists and coders
fun? lets talk!

47. LINK APPENDIX

48. STUDY LINKS
what is deep learning: http://www.jtoy.net/2016/02/14/opening-
up-deep-learning-for-everyone.html
generative models: https://en.wikipedia.org/wiki/
Generative_model
discriminative models: https://en.wikipedia.org/wiki/
Discriminative_model

49. TEST LIVE MODEL LINKS
trump char-rnn model: http://somatic.io/models/WZmmBjZ9
neural style model: http://www.somatic.io/models/5BkaqkMR
neural talk model: http://somatic.io/models/qoEGanRe
romance story telling: http://somatic.io/models/2n6g7RZQ

50. LINKS
VGG net data used: http://www.robots.ox.ac.uk/~vgg/research/
very_deep/
tensorflow version: https://github.com/anishathalye/neural-style
neural style paper: http://arxiv.org/abs/1508.06576
char-rnn code: https://github.com/somaticio/char-rnn-tensorflow
mscoco: http://mscoco.org
imagenet: http://image-net.org/

51. LINKS
char-rnn: https://github.com/somaticio/char-rnn-tensorflow
tensorflow char-rnn tutorial: https://www.tensorflow.org/
versions/r0.9/tutorials/seq2seq/index.html#recurrent-neural-
networks
neuralstyle: https://github.com/anishathalye/neural-style

52. –John Dewey
“Every great advance in science has issued from a new
audacity of imagination.”
Jason Toy
jason@somatic.io
I write here:
http://jtoy.net http://somatic.io/bog
my models here: http://somatic.io
@jtoy
QUESTIONS?