Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Slide	credit	from	Mark	Chang 1
Convolutional	Neural	Networks
• We	need	a	course	to	talk	about	this	topic
◦ http://cs231n.stanford.edu/syllabus.html
• How...
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
Image	Recognition
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
5
Image	Recognition
6
Local	Connectivity
Neurons	connect	to	a	small	
region
7
Parameter Sharing
• The	same	feature	in	different	positions
Neurons	
share	the	same	weights
8
Parameter Sharing
• Different	features	in	the	same	position
Neurons
have	different	weights
9
Convolutional	Layers
depth
widthwidth	depth
weights weights
height
shared	weight
10
Convolutional	Layers
c1
c2
b1
b2
a1
a2
a3
wb1
wb2
b1 =wb1a1+wb2a2
wb1
wb2
b2 =wb1a2+wb2a3
wc1
wc2
c1 =wc1a1+wc2a2
wc2
wc1
...
Convolutional	Layers
c1
b1
b2
a1
a2
d1
b3
a3
c2
d2
depth	=	2 depth	=	2
wc1
wc2
wc3
wc4
c1 = a1wc1 + b1wc2
+ a2wc3 + b2wc4
...
Convolutional	Layers
c1
b1
b2
a1
a2
d1
b3
a3
c2
d2
c1 = a1wc1 + b1wc2
+ a2wc3 + b2wc4
c2 = a2wc1 + b2wc2
+ a3wc3 + b3wc4
w...
Convolutional	Layers
A B C
A B C D
14
Hyper-parameters	of	CNN
• Stride • Padding
0 0
Stride	=	1
Stride	=	2
Padding	=	0
Padding	=	1
15
Example
Output
Volume	(3x3x2)
Input
Volume	(7x7x3)
Stride	=	2
Padding	=	1
http://cs231n.github.io/convolutional-networks/
...
Convolutional	Layers
http://cs231n.github.io/convolutional-networks/
17
Convolutional	Layers
http://cs231n.github.io/convolutional-networks/
18
Convolutional	Layers
http://cs231n.github.io/convolutional-networks/
19
Relationship	with	Convolution
y[n] =
X
k
x[k]w[n k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
w[0 k]
n
y[0] = x[ 2]w[2] + x[ 1]w[1] + x[...
Relationship	with	Convolution
y[n] =
X
k
x[k]w[n k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
w[1 k]
y[1] = x[ 1]w[2] + x[0]w[1] + x[2...
Relationship	with	Convolution
y[n] =
X
k
x[k]w[n k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
y[2] = x[0]w[2] + x[1]w[1] + x[2]w[0]
w[...
Relationship	with	Convolution
y[n] =
X
k
x[k]w[n k]
x[n]
w[n]
n
n
y[n]
x[k]
k
k
n
w[4 k]
y[4] = x[2]w[2] + x[3]w[1] + x[4]...
Nonlinearity
• Rectified	Linear	(ReLU)
nout =
⇢
nin if nin > 0
0 otherwise
nin n
2
6
6
4
1
4
3
1
3
7
7
5
2
6
6
4
1
4
0
1
3...
Why	ReLU?
• Easy	to	train
• Avoid	gradient	vanishing	problem
Sigmoid
saturated
gradient	≈	0 ReLU not	saturated
25
Why	ReLU?
• Biological	reason
strong stimulation
ReLU
weak stimulation
neuron t
v
strong stimulation
neuron
t
v
weak stimu...
Pooling	Layer
1 3 2 4
5 7 6 8
0 0 3 3
5 5 0 0
4 5
5 3
7 8
5 3
Maximum
Pooling
Average
Pooling
Max(1,3,5,7)	=	7 Avg(1,3,5,7...
Why	“Deep”	Learning?
28
Visual	Perception	of	Human	
http://www.nature.com/neuro/journal/v8/n8/images/nn0805-975-F1.jpg
29
Visual	Perception	of	Computer
Convolutional
Layer
Convolutional
Layer	 Pooling
Layer	
Pooling
Layer	
Receptive	Fields
Rece...
Visual	Perception	of	Computer
Input Layer
Convolutional
Layer	with
Receptive	Fields:
Max-pooling
Layer	with
Width	=3,	Heig...
Fully-Connected	Layer
• Fully-Connected	Layers	:	Global	feature	extraction
• Softmax Layer:	Classifier
Convolutional
Layer...
Visual	Perception	of	Computer
• Alexnet	
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
http://vision03.csail.mit.edu...
Training
• Forward	Propagation
n2 n1
n1(out)n2(out) n2(in) w21
34
n2(in) = w21n1(out)
n2(out) = g(n2(in)), g is activation...
Training
• Update	weights
n2 n1J
Cost	
function:
@J
@w21
=
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@w21
w21 w21 ⌘
@J
@w21
) w...
Training
• Update	weights
n2 n1J
Cost	
function:
n2(out) n2(in) w21 n1(out)
w21 w21 ⌘
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)...
Training
• Propagate	to	the	previous	layer
n2 n1J
Cost	
function:
@J
@n1(in)
=
@J
@n2(out)
@n2(out)
@n2(in)
@n2(in)
@n1(ou...
Training	Convolutional	Layers
• example:
a3
a2
a1
b2
b1
wb1
wb1
wb2
wb2
output input
Convolutional
Layer
To simplify the n...
Training	Convolutional	Layers
• Forward	propagation
a3
a2
a1
b2
b1
input
Convolutional
Layer
b1 = wb1a1 + wb2a2
b2 = wb1a2...
Training	Convolutional	Layers
• Update	weights
J
Cost	
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
wb1
wb1
@b1
@wb1
@b2
@wb1
wb...
Training	Convolutional	Layers
• Update	weights
a3
a2
a1
b2
b1 wb1
wb1
b1 = wb1a1 + wb2a2
b2 = wb1a2 + wb2a3
@b1
@wb1
= a1
...
Training	Convolutional	Layers
• Update	weights
a3
a2
a1
b2
b1
wb2 wb2 ⌘(
@J
@b1
@b1
@wb2
+
@J
@b2
@b2
@wb2
)
@b1
@wb2
@b2
...
Training	Convolutional	Layers
• Update	weights
a3
a2
a1
b2
b1
wb2
wb2
@b1
@wb2
= a2
@b2
@wb2
= a3
wb2 wb2 ⌘(
@J
@b1
a2 +
@...
Training	Convolutional	Layers
• Propagate	to	the	previous	layer
J
Cost	
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1
@a1
@b1...
Training	Convolutional	Layers
• Propagate	to	the	previous	layer
J
Cost	
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
b1 = wb1a1 ...
Max-Pooling	Layers	during	Training	
• Pooling	layers	have	no	weights
• No	need	to	update	weights
a3
a2
a1
b2
b1 a1 > a2
a2...
Max-Pooling	Layers	during	Training	
• Propagate	to	the	previous	layer
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1
@a1
= 1
a2 > a3
@b2...
Max-Pooling	Layers	during	Training	
• if	a1 =	a2	 ??
◦ Choose	the	node	with	smaller	index
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@J
...
Avg-Pooling	Layers	during	Training	
• Pooling	layers	have	no	weights
• No	need	to	update	weights
a3
a2
a1
b2
b1
b1 =
1
2
(...
Avg-Pooling	Layers	during	Training	
• Propagate	to	the	previous	layer
J
Cost	
function:
a3
a2
a1
b2
b1
@J
@b1
@J
@b2
@b1
@...
ReLU	during	Training
nout =
⇢
nin if nin > 0
0 otherwise
nin n
@nout
@nin
=
⇢
1 if nin > 1
0 otherwise
51
Training CNN
52
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
LeNet
◦ Paper:	
http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf
Yann	LeCun http://yann.lecun.com/exdb/lenet/
...
ImageNet	Challenge
• ImageNet	Large	Scale	Visual	Recognition	Challenge
◦ http://image-net.org/challenges/LSVRC/
• Dataset	...
ImageNet	Challenge
http://www.qingpingshan.com/uploads/allimg/160818/1J22QI5-0.png
56
AlexNet (2012)
• Paper:	
http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
• The	resurgence	of	Deep	Learning
Geoffrey	Hi...
VGGNet (2014)
• Paper:	https://arxiv.org/abs/1409.1556
D:	VGG16
E:	VGG19
All	filters	are	3x3
58
VGGNet
• More	layers	&	smaller	filters	(3x3)	is	better
• More	non-linearity,	fewer	parameters
One	5x5	filter
• Parameters:...
VGG	19
depth=64
3x3	conv
conv1_1
conv1_2
maxpool
depth=128
3x3	conv
conv2_1
conv2_2
maxpool
depth=256
3x3	conv
conv3_1
con...
GoogLeNet (2014)
• Paper:	
http://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf
22	layers	deep	network
Inception
Module
61
Inception Module
• Best	size?	
◦ 3x3?		5x5?
• Use	them	all,	and	combine
62
Inception Module
1x1	convolution
3x3	convolution
5x5	convolution
3x3	max-pooling
Previous	
layer
Filter
Concatenate
63
Inception Module	with	Dimension	Reduction
• Use	1x1	filters	to	reduce	dimension
64
Inception Module	with	Dimension	Reduction
Previous	
layer
1x1	convolution
(1x1x256x128)
Reduced
dimension
Input	size
1x1x2...
ResNet (2015)
• Paper:	https://arxiv.org/abs/1512.03385
• Residual	Networks
• 152	layers
66
ResNet
• Residual	learning:	a	building	block	
Residual
function
67
Residual	Learning	with	Dimension	Reduction
• using	1x1	filters
68
Pretrained	Model	Download
• http://www.vlfeat.org/matconvnet/pretrained/
◦ Alexnet:
◦ http://www.vlfeat.org/matconvnet/mod...
Using	Pretrained	Model
• Lower	layers:edge,	blob,	texture	(more	general)
• Higher	layers	:	object	part	(more	specific)
htt...
Transfer	Learning
• The	Pretrained Model	is	
trained	on	ImageNet
dataset
• If	your	data	is	similar	to	
the	ImageNet data
◦...
Transfer	Learning
• The	Pretrained	Model	is	
trained	on	ImageNet	
dataset
• If	your	data	is	far	different	
from	the	ImageN...
Tensorflow Transfer	Learning	Example
• https://www.tensorflow.org/versions/r0.11/how_tos/styl
e_guide.html
daisy
634
photo...
Tensorflow Transfer	Learning	Example
Fix	these	layers Train	this	layer
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
Visualizing	CNN
http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
76
Visualizing	CNN
CNN
CNN
flower
random
noise
filter
response
filter
response
77
filter
response:	
Gradient	Ascent
• Magnify	the	filter	response
random	
noise: x f
score: F =
X
i,j
fi,j
fi,j
lower		
scor...
filter
response:	
Gradient	Ascent
• Magnify	the	filter	response
random	
noise: x f
x
gradient:
fi,j
lower		
score
higher	
...
Gradient	Ascent
80
Different	Layers	of	Visualization
CNN
81
Multiscale	Image	Generation
visualize resize
visualize
resize
visualize
82
Multiscale	Image	Generation
83
Deep	Dream
• https://research.googleblog.com/2015/06/inceptionism-
going-deeper-into-neural.html
• Source	code:
https://gi...
Deep	Dream
85
Deep	Dream
86
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
Neural	Art
• Paper:	https://arxiv.org/abs/1508.06576
• Source	code	:	https://github.com/ckmarkoh/neuralart_tensorflow
cont...
The	Mechanism	of	Painting
BrainArtist
Scene Style ArtWork
Computer Neural	Networks
89
Misconception
90
Content	Generation
BrainArtistContent
Canvas
Minimize
the
difference
Neural	
Stimulation
Draw
91
Content	Generation
92
Filter	
ResponsesVGG19
Update	the
color	of	
the	pixels
Content
Canvas
Result
Width*Height
Depth
Mini...
Content	Generation
Layer	l’s	Filter	l	
Responses:
Layer	l’s	Filter		
Responses:
Input
Photo:
Input
Canvas:
Width*Height	(j...
Content	Generation
• Backward	Propagation
Layer	l’s	Filter	l	
Responses:
Input
Canvas:
VGG19
Update
Canvas
Learning	Rate
94
Content	Generation
95
Content	Generation
VGG19
conv1_2 conv2_2 conv3_4 conv4_4 conv5_2conv5_1
96
Style	Generation
VGG19Artwork
G
G
Filter	Responses Gram	Matrix
Width*Height
Depth
Depth
Depth
Position-
dependent
Position...
Style	Generation
1. .5
.5
.5
1.
1. .5 .25 1.
.5 .25 .5
.25 .25
1. .5 1.
Width*Height
Depth
k1 k2
k1
k2
Depth
Depth
Layer	l...
Style	Generation
Layer	l’s	
Filter	Responses
Layer	l’s	
Gram	Matrix
Layer	l’s	
Gram	Matrix
Input
Artwork:
Input
Canvas:
99
Style	Generation
VGG19
Filter
Responses
Gram	
Matrix
Minimize
the
difference
G
G
Style
Canvas
Update	the	color	of	
the	pix...
Style	Generation
101
Style	Generation
VGG19
Conv1_1 Conv1_1
Conv2_1
Conv1_1
Conv2_1
Conv3_1
Conv1_1
Conv2_1
Conv3_1
Conv4_1
Conv1_1
Conv2_1
Con...
Artwork	Generation
Filter	ResponsesVGG19
Gram	Matrix
103
Artwork	Generation
VGG19 VGG19
Conv1_1
Conv2_1
Conv3_1
Conv4_1
Conv5_1
Conv4_2
104
Artwork	Generation
105
Content	v.s.	Style
0.15 0.05
0.02 0.007
106
Neural	Doodle
• Paper:	https://arxiv.org/abs/1603.01768
• Source	code:	https://github.com/alexjc/neural-doodle
style
conte...
Neural	Doodle
• Image	analogy
108
Neural	Doodle
• Image	analogy
恐怖連結,慎入!
https://raw.githubusercontent.com/awentzonline/
image-analogies/master/examples/ima...
Real-time	Texture	Synthesis
• Paper:	https://arxiv.org/pdf/1604.04382v1.pdf
◦ GAN:	https://arxiv.org/pdf/1406.2661v1.pdf
◦...
Outline
• CNN(Convolutional	Neural	Networks)	Introduction
• Evolution	of	CNN
• Visualizing	the	Features
• CNN	as	Artist
• ...
A	Convolutional	Neural	Network	for	Modelling	
Sentences	
• Paper:	https://arxiv.org/abs/1404.2188
• Source	code:	
https://...
Drawbacks	of	Recursive	Neural	
Networks(RvNN)	
• Need	human-labeled	syntax	tree	during	training
This is a dog
Train	
RvNN
...
Drawbacks	of	Recursive	Neural	
Networks(RvNN)	
• Ambiguity	in	natural	language
http://3rd.mafengwo.cn/travels/info_wei
bo....
Element-wise	1D	operations	on	word	vectors
• 1D	Convolution	or	1D	Pooling
This is a
operationoperation
This is a
Represent...
From	RvNN to	CNN
• RvNN • CNN
This is a dog
conv3
conv2
conv1conv1conv1
conv2
Same
RvNN
Different
conv layers
This is a do...
CNN	with	Max-Pooling	Layers
• Similar	to	syntax	tree
• But	human-labeled	syntax	tree	is	not	needed
This is a dog
conv2
poo...
Sentiment	Analysis	by	CNN
• Use softmax layer to classify the sentiments
positive
This movie is awesome
conv2
pool1
conv1c...
Sentiment	Analysis	by	CNN
• Build	the “correct syntax tree” by training
negative
This movie is awesome
conv2
pool1
conv1co...
Sentiment	Analysis	by	CNN
• Build	the “correct syntax tree” by training
negative
This movie is awesome
conv2
pool1
conv1co...
Multiple	Filters
• Richer	features	than	RNN
This is
filter11 Filter13filter12
a
filter11 Filter13filter12
filter21 Filter2...
Sentence	can’t be easily resized
• Image	can	be	easily	resized • Sentence can’t be easily
resized
全台灣最高樓在台北
resize
resize
...
Various Input	Size
• Convolutional	layers	and	pooling	layers	
◦ can	handle	input	with	various	size	
This is a dog
pool1
co...
Various Input	Size
• Fully-connected	layer	and	softmax layer	
◦ need	fixed-size	input
The dog run
fc
softmax
This is a
fc
...
k-max	Pooling
• choose	the	k-max	values
• preserve	the	order	of	input	values
• variable-size	input,	fixed-size	output
3-ma...
Wide	Convolution	
• Ensures	that	all	weights	reach	the	entire	sentence	
conv conv conv conv conv convconv conv
Wide convol...
Dynamic	k-max	Pooling	
wide	convolution	&
k-max	pooling
wide	convolution
&	k-max	pooling
kl
ktop
L
s
ktop and L are consta...
Dynamic	k-max	Pooling	
s = 10
L = 2
k1 = max(3, d
2 1
2
⇥ 10e) = 5
ktop = 3
kl = max(ktop, d
L l
L
se)
conv &	pooling
conv...
Dynamic	k-max	Pooling	
conv &	pooling
conv &	pooling
L = 2
ktop = 3
kl = max(ktop, d
L l
L
se)
k1 = max(3, d
2 1
2
⇥ 14e) ...
Dynamic	k-max	Pooling	
conv &	pooling
conv &	pooling
L = 2
ktop = 3
kl = max(ktop, d
L l
L
se)
s = 8
k1 = max(3, d
2 1
2
⇥...
Dynamic	k-max	Pooling	
Wide	convolution	&
Dynamic	k-max	pooling
131
Convolutional	Neural	Networks	for	Sentence	
Classification	
• Paper:	http://www.aclweb.org/anthology/D14-1181
• Sourcee co...
Static	&	Non-Static	Channel
• Pretrained by	word2vec
• Static:	fix	the	values	during	training
• Non-Static:	update	the	val...
About	the	Lecturer
Mark	Chang
• Email:	ckmarkoh at gmail dot com
• Blog: http://cpmarkchang.logdown.com
• Github:	https://...
Nächste SlideShare
Wird geladen in …5
×

Applied Deep Learning 11/03 Convolutional Neural Networks

10.150 Aufrufe

Veröffentlicht am

Applied Deep Learning 11/03 Convolutional Neural Networks
https://www.csie.ntu.edu.tw/~yvchen/f105-adl/

Veröffentlicht in: Technologie
  • Develops your Dog's "Hidden Intelligence" To eliminate bad behavior and Create the obedient, well-behaved pet of your dreams... ★★★ http://ishbv.com/brainydogs/pdf
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download Full doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download PDF EBOOK here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download EPUB Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... Download doc Ebook here { http://shorturl.at/mzUV6 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Applied Deep Learning 11/03 Convolutional Neural Networks

  1. 1. Slide credit from Mark Chang 1
  2. 2. Convolutional Neural Networks • We need a course to talk about this topic ◦ http://cs231n.stanford.edu/syllabus.html • However, we only have a lecture 2
  3. 3. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 3
  4. 4. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 4
  5. 5. Image Recognition http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf 5
  6. 6. Image Recognition 6
  7. 7. Local Connectivity Neurons connect to a small region 7
  8. 8. Parameter Sharing • The same feature in different positions Neurons share the same weights 8
  9. 9. Parameter Sharing • Different features in the same position Neurons have different weights 9
  10. 10. Convolutional Layers depth widthwidth depth weights weights height shared weight 10
  11. 11. Convolutional Layers c1 c2 b1 b2 a1 a2 a3 wb1 wb2 b1 =wb1a1+wb2a2 wb1 wb2 b2 =wb1a2+wb2a3 wc1 wc2 c1 =wc1a1+wc2a2 wc2 wc1 c2 =wc1a2+wc2a3 depth = 2depth = 1 11
  12. 12. Convolutional Layers c1 b1 b2 a1 a2 d1 b3 a3 c2 d2 depth = 2 depth = 2 wc1 wc2 wc3 wc4 c1 = a1wc1 + b1wc2 + a2wc3 + b2wc4 wc1 wc2 wc3 wc4 c2 = a2wc1 + b2wc2 + a3wc3 + b3wc4 12
  13. 13. Convolutional Layers c1 b1 b2 a1 a2 d1 b3 a3 c2 d2 c1 = a1wc1 + b1wc2 + a2wc3 + b2wc4 c2 = a2wc1 + b2wc2 + a3wc3 + b3wc4 wd1 wd2 wd3 wd4 d1 = a1wd1 + b1wd2 + a2wd3 + b2wd4 wd1 wd2 wd3 wd4 d2 = a2wd1 + b2wd2 + a3wd3 + b3wd4 depth = 2 depth = 2 13
  14. 14. Convolutional Layers A B C A B C D 14
  15. 15. Hyper-parameters of CNN • Stride • Padding 0 0 Stride = 1 Stride = 2 Padding = 0 Padding = 1 15
  16. 16. Example Output Volume (3x3x2) Input Volume (7x7x3) Stride = 2 Padding = 1 http://cs231n.github.io/convolutional-networks/ Filter (3x3x3) 16
  17. 17. Convolutional Layers http://cs231n.github.io/convolutional-networks/ 17
  18. 18. Convolutional Layers http://cs231n.github.io/convolutional-networks/ 18
  19. 19. Convolutional Layers http://cs231n.github.io/convolutional-networks/ 19
  20. 20. Relationship with Convolution y[n] = X k x[k]w[n k] x[n] w[n] n n y[n] x[k] k k w[0 k] n y[0] = x[ 2]w[2] + x[ 1]w[1] + x[0]w[0] 20
  21. 21. Relationship with Convolution y[n] = X k x[k]w[n k] x[n] w[n] n n y[n] x[k] k k n w[1 k] y[1] = x[ 1]w[2] + x[0]w[1] + x[2]w[0] 21
  22. 22. Relationship with Convolution y[n] = X k x[k]w[n k] x[n] w[n] n n y[n] x[k] k k n y[2] = x[0]w[2] + x[1]w[1] + x[2]w[0] w[2 k] 22
  23. 23. Relationship with Convolution y[n] = X k x[k]w[n k] x[n] w[n] n n y[n] x[k] k k n w[4 k] y[4] = x[2]w[2] + x[3]w[1] + x[4]w[0] 23
  24. 24. Nonlinearity • Rectified Linear (ReLU) nout = ⇢ nin if nin > 0 0 otherwise nin n 2 6 6 4 1 4 3 1 3 7 7 5 2 6 6 4 1 4 0 1 3 7 7 5ReLU 24
  25. 25. Why ReLU? • Easy to train • Avoid gradient vanishing problem Sigmoid saturated gradient ≈ 0 ReLU not saturated 25
  26. 26. Why ReLU? • Biological reason strong stimulation ReLU weak stimulation neuron t v strong stimulation neuron t v weak stimulation 26
  27. 27. Pooling Layer 1 3 2 4 5 7 6 8 0 0 3 3 5 5 0 0 4 5 5 3 7 8 5 3 Maximum Pooling Average Pooling Max(1,3,5,7) = 7 Avg(1,3,5,7) = 4 no overlap no weights depth = 1 Max(0,0,5,5) = 5 27
  28. 28. Why “Deep” Learning? 28
  29. 29. Visual Perception of Human http://www.nature.com/neuro/journal/v8/n8/images/nn0805-975-F1.jpg 29
  30. 30. Visual Perception of Computer Convolutional Layer Convolutional Layer Pooling Layer Pooling Layer Receptive Fields Receptive Fields Input Layer 30
  31. 31. Visual Perception of Computer Input Layer Convolutional Layer with Receptive Fields: Max-pooling Layer with Width =3, Height = 3 Filter Responses Filter Responses Input Image 31
  32. 32. Fully-Connected Layer • Fully-Connected Layers : Global feature extraction • Softmax Layer: Classifier Convolutional Layer Convolutional Layer Pooling Layer Pooling Layer Input Layer Input Image Fully-Connected Layer Softmax Layer 5 7 Class Label 32
  33. 33. Visual Perception of Computer • Alexnet http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf http://vision03.csail.mit.edu/cnn_art/data/single_layer.png 33
  34. 34. Training • Forward Propagation n2 n1 n1(out)n2(out) n2(in) w21 34 n2(in) = w21n1(out) n2(out) = g(n2(in)), g is activation function
  35. 35. Training • Update weights n2 n1J Cost function: @J @w21 = @J @n2(out) @n2(out) @n2(in) @n2(in) @w21 w21 w21 ⌘ @J @w21 ) w21 w21 ⌘ @J @n2(out) @n2(out) @n2(in) @n2(in) @w21 n2(out) n2(in) w21 n1(out) 35 @J @w21 = @J @n2(out) @n2(out) @n2(in) @n2(in) @w21 w21 w21 ⌘ @J @w21 ) w21 w21 ⌘ @J @n2(out) @n2(out) @n2(in) @n2(in) @w21
  36. 36. Training • Update weights n2 n1J Cost function: n2(out) n2(in) w21 n1(out) w21 w21 ⌘ @J @n2(out) @n2(out) @n2(in) @n2(in) @w21 ) w21 w21 ⌘ @J @n2(out) g0 (n2(in))n1(out) n2(out) = g(n2(in)), n2(in) = w21n1(out) ) @n2(out) @n2(in) = g0 (n2(in)), @n2(in) @w21 = n1(out) 36
  37. 37. Training • Propagate to the previous layer n2 n1J Cost function: @J @n1(in) = @J @n2(out) @n2(out) @n2(in) @n2(in) @n1(out) @n1(out) @n1(in) n2(out) n2(in) n1(out) n1(in) 37
  38. 38. Training Convolutional Layers • example: a3 a2 a1 b2 b1 wb1 wb1 wb2 wb2 output input Convolutional Layer To simplify the notations, in the following slides, we make: b1 means b1(in), a1 means a1(out), and so on. 38
  39. 39. Training Convolutional Layers • Forward propagation a3 a2 a1 b2 b1 input Convolutional Layer b1 = wb1a1 + wb2a2 b2 = wb1a2 + wb2a3 wb1 wb1 wb2 wb2 39
  40. 40. Training Convolutional Layers • Update weights J Cost function: a3 a2 a1 b2 b1 @J @b1 @J @b2 wb1 wb1 @b1 @wb1 @b2 @wb1 wb1 wb1 ⌘( @J @b1 @b1 @wb1 + @J @b2 @b2 @wb1 ) 40
  41. 41. Training Convolutional Layers • Update weights a3 a2 a1 b2 b1 wb1 wb1 b1 = wb1a1 + wb2a2 b2 = wb1a2 + wb2a3 @b1 @wb1 = a1 @b2 @wb1 = a2 wb1 wb1 ⌘( @J @b1 a1 + @J @b2 a2) @J @b1 @J @b2 J Cost function: 41
  42. 42. Training Convolutional Layers • Update weights a3 a2 a1 b2 b1 wb2 wb2 ⌘( @J @b1 @b1 @wb2 + @J @b2 @b2 @wb2 ) @b1 @wb2 @b2 @wb2 wb2 wb2 @J @b1 @J @b2 J Cost function: 42
  43. 43. Training Convolutional Layers • Update weights a3 a2 a1 b2 b1 wb2 wb2 @b1 @wb2 = a2 @b2 @wb2 = a3 wb2 wb2 ⌘( @J @b1 a2 + @J @b2 a3) b1 = wb1a1 + wb2a2 b2 = wb1a2 + wb2a3 @J @b1 @J @b2 J Cost function: 43
  44. 44. Training Convolutional Layers • Propagate to the previous layer J Cost function: a3 a2 a1 b2 b1 @J @b1 @J @b2 @b1 @a1 @b1 @a2 @b2 @a2 @b2 @a3 @J @b1 @b1 @a1 @J @b2 @b2 @a3 @J @b1 @b1 @a2 + @J @b2 @b2 @a2 44
  45. 45. Training Convolutional Layers • Propagate to the previous layer J Cost function: a3 a2 a1 b2 b1 @J @b1 @J @b2 b1 = wb1a1 + wb2a2 b2 = wb1a2 + wb2a3 @b1 @a1 = wb1 @b1 @a2 = wb2 @b2 @a2 = wb1 @b2 @a3 = wb2 @J @b1 wb1 @J @b1 wb1 + @J @b2 wb2 @J @b2 wb2 45
  46. 46. Max-Pooling Layers during Training • Pooling layers have no weights • No need to update weights a3 a2 a1 b2 b1 a1 > a2 a2 > a3 b2 = max(a2, a3) b1 = max(a1, a2) Max-pooling 46 = ⇢ a2 if a2 a3 a3 otherwise @b2 @a2 = ⇢ 1 if a2 a3 0 otherwise
  47. 47. Max-Pooling Layers during Training • Propagate to the previous layer a3 a2 a1 b2 b1 @J @b1 @J @b2 @b1 @a1 = 1 a2 > a3 @b2 @a2 = 1 a1 > a2 @J @b1 @J @b2 @b1 @a2 = 0 @b2 @a3 = 0 J Cost function: 47
  48. 48. Max-Pooling Layers during Training • if a1 = a2 ?? ◦ Choose the node with smaller index a3 a2 a1 b2 b1 @J @b1 @J @b2 @J @b1 @J @b2 J Cost function: a1 = a2 = a3 48
  49. 49. Avg-Pooling Layers during Training • Pooling layers have no weights • No need to update weights a3 a2 a1 b2 b1 b1 = 1 2 (a1 + a2) b2 = 1 2 (a2 + a3) @b2 @a2 = 1 2 @b2 @a3 = 1 2 Avg-pooling 49
  50. 50. Avg-Pooling Layers during Training • Propagate to the previous layer J Cost function: a3 a2 a1 b2 b1 @J @b1 @J @b2 @b1 @a1 = @b1 @a2 = 1 2 @b2 @a2 = @b2 @a3 = 1 2 1 2 @J @b1 1 2 ( @J @b1 + @J @b2 ) 1 2 @J @b2 50
  51. 51. ReLU during Training nout = ⇢ nin if nin > 0 0 otherwise nin n @nout @nin = ⇢ 1 if nin > 1 0 otherwise 51
  52. 52. Training CNN 52
  53. 53. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 53
  54. 54. LeNet ◦ Paper: http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf Yann LeCun http://yann.lecun.com/exdb/lenet/ 54
  55. 55. ImageNet Challenge • ImageNet Large Scale Visual Recognition Challenge ◦ http://image-net.org/challenges/LSVRC/ • Dataset : ◦ 1000 categories ◦ Training: 1,200,000 ◦ Validation: 50,000 ◦ Testing: 100,000 http://vision.stanford.edu/Datasets/collage_s.png 55
  56. 56. ImageNet Challenge http://www.qingpingshan.com/uploads/allimg/160818/1J22QI5-0.png 56
  57. 57. AlexNet (2012) • Paper: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf • The resurgence of Deep Learning Geoffrey HintonAlex Krizhevsky 57
  58. 58. VGGNet (2014) • Paper: https://arxiv.org/abs/1409.1556 D: VGG16 E: VGG19 All filters are 3x3 58
  59. 59. VGGNet • More layers & smaller filters (3x3) is better • More non-linearity, fewer parameters One 5x5 filter • Parameters: 5x5 = 25 • Non-linear:1 Two 3x3 filters • Parameters: 3x3x2 = 18 • Non-linear:2 59
  60. 60. VGG 19 depth=64 3x3 conv conv1_1 conv1_2 maxpool depth=128 3x3 conv conv2_1 conv2_2 maxpool depth=256 3x3 conv conv3_1 conv3_2 conv3_3 conv3_4 depth=512 3x3 conv conv4_1 conv4_2 conv4_3 conv4_4 depth=512 3x3 conv conv5_1 conv5_2 conv5_3 conv5_4 maxpool maxpool maxpool size=4096 FC1 FC2 size=1000 softmax 60
  61. 61. GoogLeNet (2014) • Paper: http://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf 22 layers deep network Inception Module 61
  62. 62. Inception Module • Best size? ◦ 3x3? 5x5? • Use them all, and combine 62
  63. 63. Inception Module 1x1 convolution 3x3 convolution 5x5 convolution 3x3 max-pooling Previous layer Filter Concatenate 63
  64. 64. Inception Module with Dimension Reduction • Use 1x1 filters to reduce dimension 64
  65. 65. Inception Module with Dimension Reduction Previous layer 1x1 convolution (1x1x256x128) Reduced dimension Input size 1x1x256 128256 Output size 1x1x128 65
  66. 66. ResNet (2015) • Paper: https://arxiv.org/abs/1512.03385 • Residual Networks • 152 layers 66
  67. 67. ResNet • Residual learning: a building block Residual function 67
  68. 68. Residual Learning with Dimension Reduction • using 1x1 filters 68
  69. 69. Pretrained Model Download • http://www.vlfeat.org/matconvnet/pretrained/ ◦ Alexnet: ◦ http://www.vlfeat.org/matconvnet/models/imagenet-matconvnet- alex.mat ◦ VGG19: ◦ http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep- 19.mat ◦ GoogLeNet: ◦ http://www.vlfeat.org/matconvnet/models/imagenet-googlenet-dag.mat ◦ ResNet ◦ http://www.vlfeat.org/matconvnet/models/imagenet-resnet-152-dag.mat 69
  70. 70. Using Pretrained Model • Lower layers:edge, blob, texture (more general) • Higher layers : object part (more specific) http://vision03.csail.mit.edu/cnn_art/data/single_layer.png 70
  71. 71. Transfer Learning • The Pretrained Model is trained on ImageNet dataset • If your data is similar to the ImageNet data ◦ Fix all CNN Layers ◦ Train FC layer Conv layer FC layer FC layer Labeled dataLabeled dataLabeled dataLabeled dataLabeled dataImageNet data Your data … Conv layer Conv layer … Conv layer Your data … … 71
  72. 72. Transfer Learning • The Pretrained Model is trained on ImageNet dataset • If your data is far different from the ImageNet data ◦ Fix lower CNN Layers ◦ Train higher CNN and FC layers Conv layer FC layer FC layer Labeled dataLabeled dataLabeled dataLabeled dataLabeled dataImageNet data Your data … Conv layer Conv layer … Conv layer Your data … … 72
  73. 73. Tensorflow Transfer Learning Example • https://www.tensorflow.org/versions/r0.11/how_tos/styl e_guide.html daisy 634 photos dandelion 899 photos roses 642 photos tulips 800 photos sunflowers 700 photos http://download.tensorflow.org/example_images/flower_photos.tgz
  74. 74. Tensorflow Transfer Learning Example Fix these layers Train this layer
  75. 75. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 75
  76. 76. Visualizing CNN http://vision03.csail.mit.edu/cnn_art/data/single_layer.png 76
  77. 77. Visualizing CNN CNN CNN flower random noise filter response filter response 77
  78. 78. filter response: Gradient Ascent • Magnify the filter response random noise: x f score: F = X i,j fi,j fi,j lower score higher score x F gradient: @F @x 78
  79. 79. filter response: Gradient Ascent • Magnify the filter response random noise: x f x gradient: fi,j lower score higher score F @F @x update x learning rate x x + ⌘ @F @x 79
  80. 80. Gradient Ascent 80
  81. 81. Different Layers of Visualization CNN 81
  82. 82. Multiscale Image Generation visualize resize visualize resize visualize 82
  83. 83. Multiscale Image Generation 83
  84. 84. Deep Dream • https://research.googleblog.com/2015/06/inceptionism- going-deeper-into-neural.html • Source code: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/ex amples/tutorials/deepdream/deepdream.ipynb http://download.tensorflow.org/example_images /flower_photos.tgz 84
  85. 85. Deep Dream 85
  86. 86. Deep Dream 86
  87. 87. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 87
  88. 88. Neural Art • Paper: https://arxiv.org/abs/1508.06576 • Source code : https://github.com/ckmarkoh/neuralart_tensorflow content style artwork http://www.taipei- 101.com.tw/upload/news/201502/2015 021711505431705145.JPG https://github.com/andersbll/neural_ar tistic_style/blob/master/images/starry_ night.jpg?raw=true 88
  89. 89. The Mechanism of Painting BrainArtist Scene Style ArtWork Computer Neural Networks 89
  90. 90. Misconception 90
  91. 91. Content Generation BrainArtistContent Canvas Minimize the difference Neural Stimulation Draw 91
  92. 92. Content Generation 92 Filter ResponsesVGG19 Update the color of the pixels Content Canvas Result Width*Height Depth Minimize the difference
  93. 93. Content Generation Layer l’s Filter l Responses: Layer l’s Filter Responses: Input Photo: Input Canvas: Width*Height (j) Depth (i) Width*Height (j) Depth (i) 93
  94. 94. Content Generation • Backward Propagation Layer l’s Filter l Responses: Input Canvas: VGG19 Update Canvas Learning Rate 94
  95. 95. Content Generation 95
  96. 96. Content Generation VGG19 conv1_2 conv2_2 conv3_4 conv4_4 conv5_2conv5_1 96
  97. 97. Style Generation VGG19Artwork G G Filter Responses Gram Matrix Width*Height Depth Depth Depth Position- dependent Position- independent 97
  98. 98. Style Generation 1. .5 .5 .5 1. 1. .5 .25 1. .5 .25 .5 .25 .25 1. .5 1. Width*Height Depth k1 k2 k1 k2 Depth Depth Layer l’s Filter Responses Gram Matrix G 98
  99. 99. Style Generation Layer l’s Filter Responses Layer l’s Gram Matrix Layer l’s Gram Matrix Input Artwork: Input Canvas: 99
  100. 100. Style Generation VGG19 Filter Responses Gram Matrix Minimize the difference G G Style Canvas Update the color of the pixelsResult 100
  101. 101. Style Generation 101
  102. 102. Style Generation VGG19 Conv1_1 Conv1_1 Conv2_1 Conv1_1 Conv2_1 Conv3_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1 102
  103. 103. Artwork Generation Filter ResponsesVGG19 Gram Matrix 103
  104. 104. Artwork Generation VGG19 VGG19 Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1 Conv4_2 104
  105. 105. Artwork Generation 105
  106. 106. Content v.s. Style 0.15 0.05 0.02 0.007 106
  107. 107. Neural Doodle • Paper: https://arxiv.org/abs/1603.01768 • Source code: https://github.com/alexjc/neural-doodle style content resultsemantic maps 107
  108. 108. Neural Doodle • Image analogy 108
  109. 109. Neural Doodle • Image analogy 恐怖連結,慎入! https://raw.githubusercontent.com/awentzonline/ image-analogies/master/examples/images/trump- image-analogy.jpg 109
  110. 110. Real-time Texture Synthesis • Paper: https://arxiv.org/pdf/1604.04382v1.pdf ◦ GAN: https://arxiv.org/pdf/1406.2661v1.pdf ◦ VAE: https://arxiv.org/pdf/1312.6114v10.pdf • Source Code : https://github.com/chuanli11/MGANs 110
  111. 111. Outline • CNN(Convolutional Neural Networks) Introduction • Evolution of CNN • Visualizing the Features • CNN as Artist • Sentiment Analysis by CNN 111
  112. 112. A Convolutional Neural Network for Modelling Sentences • Paper: https://arxiv.org/abs/1404.2188 • Source code: https://github.com/FredericGodin/DynamicCNN 112
  113. 113. Drawbacks of Recursive Neural Networks(RvNN) • Need human-labeled syntax tree during training This is a dog Train RvNN Word vector This is a dog RvNN RvNN RvNN 113
  114. 114. Drawbacks of Recursive Neural Networks(RvNN) • Ambiguity in natural language http://3rd.mafengwo.cn/travels/info_wei bo.php?id=2861280 114 http://www.appledaily.com.tw/realtimen ews/article/new/20151006/705309/
  115. 115. Element-wise 1D operations on word vectors • 1D Convolution or 1D Pooling This is a operationoperation This is a Represented by 115
  116. 116. From RvNN to CNN • RvNN • CNN This is a dog conv3 conv2 conv1conv1conv1 conv2 Same RvNN Different conv layers This is a dog RvNN RvNN RvNN 116
  117. 117. CNN with Max-Pooling Layers • Similar to syntax tree • But human-labeled syntax tree is not needed This is a dog conv2 pool1 conv1conv1conv1 pool1 This is a dog conv2 pool1 conv1conv1 pool1 Max Pooling 117
  118. 118. Sentiment Analysis by CNN • Use softmax layer to classify the sentiments positive This movie is awesome conv2 pool1 conv1conv1conv1 pool1 softmax negative This movie is awful conv2 pool1 conv1conv1conv1 pool1 softmax 118
  119. 119. Sentiment Analysis by CNN • Build the “correct syntax tree” by training negative This movie is awesome conv2 pool1 conv1conv1conv1 pool1 softmax negative This movie is awesome conv2 pool1 conv1conv1conv1 pool1 softmax Backward propagation error 119
  120. 120. Sentiment Analysis by CNN • Build the “correct syntax tree” by training negative This movie is awesome conv2 pool1 conv1conv1conv1 pool1 softmax positive This movie is awesome conv2 pool1 conv1conv1conv1 pool1 softmax Update the weights 120
  121. 121. Multiple Filters • Richer features than RNN This is filter11 Filter13filter12 a filter11 Filter13filter12 filter21 Filter23filter22 121
  122. 122. Sentence can’t be easily resized • Image can be easily resized • Sentence can’t be easily resized 全台灣最高樓在台北 resize resize 全台灣最高的高樓在台北市 全台灣最高樓在台北市 台灣最高樓在台北 122
  123. 123. Various Input Size • Convolutional layers and pooling layers ◦ can handle input with various size This is a dog pool1 conv1conv1conv1 pool1 the dog run pool1 conv1conv1 123
  124. 124. Various Input Size • Fully-connected layer and softmax layer ◦ need fixed-size input The dog run fc softmax This is a fc softmax dog 124
  125. 125. k-max Pooling • choose the k-max values • preserve the order of input values • variable-size input, fixed-size output 3-max pooling 13 4 1 7 812 5 21 15 7 4 9 3-max pooling 12 21 15 13 7 8 125
  126. 126. Wide Convolution • Ensures that all weights reach the entire sentence conv conv conv conv conv convconv conv Wide convolutionNarrow convolution 126
  127. 127. Dynamic k-max Pooling wide convolution & k-max pooling wide convolution & k-max pooling kl ktop L s ktop and L are constants l : index of current layer kl : k of current layer ktop : k of top layer L : total number of layers s : length of input sentence kl = max(ktop, d L l L se) 127
  128. 128. Dynamic k-max Pooling s = 10 L = 2 k1 = max(3, d 2 1 2 ⇥ 10e) = 5 ktop = 3 kl = max(ktop, d L l L se) conv & pooling conv & pooling 128
  129. 129. Dynamic k-max Pooling conv & pooling conv & pooling L = 2 ktop = 3 kl = max(ktop, d L l L se) k1 = max(3, d 2 1 2 ⇥ 14e) = 7 s = 14 129
  130. 130. Dynamic k-max Pooling conv & pooling conv & pooling L = 2 ktop = 3 kl = max(ktop, d L l L se) s = 8 k1 = max(3, d 2 1 2 ⇥ 8e) = 4 130
  131. 131. Dynamic k-max Pooling Wide convolution & Dynamic k-max pooling 131
  132. 132. Convolutional Neural Networks for Sentence Classification • Paper: http://www.aclweb.org/anthology/D14-1181 • Sourcee code: https://github.com/yoonkim/CNN_sentence 132
  133. 133. Static & Non-Static Channel • Pretrained by word2vec • Static: fix the values during training • Non-Static: update the values during training 133
  134. 134. About the Lecturer Mark Chang • Email: ckmarkoh at gmail dot com • Blog: http://cpmarkchang.logdown.com • Github: https://github.com/ckmarkoh • Slideshare: http://www.slideshare.net/ckmarkohchang • Youtube: https://www.youtube.com/channel/UCckNPGDL21aznRhl3EijRQw 134 HTC Research & Healthcare Deep Learning Algorithms Research Engineer

×