for 実践者向けディープラーニング勉強会 第二回 2019-04-17 https://dl4-practitioners.connpass.com/event/124498/

1. 畳み込みニューラル
ネットワークの基礎と応⽤
実践者向けディープラーニング勉強会 第⼆回
https://dl4-practitioners.connpass.com/event/124498/
2019-04-17 (Wed) @ SB C&S

2. Agenda
19:00 〜 19:05 opening
19:05 〜 19:20 機械学習の導⼊
19:20 〜 19:30 walkthrough Kerasを⽤いたコーディングサンプル
19:30 〜 19:45 畳み込みニューラルネットワークの基礎
19:45 〜 20:00 walkthrough CNNのフィルタの理解
20:00 〜 20:10 休憩
20:10 〜 20:25 breakout CNNアーキテクチャの紹介
20:25 〜 20:40 Skymind Intelligent Layer (SKIL) の解説
20:40 〜 20:55 walkthrough SKILを⽤いたモデルのデプロイ
20:55 〜 21:00 closing
!2

3. !3
15

4. ニューラルネットワークの基礎
!4

5. !5
https://www.coursera.org/learn/introduction-tensorflow

6. Activity Recognition
!6
https://www.coursera.org/learn/introduction-tensorflow

7. Activity Recognition
!7
https://www.coursera.org/learn/introduction-tensorflow

8. Activity Recognition
!8
https://www.coursera.org/learn/introduction-tensorflow

9. Activity Recognition
!9
https://www.coursera.org/learn/introduction-tensorflow

10. !10
https://www.coursera.org/learn/introduction-tensorflow

11. Activity Recognition
!11
https://www.coursera.org/learn/introduction-tensorflow

12. 機械学習の"Hello World”
!12
x = -2, -1, 0, 1, 2, 3, 4
y = -3, -1, 1, 3, 5, 7, 9
y = f(x)

13. 機械学習の"Hello World”
!13
x = -2, -1, 0, 1, 2, 3, 4
y = -3, -1, 1, 3, 5, 7, 9
y = f(x) = 2x + 1

14. 機械学習のアプローチ
•適当にモデルを初期化 (y_=ax+b)
!14

15. 機械学習のアプローチ
• 適当にモデルを初期化 (y_=ax+b)
•誤差（損失）を計算 (L=1/N Σ(y-y_)2)
!15

16. 機械学習のアプローチ
• 適当にモデルを初期化 (y_=ax+b)
• 誤差（損失）を計算 (L=1/N Σ(y-y_)2)
•誤差が⼩さくなるようにパラメータ (a, b)
を少し更新 (a ← a - η ∂L/∂a)
!16

17. 機械学習のアプローチ
!17
• 適当にモデルを初期化 (y_=ax+b)
• 誤差（損失）を計算 (L=1/N Σ(y-y_)2)
• 誤差が⼩さくなるようにパラメータ (a, b)
を少し更新 (a ← a - η ∂L/∂a)
•誤差（損失）を計算 (L=1/N Σ(y-y_)2)

18. 機械学習のアプローチ
!18
• 適当にモデルを初期化 (y_=ax+b)
• 誤差（損失）を計算 (L=1/N Σ(y-y_)2)
• 誤差が⼩さくなるようにパラメータ (a, b)
を少し更新 (a ← a - η ∂L/∂a)
• 誤差（損失）を計算 (L=1/N Σ(y-y_)2)
•誤差が⼩さくなるようにパラメータ (a, b)
を少し更新 (a ← a - η ∂L/∂a)
• …

19. !19

20. !20
x
a
+ b
y_

21. !21

22. !22

23. 勾配降下法
!23
「⽬隠しで⾜元の勾配情報のみを使って⼭の頂上を⽬指すようなもの」
学習率 η は歩幅のイメージ（η⼩＝すり⾜、η⼤＝巨⼈の⼀歩）
https://twitter.com/momiji_fullmoon/status/1110316960611368960
https://www.yamakei-online.com/journal/detail.php?id=3185

24. Fashion MNIST Dataset
• 7万画像
• 10カテゴリ
• 28×28 pixels
• 実験⽤データセット
!24
https://github.com/zalandoresearch/fashion-mnist

25. !25

26. !26

27. !27

28. !28
…
…………
…
…
Flatten: (28, 28) => (784)
0 1 2 3 4 5 6 7 125 126 127
0 1 2 3 4 5 6 7 8 9

29. Dense Layer の⽋点
• ⼊⼒のベクトルの全要素の相関をみている
> 住宅価格予測みたいな話ならまだいい
> もう作ってる特徴量と特徴量の組み合わせ
- 例）東京墨⽥区 & 床⾯積 30m2 & 1K & ⾵呂トイレ別 & 新築 => 家賃⽉10万円
• 「画像の特徴量」を抽出してからDense Layerに渡せば効率的
!29
畳込みニューラルネットワーク (Convolutional Neural Network; CNN)

30. 畳み込みニューラルネットワークの基礎
!30

31. 畳み込み (Convolution)
CURRENT_PIXEL_VALUE = 82
NEW_PIXEL_VALUE =
(-1 * 144) + (0 * 60) + (-2 * 19)
+ (0.5 * 188) + (4.5 * 82) + (-1.5 * 32)
+ (1.5 * 156) + (2 * 55) + (-3 * 27)
!31
144 60 19
188 82 32
156 55 27
-1 0 -2
0.5 4.5 -1.5
1.5 2 -3
uijm =
K 1X
k=0
W 1X
p=0
H 1X
q=0
z
(l 1)
i+p,j+q,khpqkm + bijm
<latexit sha1_base64="bp48ep/Dp5jA49Knv3mGnitdbqk=">AAACwnichVHLShxBFD12Xjp5OEk2ATfiYDCMDreNoAiCGBdCNj4yjqBj090ptaaf048BbfsH/IEsXBkIIfgZbvIDWfgJkp0jZOPCO91NQiJJbtNV55y659atKsO3ZRgRnfcpd+7eu/+gf6D08NHjJ4Plp8/WQy8OTFE3PdsLNgw9FLZ0RT2SkS02/EDojmGLhmG96a03OiIIpee+i/Z90XT0XVfuSFOPWNLKzVhLZMtJ57bC2NlO3k6oqZZYc5TmvJFx/ydfynib+cF2MmZPqK+Yyqo/3qq2x610j3PblpNWjbyqVq5QjbIYvg3UAlRQxLJX/owtvIcHEzEcCLiIGNvQEfK3CRUEn7UmEtYCRjJbF0hRYm/MWYIzdFYtHneZbRaqy7xXM8zcJu9i8x+wcxij9I2+UJe+0ild0PVfayVZjV4v+zwbuVf42uDRi7Uf/3U5PEfY++X6Z88RdjCT9Sq5dz9Teqcwc3/n4EN3bXZ1NHlJH+k7939C53TGJ3A7V+anFbF6jBI/gPrndd8G65M19XVtcmWqMr9QPEU/hjCCMb7vacxjCcuo875nuMAlusqi0lLaSpinKn2F5zl+C+XwBo4trvM=</latexit>
https://news.yahoo.co.jp/byline/yuasamakoto/20190403-00120722/

32. !32
1
2
1
0
0
0
-1
-2
-1
-1
-2
-1
0
0
0
1
2
1
-2
0
2
-1
0
-1
-1
0
1
https://www.bbkong.net/fs/alleyoop/molten_BGL7

33. stride=(1, 1), padding=ʻvalidʼ
!33

34. stride=(1, 1), padding=ʻvalidʼ
!34

35. stride=(1, 1), padding=ʻvalidʼ
!35

36. stride=(1, 1), padding=ʻvalidʼ
!36

37. stride=(1, 1), padding=ʻvalidʼ
!37

38. stride=(1, 1), padding=ʻvalidʼ
!38

39. stride=(1, 1), padding=ʻvalidʼ
!39

40. stride=(2, 2), padding=ʻvalidʼ
!40

41. stride=(2, 2), padding=ʻvalidʼ
!41

42. stride=(2, 2), padding=ʻvalidʼ
!42

43. stride=(1, 1), padding=ʻsameʼ
!43

44. stride=(1, 1), padding=ʻsameʼ
!44

45. stride=(1, 1), padding=ʻsameʼ
!45

46. stride=(1, 1), padding=ʻsameʼ
!46

47. Max Pooling
!47
https://www.coursera.org/learn/introduction-tensorflow

48. AlexNet
!48

49. !49slide from Keynote Speech by Laurence Moroney at Deep Learning Day 2019

50. !50
https://medium.com/@lmoroney_40129/codelabs-from-googlemlsummit-f9d53cac8d24

51. !51
https://medium.com/@lmoroney_40129/codelabs-from-googlemlsummit-f9d53cac8d24

52. 応⽤的なCNNアーキテクチャの紹介
!52

53. ImageNet コンペの優勝モデル
!53
https://www.slideshare.net/ren4yu/ss-84282514

54. ResNet
• 2015年のImageNetコンペ (ILSVRC) 優勝モデル
• Residualモジュール（ショートカット機構）の導⼊
!54
http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf
Revolution of Depth
3.57
6.7 7.3
11.7
16.4
25.8
28.2
ILSVRC'15
ResNet
ILSVRC'14
GoogleNet
ILSVRC'14
VGG
ILSVRC'13 ILSVRC'12
AlexNet
ILSVRC'11 ILSVRC'10
ImageNet Classification top-5 error (%)
shallow8 layers
19 layers22 layers
152 layers
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
8 layers

55. !55
Revolution of Depth
ResNet, 152 layers
1x1 conv, 64
3x3 conv, 64
1x1 conv, 256
1x1 conv, 64
3x3 conv, 64
1x1 conv, 256
1x1 conv, 64
3x3 conv, 64
1x1 conv, 256
1x2 conv, 128, /2
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 256, /2
3x3 conv, 256
7x7 conv, 64, /2, pool/2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
(there was an animation here)

56. !56
Revolution of Depth
ResNet, 152 layers
1x1 conv, 512
1x1 conv, 128
3x3 conv, 128
1x1 conv, 512
1x1 conv, 256, /2
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
(there was an animation here)

57. !57
Revolution of Depth
ResNet, 152 layers
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
(there was an animation here)

58. !58
Revolution of Depth
ResNet, 152 layers
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 256
3x3 conv, 256
1x1 conv, 1024
1x1 conv, 512, /2
3x3 conv, 512
1x1 conv, 2048
1x1 conv, 512
3x3 conv, 512
1x1 conv, 2048
1x1 conv, 512
3x3 conv, 512
1x1 conv, 2048
ave pool, fc 1000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
(there was an animation here)

59. ResNet
!59
Deep Residual Learning
• Plaint net
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.
any two
stacked layers
𝑥
𝐻(𝑥)
weight layer
weight layer
relu
relu
𝐻 𝑥 is any desired mapping,
hope the 2 weight layers fit 𝐻(𝑥)
Deep Residual Learning
• Residual net
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian
𝐻
hop
hopweight layer
weight layer
relu
relu
𝑥
𝐻 𝑥 = 𝐹 𝑥 + 𝑥
identity
𝑥
𝐹(𝑥)
http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf

60. !60
CIFAR-10 experiments
0 1 2 3 4 5 6
0
5
10
20
iter. (1e4)
error(%)
plain-20
plain-32
plain-44
plain-56
20-layer
32-layer
44-layer
56-layer
CIFAR-10 plain nets
0 1 2 3 4 5 6
0
5
10
20
iter. (1e4)
error(%)
ResNet-20
ResNet-32
ResNet-44
ResNet-56
ResNet-110
CIFAR-10 ResNets
56-layer
44-layer
32-layer
20-layer
110-layer
• Deep ResNets can be trained without difficulties
• Deeper ResNets have lower training error, and also lower test error
solid: test
dashed: train
Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

61. Further Reading
• ⼀年半前 (2017-12) の資料ですが素晴らしいスライドです
• 畳み込みニューラルネットワークの研究動向 by DeNA 内⽥さん
• https://www.slideshare.net/ren4yu/ss-84282514
!61

62. YOLO (You Only Look Once)
!62
https://www.youtube.com/watch?v=MPU2HistivI

63. YOLOのアーキテクチャ
!63

64. Labeling / Annotation
!64
https://github.com/Microsoft/VoTT

65. AIモデルの運⽤・デプロイ
!65

66. !66 https://www.programming-fun.net/article/article_40.html

67. !67
https://www.youtube.com/watch?v=-UYyyeYJAoQ

68. モデルの利⽤
!68
n月間ラムダノートVol.1, No.1より

69. モデルの利⽤
!69
n月間ラムダノートVol.1, No.1より

70. !70

71. !71

72. !72

73. !73

74. !74

75. CNNモデルのデプロイ
!75

76. !76

77. !77

78. !78
curl --request POST
--url http://<SKILhostIP>:9008/login
--header 'accept: application/json'
--header 'content-type: application/json'
--data '{"userId":"<userId>","password":"<password>"}'

79. Links
• SKIL Download Page
• https://skymind.ai/download
• SKIL Documentation
• https://docs.skymind.ai/docs
• SKIL REST API reference
• https://docs.skymind.ai/reference?newApiExplorer=true
• SKIL Clients
• https://github.com/SkymindIO/skil-clients
!79

80. まとめ
!80