Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

0

Teilen

Herunterladen, um offline zu lesen

Capsule networks

Herunterladen, um offline zu lesen

2018-05-15 presentation in SNU AI Study
캡슐 네트워크

Ähnliche Bücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen
  • Gehören Sie zu den Ersten, denen das gefällt!

Capsule networks

  1. 1. Capsule networks Intelligent Control and Systems Laboratory J.hyeon Park 2018-08-07 SNU AI Study
  2. 2. Covering range • Original capsule network Sabour 2017. "Dynamic routing between capsules." • EM-routing Hinton 2018. "Matrix capsules with EM routing." • Unsupervised training Rawlinson 2018. "Sparse unsupervised capsules generalize better.” • Stable training - Zhao 2018. "Investigating Capsule Networks with Dynamic Routing for Text Classification."
  3. 3. Traditional CNN : Conv + Pooling http://cs231n.github.io/convolutional-networks/
  4. 4. Traditional CNN : Conv + Pooling http://cs231n.github.io/convolutional-networks/ • Conv extracts features.
  5. 5. Traditional CNN : Conv + Pooling http://cs231n.github.io/convolutional-networks/ • Conv extracts features. • Pool abstracts features
  6. 6. Traditional CNN : Conv + Pooling http://cs231n.github.io/convolutional-networks/ • Conv extracts features. • Pool abstracts feature • without spatial relationship!
  7. 7. Drawbacks of traditional CNN Understanding Capsule Networks — AI’s Alluring New Architecture
  8. 8. Drawbacks of traditional CNN Jaeyun’s Blog : 캡슐 네트워크(캡스넷 - Capsnet) – 1
  9. 9. Capsules
  10. 10. Capsules • A capsule is a vector
  11. 11. Capsules • A capsule is a vector • Each capsule represents an entity (nose, eye ...) capsule1 (faceline) capsule2 (left eye) capsule3 (right eye) capsule4 (nose) capsule5 (mouse)
  12. 12. Capsules • A capsule is a vector • Each capsule represents an entity (nose, eye ...) • The direction of the capsule represents the property of entity capsule3 (right eye) ... various status of eyes...
  13. 13. Capsules • A capsule is a vector • Each capsule represents an entity (nose, eye ...) • The direction of the capsule represents the property of entity • The norm of the capsule represents the presence of entity capsule3 (right eye) ∥ 𝑐𝑝𝑎𝑠𝑢𝑙𝑒3 ∥ is logit of the presence of eye
  14. 14. Capsules • A capsule is a vector • Each capsule represents an entity (nose, eye ...) • The direction of the capsule represents the property of entity • The norm of the capsule represents the presence of entity • The lower capsules activate the higher capsules according to its spatial hierarchy face capsule face capsule X
  15. 15. Why I study a capsule? A task visually demonstrated by human Robot will learn the task
  16. 16. Why I study a capsule? Object segment by a region proposal network (we need object-centric information for a robot)
  17. 17. Why I study a capsule? Feature extraction by an Alexnet pre-trained with imagenet
  18. 18. Object spatial relationship determines task features Why I study a capsule? task features ?
  19. 19. Capsule network 28× 28 Sabour 2017. "Dynamic routing between capsules."
  20. 20. Capsule network convolutional kernel kernel = 9 × 9 × 256 strides = 1 28× 28
  21. 21. Capsule network convolutional kernel kernel = 9 × 9 × 256 strides = 2 28× 28
  22. 22. Capsule network 28× 28 1 1 8 primary capsule 𝑢𝑖 (dim = 8, total number = 6*6*32=1152) 1 16 digit capsule 𝑣𝑗 (dim = 16, total number = 10)
  23. 23. Capsule network 28× 28 prediction 𝑢𝑗|𝑖 = 𝑊𝑖𝑗 𝑢𝑖 𝑊 =[1152, 10, 8, 16] Training parameter (1) : # of PrimaryCaps (2) : # of DigitCpas (3) : Dim of PrimarCaps (4) : Dim of DigitCaps (1) (2) (3) (4) 𝑊𝑖𝑗 = [8,16] pick 𝑖, 𝑗 components 1 1 8 primary capsule 𝑢𝑖 (dim = 8, total number = 6*6*32=1152) 1 16 digit capsule 𝑣𝑗 (dim = 16, total number = 10)
  24. 24. Capsule network 28× 28 dynamic routing 𝑣𝑗 = 𝑑𝑦𝑛𝑎𝑚𝑖𝑐 𝑟𝑜𝑢𝑡𝑖𝑛𝑔( 𝑢𝑗|𝑖) 1 1 8 primary capsule 𝑢𝑖 (dim = 8, total number = 6*6*32=1152) 1 16 digit capsule 𝑣𝑗 (dim = 16, total number = 10)
  25. 25. Capsule network 𝑢1|𝑖 𝑣1 𝑢2|𝑖 𝑢…|𝑖 𝑣2 𝑣… 𝑣10𝑢10|𝑖 Primary capsule 𝑖 ... ... 𝑐1𝑖 𝑐2𝑖 𝑐…𝑖 𝑐10𝑖 Digit capsules 𝑣𝑗 = 𝑠𝑞𝑢𝑎𝑠ℎ Σ𝑖 𝑐𝑖𝑗 𝑢𝑗|𝑖 where 𝑠𝑞𝑢𝑎𝑠ℎ 𝑥 = ∥𝑥∥2 1+∥𝑥∥2 𝑥 ∥𝑥∥ • Dynamic routing
  26. 26. Capsule network 𝑢1|𝑖 𝑣1 𝑢2|𝑖 𝑢…|𝑖 𝑣2 𝑣… 𝑣10𝑢10|𝑖 Primary capsule 𝑖 ... ... 𝑐1𝑖 𝑐2𝑖 𝑐…𝑖 𝑐10𝑖 Digit capsules Σ𝑗 𝑐𝑖𝑗 = 1 increase 𝑐𝑖𝑗 if 𝑢𝑗|𝑖 has similar direction with 𝑣𝑗 • Dynamic routing
  27. 27. Capsule network • Dynamic routing Charles Martin, Capsule Networks (slide share)
  28. 28. Capsule network
  29. 29. Capsule network Loss = margin loss + reconstruction loss • margin loss : • reconstruction loss : 𝑇𝑘 = 1 if the label is 𝑘 otherwise 0 𝑚+ : target capsule length if activated 𝑚−: target capsule length if not activated
  30. 30. Capsule network
  31. 31. Capsule network
  32. 32. EM routing Hinton 2018. "Matrix capsules with EM routing." (𝑀: 4 × 4) (𝑎: scalar) prediction : 𝑉𝑖𝑗 = 𝑀𝑖 𝑊𝑖𝑗 (𝑊𝑖𝑗 : 4×4 trainable parameters connecting between each capsule 𝑖 and 𝑗) routing 𝑉𝑖𝑗 : EM-routing
  33. 33. Recall : dynamic routing
  34. 34. EM routing (gif) Jonathan hui : Understanding Matrix capsules with EM Routing (Based on Hinton's Capsule Networks)
  35. 35. EM routing • 4 × 4 Gaussian clusters = 𝜇ℎ , 𝜎ℎ (ℎ = 1, … , 16)
  36. 36. EM routing • 4 × 4 Gaussian clusters = 𝜇ℎ , 𝜎ℎ (ℎ = 1, … , 16) • For each Gaussian components ℎ of , computes the probability of 𝑣𝑖𝑗 ℎ belonging to capsule 𝑗′ 𝑠 Gaussian model 𝑝𝑖|𝑗 ℎ = 1 2𝜋 𝜎𝑗 ℎ 2 exp − 𝑉𝑖𝑗 ℎ − 𝜇 𝑗 ℎ 2 2 𝜎𝑗 ℎ 2
  37. 37. EM routing • cost : the lower the cost, the more likely a capsule will be activated 𝑐𝑜𝑠𝑡𝑖𝑗 ℎ = − ln 𝑃𝑖|𝑗 ℎ 𝑐𝑜𝑠𝑡𝑗 ℎ = 𝑖 𝑅𝑖𝑗 𝑐𝑜𝑠𝑡𝑖𝑗 ℎ where 𝑅𝑖𝑗 : assignment probability (the amount of data assigned to 𝑗)
  38. 38. EM routing • cost : the lower the cost, the more likely a capsule will be activated 𝑐𝑜𝑠𝑡𝑖𝑗 ℎ = − ln 𝑃𝑖|𝑗 ℎ 𝑐𝑜𝑠𝑡𝑗 ℎ = 𝑖 𝑅𝑖𝑗 𝑐𝑜𝑠𝑡𝑖𝑗 ℎ where 𝑅𝑖𝑗 : assignment probability (the amount of data assigned to 𝑗) • activation : 𝑎𝑗 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝜆 𝑏𝑗 − ℎ 𝑐𝑜𝑠𝑡𝑗 ℎ
  39. 39. EM routing • E-step : determine 𝑅𝑖𝑗 • M-step : recalculate 𝜇 𝑗, 𝜎𝑗, 𝑎𝑗 to reduce cost
  40. 40. EM routing • E-step : 𝑝𝑗 = 1 2𝜋 𝜎𝑗 ℎ 2 exp − ℎ 𝐻 𝑉𝑖𝑗 ℎ −𝜇 𝑗 ℎ 2 2 𝜎𝑗 ℎ 2 𝑅𝑖𝑗 = 𝑎 𝑗 𝑝 𝑗 𝑎 𝑘 𝑝 𝑘
  41. 41. EM routing • M-step 𝑅𝑖𝑗 = 𝑅𝑖𝑗 ∗ 𝑎𝑖 𝜇 𝑗 ℎ = 𝑖 𝑅𝑖𝑗 𝑉𝑖𝑗 ℎ 𝑖 𝑅𝑖𝑗 𝜎𝑗 ℎ 2 = 𝑖 𝑅𝑖𝑗 𝑉𝑖𝑗 ℎ − 𝜇 𝑗 ℎ 2 𝑖 𝑅𝑖𝑗 𝑐𝑜𝑠𝑡𝑖𝑗 ℎ = − ln 𝑃𝑖|𝑗 ℎ 𝑎𝑗 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 𝜆 𝑏𝑗 − ℎ 𝑐𝑜𝑠𝑡𝑗 ℎ
  42. 42. EM routing
  43. 43. EM routing spread loss :
  44. 44. EM routing
  45. 45. EM routing
  46. 46. Unsupervised training Rawlinson 2018. "Sparse unsupervised capsules generalize better.“
  47. 47. Unsupervised training
  48. 48. Unsupervised training
  49. 49. Unsupervised training • Add sparsity to capsule 𝜓𝑗𝑘 : weight connecting capsules 𝑔𝑗 : boosting value 𝑟𝑗𝑘 : activation raking of j-th capsule 𝑚𝑗𝑘 : normalized ranking 𝑣 : original capsule 𝑣′ : sparsity-added capsule
  50. 50. • Count the # of activation for each capsule Unsupervised training 𝑟𝑗𝑘 : activation raking of j-th capsule of k-th data K : batch size 𝐽 : number of capsule 𝜖𝑗 : count of activation of j-th capsule 𝜇 𝑗 : moving average of 𝜖𝑗
  51. 51. Unsupervised training • Boost capsule based on the count 𝑑 : boosting step size 𝑔𝑗 : boosting value 𝜇 𝑚𝑖𝑛, 𝜇 𝑚𝑎𝑥 : target frequency of activation
  52. 52. Unsupervised training
  53. 53. Unsupervised training
  54. 54. Stable training Zhao 2018. "Investigating Capsule Networks with Dynamic Routing for Text Classification."
  55. 55. Thank you! Q&A

2018-05-15 presentation in SNU AI Study 캡슐 네트워크

Aufrufe

Aufrufe insgesamt

595

Auf Slideshare

0

Aus Einbettungen

0

Anzahl der Einbettungen

1

Befehle

Downloads

49

Geteilt

0

Kommentare

0

Likes

0

×