Categorical reparameterization with gumbel softmax

1. Categorical Reparameterization with Gumbel-Softmax
Semi-supervised Learning with Deep Generative Models

2. n Gumbel-softmax
n VAE
n VAE Gumbel-softmax

3. : Categorical Reparameterization with Gumbel-Softmax
n ICLR 2017 accepted
n : Eric Jang(google brain)
n : Gumbel-Softmax
[Kingma, 2013]
[Gumbel, 1954]
GS ( )

4. : Gumbel-Softmax
n p(z)
[Gumbel, 1954]
n Gumbel-Max trick
n g Gumbel
n π z
n ex) z = (0, 1, 0, ..., 0)
n argmax Softmax
n y Gumbel-Softmax
n τ
n k
n ex) y = (0.1, 0.2, 0.5, …, 0.1)
y
y
z

5. : Gumbel-Softmax
n τ
n
n y
p(z) p(y)

6. : Categorical/Gumbel-Softmax

7. : Gumbel-Softmax

8. :
n
(1)
n
(2)→

9. : Gumbel-Softmax
n [Kingma, 2013]
n
n
n
n z argmax
n y π
→

10. : Semi-supervised Learning with Deep Generative Models
n NIPS 2014 accepted
n : Kingma
n : VAE(CVAE)
!
"
#$("|', !)
#(!)
!
"
*+(!|", ')
'
#(')
*+('|")
'

11. : M1 M2
n M1: VAE
n M2: VAE
!
"
M1
#$("|!)
#(!)
!
"
()(!|")
!
"
M2
#$("|*, !)
#(!)
!
"
()(!|", *)
*
#(*)
()(*|")
*

12. : M1 (VAE)
n
n M1 z y
!
"
#$("|!)
#(!)
!
"
()(!|")
*
(+(*|!)

13. : M1
n M1 100 z y
n unlabel data

14. : M1
n

15. : M2
n = / +
n /
!
"
#$("|', !)
#(!)
!
"
*+(!|", ')
'
#(')
'
*+('|")

16. : →
n y
( a,b)
n gumbel-softmax
y
( c)

17. : M2
n M2 0.91
n 0.5 ( )
n 100 0.86 0.05
M2 2M2 1

18. : M2
n 100
n disentangle

19. : M1 + M2
n M1 M2
n M1 z1 M2 (end-to-end )
!"
#$(!&|(, !")
#(!")
!"
+
,-(!"|!&, ()
(
#(()
+
!& !&
(
,- ( !&
,-(!&|+)#$(+|!&)

20. : M1 + M2
n 0.81
n 0.96

21. : M1 + M2
n disentangle

22. gumbel-softmax /
n Learning Disentangled Joint Continuous and Discrete Representations(NIPS, 2018)
n Learning Disentangled Representations with Semi-Supervised Deep Generative Models(NIPS,
2017)
n Auxiliary Deep Generative Models (ICML, 2016)
n GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution(NIPS
workshop, 2016)

23. n pixyz
n : https://qiita.com/kogepan102/items/22b685ce7e9a51fbab98
n : https://github.com/HironoOkamoto/practice_pixyz
n pixyz : https://github.com/masa-su/pixyz