論文輪読資料「Gated Feedback Recurrent Neural Networks」

1. Gated Feedback Recurrent Neural Networks Matsuo lab. paper reading session Jul.17 2015 School of Engineering, The University of Tokyo Hiroki Kurotaki kurotaki@weblab.t.u-tokyo.ac.jp

2. Contents 2 ・Paper information ・Introduction ・Related Works ・Proposed Methods ・Experiments, Results and Analysis ・Conclusion

4. Paper Information ・Gated Feedback Recurrent Neural Networks ・Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio Dept. IRO, Universite de Montreal, CIFAR Senior Fellow ・Proceedings of The 32nd International Conference on Machine Learning, pp. 2067–2075, 2015 (ICML 2015) ・1st submission to arXiv.org is on 9 Feb 2015 ・Cited by 9 (Google Scholar, Jul 17 2015) ・http://jmlr.org/proceedings/papers/v37/chung15.html 4

6. Introduction 1/3 ・They propose a novel recurrent neural network(RNN) architecture, Gated-feedback RNN (GN-RNN). ・GN-RNN allows connections from upper layers to lower layers, then controlls the signals by global gating unit. 6 (Each circle represents a layer consists of recurrent units (i.e. LSTM-Cell))

7. Introduction 2/3 ・The proposed GF-RNN outperforms the baseline methods in these tasks. 7 1. Character-level Lauguage Modeling (from a subsequence of structured data, predict the rest of the characters.)

8. Introduction 3/3 ・The proposed GF-RNN outperforms the baseline methods in these tasks. 8 2. Python Program Evaluation (predict script execution results from the input as a raw charater sequence.) ( [Zaremba 2014] Figure 1)

10. Related works(Unit) : Long short-term memory ・A LSTM cell is just a neuron ・but it decides when to memorize, forget and expose the content value 10 ( [Zaremba 2014] Figure 1) (The notation used in the figure is slightly different from this paper)

11. Related works(Unit) : Gated recurrent unit ・Cho et al. 2014 ・Like LSTM, adaptively reset(forget) or update(input) its memory content. ・But unlike LSTM, no output gate ・balances between the previous and new memory contents adaptively 11( [Cho 2014] Figure 2)

12. Related works(Architecture) : Conventional Stacked RNN ・Each circle represents a layer consists of many recurrent units ・Several hidden recurrent layers are stacked to model and capture hierahchical structure between short and long-term dependencies. 12

13. Related works(Architecture) : Clockwork RNN ・i-th hidden module is only updated at the rate of 2^(i-1) ・Neurons in faster module i are connected to neurons in a slower module j only if a clock period T_i < T_j. 13( [Koutnik 2014] Figure 1)

15. Proposed Method : Gated Feedback RNN ・Generalize the Clockwork RNN in both connection and work rate ・Flows back from the upper recurrent layers into the lower layers ・Adaptively control when to connect each layer with "global reset gates". (Small bullets on the edges) 15 : the concatenation of all the hidden states from the previous timestep (t-1) : from layer i in timestep (t-1) to layer j in timestep t global reset gate

16. Proposed Method : GF-RNN with LSTM unit ・Only used when computing new memory state 16 ( [Zaremba 2014] Figure 1) (The notation used in the figure is slightly different from this paper)

17. Proposed Method : GF-RNN with GRN unit ・Only used when computing new memory state 17( [Cho 2014] Figure 2)

19. Experiment : Tasks (Lauguage Modeling) ・From a subsequence of structured data, predict the rest of the characters.) 19

20. Experiment : Tasks (Lauguage Modeling) ・Hutter dataset ・English Wikipedia, contains 100 MBytes of characters which include Latin alphabets, non-Latin alphabets, XML markups and special characters ・Training set : the first 90 MBytes Validation set : the next 5 MBytes Test set : the last 10 MBytes ・Performance measure : the average number of bits-per-character (BPC) 20

21. Experiment : Models (Lauguage Modeling) ・3 RNN architectures : single, (conventional) stacked, Gated-feedback ・3 recurrent units : tanh, LSTM, Gated Recurrent Unit (GRU) ・The number of parameters are constrained to be roughly 1000 ・Detail - RMSProp & momentum - 100 epochs - learning rate : 0.001 (GRU, LSTM) 5×10^(-5) (tanh) - momentum coef. : 0.9 - Each update is done using a minibatch of 100 subsequences of length 100 each. 21

22. Experiment : Results and Analysis (Lauguage Modeling) 22 ・GF-RNN is good when used together with GRU and LSTM ・But failed to improve the performance with tanh units ・GF-RNN with LSTM is better than the Non-Gated (The undermost)

23. Experiment : Results and Analysis (Lauguage Modeling) 23 ・the stacked LSTM failed to close the tags with </username> and </contributor> in both trials ・However, the GF-LSTM succeeded to close both of them, which shows that it learned about the structure of XML tags

24. Experiment : Additional results (Lauguage Modeling) 24 ・They trained another GF-RNN with LSTM which includes larger number of parameters, and obtained comparable results. ・(They wrote it's better than the previously reported best results, but there is a non-RNN work that acheived 1.278)

25. Experiment : Tasks (Python Program Evaluation) ・input : a python program ends with a print statement, 41symbols output : the result of a print statement, 13 symbols ・Scripts used in this task include addition, multiplication, subtraction, for-loop, variable assignment, logical comparison and if-else statement. ・Both the input & output are sequences of characters. ・Nesting : [1,5] ・length : [1, 1^10] 25( [Zaremba 2014] Figure 1)

26. Experiment : Models (Python Program Evaluation) ・RNN encoder-decoder approach, used for translation task previously ・Encoder RNN : the hidden state of the encoder RNN is unfolded for 50 timesteps. ・Decoder RNN : initial hidden state is initialized with the last hidden state of the encoder RNN. ・Detail - GRU & LSTM with and without Gated - 3 hidden layers for each Encoder & Decoder RNN - hidden layer contains : 230 units(GRM) 200 units(LSTM) - mixed curriculum strategy [Zaremba '14] - Adam [Kingma '14] - minibatch with 128 sequences - 30 epochs 26( [Cho 2014] Figure 1)

27. Experiment : Results & Analysis (Python Program Evaluation) ・From the 3rd column, GF-RNN is better with almost all target script. 27 GRULSTM

29. Conclusion ・They proposed a novel architecture for deep stacked RNNs which uses gated- feedback connections between different layers. ・The proposed method outperformed previous results in the tasks of character-level language modeling and Python program evaluation. ・Gated-feedback architecture is faster and better (in performance) than the standard stacked RNN even with a same amount of capacity. ・More thorough investigation into the interaction between the gated- feedback connections and the role of recurrent activation function is required in the future. (because the proposed gated-feedback architecture works bad with the tanh activation function) 29

30. References [Cho 2014] Cho, Kyunghyun, Van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. [Koutnik 2014] Koutnik, Jan, Greff, Klaus, Gomez, Faustino, and Schmidhuber, Ju ̈rgen. A clockwork rnn. In Proceedings of the 31st International Conference on Machine Learning (ICML’14), 2014. [Schmidhuber 1992] Schmidhuber, Jurgen. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242, 1992. [Stollenga 2014] Stollenga, Marijn F, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Ju ̈rgen. Deep networks with internal selective attention through feedback connections. In Ad- vances in Neural Information Processing Systems, pp. 3545–3553, 2014. [Zaremba 2014] Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. 30