Dynamic RNNs Adapt Computation
•
1 gefällt mir•3,474 views
The document discusses SkipRNN and RepeatRNN, which are recurrent neural network models that can dynamically control computation. SkipRNN learns to skip state updates in RNNs by introducing a binary gate that decides whether to update or copy the state. It was shown to achieve better accuracy with less computation on sequential tasks. RepeatRNN reproduces adaptive computation time, allowing the network to repeat computations for difficult examples. It was found to train and infer faster than baselines on an addition task. Code for both models is available online in PyTorch and TensorFlow.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Dynamic RNNs Adapt Computation
Ähnlich wie Dynamic RNNs Adapt Computation (20)
Mehr von Universitat Politècnica de Catalunya
Mehr von Universitat Politècnica de Catalunya (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Dynamic RNNs Adapt Computation
- 1. Xavier Giro-i-Nieto xavier.giro@upc.edu Associate Professor Intelligent Data Science and Artificial Intelligence Center Universitat Politecnica de Catalunya (UPC) DEEP LEARNING WORKSHOP Dublin City University 21-22 May 2018 Skipping and Repeating Samples in RNNs Case study III #InsightDL2018
- 2. bit.ly/InsightDL2018 #InsightDL2018 Our team @ UPC IDEAI Barcelona Victor Campos Amaia Salvador Amanda Duarte Dani Fojo Görkem Çamli Akis Linardos Santiago Pascual Xavi Giró Miriam Bellver Marta R. Costa.jussà Ferran Marqués Jordi Torres (BSC) Marc Assens Sandra Roca Miquel LLobet Antonio Bonafonte
- 3. bit.ly/InsightDL2018 #InsightDL2018 Kevin McGuinness Shih-Fu Chang Noel E. O’Connor Cathal Gurrin Eva Mohedano Carles Ventura Our partners @ academia Francisco Roldan Marta Coll Paula Gómez Alejandro Woodward Marc Górriz
- 5. 2016 20172015 Convolutional Neural Networks (CNNs) Evolution Strategies Recurrent Neural Networks (RNN) Adversarial Training 2018 Unsupervised Learning Reinforcement Learning
- 6. 2016 20172015 Convolutional Neural Networks (CNNs) Evolution Strategies Recurrent Neural Networks (RNN) Adversarial Training 2018 Unsupervised Learning Reinforcement Learning
- 8. bit.ly/InsightDL2018 #InsightDL2018 Outline Recurrent Neural Networks (RNNs) SkipRNN [ICLR 2018] RepeatRNN [ICLRW 2018]
- 9. bit.ly/InsightDL2018 #InsightDL2018 Outline Recurrent Neural Networks (RNNs) SkipRNN [ICLR 2018] RepeatRNN [ICLRW 2018]
- 10. bit.ly/InsightDL2018 #InsightDL2018 Feed-forward Neural Network INPUT(x) OUTPUT(y) FeedForward Hidden States h1 & h2 Feed-forward Weights (Wi ) Figure: Hugo Larochelle
- 12. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network Recurrent Weights (U) Feed-forward Weights (W)
- 13. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network Updated state Previous state INPUT(x)
- 14. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network time time Unfold (Rotation 90o ) Unfold (Rotation 90o )
- 16. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network CNN CNN CNN... RNN RNN RNN... Video sequences (eg. action recognition)
- 17. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network Word sequences (eg. Machine Translation)
- 18. bit.ly/InsightDL2018 #InsightDL2018 Recurrent Neural Network Spectrograms sequence (eg. speech recognition) T H _ _ E … D _ O _ _G
- 19. bit.ly/InsightDL2018 #InsightDL2018 Outline Recurrent Neural Networks (RNNs) SkipRNN [ICLR 2018] RepeatRNN [ICLRW 2018]
- 20. bit.ly/InsightDL2018 #InsightDL2018 Skip RNN: Learning to Skip State Updates in RNNs Víctor Campos Jordi Torres Xavier Giró-i-Nieto Shih-Fu ChangBrendan Jou Victor Campos, Brendan Jou, Xavier Giro-i-Nieto, Jordi Torres, and Shih-Fu Chang. “Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks”, ICLR 2018.
- 21. bit.ly/InsightDL2018 #InsightDL2018 Motivation Used Unused CNN CNN CNN... RNN RNN RNN...
- 28. bit.ly/InsightDL2018 #InsightDL2018 Limiting computation cost per sample (hyperparameter) 1 if sample used 0 otherwise Intuition: the network can be encouraged to perform fewer updates by adding a penalization when ut = 1
- 30. bit.ly/InsightDL2018 #InsightDL2018 Experiments: Sequential MNIST Epochs for Skip LSTM (λ = 10-4 ) ~30% acc ~50% acc ~70% acc ~95% acc 11 epochs 51 epochs 101 epochs 400 epochs Used Unused
- 31. bit.ly/InsightDL2018 #InsightDL2018 Experiments: Sequential MNIST cost per sample (hyperparameter) Random baseline
- 33. bit.ly/InsightDL2018 #InsightDL2018 Experiments: Sequential MNIST ...with less computation Ours
- 34. bit.ly/InsightDL2018 #InsightDL2018 Experiments: Action Localization CNN CNN CNN... RNN RNN RNN... better
- 35. bit.ly/InsightDL2018 #InsightDL2018 Experiments: Action Localization CNN CNN CNN... RNN RNN RNN... faster
- 37. bit.ly/InsightDL2018 #InsightDL2018 Outline Recurrent Neural Networks (RNNs) SkipRNN [ICLR 2018] RepeatRNN [ICLRW 2018]
- 38. Reproducing and Analyzing Adaptive Computation Time in PyTorch and TensorFlow Víctor Campos Xavier Giró-i-NietoDani Fojo Fojo, Daniel, Víctor Campos, and Xavier Giró-i-Nieto. "Comparing Fixed and Adaptive Computation Time for Recurrent Neural Networks." ICLR Workshop 2018.
- 39. bit.ly/InsightDL2018 #InsightDL2018 Adaptive Computation Time (ACT) Graves, Alex. "Adaptive computation time for recurrent neural networks." arXiv preprint arXiv:1603.08983 (2016). REPEAT 3 REPEAT 2
- 44. bit.ly/InsightDL2018 #InsightDL2018 Experiment: One-hot addition Interpretable hyperparameter Ours
- 45. bit.ly/InsightDL2018 #InsightDL2018 Experiment: One-hot addition Faster training…. Ours
- 46. bit.ly/InsightDL2018 #InsightDL2018 Experiment: One-hot addition ...and faster inference Ours
- 48. bit.ly/InsightDL2018 #InsightDL2018 Outline Recurrent Neural Networks (RNNs) SkipRNN [ICLR 2018] RepeatRNN [ICLRW 2018]
- 50. bit.ly/InsightDL2018 #InsightDL2018Deep Learning courses @ UPC (videos) ● MSc course (2017) ● BSc course (2018) ● 1st edition (2016) ● 2nd edition (2017) ● 3rd edition (2018) ● 1st edition (2017) ● 2nd edition (2018) Next edition Autumn 2018 Next edition Winter/Spring 2019Summer School (starts 28/06)
- 51. @DocXavi Xavier Giro-i-Nieto Download slides from: http://bit.ly/InsightDL2018 We are looking for both industrial & academic partners: xavier.giro@upc.edu #InsightDL2018 Click to comment:
- 52. Questions?
- 53. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Intuition: introduce a binary update state gate, ut , deciding whether the RNN state is updated or copied S(xt , ht-1 ) if ut = 1 // update operation st-1 if ut = 0 // copy operation st =S xt st
- 55. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Update state gate ∈ {0, 1} Update state probability ∈ [0, 1]
- 56. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Update state gate ∈ {0, 1} Update state probability ∈ [0, 1] Increment for the update state probability
- 57. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Update state gate ∈ {0, 1} Update state probability ∈ [0, 1] Increment for the update state probability
- 58. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Update state (ut = 1) Copy state (ut = 0)
- 59. bit.ly/InsightDL2018 #InsightDL2018 SkipRNN Update state (ut = 1) Copy state (ut = 0)
- 60. bit.ly/InsightDL2018 #InsightDL2018 Straight Though Estimator fbinarize (x) x Forward pass 1 10 G. Hinton, Neural Networks for Machine Learning, Coursera video lectures 2012
- 61. bit.ly/InsightDL2018 #InsightDL2018 fbinarize (x) x Forward pass Backward pass G. Hinton, Neural Networks for Machine Learning, Coursera video lectures 2012 1 10 Straight Though Estimator