Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (12)
Ähnlich wie Deep learning: what? how? why? How to win a Kaggle competition
Ähnlich wie Deep learning: what? how? why? How to win a Kaggle competition (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Deep learning: what? how? why? How to win a Kaggle competition
- 2. In about 45 minutes ZMUV
- 3. In about 30 minutes Bigger & Deeper Is better
- 4. In about 15 minutes
- 5. Who works in Machine learning?
- 6. Who am I Jonas Degrave Phd student UGent
- 8. Who are we? former Reservoir Lab Data Science Lab IDLab
- 10. What do we do? Machine learning Robotics Brain-inspired computing
- 11. What did we do? Totalling $160k in prizes
- 12. Testimonials
- 13. Neural networks in 5 minutes
- 14. Neural networks in 5 minutes Input layer Hidden layers Output layer
- 15. Gradient descent
- 16. Backpropagation
- 17. Deep learning Input layer Hidden layer Output layer Hidden layer Hidden layer
- 18. History Artificial Neural Net: 1949 Backpropagation: 1975 Deep Learning: 2012
- 19. What used to be the problem Input layer Hidden layer Output layer Hidden layer Hidden layer
- 21. For long, we didn’t know
- 24. State of the art for all problems with spatially correlated data No more feature engineering!
- 25. Old school bingo Boltzmann Machines Energy Tanh or sigmoid activation Feature engineering Deep belief networks
- 27. Train set Make the sets, make them well Validation set & Test set
- 28. Choose your error function Always optimize the error function where possible! Use error function for your problem
- 32. Regularize Bigger & Deeper “Larger networks tend to work better. Make your network bigger and bigger until the accuracy stops increasing. Then regularize the hell out of it. Then make it bigger still.” – Yoshua Bengio
- 33. My first architecture Start with standard components: Conv-layers, dense layers, max-pooling, dropout
- 34. Sparsity Make sure, that for each sample, only a few parameters are used
- 36. Dropout
- 37. Maxpool y x
- 40. No bigger than 3x3 3x3 layer 9 parameters 3x3 receptive field 5x5 layer 25 parameters 5x5 receptive field 2 stacked 3x3 layer 19 parameters 5x5 receptive field
- 41. Output function
- 42. My first architecture ~ 1 million parameters
- 43. Let us optimize
- 47. Local minimum You want generalization, not the global minimum on the train set!
- 48. My first architecture ~ 1 million parameters
- 49. Initialization
- 50. Weight matrices Random orthogonal initialization With correct amplitude Most libraries provide this Does not lose information
- 51. Output layer You have prior information! Initialize with zeros!
- 52. Bias Bias sets the initial sparsity!
- 54. My first architecture ~ 1 million parameters
- 55. Does Train on 1 sample it work ? Train on 2 samples
- 56. Learning rate Overshooting Learn too slow
- 57. Learning rate
- 58. Data preprocessing ZMUV your data Zero mean Unit Variance
- 60. Input layer Hidden layer Output layer Hidden layer Hidden layer Batch normalization batchnorm batchnorm batchnorm batchnorm
- 63. Unsupervised learning Learn on the test set Pseudo-labeling Ladder networks
- 64. Insert a priori information into architecture
- 65. Insert a priori information
- 66. It’s an art
- 67. Regularize Bigger & Deeper “Rinse and repeat” – Jonas Degrave
- 68. Ensemble The average prediction will always be better than the worst prediction.
- 70. Submit
- 71. Computing time Deadlines are fixed End performance is proportional to number of iterations, NOT training time per model
- 72. Major take-aways Everything has a reason Don’t buy into hypes If it can’t be explained in 1 minute why it works, it probably isn’t working.