Fast and Efficient Training of Neural Networks

February 5, 2020

---

References in the Video

• CodeEmporium
• Code for the video
• Code behind the DCGAN with Apex
• Mixed Precision Training
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Don't Decay the Learning Rate, Increase the Batch Size
• On the Variance of the Adaptive Learning Rate and Beyond
• Cyclical Learning Rates for Training Neural Networks
• The 1cycle policy
• Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Deep Double Descent
• Deep Double Descent: Where Bigger Models and More Data Hurt
• Reconciling modern machine learning practice and the bias-variance trade-of

---