Don T Decay The Learning Rate Increase The Batch Size
Don T Decay The Learning Rate Increase The Batch Size . Don’t decay the learning rate, increase the batch size. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. [Tip] Reduce the batch size to generalize your model from forums.fast.ai In this sense decaying learning rate during training is very similar to simulated annealing. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. It is common practice to decay the learning rate.