论文标题
动量改善了标准化的SGD
Momentum Improves Normalized SGD
论文作者
论文摘要
我们提供了对标准化SGD的改进分析，表明添加动量可以消除对非convex目标上的大批量大小的需求。然后，我们考虑具有有界第二个衍生物的目标的情况，并表明对动量公式进行了小调整，可以使标准化的SGD具有动量，以找到$ O（1/ε^{3.5}）$迭代中的$ε$ - 关键点，与迭代率匹配，与最不知名的速率相匹配而无需计算任何差异因素或依赖性依赖性。我们还提供了一种自适应方法，该方法可以自动提高梯度方差时，会自动提高收敛速率。最后，我们表明，当在流行的大规模任务（例如Resnet-50和Bert预审计）上使用时，我们的方法是有效的，与用于在这两个任务上获得最先进结果的不同方法的性能相匹配。
We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives. Then, we consider the case of objectives with bounded second derivative and show that in this case a small tweak to the momentum formula allows normalized SGD with momentum to find an $ε$-critical point in $O(1/ε^{3.5})$ iterations, matching the best-known rates without accruing any logarithmic factors or dependence on dimension. We also provide an adaptive method that automatically improves convergence rates when the variance in the gradients is small. Finally, we show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining, matching the performance of the disparate methods used to get state-of-the-art results on both tasks.
