论文标题
弱凸随机优化问题的自适应第一和零阶方法
Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problems
论文作者
论文摘要
在本文中，我们设计和分析了一种新的自适应亚级别方法，以解决一类重要的弱凸（可能是非平滑）随机优化问题。使用指数移动平均值来更新搜索方向和学习率的自适应方法最近引起了很多关注，以解决机器学习中出现的优化问题。然而，它们的收敛分析几乎只需要目标函数的平滑度和/或凸度。相比之下，我们建立了对第一和零阶自适应方法的收敛速率及其近端变体，用于相当广泛的非平滑\和非convex优化问题。实验结果表明，所提出的算法在经验上如何优于随机梯度下降及其用于解决此类优化问题的零级变体。
In this paper, we design and analyze a new family of adaptive subgradient methods for solving an important class of weakly convex (possibly nonsmooth) stochastic optimization problems. Adaptive methods that use exponential moving averages of past gradients to update search directions and learning rates have recently attracted a lot of attention for solving optimization problems that arise in machine learning. Nevertheless, their convergence analysis almost exclusively requires smoothness and/or convexity of the objective function. In contrast, we establish non-asymptotic rates of convergence of first and zeroth-order adaptive methods and their proximal variants for a reasonably broad class of nonsmooth \& nonconvex optimization problems. Experimental results indicate how the proposed algorithms empirically outperform stochastic gradient descent and its zeroth-order variant for solving such optimization problems.
