论文标题
两层神经网络中$ L_1 $正则化的功效
The Efficacy of $L_1$ Regularization in Two-Layer Neural Networks
论文作者
论文摘要
神经网络中的一个关键问题是选择最适当的隐藏神经元并获得严格的统计风险范围。在这项工作中，我们对神经网络中的偏见变化权衡提出了新的看法。作为选择神经元数量的替代方法，我们从理论上表明$ L_1 $正则化可以控制概括错误并占用输入维度。特别是，在输出层上有适当的$ L_1 $正则化，该网络可以产生统计风险，而最小值接近最佳。此外，输入层上适当的$ L_1 $正则化会导致不涉及输入数据维度的风险界限。我们的分析基于基于维度和基于规范的复杂性分析的新合并，以限制概括误差。从我们的结果中看的一个结果是，在适当的正则化下，过多的神经元不一定会膨胀泛化错误。
A crucial problem in neural networks is to select the most appropriate number of hidden neurons and obtain tight statistical risk bounds. In this work, we present a new perspective towards the bias-variance tradeoff in neural networks. As an alternative to selecting the number of neurons, we theoretically show that $L_1$ regularization can control the generalization error and sparsify the input dimension. In particular, with an appropriate $L_1$ regularization on the output layer, the network can produce a statistical risk that is near minimax optimal. Moreover, an appropriate $L_1$ regularization on the input layer leads to a risk bound that does not involve the input data dimension. Our analysis is based on a new amalgamation of dimension-based and norm-based complexity analysis to bound the generalization error. A consequent observation from our results is that an excessively large number of neurons do not necessarily inflate generalization errors under a suitable regularization.
