论文标题
代表学习的功能正则化：统一的理论观点
Functional Regularization for Representation Learning: A Unified Theoretical Perspective
论文作者
论文摘要
无监督和自我监督的学习方法已成为学习下游预测任务表示表示的关键工具。尽管这些方法在实践中被广泛使用并获得令人印象深刻的经验收益，但它们的理论理解在很大程度上落后。为了弥合这一差距，我们提出了一个统一的观点，可以将几种此类方法视为使用未标记的数据通过可学习的函数对表示形式施加正则化。我们提出了一个歧视性理论框架，用于分析这些方法的样本复杂性，该框架概括了（Balcan和Blum，2010）的框架以允许可学习的正则化功能。我们的样本复杂性界限表明，通过精心选择的假设类别利用数据中的结构，这些可学习的正则化功能可以修剪假设空间，并有助于减少所需的标记数据量。然后，我们提供了两个具体的函数正则化示例，一个使用自动编码器，另一个使用掩盖的自我安排，并应用我们的框架来量化标记数据的样本复杂性界限的降低。我们还提供互补的经验结果来支持我们的分析。
Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. While these approaches are widely used in practice and achieve impressive empirical gains, their theoretical understanding largely lags behind. Towards bridging this gap, we present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions. Our sample complexity bounds show that, with carefully chosen hypothesis classes to exploit the structure in the data, these learnable regularization functions can prune the hypothesis space, and help reduce the amount of labeled data needed. We then provide two concrete examples of functional regularization, one using auto-encoders and the other using masked self-supervision, and apply our framework to quantify the reduction in the sample complexity bound of labeled data. We also provide complementary empirical results to support our analysis.
