论文标题
对为什么过度参数化加剧虚假相关性的调查
An Investigation of Why Overparameterization Exacerbates Spurious Correlations
论文作者
论文摘要
我们研究了为什么过度参数化 - 增加模型大小远远超出了零训练错误的位置 - 尽管数据中存在虚假相关性时，但仍会损害少数群体的测试错误。通过在两个图像数据集上的模拟和实验，我们确定了训练数据的两个关键特性，这些训练数据驱动了这种行为：多数群体与少数群体的比例以及虚假相关性的信噪比。然后，我们分析了线性设置，并从理论上展示了模型对“记忆”较少示例的电感偏差如何导致过度参数化受到伤害。我们的分析导致了对多数族群进行亚采样的违反直觉方法，即使在过重少数群体失败的标准方法中，在经验上实现了低少数群体错误。总体而言，我们的结果表明，使用过度参数化的模型与使用所有训练数据之间的张力有张力。
We study why overparameterization -- increasing model size well beyond the point of zero training error -- can hurt test error on minority groups despite improving average test error when there are spurious correlations in the data. Through simulations and experiments on two image datasets, we identify two key properties of the training data that drive this behavior: the proportions of majority versus minority groups, and the signal-to-noise ratio of the spurious correlations. We then analyze a linear setting and theoretically show how the inductive bias of models towards "memorizing" fewer examples can cause overparameterization to hurt. Our analysis leads to a counterintuitive approach of subsampling the majority group, which empirically achieves low minority error in the overparameterized regime, even though the standard approach of upweighting the minority fails. Overall, our results suggest a tension between using overparameterized models versus using all the training data for achieving low worst-group error.
