论文标题
超越校准：估计现代神经网络的分组损失
Beyond calibration: estimating the grouping loss of modern neural networks
论文作者
论文摘要
确保分类器给出可靠的信心分数的能力对于确保明智的决策至关重要。为此，最近的工作集中于误解，即模型得分的过度或置信度。然而，校准还不够：即使是具有最佳准确性的完美校准分类器也可以具有远离真正的后验概率的置信度得分。这是由于分组损失造成的，该损失由具有相同置信度得分的样本创建，但实际的后验概率不同。适当的评分规则理论表明，鉴于校准损失，表征单个错误的缺失部分是分组损失。尽管校准损失的估计值很多，但在标准设置中的分组损失不存在。在这里，我们提出一个估算器以近似分组损失。我们表明，视觉和NLP中的现代神经网络体系结构表现出分组损失，尤其是在分配转移设置中，这突出了预生产验证的重要性。
The ability to ensure that a classifier gives reliable confidence scores is essential to ensure informed decision-making. To this end, recent work has focused on miscalibration, i.e., the over or under confidence of model scores. Yet calibration is not enough: even a perfectly calibrated classifier with the best possible accuracy can have confidence scores that are far from the true posterior probabilities. This is due to the grouping loss, created by samples with the same confidence scores but different true posterior probabilities. Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. While there are many estimators of the calibration loss, none exists for the grouping loss in standard settings. Here, we propose an estimator to approximate the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings, which highlights the importance of pre-production validation.
