论文标题
注意力解释吗？集合的定量评估
Is Attention Interpretation? A Quantitative Assessment On Sets
论文作者
论文摘要
围绕注意机制的解释性的争论集中在是否可以将注意力评分用作数据的相对信号量的代理。我们建议在设置机器学习的背景下研究注意力的解释性，其中每个数据点由带有全球标签的无序集合组成。对于经典的多个实体学习问题和简单的扩展，有一个明确定义的“重要性”地面真理，可以将解释作为二进制分类问题，我们可以对此进行定量评估。通过在几种数据方式上构建合成数据集，我们对基于注意力的解释进行系统评估。我们发现，注意力分布确实通常反映了各个实例的相对重要性，但是在模型具有很高的分类性能而却不与期望不符的注意力模式的情况下发生沉默失败。基于这些观察结果，我们建议使用结合来最大程度地减少误导基于注意力的解释的风险。
The debate around the interpretability of attention mechanisms is centered on whether attention scores can be used as a proxy for the relative amounts of signal carried by sub-components of data. We propose to study the interpretability of attention in the context of set machine learning, where each data point is composed of an unordered collection of instances with a global label. For classical multiple-instance-learning problems and simple extensions, there is a well-defined "importance" ground truth that can be leveraged to cast interpretation as a binary classification problem, which we can quantitatively evaluate. By building synthetic datasets over several data modalities, we perform a systematic assessment of attention-based interpretations. We find that attention distributions are indeed often reflective of the relative importance of individual instances, but that silent failures happen where a model will have high classification performance but attention patterns that do not align with expectations. Based on these observations, we propose to use ensembling to minimize the risk of misleading attention-based explanations.
