论文标题
迈向自动化错误分析：学会表征错误
Towards Automated Error Analysis: Learning to Characterize Errors
论文作者
论文摘要
表征系统制造的错误模式，可以帮助研究人员将未来的发展集中在提高其准确性和鲁棒性上。我们提出了一种新颖的“元学习”形式，该形式会自动学习可解释的规则，这些规则表征了系统造成的错误类型，并演示了这些规则的能力，可以帮助理解和改善两个NLP系统。我们的方法是通过在验证数据上收集错误案例，提取描述这些样本的元功能的错误情况，最后学习使用这些功能来表征错误的规则。我们将我们的方法应用于Vilbert，以进行视觉问答和Roberta，以回答常识。我们的系统学习可解释的规则，这些规则可以洞悉这些系统对给定任务的系统错误。使用这些见解，我们还能够“关闭循环”并适度提高这些系统的性能。
Characterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes, and demonstrate these rules' ability to help understand and improve two NLP systems. Our approach works by collecting error cases on validation data, extracting meta-features describing these samples, and finally learning rules that characterize errors using these features. We apply our approach to VilBERT, for Visual Question Answering, and RoBERTa, for Common Sense Question Answering. Our system learns interpretable rules that provide insights into systemic errors these systems make on the given tasks. Using these insights, we are also able to "close the loop" and modestly improve performance of these systems.
