论文标题
基准测试机器阅读理解模型的鲁棒性
Benchmarking Robustness of Machine Reading Comprehension Models
论文作者
论文摘要
机器阅读理解（MRC）是评估模型自然语言理解（NLU）能力的重要测试。在这一领域取得了迅速的进步，新的模型在各种基准测试中取得了令人印象深刻的性能。但是，现有基准仅评估在内域测试集上的模型，而无需考虑其在测试时间扰动或对抗攻击下的稳健性。为了填补这一重要差距，我们构建了Advrace（对抗种族），这是一种新的模型无关基准，用于评估MRC模型在四种不同类型的对抗性攻击下的鲁棒性，包括我们的新型干扰器提取和一代攻击。我们表明，最新的（SOTA）模型容易受到所有这些攻击的影响。我们得出的结论是，建立更强大的MRC模型的空间很大，我们的基准可以帮助激励和衡量该领域的进度。我们在https://github.com/noviscl/advrace上发布数据和代码。
Machine Reading Comprehension (MRC) is an important testbed for evaluating models' natural language understanding (NLU) ability. There has been rapid progress in this area, with new models achieving impressive performance on various benchmarks. However, existing benchmarks only evaluate models on in-domain test sets without considering their robustness under test-time perturbations or adversarial attacks. To fill this important gap, we construct AdvRACE (Adversarial RACE), a new model-agnostic benchmark for evaluating the robustness of MRC models under four different types of adversarial attacks, including our novel distractor extraction and generation attacks. We show that state-of-the-art (SOTA) models are vulnerable to all of these attacks. We conclude that there is substantial room for building more robust MRC models and our benchmark can help motivate and measure progress in this area. We release our data and code at https://github.com/NoviScl/AdvRACE .
