论文标题
测量测量工具：自动评估文本语料库的语义指标
Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora
论文作者
论文摘要
在各种自然语言处理应用中，比较文本语料库之间语义相似性的能力很重要。但是，尚未确定评估这些指标的标准方法。我们提出了一组自动且可解释的措施，以评估语料库级别相似性指标的特征，从而可以对其行为进行明智的比较。我们通过在经典和最先进的指标中评估评估措施来捕获基本特征的有效性。我们的措施表明，最近开发的指标在识别语义分布不匹配方面变得更好，而经典指标对表面文本水平的扰动更为敏感。
The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.
