论文标题
我们到了吗？使用EUPEG作为基准平台评估基于神经网络的最先进的地理赛车
Are We There Yet? Evaluating State-of-the-Art Neural Network based Geoparsers Using EUPEG as a Benchmarking Platform
论文作者
论文摘要
地理信息检索是一项重要任务。一个被称为Geoparser的地理系统将一些文本作为输入，并输出公认的位置提及及其位置坐标。 2019年6月，一场地理竞赛（科学论文中的高级决议）被视为Semeval 2019年任务之一。获胜的团队开发了基于神经网络的地理赛车，可以取得出色的表现（超过90％的精确度，回忆和F1得分以获得最高的识别）。这个令人兴奋的结果带来了一个问题：“我们还在吗？”，即我们取得了足够高的表现，可以将地理验证的问题视为解决的问题吗？这项竞争的一个局限性是，仅在一个数据集上测试了开发的地理阶层，该数据集中有45个研究文章从特定的生物医学领域收集。众所周知，相同的地理阶段器可以在不同的数据集上具有非常不同的性能。因此，这项工作使用我们最近开发的基准测试平台Eupeg对这些最先进的地理阶层进行了系统的评估，该平台EUPEG具有八个带注释的数据集，九个基线地理票据和八个性能指标。评估结果表明，尽管仍然存在一些挑战，但这些新的地理赛车确实改善了在多个数据集上进行地理验证的性能。
Geoparsing is an important task in geographic information retrieval. A geoparsing system, known as a geoparser, takes some texts as the input and outputs the recognized place mentions and their location coordinates. In June 2019, a geoparsing competition, Toponym Resolution in Scientific Papers, was held as one of the SemEval 2019 tasks. The winning teams developed neural network based geoparsers that achieved outstanding performances (over 90% precision, recall, and F1 score for toponym recognition). This exciting result brings the question "are we there yet?", namely have we achieved high enough performances to possibly consider the problem of geoparsing as solved? One limitation of this competition is that the developed geoparsers were tested on only one dataset which has 45 research articles collected from the particular domain of Bio-medicine. It is known that the same geoparser can have very different performances on different datasets. Thus, this work performs a systematic evaluation of these state-of-the-art geoparsers using our recently developed benchmarking platform EUPEG that has eight annotated datasets, nine baseline geoparsers, and eight performance metrics. The evaluation result suggests that these new geoparsers indeed improve the performances of geoparsing on multiple datasets although some challenges remain.
