论文标题
通过绩效预测解剖跨度标识任务
Dissecting Span Identification Tasks with Performance Prediction
论文作者
论文摘要
SPAN标识（简而言之，跨度ID）任务，例如块，NER或代码转换检测，请模型以识别和对文本中的相关跨度进行分类。尽管是NLP的主食，并共享共同的结构，但对这些任务的属性如何影响它们的难度几乎没有见解，因此几乎没有指导哪些模型家族在跨度ID任务以及原因方面效果很好。我们通过性能预测分析跨度ID任务，估计神经体系结构在不同任务上的表现。我们的贡献是：（a）我们确定可以为性能预测提供信息的跨度ID任务的关键属性； （b）我们对英语数据进行了大规模实验，建立了一个模型，以预测可以支持体系结构选择的看不见的跨度ID任务； （c），我们研究了元模型的参数，对模型和任务属性如何相互作用以影响跨度ID性能产生了新的见解。我们发现，例如，跨度频率对LSTM尤为重要，并且当跨度很少且边界不固定时，CRF会有所帮助。
Span identification (in short, span ID) tasks such as chunking, NER, or code-switching detection, ask models to identify and classify relevant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks' properties influence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, building a model to predict performance for unseen span ID tasks that can support architecture choices; (c), we investigate the parameters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive.
