论文标题
使用语义启动测试探索Bert对词汇提示的敏感性
Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming
论文作者
论文摘要
在自然语言处理中，经过培训以估计上下文中概率的模型已变得无处不在。这些模型如何在上下文中使用词汇提示来告知其单词概率？为了回答这个问题，我们提出了一个案例研究，分析了预先训练的BERT模型，并通过语义启动告知的测试。使用显示人类启动的英语词汇刺激，我们发现Bert也显示出“启动”，当上下文包含一个相关单词与无关的单词时，可以预测一个概率更大的单词。随着上下文提供的信息量的增加，此效果会降低。后续分析表明，随着上下文变得更加有用，伯特越来越分心，将相关的概率分配给相关词。我们的发现强调了在研究这些模型中研究单词预测时考虑上下文约束效应的重要性，并强调了与人类处理的可能相似之处。
Models trained to estimate word probabilities in context have become ubiquitous in natural language processing. How do these models use lexical cues in context to inform their word probabilities? To answer this question, we present a case study analyzing the pre-trained BERT model with tests informed by semantic priming. Using English lexical stimuli that show priming in humans, we find that BERT too shows "priming," predicting a word with greater probability when the context includes a related word versus an unrelated one. This effect decreases as the amount of information provided by the context increases. Follow-up analysis shows BERT to be increasingly distracted by related prime words as context becomes more informative, assigning lower probabilities to related words. Our findings highlight the importance of considering contextual constraint effects when studying word prediction in these models, and highlight possible parallels with human processing.
