论文标题
用语义类探测量化单词表示的上下文化
Quantifying the Contextualization of Word Representations with Semantic Class Probing
论文作者
论文摘要
预审前的语言模型已在许多NLP任务上实现了新的技术状态，但是关于它们如何以及为什么它们如此良好的工作仍然存在许多开放问题。我们研究了伯特中单词的上下文化。我们通过研究可以从其上下文化的嵌入中推断出单词的语义类别的程度来量化上下文化的数量，即在上下文中解释的量。量化上下文化有助于理解和利用验证的语言模型。我们表明，顶层表示能够高精度推断语义类别；最强的情境化效应发生在下层。这种本地环境大多足以进行语义类别推论。而且，在登录后，该顶层表示更为特定于任务，而下层表示则更可转移。 Finetuning发现与任务相关的功能，但预处理的知识仍在很大程度上保留。
Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We investigate the contextualization of words in BERT. We quantify the amount of contextualization, i.e., how well words are interpreted in context, by studying the extent to which semantic classes of a word can be inferred from its contextualized embeddings. Quantifying contextualization helps in understanding and utilizing pretrained language models. We show that top layer representations achieve high accuracy inferring semantic classes; that the strongest contextualization effects occur in the lower layers; that local context is mostly sufficient for semantic class inference; and that top layer representations are more task-specific after finetuning while lower layer representations are more transferable. Finetuning uncovers task related features, but pretrained knowledge is still largely preserved.
