论文标题

将明确的知识纳入预先训练的语言模型以重新排序

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

论文作者

Dong, Qian, Liu, Yiding, Cheng, Suqi, Wang, Shuaiqiang, Cheng, Zhicong, Niu, Shuzi, Yin, Dawei

论文摘要

通过重新排列的段落是为了从检索阶段获得候选段落的排列。由于自然语言理解的优势,预先训练的语言模型(PLM)蓬勃发展。但是,现有的基于PLM的重新级别可能很容易遭受词汇不匹配和缺乏域特定知识的困扰。为了减轻这些问题,我们的工作中仔细介绍了知识图中包含的明确知识。具体来说,我们采用了不完整且嘈杂的现有知识图,并首先将其应用于段落重新排列任务。为了利用可靠的知识,我们提出了一种新颖的知识图蒸馏方法,并获得知识元图作为查询和通过之间的桥梁。为了使两种嵌入在潜在空间中,我们将PLM作为文本编码器和图形神经网络而不是知识元图作为知识编码器。此外,新颖的知识喷油器设计用于文本和知识编码器之间的动态相互作用。实验结果证明了我们方法的有效性,尤其是在需要深入领域知识的查询中。

Passage re-ranking is to obtain a permutation over the candidate passage set from retrieval stage. Re-rankers have been boomed by Pre-trained Language Models (PLMs) due to their overwhelming advantages in natural language understanding. However, existing PLM based re-rankers may easily suffer from vocabulary mismatch and lack of domain specific knowledge. To alleviate these problems, explicit knowledge contained in knowledge graph is carefully introduced in our work. Specifically, we employ the existing knowledge graph which is incomplete and noisy, and first apply it in passage re-ranking task. To leverage a reliable knowledge, we propose a novel knowledge graph distillation method and obtain a knowledge meta graph as the bridge between query and passage. To align both kinds of embedding in the latent space, we employ PLM as text encoder and graph neural network over knowledge meta graph as knowledge encoder. Besides, a novel knowledge injector is designed for the dynamic interaction between text and knowledge encoder. Experimental results demonstrate the effectiveness of our method especially in queries requiring in-depth domain knowledge.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源