论文标题
Pangu-bot：预先训练的语言模型的有效生成对话预训练
PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
论文作者
论文摘要
在本文中，我们介绍了基于大型预训练的语言模型（PLM）pangu-alpha（Zeng等，2021）的中国预训练的开放域对话生成模型Pangu-bot。不同于其他预训练的对话模型，该模型通过从头开始进行大量对话数据训练，我们旨在通过继承PLMS的有价值的语言能力和知识来构建强大的对话模型，其数据和计算成本相对较少。为此，我们训练大型PLM Pangu-Alpha的Pangu-bot，该机器人已被证明在各种中国自然语言任务上表现出色。我们研究了pangu-bot产生的响应的不同方面，包括响应质量，知识和安全性。我们表明，Pangu-Bot优于最先进的中国对话系统（CDIALGPT（Wang等，2020），Eva（Zhou等，2021），EVA2.0（Gu等，2022））W.R.T.以上三个方面。我们还证明，可以轻松地部署pangu-bot，以在没有进一步训练的情况下产生情感反应。在我们的经验分析中，我们还指出，pangu-bot响应质量，知识正确性和安全性仍然远非完美，进一步的探索对于构建可靠且智能的对话系统是必不可少的。我们的模型和代码将在https://github.com/huawei-noah/pretaining-language-model/tree/master/master/pangu-bot上提供。
In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive amount of dialogue data from scratch, we aim to build a powerful dialogue model with relatively fewer data and computation costs by inheriting valuable language capabilities and knowledge from PLMs. To this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been proven well-performed on a variety of Chinese natural language tasks. We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety. We show that PanGu-Bot outperforms state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA (Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects. We also demonstrate that PanGu-Bot can be easily deployed to generate emotional responses without further training. Throughout our empirical analysis, we also point out that the PanGu-Bot response quality, knowledge correctness, and safety are still far from perfect, and further explorations are indispensable to building reliable and smart dialogue systems. Our model and code will be available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/PanGu-Bot soon.
