论文标题
违反：因果语言模型的对比度学习
ContraCLM: Contrastive Learning For Causal Language Model
论文作者
论文摘要
尽管因果语言模型令人兴奋，但由于歧视能力差，表示形式的表现力在很大程度上受到限制。为了解决这个问题，我们提出了Contraclm，这是一个在令牌级别和序列级别上的新型对比学习框架。我们对各种下游任务进行评估。我们表明，违规行为增强了对表示形式的歧视，并与仅编码模型的差距弥合了差距，从而使因果语言模型更适合于语言生成以外的任务。具体来说，我们在语义文本相似性任务上获得了$ 44 \％$的相对改进，而代码对代码搜索任务的$ 34 \％$。此外，通过提高表示形式的表现力，contraclm还以$ 9 \％$ $ $ $ $的相对提高人类基准的执行准确性提高了源代码生成能力。
Despite exciting progress in causal language models, the expressiveness of the representations is largely limited due to poor discrimination ability. To remedy this issue, we present ContraCLM, a novel contrastive learning framework at both token-level and sequence-level. We assess ContraCLM on a variety of downstream tasks. We show that ContraCLM enhances discrimination of the representations and bridges the gap with the encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain $44\%$ relative improvement on the Semantic Textual Similarity tasks and $34\%$ on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of the representations, ContraCLM also boosts the source code generation capability with $9\%$ relative improvement on execution accuracy on the HumanEval benchmark.
