论文标题
值得（环境）成本吗？通过连续培训进行时间适应的有限证据
Is It Worth the (Environmental) Cost? Limited Evidence for Temporal Adaptation via Continuous Training
论文作者
论文摘要
语言正在不断变化和发展，使语言模型变得快速过时。因此，我们应该不断使用新数据更新我们的模型，以使其暴露于新事件和事实中。但是，这需要其他计算，这意味着新的碳排放。有任何可衡量的好处证明这一费用是合理的吗？本文寻求经验证据来支持持续培训。我们复制现有的基准测试，并将其扩展到包括额外的时间段，模型和任务。我们的结果表明，随着时间的推移，暂时适应的英语模型的下游任务性能不会改善。实际上，没有时间适应的预处理模型实际上更加有效。但是，我们还注意到缺乏合适的时间基准。我们的发现引起了人们对何时以及如何暂时适应语言模型的批判性思考，从而考虑了可持续性。
Language is constantly changing and evolving, leaving language models to become quickly outdated. Consequently, we should continuously update our models with new data to expose them to new events and facts. However, that requires additional computing, which means new carbon emissions. Do any measurable benefits justify this cost? This paper looks for empirical evidence to support continuous training. We reproduce existing benchmarks and extend them to include additional time periods, models, and tasks. Our results show that the downstream task performance of temporally adapted English models for social media data do not improve over time. Pretrained models without temporal adaptation are actually significantly more effective and efficient. However, we also note a lack of suitable temporal benchmarks. Our findings invite a critical reflection on when and how to temporally adapt language models, accounting for sustainability.
