论文标题
部分可观测时空混沌系统的无模型预测
CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Control
论文作者
论文摘要
内在动机是一种有前途的探索技术，用于解决稀疏或缺乏外在奖励的增强学习任务。实施内在动机有两个技术挑战：1）如何设计一个适当的内在目标以促进有效的探索； 2）如何将固有目标与外在目标结合起来，以帮助找到更好的解决方案。在当前的文献中，固有的目标都是以任务不合稳定的方式设计的，并通过简单添加（或自身用于无奖励的预训练）与外部目标结合。在这项工作中，我们表明这些设计将在典型的稀疏奖励连续控制任务中失败。为了解决该问题，我们提出了受限的内在动机（CIM），以利用易于达到的任务先验来构建一个约束的内在目标，同时利用Lagrangian方法来适应通过同时最大化框架来适应内在和外在目标。我们从经验上表明，在多个稀疏奖励连续的控制任务上，我们的CIM方法实现了对最先进方法的性能和样本效率的大大提高。此外，我们的CIM的关键技术也可以插入现有方法以提高其性能。
Intrinsic motivation is a promising exploration technique for solving reinforcement learning tasks with sparse or absent extrinsic rewards. There exist two technical challenges in implementing intrinsic motivation: 1) how to design a proper intrinsic objective to facilitate efficient exploration; and 2) how to combine the intrinsic objective with the extrinsic objective to help find better solutions. In the current literature, the intrinsic objectives are all designed in a task-agnostic manner and combined with the extrinsic objective via simple addition (or used by itself for reward-free pre-training). In this work, we show that these designs would fail in typical sparse-reward continuous control tasks. To address the problem, we propose Constrained Intrinsic Motivation (CIM) to leverage readily attainable task priors to construct a constrained intrinsic objective, and at the same time, exploit the Lagrangian method to adaptively balance the intrinsic and extrinsic objectives via a simultaneous-maximization framework. We empirically show, on multiple sparse-reward continuous control tasks, that our CIM approach achieves greatly improved performance and sample efficiency over state-of-the-art methods. Moreover, the key techniques of our CIM can also be plugged into existing methods to boost their performances.
