Adaptive Advantage Estimation for Actor-Critic Algorithms

摘要

Critics are applied to estimating policy gradients in the actor-critic framework, an essential part of Reinforcement Learning methods. An appropriate critic is supposed to balance variance from sample returns and bias introduced by parameterized value functions. A typical critic of balancing variance and bias, the generalized advantage estimator (GAE), combines sample returns and value functions with a fixed weight parameter. However, such a parameter is hard to fit, which results in no promised stability for GAE. In this paper, indicators of variance and bias are proposed to get adaptive weight parameters, with which adaptive advantage estimators are obtained. Empirical results on both 2D and 3D simulated robotic locomotion tasks show that the adaptive advantage estimators achieve similar or superior performance compared to GAE.