Balanced Sampling and Reusing Imaginary Data for World Models in Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) has shown significant success in domains such as computer vision and robot control. However, DRL agents often suffer from low sample efficiency, limiting their practical applicability in industrial settings. Recent advances in model-based DRL, particularly model-based approaches, have sought to address this issue by leveraging imaginary data to improve decision-making and sampling efficiency. Despite their promise, these methods face challenges such as overreliance on early experiences in the replay buffer and under-utilization of imaginary data, which can lead to overfitting and suboptimal policy optimization. To overcome these limitations, we propose a novel reinforcement learning framework, balanced sampling and reusing imaginary data (BSRID), which introduces two key innovations: (1) a balanced sampling (BS) mechanism that ensures uniform sampling rates to mitigate bias toward early experiences and (2) a reusing imaginary data (RID) strategy that enhances policy optimization by increasing update frequency and maximizing the utility of imaginary data. The experimental results on the Atari 100k benchmark demonstrate that BSRID significantly improves sample efficiency and achieves state-of-the-art performance. This work provides a robust and efficient solution for DRL applications in scenarios requiring high sample efficiency and reliable decision making. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wwwqqyy/BSRID</uri>.