RAC-SAC: An improved Actor-Critic algorithm for Continuous Multi-task manipulation on Robot Arm Control

摘要

Controlling a robot arm in a complex environment is a challenging task. In this problem, the robot arm interacts with objects of uncertain shapes. To move an object to a required position where it can be grasped, the robot arm needs the ability to learn autonomously. It must be agile and intelligent enough to identify interaction points with the object, move it, and rotate it in a sensible direction without prior learning in complex environmental conditions.In this paper, we propose Deep Reinforcement Learning approach to tackle this problem, utilizing model-free method called Soft Actor-Critic (SAC) and Realistic Actor Critic (RAC). To enhance accuracy and expedite the learning process, we fine-tune the model by redesigning the reward function to enable the agent to learn more effectively. Additionally, we combine it with methods such as Hindsight Experience Replay (HER), Relay Hindsight Experience Replay (RHER), and Projected Conflicting Gradients (PCGrad) to increase stability in executing sequential actions over extended periods.The experimental results were simulated on a Kuka 7-degree-of-freedom robot arm. The results demonstrate that the proposed method achieves stable and faster convergence with an accuracy of up to 87-97% when pushing objects such as cubes and scissors. Comparative results also indicate that the proposed method is more effective than previous state-of-the-art methods. Additionally, we have built a framework that supports various RL algorithms and can be applied to different objects when addressing the problem.