On-policy and Off-policy Q-learning algorithms with policy iteration for two-wheeled inverted pendulum systems

Abstract

This article delves into the investigation of On-policy and Off-policy Q-learning algorithms for controlling two-wheeled inverted pendulum (TWIP) robots in situations where knowledge about the dynamic system is uncertain. Both on-policy and off-policy Q-learning algorithms ensure optimal and model-free control by employing a data collection approach without the knowledge of model. The On-policy algorithm performs real-time data collection, continuously gathering data and iteratively calculating a new control policy until it converges to the optimal value. In contrast, the Off-policy algorithm collects data only once and applies it to the system after completing the learning process. To enhance computational efficiency and minimize the amount of data required, the TWIP system is divided into two Sub-systems. These Sub-systems consist of smaller system matrices that can be controlled independently. This division reduces the data collection burden and accelerates the calculation speed of the algorithms. The utilization of Off-policy techniques proves to be advantageous in developing algorithms with data efficiency and achieving higher accuracy. The influence of probing noise on the Q-function is comprehensively considered in both proposed algorithms. By utilizing a single data set and eliminating the influence of noise, the Off-policy techniques enhance algorithm performance. Finally, the article presents simulation results of the TWIP system to validate the effectiveness of the two proposed control schemes.