Improving Robotics Competitions for Real-World Evaluation of AI

Abstract

While embodied robotic applications have been a strong influence on moving artificial intelligence toward fo-cussing on broad, robust solutions that operate in the real world, evaluating such systems remains dif-ficult. Competition-based evaluation, using common challenge problems, is one of the major methods for comparing AI systems employing robotic embodiment. Competitions unfortunately tend to influence the cre-ation of specific solutions that exploit particular rules rather than the broad and robust techniques that are hoped for, however, and physical embodiment in the real world also creates difficulties in control and re-peatability. In this paper we discuss the positive and negative influences of competitions as a means of eval-uating AI systems, and present recent work designed to improve such evaluations. We describe how improved control and repeatability can be achieved with mixed re-ality applications for challenge problems, and how com-petitions themselves can encourage breadth and robust-ness, using our rules for the FIRA HuroCup as an ex-ample.