Performance regression testing and run-time verification of components in robotics systems

Abstract

Unintended changes in the utilization of resources like CPU and memory can lead to severe problems for the operation of robotics and intelligent systems. Nevertheless, systematic testing for such performance regressions has largely been ignored in this domain. We present a method to specify and execute performance tests for individual components of component-based robotics systems based on their middleware interfaces. The method includes an automatic analysis of each component revision against previous ones that reports potential changes to the resource utilization characteristics. This informs developers about the impact of their changes. We describe the design of the framework and present evaluation results for the automatic detection of performance changes based on tests for a variety of robotics components. Additionally, we demonstrate how performance tests can be used as a basis for learning resource utilization models of components. These models can be used to detect faults at system run-time, which provides an additional level of safety for the systems besides the offline testing.