ISTA: Automatic Test Case Generation and Optimization for Intelligent Systems based on Coverage Analysis

Abstract

With the applications of intelligent systems in areas (such as self-driving cars, robotics, and smart cities), the impact of these intelligent systems’ defects cannot be ignored. For example, in a recent report, the self-driving car collided with another self-driving car because it incorrectly identified a roadblock. Therefore, it is necessary to conduct adequate testing of intelligent systems to avoid dangerous behaviors as much as possible. However, due to the particularity of its own structure, the low efficiency, and the high cost of manual collection the large-scale test cases, it is important and challenging to design tools to test the adequacy of intelligent systems.To overcome the above problems, we propose an intelligent system test adequacy evaluation tool ISTA. ISTA implements the automatic generation and optimization of test cases based on coverage analysis, which can improve the test adequacy of the intelligent system while expanding the dataset. To evaluate the usefulness of our developed tool, we analyze the application of ISTA on the five-layer fully-connected dnn model and german credit dataset (text data type) for binary classification as well as on the Rambo model and hmb dataset (image data type) for self-driving car. The evaluation results show that the test dataset is expanded and the models are more fully tested after ISTA’s test case generation and optimization for both text and image data types, with a corresponding increase in the average 80% coverage criteria used.