| 摘要: |
| 深度神经网络目前已被广泛应用于自动驾驶、医疗诊断、语音识别、人脸识别等安全攸关领域, 因此深度神经网络测试对于保证其质量非常关键. 然而, 为判断DNN模型预测是否正确而对测试用例进行标注的成本很高. 因此, 筛选出能够揭示DNN模型错误行为的测试用例并优先对其进行标注, 能够尽快修复模型缺陷, 从而提升DNN测试的效率、保证DNN模型质量. 提出一种基于数据变异的测试用例选择方法DMS. 该方法设计并实现数据变异算子生成变异模型, 以模拟模型缺陷并捕获测试用例揭错时的动态模式, 从而评估测试用例的揭错能力. 在25个深度学习测试集和模型的组合上进行实验, 结果表明, 无论是筛选出的样本中揭错用例的比例还是揭错方向的多样性, DMS都要显著优于现有的测试用例选择方法. 具体来说, 以原始测试集作为候选集时, 在选择10%的测试用例时, DMS能够筛选出候选集中53.85%–99.22%的揭错用例, 在选择5%的测试用例时, DMS筛选出的测试用例已经几乎能覆盖所有的揭错方向. 相较于8种对比方法, DMS平均多找出12.38%–71.81%的揭错用例, 证明了DMS在测试用例选择任务中的显著有效性. |
| 关键词: 深度学习 软件测试 测试用例选择 数据变异 |
| DOI：10.13328/j.cnki.jos.007005 |
| 分类号:TP311 |
| 基金项目:国家自然科学基金(62002256) |
|
| Test Case Selection for Neural Network via Data Mutation |
|
CAO Xue-Jie1, CHEN Jun-Jie1, YAN Ming1, YOU Han-Mo1, WU Zhuo2, WANG Zan1,2
|
|
1.College of Intelligence and Computing, Tianjin University, Tianjin 300350, China;2.School of New Media and Communication, Tianjin University, Tianjin 300350, China
|
| Abstract: |
| Nowadays, deep neural network (DNN) is widely used in autonomous driving, medical diagnosis, speech recognition, face recognition, and other safety-critical fields. Therefore, DNN testing is critical to ensure the quality of DNN. However, labeling test cases to judge whether the DNN model predictions are correct is costly. Therefore, selecting test cases that reveal incorrect behavior of DNN models and labeling them earlier can help developers debug DNN models as soon as possible, thus improving the efficiency of DNN testing and ensuring the quality of DNN models. This study proposes a test case selection method based on data mutation, namely DMS. In this method, a data mutation operator is designed and implemented to generate a mutation model to simulate model defects and capture the dynamic pattern of test case bug-revealing, so as to evaluate the ability of test case bug-revealing. Experiments are conducted on the combination of 25 deep learning test sets and models. The results show that DMS is significantly better than the existing test case selection methods in terms of both the proportion of bug-revealing and the diversity of bug-revealing directions in the selected samples. Specifically, taking the original test set as the candidate set, DMS can filter out 53.85%–99.22% of all bug-revealing test cases when selecting 10% of the test cases. Moreover, when 5% of the test cases are selected, the selected cases by DMS can cover almost all bug-revealing directions. Compared with the eight comparison methods, DMS finds 12.38%–71.81% more bug-revealing cases on average, which proves the significant effectiveness of DMS in the task of test case selection. |
| Key words: deep learning software testing test case selection data mutation |
