| 摘要: |
| 变异测试是一种有效的软件测试技术, 通过生成变异体来模拟软件缺陷, 帮助提升现有测试套件的缺陷检测能力. 变异体的质量对于变异测试的有效性具有显著影响. 传统的变异测试方法通常采用人工设计的基于语法规则的变异算子生成变异体, 并已取得一定的研究成果. 近年来, 许多研究开始结合深度学习技术, 通过学习开源项目历史代码生成变异体. 目前, 该新方法在变异体生成方面取得了初步的成果. 基于语法规则和基于学习的两种变异技术, 其机理不同, 但其目标均是通过生成变异体来提高测试套件的缺陷检测能力, 因此, 全面比较这两种变异技术对于变异测试及其下游任务至关重要. 针对这一问题, 设计实现了一项针对基于语法规则和基于学习的变异技术的实证研究, 旨在了解不同机理的变异技术在变异测试任务上的性能以及生成的变异体在程序语义上的差异性. 具体地, 以Defect4J v1.2.0数据集为实验对象, 比较以MAJOR和PIT为代表的基于语法规则的变异技术和以DeepMutation、μBERT和LEAM为代表的基于深度学习的变异技术. 实验结果表明: 基于规则与学习的变异技术均可有效支持变异测试实践, 但MAJOR的测试效果最优, 能够检测85.4%的真实缺陷. 在语义表示上, MAJOR具有最强的语义代表能力, 基于其构造的测试套件能够杀死其余变异技术生成的超过95%占比的变异体. 在缺陷表征上, 两类技术均具有独特性. |
| 关键词: 变异测试 变异分析 实证研究 缺陷检测 |
| DOI：10.13328/j.cnki.jos.007113 |
| 分类号: |
| 基金项目:国家重点研发计划(2022YFB4501902);国家自然科学基金(62232001) |
|
| Comparison Research on Rule-based and Learning-based Mutation Techniques |
|
GONG Zhi-Hao1,2, CHEN Yi-Zhou1,2, CHEN Jun-Jie3, HAO Dan1,2
|
|
1.Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, Beijing 100871, China;2.School of Computer Science, Peking University, Beijing 100871, China;3.College of Intelligence and Computing, Tianjin University, Tianjin 300350, China
|
| Abstract: |
| Mutation testing is an effective software testing technique. It helps improve the defect detection capability of an existing test suite by generating mutants that simulate software defects. The quality of mutants has a significant impact on the effectiveness of mutation testing. The traditional mutation testing approach typically employs manually designed syntactic rule-based mutation operators to generate mutants, and has achieved some academic success. In recent years, many studies have started to incorporate deep learning techniques to generate mutants by learning historical code from open source projects. This new approach has achieved preliminary progress in mutant generation. A comprehensive comparison of the two mutation techniques, i.e. rule-based and learning-based, which have different mechanisms but both aim to improve the defect detection capability of the test suite by mutation, is crucial for mutation testing and its downstream tasks. To handle the problem, this study designs and implements an empirical study of rule-based and learning-based mutation techniques, aiming to understand the performance of mutation techniques with different mechanisms on the task of mutation testing, as well as the variability of the generated mutants in terms of program semantics. Specifically, this study uses the Defect4J v1.2.0 dataset to compare the syntactic rule-based mutation techniques represented by MAJOR and PIT with the deep learning-based mutation techniques represented by DeepMutation, μBERT, and LEAM. The experimental results show that both rule-based and learning-based mutation techniques can effectively support mutation testing practices, but MAJOR has the best testing performance and is able to detect 85.4% of real defects. In terms of semantic representation, MAJOR has the strongest semantic representation capability, and its constructed test suite is able to kill more than 95% of the mutants generated by other mutation techniques. In terms of defect representation, both types of techniques are unique. |
| Key words: mutation testing mutation analysis empirical study defect detection |
