| 摘要: |
| GitHub是目前最流行的开源项目管理平台之一. 由于团队协作的需要, GitHub引入了问题报告跟踪功能以方便项目使用者提交和追踪项目中出现的问题或新功能请求. 问题报告贡献者在解决问题报告时, 通常需要执行故障复现测试用例来复现问题报告中提到的问题并验证问题报告是否解决. 然而, 在SWE-bench Lite数据集上进行实证研究发现, 有近90%的问题报告在用户提交时没有附带故障复现测试用例, 这导致问题报告贡献者在解决问题报告时还需额外编写故障复现测试用例, 带来了额外的工作负担. 现有的故障复现测试用例生成技术通常依赖错误栈信息, 然而GitHub问题报告中并未明确要求有这类信息. 因此, 提出基于大语言模型的故障复现测试用例生成方法, 旨在自动化地为GitHub问题报告生成故障复现测试用例, 帮助问题报告贡献者复现、理解并验证问题报告, 提升问题报告解决效率. 该方法首先通过检索与问题报告相关的多样化代码上下文信息, 包括报错根函数、import语句和测试用例样本, 随后构建精确的prompt, 以引导大语言模型生成有效的故障复现测试用例. 开展对比实验和消融实验, 验证所提方法在面向GitHub问题报告的故障复现测试用例生成任务上的有效性. |
| 关键词: 故障复现 测试用例生成 大语言模型 检索增强生成 |
| DOI：10.13328/j.cnki.jos.007474 |
| 分类号: |
| 基金项目: |
|
| Failure Reproducing Test Case Generation Method Based on Large Language Model |
|
WANG Ying, ZI Qian-Cheng, PENG Xin, LOU Yi-Ling
|
|
School of Computer Science, Fudan University, Shanghai 200438, China
|
| Abstract: |
| GitHub is one of the most popular open-source project management platforms. Due to the need for team collaboration, GitHub introduced an issue tracking function to facilitate project users in submitting and tracking problems or new feature requests. When resolving issues, contributors of open-source projects typically need to execute failure reproducing test cases to reproduce the problems mentioned in the issue and verify whether the issue has been resolved. However, empirical research conducted on the SWE-bench Lite dataset reveals that nearly 90% of issues are submitted without failure reproducing test cases, leading contributors to write additional failure reproducing test cases when resolving the issues, bringing additional work burden. Existing failure reproducing test case generation methods usually rely on stack trace information, but GitHub issues do not explicitly require such information. Therefore, this study proposes a failure reproducing test case generation method based on a large language model, aimed at automatically generating failure reproducing test cases for GitHub issues, assisting issue contributors in reproducing, understanding, and verifying issues, and improving the efficiency of issue resolution. This method first retrieves diverse code context information related to the issue, including error root functions, import statements, and test case examples, then constructs precise prompts to guide the large language model in generating effective failure reproducing test cases. This study conducts comparative and ablation experiments to verify the effectiveness of this method in generating failure reproducing test cases for GitHub issues. |
| Key words: failure reproducing test case generation large language model (LLM) retrieval-augmented generation (RAG) |
