| 引用本文: | 曲慕子,亢良伊,刘杰,王帅,叶丹,黄涛.CodeLLMTuner: 基于样本重用的代码大模型选择与解码参数调优框架.软件学报,2026,37(5):2131-2150 |
| |
|
| |
|
|
| 本文已被：浏览 911次 下载 1054次 |
码上扫一扫！ |
|
|
| CodeLLMTuner: 基于样本重用的代码大模型选择与解码参数调优框架 |
|
曲慕子1,2, 亢良伊1,2, 刘杰1,2,3,4,5, 王帅1,2, 叶丹1,2,3,4, 黄涛1,2,3,4
|
|
1.中国科学院 软件研究所, 北京 100190;2.中国科学院大学, 北京 100049;3.基础软件与系统重点实验室(中国科学院 软件研究所), 北京 100190;4.计算机科学国家重点实验室 (中国科学院 软件研究所), 北京 100190;5.中国科学院大学南京学院, 江苏 南京 211135
|
|
| 摘要: |
| 随着大语言模型(large language model, LLM)技术的迅速发展, 涌现了众多代码大模型(Code LLM), 以支持代码生成、代码补全、代码测试和代码重构等任务. 不同模型在处理相同任务时可能表现出显著的性能差异, 且推理阶段的解码参数也会对模型性能产生重要影响. 研究如何为特定代码开发任务高效地选择最佳模型及其最优解码参数. 现有方法通常将模型选择和参数调优分为两个独立阶段, 由于不同阶段的采样策略差异导致无法共享样本数据, 采样与评估计算成本较高. 考虑到不同代码大模型解码参数空间相同, 提出利用倾向评分匹配(propensity score matching, PSM)算法加权调整和对齐不同分布的样本数据, 以提高样本数据复用效率、降低计算成本. 由此提出了一个基于样本重用的代码大模型选择与解码参数调优框架CodeLLMTuner. 该框架包含3个阶段: (1)独立采样阶段, 对多个代码大模型并行执行解码参数调优(如贝叶斯优化)并进行数据采样与评估以收集样本数据; (2)模型选择阶段, 利用PSM技术对齐不同模型的样本数据, 从中选出性能期望最优的模型; (3)获选模型的解码参数调优阶段, 复用获选模型的样本数据, 并在其基础上继续进行解码参数调优, 以全面探索性能空间并显著降低采样成本. 实验结果表明, 在代码生成、代码摘要和测试用例生成这3项任务上, CodeLLMTuner相比于基线方法在相同成本下性能提升10%–15%, 或在达到相同性能下成本降低超过20%. |
| 关键词: 代码大模型 自动代码生成 模型选择 解码参数调优 倾向评分匹配 |
| DOI：10.13328/j.cnki.jos.007508 |
| 分类号:TP311 |
| 基金项目:中国科学院软件研究所基础研究项目(ISCAS-JCMS-202405) |
|
| CodeLLMTuner: Code LLM Selection and Decoding Parameter Tuning Framework Based on Sample Reusing |
|
QU Mu-Zi1,2, KANG Liang-Yi1,2, LIU Jie1,2,3,4,5, WANG Shuai1,2, YE Dan1,2,3,4, HUANG Tao1,2,3,4
|
|
1.Institute of Software, Chinese Academy of Sciences, Beijing 100190, China;2.University of Chinese Academy of Sciences, Beijing 100049, China;3.Key Laboratory of Systems Software (Institute of Software, Chinese Academy of Sciences), Beijing 100190, China;4.State Key Laboratory of Computer Science (Institute of Software, Chinese Academy of Sciences), Beijing 100190, China;5.University of Chinese Academy of Sciences, Nanjing, Nanjing 211135, China
|
| Abstract: |
| With the rapid development of large language model (LLM) technology, many Code LLMs have emerged to support tasks such as code generation, code completion, code testing, and code refactoring. Different models may show significant performance differences when processing the same task, and the decoding parameters at the inference stage will also have an important influence on model performance. This study investigates how to efficiently select the best model and its optimal decoding parameters for a specific code development task. Existing methods generally divide model selection and parameter tuning into two independent stages. Due to the differences in sampling strategies at different stages, sample data cannot be shared, and the computational cost of sampling and evaluation is high. Considering that the decoding parameter space of different Code LLMs is the same, this study proposes the utilization of the propensity score matching (PSM) algorithm for conducting weighted adjustment and aligning sample data of different distributions to improve the reuse efficiency of sample data and reduce computational costs. Therefore, this study proposes a framework CodeLLMTuner for Code LLM selection and decoding parameter tuning based on sample reuse. The framework includes three stages, including the independent sampling stage, which performs decoding parameter tuning (such as Bayesian optimization) on multiple Code LLMs in parallel and conducts data sampling and evaluation to collect sample data. Additionally, the model selection stage adopts PSM technology to align the sample data of different models and selects the model with the optimal performance expectations. The decoding parameter tuning stage of the selected model reuses the sample data of the selected model and continues decoding parameter tuning based on it to fully explore the performance space and significantly reduce sampling costs. Experimental results show that in the three tasks of code generation, code summarization and test case generation, CodeLLMTuner improves performance by 10% to 15% at the same cost compared to the baseline methods, or reduces the cost by more than 20% under the same performance. |
| Key words: Code LLM automatic code generation model selection decoding parameter tuning propensity score matching (PSM) |
|
|
|
|
