| 摘要: |
| 在大规模在线服务系统中, 为了适应快速变化的用户需求和信息技术如连续集成/交付等, 软件变更频繁发生且呈现上升趋势. 尽管工程师会在软件变更上线之前对新版本进行严格的测试, 但由于测试环境与生产环境之间在负载、规模、用户等方面存在诸多差异, 导致部分隐蔽缺陷未能被及时发现, 随新版本发布带入生产环境, 对系统的可用性和稳定性造成影响. 为了更深入地了解缺陷变更在部署到生产环境后的影响和行为, 基于来自全球大规模即时通信系统微信的真实变更故障数据进行了实证分析, 并得出5个关于缺陷变更的关键发现. 基于实证研究的发现和结论, 提出一种轻量级故障根因变更识别方法. 该方法旨在自动化地识别导致变更故障的根因变更, 从而帮助运维工程师完成根因定位和故障修复工作. 为了验证提出的故障根因变更识别方法的有效性, 在微信的生产环境中采集了包含多种类型缺陷变更的真实数据集, 同时还构建一个微服务基准测试系统的模拟变更数据集, 然后对提出的方法进行系统性评估. 实验结果表明, 所提方法在微信生产环境数据集和模拟变更数据上的故障根因变更Top-3命中率分别达到80%和84%, 并且故障根因变更识别效果显著优于当前最先进的缺陷变更检测方法. 此外, 从工程实践角度, 系统在处理典型规模故障时内存占用仅为2.3 GB, 平均分析时延28.6 s, 满足实际生产环境需求. |
| 关键词: 软件变更 根因定位 故障 在线系统 |
| DOI：10.13328/j.cnki.jos.007482 |
| 分类号:TP311 |
| 基金项目:国家重点研发计划(2024YFB4505904); 国家自然科学基金(62272495); 广东省基础与应用基础研究基金(2023B1515020054) |
|
| Root Cause Change Identification for Failures in Large-scale Online System |
|
YU Guang-Ba1, CHEN Peng-Fei1, TANG Xi-Tao1, ZHENG Zi-Bin2
|
|
1.School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China;2.School of Software Engineering, Sun Yat-sen University, Zhuhai 519082, China
|
| Abstract: |
| In large-scale online service systems, software changes occur frequently and are on the rise due to the need to adapt to rapidly changing user demands and information technologies, such as continuous integration and delivery. Although engineers rigorously test new software versions before deployment, some defects may go unnoticed during testing and end up being deployed to the production environment. This primarily occurs because significant differences exist between the testing and production environments in terms of load, scale, and user characteristics. As a result, these defects can impact the system’s availability and stability. To better understand the impact and behavior of defective changes after deployment to the production environment, this study conducts an empirical analysis using real change failure data from WeChat, a large-scale global instant messaging system. Five key findings related to defective changes are derived from this analysis. Based on these empirical findings and conclusions, this study proposes a lightweight root cause change identification method. This method aims to automatically identify the root cause changes that lead to failure, assisting operations and maintenance engineers in root cause localization and trouble shooting efforts. To validate the effectiveness of the proposed method, a real dataset containing various types of defective changes from WeChat’s production environment is collected, along with a simulated change dataset based on a microservice benchmark system. A systematic evaluation of the proposed method is then conducted. The experimental results show that the proposed method achieves Top-3 root cause change hit rates of 80% and 84% for the WeChat production environment dataset and simulated change data, respectively, significantly outperforming the state-of-the-art defective change detection methods. Moreover, from an engineering practice perspective, the system uses only 2.3 GB of memory and has an average analysis latency of 28.6 s when processing typical-scale failures, thus meeting the requirements of actual production environments. |
| Key words: software change root cause location failure online system |
