面向大规模在线系统的故障根因变更识别
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

TP311

基金项目:

国家重点研发计划(2024YFB4505904); 国家自然科学基金(62272495); 广东省基础与应用基础研究基金(2023B1515020054)


Root Cause Change Identification for Failures in Large-scale Online System
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    在大规模在线服务系统中, 为了适应快速变化的用户需求和信息技术如连续集成/交付等, 软件变更频繁发生且呈现上升趋势. 尽管工程师会在软件变更上线之前对新版本进行严格的测试, 但由于测试环境与生产环境之间在负载、规模、用户等方面存在诸多差异, 导致部分隐蔽缺陷未能被及时发现, 随新版本发布带入生产环境, 对系统的可用性和稳定性造成影响. 为了更深入地了解缺陷变更在部署到生产环境后的影响和行为, 基于来自全球大规模即时通信系统微信的真实变更故障数据进行了实证分析, 并得出5个关于缺陷变更的关键发现. 基于实证研究的发现和结论, 提出一种轻量级故障根因变更识别方法. 该方法旨在自动化地识别导致变更故障的根因变更, 从而帮助运维工程师完成根因定位和故障修复工作. 为了验证提出的故障根因变更识别方法的有效性, 在微信的生产环境中采集了包含多种类型缺陷变更的真实数据集, 同时还构建一个微服务基准测试系统的模拟变更数据集, 然后对提出的方法进行系统性评估. 实验结果表明, 所提方法在微信生产环境数据集和模拟变更数据上的故障根因变更Top-3命中率分别达到80%和84%, 并且故障根因变更识别效果显著优于当前最先进的缺陷变更检测方法. 此外, 从工程实践角度, 系统在处理典型规模故障时内存占用仅为2.3 GB, 平均分析时延28.6 s, 满足实际生产环境需求.

    Abstract:

    In large-scale online service systems, software changes occur frequently and are on the rise due to the need to adapt to rapidly changing user demands and information technologies, such as continuous integration and delivery. Although engineers rigorously test new software versions before deployment, some defects may go unnoticed during testing and end up being deployed to the production environment. This primarily occurs because significant differences exist between the testing and production environments in terms of load, scale, and user characteristics. As a result, these defects can impact the system’s availability and stability. To better understand the impact and behavior of defective changes after deployment to the production environment, this study conducts an empirical analysis using real change failure data from WeChat, a large-scale global instant messaging system. Five key findings related to defective changes are derived from this analysis. Based on these empirical findings and conclusions, this study proposes a lightweight root cause change identification method. This method aims to automatically identify the root cause changes that lead to failure, assisting operations and maintenance engineers in root cause localization and trouble shooting efforts. To validate the effectiveness of the proposed method, a real dataset containing various types of defective changes from WeChat’s production environment is collected, along with a simulated change dataset based on a microservice benchmark system. A systematic evaluation of the proposed method is then conducted. The experimental results show that the proposed method achieves Top-3 root cause change hit rates of 80% and 84% for the WeChat production environment dataset and simulated change data, respectively, significantly outperforming the state-of-the-art defective change detection methods. Moreover, from an engineering practice perspective, the system uses only 2.3 GB of memory and has an average analysis latency of 28.6 s when processing typical-scale failures, thus meeting the requirements of actual production environments.

    参考文献
    相似文献
    引证文献
引用本文

余广坝,陈鹏飞,唐锡涛,郑子彬.面向大规模在线系统的故障根因变更识别.软件学报,2026,37(2):641-661

复制
相关视频

分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2024-06-07
  • 最后修改日期:2025-03-25
  • 录用日期:
  • 在线发布日期: 2025-12-03
  • 出版日期: 2026-02-06
文章二维码
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号