Abstract:In large-scale online service systems, software changes occur frequently and are on the rise due to the need to adapt to rapidly changing user demands and information technologies, such as continuous integration and delivery. Although engineers rigorously test new software versions before deployment, some defects may go unnoticed during testing and end up being deployed to the production environment. This primarily occurs because significant differences exist between the testing and production environments in terms of load, scale, and user characteristics. As a result, these defects can impact the system’s availability and stability. To better understand the impact and behavior of defective changes after deployment to the production environment, this study conducts an empirical analysis using real change failure data from WeChat, a large-scale global instant messaging system. Five key findings related to defective changes are derived from this analysis. Based on these empirical findings and conclusions, this study proposes a lightweight root cause change identification method. This method aims to automatically identify the root cause changes that lead to failure, assisting operations and maintenance engineers in root cause localization and trouble shooting efforts. To validate the effectiveness of the proposed method, a real dataset containing various types of defective changes from WeChat’s production environment is collected, along with a simulated change dataset based on a microservice benchmark system. A systematic evaluation of the proposed method is then conducted. The experimental results show that the proposed method achieves Top-3 root cause change hit rates of 80% and 84% for the WeChat production environment dataset and simulated change data, respectively, significantly outperforming the state-of-the-art defective change detection methods. Moreover, from an engineering practice perspective, the system uses only 2.3 GB of memory and has an average analysis latency of 28.6 s when processing typical-scale failures, thus meeting the requirements of actual production environments.