Abstract:In the era of artificial intelligence, efficiently completing the pre-training of large language models to meet requirements for scalability, performance, and stability presents a critical challenge. These systems leverage accelerators and high-speed network interfaces to execute parallel tensor computations and communications, significantly enhancing training efficiency. However, these advancements bring a series of unresolved system design challenges. Based on an analysis of the pre-training process, this study first outlines the training procedures and workload characteristics of large language models. It then reviews system technologies from the perspectives of scalability, performance, and reliability, covering their classifications, underlying principles, current research progress, and key challenges. Finally, this study provides an in-depth analysis of the broader challenges facing large language model pre-training systems and discusses potential directions for future development.

Abstract:In distributed system environments, ensuring high availability of databases poses multiple challenges, including network latency, node failures, and the maintenance of data consistency. Addressing these challenges requires not only advanced technical solutions but also flexible architectural design and refined management strategies. High availability plays a crucial role in maintaining data integrity and consistency, as well as in improving system performance and enhancing fault tolerance. This study provides a comprehensive review of the current challenges and issues associated with high availability in distributed databases. Important concepts, theoretical foundations, and technical approaches are examined, and the current state of research is analyzed across three levels: system and network, data and computing, and application and service. The study aims to deepen the understanding of the difficulties to be addressed and the existing solutions while offering recommendations for future research and technological advancements in the field.

Abstract:Root cause analysis refers to identifying the underlying factors that lead to abnormal failures in complex systems. Causal-based backward reasoning methods, founded on structural causal models, are among the optimal approaches for implementing root cause analysis. Most current causality-driven root cause analysis methods require the prior discovery of the causal structure from data as a prerequisite, making the effectiveness of the analysis heavily dependent on the success of this causal discovery task. Recently, score function-based intervention identification has gained significant attention. By comparing the variance of score function derivatives before and after interventions, this approach detects the set of intervened variables, showing potential to overcome the constraints of causal discovery in root cause analysis. However, mainstream score function-based intervention identification is often limited by the score function estimation step. The analytical solutions used in existing methods struggle to effectively model the real distribution of high-dimensional complex data. In light of recent advances in data generation, this study proposes a diffusion model-guided root cause analysis strategy. Specifically, the proposed method first estimates the score functions corresponding to data distributions before and after the anomaly using diffusion models. It then identifies the set of root cause variables by observing the variance of the first-order derivatives of the overall score function after weighted fusion. Furthermore, to solve the issue of computational overhead raised by the pruning operation, an acceleration strategy is proposed to estimate the score function from the initially trained diffusion model, avoiding the re-training cost of the diffusion model after each pruning operation. Experimental results on simulated and real-world datasets demonstrate that the proposed method accurately identifies the set of root cause variables. Furthermore, ablation studies show that the guidance provided by the diffusion model is critical to the improved performance.

Abstract:GitHub is one of the most popular open-source project management platforms. Due to the need for team collaboration, GitHub introduced an issue tracking function to facilitate project users in submitting and tracking problems or new feature requests. When resolving issues, contributors of open-source projects typically need to execute failure reproducing test cases to reproduce the problems mentioned in the issue and verify whether the issue has been resolved. However, empirical research conducted on the SWE-bench Lite dataset reveals that nearly 90% of issues are submitted without failure reproducing test cases, leading contributors to write additional failure reproducing test cases when resolving the issues, bringing additional work burden. Existing failure reproducing test case generation methods usually rely on stack trace information, but GitHub issues do not explicitly require such information. Therefore, this study proposes a failure reproducing test case generation method based on a large language model, aimed at automatically generating failure reproducing test cases for GitHub issues, assisting issue contributors in reproducing, understanding, and verifying issues, and improving the efficiency of issue resolution. This method first retrieves diverse code context information related to the issue, including error root functions, import statements, and test case examples, then constructs precise prompts to guide the large language model in generating effective failure reproducing test cases. This study conducts comparative and ablation experiments to verify the effectiveness of this method in generating failure reproducing test cases for GitHub issues.

Abstract:In privacy-preserving inference using convolutional neural network (CNN) models, previous research has employed methods such as homomorphic encryption and secure multi-party computation to protect client data privacy. However, these methods typically suffer from excessive prediction time overhead. To address this issue, an efficient privacy-preserving CNN prediction scheme is proposed. This scheme exploits the different computational characteristics of the linear and non-linear layers in CNNs and designs a matrix decomposition computation protocol and a parameterized quadratic polynomial approximation for the ReLU activation function. This enables efficient and secure computation of both the linear and non-linear layers, while mitigating the prediction accuracy loss caused by the approximations. The computations in both the linear and non-linear layers can be performed using lightweight cryptographic primitives, such as secret sharing. Theoretical analysis and experimental results show that, while ensuring security, the proposed scheme improves prediction speed by a factor of 2 to 15, with only about a 2% loss in prediction accuracy.

Abstract:The black-box vulnerability scanner is an essential tool for Web application vulnerability detection, capable of identifying potential security threats effectively before a Web application is launched, thus enhancing the overall security of the application. However, most current black-box scanners primarily collect the attack surface through user operation simulation and regular expression matching. The simulation of user operations is vulnerable to interception by input validation mechanisms and struggles with handling complex event operations, while regular expression matching is ineffective in processing dynamic content. As a result, the scanner cannot effectively address hidden attack surfaces within JavaScript code or dynamically generated attack surfaces, leading to suboptimal vulnerability detection in some Web applications. To resolve these issues, this study proposes a JavaScript Exposure Scanner (JSEScan), a vulnerability scanner enhancement framework based on JavaScript code analysis. The framework integrates static and dynamic code analysis techniques, bypassing form validation and event-triggering restrictions. By extracting attack surface features from JavaScript code, JSEScan identifies attack surfaces and synchronizes them across multiple scanners, enhancing their vulnerability detection capabilities. The experimental results demonstrate that JSEScan increases coverage by 81.02% to 242.15% compared to using a single scanner and uncovers an additional 239 security vulnerabilities when compared to multiple scanners working concurrently, showing superior attack surface collection and vulnerability detection capabilities.

Abstract:As the foundation of AI, deep learning frameworks play a vital role in driving the rapid progress of AI technologies. However, due to the lack of unified standards, compatibility across different frameworks remains limited. Faithful model transformation enhances interoperability by converting a source model into an equivalent model in the target framework. However, the large number and diversity of deep learning frameworks, combined with the increasing demand for custom frameworks, lead to high conversion costs. To address this issue, this study proposes an automatic AI source code migration method between frameworks based on a domain knowledge graph. The method integrates domain knowledge graphs and abstract syntax trees to systematically manage migration challenges. First, the source code is transformed into a framework-specific abstract syntax tree, from which general dependency information and operator-specific details are extracted. By applying the operator and parameter mappings stored in the domain knowledge graph, the code is migrated to the target framework, generating equivalent target model code while significantly reducing engineering complexity. Compared with existing code migration tools, the proposed method supports mutual migration among widely used deep learning frameworks, such as PyTorch, PaddlePaddle, and MindSpore. The approach has proven to be both mature and reliable, with part of its implementation open-sourced in Baidu’s official migration tool, PaConvert.

Abstract:With the rapid development of merchant review websites, the volume of content on these websites has increased significantly, making it challenging for users to quickly find valuable reviews. This study introduces a new task, “multimodal customized review generation”. The task aims to generate customized reviews for specific users about products they have not yet reviewed, thus providing valuable insights into these products. To achieve this goal, this study explores a multimodal review generation framework based on a pre-trained language model. Specifically, a multimodal pre-trained language model is employed, which takes product images and user preferences as inputs. The visual and textual features are then fused to generate customized reviews. Experimental results demonstrate that the proposed model is effective in generating high-quality customized reviews.

Abstract:As core programmable components of blockchain, smart contracts are responsible for asset management and the execution of complex business logic, forming the foundation of decentralized finance (DeFi) protocols. However, with the rapid advancement of blockchain technology, security issues related to smart contracts and DeFi protocols have become increasingly prominent, attracting numerous attackers seeking to exploit vulnerabilities for illicit gains. In recent years, several major security incidents involving smart contracts and DeFi protocols have highlighted the importance of vulnerability detection research, making it a critical area for security defense. This study systematically reviews existing literature and proposes a comprehensive framework for research on vulnerability detection in smart contracts and DeFi protocols. Specifically, vulnerabilities and detection techniques are categorized and analyzed for both domains. For smart contracts, the study focuses on the application of large language models (LLM) as primary detection engines and their integration with traditional methods. For DeFi protocols, it categorizes and details various protocol-level vulnerabilities and their detection methods, analyzing the strengths and limitations of detection strategies before and after attacks, addressing gaps in existing reviews on DeFi vulnerability detection. Finally, this study summarizes the challenges faced by current detection approaches and outlines future research directions, aiming to provide new insights and theoretical support for the security detection of smart contracts and DeFi protocols.

Abstract:Contrastive learning is a self-supervised learning technique widely used in various fields such as computer vision and natural language processing. Graph contrastive learning (GCL) refers to methods that apply contrastive learning techniques to graph data. A review is presented on the basic concepts, methods, and applications of graph contrastive learning. First, the background and significance of GCL, as well as its basic concepts on graph data, are introduced. Then, the mainstream GCL methods are elaborated in detail, including methods with different graph data augmentation strategies, methods with different graph neural network (GNN) encoder structures, and methods with different contrastive loss objectives. Finally, three research ideas for GCL are proposed. Research findings demonstrate that graph contrastive learning is an effective approach for addressing various downstream tasks, including node classification and graph classification.