| 摘要: |
| 在人工智能时代, 如何高效地完成大语言模型的预训练, 以满足其在扩展性、性能与稳定性方面的需求, 是亟需解决的重要问题. 大语言模型系统充分利用加速器和高速网卡进行并行张量计算和通信, 极大地提高了模型训练的性能, 这一进展伴随着一系列尚待解决的系统设计问题. 首先, 在分析大语言模型预训练过程的基础上, 介绍了其训练流程与负载特点. 其次, 从预训练系统的扩展性、性能和可靠性角度出发, 分别介绍了各类系统技术的分类、原理、研究现状及热点问题. 最后, 从总体层面深入分析了大型语言预训练系统面临的挑战, 并展望了其未来的发展前景. |
| 关键词: 人工智能 大语言模型 大语言模型预训练系统 |
| DOI：10.13328/j.cnki.jos.007438 |
| 分类号:TP18 |
| 基金项目:国家自然科学基金(62272466, U24A20233); 中国人民大学国家治理大数据和人工智能创新平台 |
|
| Survey on Key Technologies for Large Language Model Pre-training Systems |
|
GAO Yan-Jie1, CHEN Yue-Guo2
|
|
1.School of Information, Renmin University of China, Beijing 100872, China;2.Key Laboratory of Data Engineering and Knowledge Engineering (Renmin University of China), Beijing 100872, China
|
| Abstract: |
| In the era of artificial intelligence, efficiently completing the pre-training of large language models to meet requirements for scalability, performance, and stability presents a critical challenge. These systems leverage accelerators and high-speed network interfaces to execute parallel tensor computations and communications, significantly enhancing training efficiency. However, these advancements bring a series of unresolved system design challenges. Based on an analysis of the pre-training process, this study first outlines the training procedures and workload characteristics of large language models. It then reviews system technologies from the perspectives of scalability, performance, and reliability, covering their classifications, underlying principles, current research progress, and key challenges. Finally, this study provides an in-depth analysis of the broader challenges facing large language model pre-training systems and discusses potential directions for future development. |
| Key words: artificial intelligence large language model (LLM) large language model pre-training system |
