| 引用本文: | 高赫然,吴恒,许源佳,李修和,王焘,张文博.面向深度学习训练的内存交换机制综述.软件学报,2023,34(12):5862-5886 |
| |
|
| |
|
|
| 本文已被：浏览 2337次 下载 6287次 |
码上扫一扫！ |
|
|
| 面向深度学习训练的内存交换机制综述 |
|
高赫然1,2, 吴恒2,3,4, 许源佳1,2, 李修和5, 王焘2,6, 张文博2,6,3,4
|
|
1.中国科学院大学, 北京 100049;2.中国科学院 软件研究所 软件工程技术研究开发中心, 北京 100190;3.中国科学院大学南京学院, 江苏 南京 211135;4.中科南京软件技术研究院, 江苏 南京 210000;5.国防科技大学 电子对抗学院, 安徽 合肥 230037;6.计算机科学国家重点实验室(中国科学院 软件研究所), 北京 100190
|
|
| 摘要: |
| 随着深度学习技术的快速发展和深入应用, 深度学习训练规模持续增大, 内存不足已成为影响深度学习可用性的主要瓶颈之一. 内存交换机制是应对深度学习训练内存问题的关键技术, 该机制利用深度学习训练内存需求的“时变”特征, 在专用计算加速设备内存与外部存储之间按需移动数据, 通过瞬时内存需求替代累积内存需求, 保障深度学习训练任务的运行. 对面向深度学习训练的内存交换机制进行综述, 以深度学习训练内存需求的时变特征为研究视角, 分别针对基于算子运行特征的内存换出机制、基于数据依赖关系的内存换入机制以及效能驱动的联合换出与换入决策等重要研究工作进行了总结分析, 并针对该技术领域的发展方向进行了展望. |
| 关键词: 深度学习训练 内存交换 内存需求特征 |
| DOI：10.13328/j.cnki.jos.006800 |
| 分类号: |
| 基金项目:国家重点研发计划(2018YFB1402803); 国家自然科学基金(61872344, 61972386); 山东省重大研发计划(2021CXGC010101) |
|
| Survey on Memory Swapping Mechanism for Deep Learning Training |
|
GAO He-Ran1,2, WU Heng2,3,4, XU Yuan-Jia1,2, LI Xiu-He5, WANG Tao2,6, ZHANG Wen-Bo2,6,3,4
|
|
1.University of Chinese Academy of Sciences, Beijing 100049, China;2.Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China;3.University of Chinese Academy of Sciences, Nanjing, Nanjing 211135, China;4.Nanjing Institute of Software Technology, Nanjing 210000, China;5.Institute of Electronic Countermeasures, National University of Defense Technology, Hefei 230037, China;6.State Key Laboratory of Computer Science (Institute of Software, Chinese Academy of Sciences), Beijing 100190, China
|
| Abstract: |
| With the rapid growth and further application of deep learning (DL), the scale of DL training continues to expand, and memory insufficiency has become one of the major bottlenecks threatening DL availability. Memory swapping mechanism is the key mechanism to alleviate the memory problem of DL training. This mechanism leverages the “time-varying” memory requirement of DL training and moves the data between specific computing accelerating device memory and external storage according to demands. The operation of DL training tasks can be ensured by replacing an accumulated memory requirement with an instant one. This study surveys the memory swapping mechanism for DL training from the aspect of time-varying memory requirements. Key studies of an operator feature-based memory swapping-out mechanism, a data dependency based swapping-in mechanism, and efficiency-driven joint swapping-in and swapping-out decisions are summarized. Finally, the development prospect of this technology is pointed out. |
| Key words: deep learning (DL) training memory swapping memory requirement characteristic |
|
|
|
|
