| 摘要: |
| 深度神经网络作为机器学习领域的热门研究方向，在训练中容易出现梯度不稳定现象，是制约其发展的重要因素，控制和避免深度神经网络的梯度不稳定现象是深度神经网络的重要研究内容.分析了梯度不稳定现象的成因和影响，并综述了目前解决梯度不稳定现象的关键技术和主要方法.最后展望了梯度不稳定现象的未来研究方向. |
| 关键词: 深度神经网络 梯度不稳定现象 梯度衰减 梯度爆炸 |
| DOI：10.13328/j.cnki.jos.005561 |
| 分类号: |
| 基金项目:国家重点基础研究发展计划（973）（2014CB340404）；国家自然科学基金（71571136）；上海市科委基础研究项目（16JC403000） |
|
| Survey of Unstable Gradients in Deep Neural Network Training |
|
CHEN Jian-Ting, XIANG Yang
|
|
College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China
|
| Abstract: |
| As a popular research direction in the field of machine learning, deep neural networks are prone to the phenomenon of unstable gradients in training, which has become an important element that restricts their development. How to avoid and control unstable gradients is an important research topic of deep neural networks. This paper analyzes the cause and effect of the unstable gradients, and reviews the main models and methods of solving the unstable gradients. Furthermore, the future research trends in the unstable gradients is discussed. |
| Key words: deep neural network unstable gradient vanishing gradient exploding gradient |
