Journal of Software:2018.29(7):2071-2091

(同济大学 电子信息与工程学院, 上海 201804)
Survey of Unstable Gradients in Deep Neural Network Training
CHEN Jian-Ting,XIANG Yang
(College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China)
Chart / table
Similar Articles
Article :Browse 2133   Download 2055
Received:September 27, 2017    Revised:November 10, 2017
> 中文摘要: 深度神经网络作为机器学习领域的热门研究方向,在训练中容易出现梯度不稳定现象,是制约其发展的重要因素,控制和避免深度神经网络的梯度不稳定现象是深度神经网络的重要研究内容.分析了梯度不稳定现象的成因和影响,并综述了目前解决梯度不稳定现象的关键技术和主要方法.最后展望了梯度不稳定现象的未来研究方向.
Abstract:As a popular research direction in the field of machine learning, deep neural networks are prone to the phenomenon of unstable gradients in training, which has become an important element that restricts their development. How to avoid and control unstable gradients is an important research topic of deep neural networks. This paper analyzes the cause and effect of the unstable gradients, and reviews the main models and methods of solving the unstable gradients. Furthermore, the future research trends in the unstable gradients is discussed.
文章编号:     中图分类号:    文献标志码:
基金项目:国家重点基础研究发展计划(973)(2014CB340404);国家自然科学基金(71571136);上海市科委基础研究项目(16JC403000) 国家重点基础研究发展计划(973)(2014CB340404);国家自然科学基金(71571136);上海市科委基础研究项目(16JC403000)
Foundation items:National Basic Research Program of China (973) (2014CB340404); National Natural Science Foundation of China (71571136); Project of Science and Technology Commission of Shanghai Municipality (16JC403000)
Reference text:


CHEN Jian-Ting,XIANG Yang.Survey of Unstable Gradients in Deep Neural Network Training.Journal of Software,2018,29(7):2071-2091