| 摘要: |
| 深度神经网络目前在许多任务中的表现已经达到甚至超越了人类的水平，但是其泛化能力和人类相比还是相去甚远.如何提高网络的泛化性，一直是重要的研究方向之一.围绕这个方向开展的大量卓有成效的研究，从扩展增强训练数据、通过正则化抑制模型复杂度、优化训练策略等角度，提出了很多行之有效的方法.这些方法对于训练数据集来说都是某种全局性质的策略，每一个样本数据都会被平等的对待.但是，每一个样本数据由于其携带的信息量、噪声等的不同，在训练过程中，对模型的拟合性能和泛化性能的影响也应该是有差异性的.针对是否一些样本在反复的迭代训练中更倾向于使得模型过度拟合，如何找到这些样本，是否可以通过对不同的样本采用差异化的抗过拟合策略使得模型获得更好的泛化性能等问题，提出了一种依据样本数据的差异性来训练深度神经网络的方法，首先使用预训练模型对每一个训练样本进行评估，判断每个样本对该模型的拟合效果；然后依据评估结果将训练集分为易使得模型过拟合的样本和普通的样本两个子集；最后，再使用两个子集的数据对模型进行交替训练，过程中对易使得模型过拟合的子集采用更强有力的抗过拟合策略.通过在不同的数据集上对多种深度模型进行的一系列实验，验证了该方法在典型的分类任务和细粒度分类任务中的效果. |
| 关键词: 深度神经网络 泛化性 正则化 权重衰减 |
| DOI：10.13328/j.cnki.jos.006371 |
| 分类号:TP181 |
| 基金项目:贵州省科技计划（黔科合支撑[2020]4Y058） |
|
| Deep Neural Network Training Method Based on Individual Differences of Training Samples |
|
LI Xiang1,2, LIU Ming1, LIU Ming-Hui1, JIANG Qing2, CAO Yang2
|
|
1.School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China;2.CETC Big Data Research Institute Co., Ltd., Guiyang 550022, China
|
| Abstract: |
| In recent years, the performance of deep neural networks in many tasks has been comparable to or even surpassed that of humans, but its generalization ability is still far from that of humans. How to improve the generalization of the network has always been an important research direction, and a lot of fruitful research has been carried out around this direction. Many effective methods have been proposed from the perspectives of expanding and enhancing training data, suppressing model complexity through regularization, and optimizing training strategies. These methods are a global strategy for the training data set, and each sample data will be treated equally. However, due to the difference in the amount of information and noise carried by each sample data, the impact on the fitting performance and generalization performance of the model during the training process should also be different. Are some samples more likely to overfit the model during repeated iterative training? How to find these samples? Can the model obtain better generalization performance by adopting a differentiated anti-overfitting strategy for different samples? In response to these problems, a method for training deep neural networks is proposed based on individual differences in sample data. First, the pre-training model is used to evaluate each training sample to determine the fit effect of each sample to the model. Then, according to the evaluation results, the training set is divided into two subsets:samples that are easy to overfit the model and the remaining ordinary samples. Finally, two subsets of data are used to train the model. In the process, a stronger anti-overfitting strategy is adopted for the subset that is more likely to overfit the model. Through a series of experiments on various deep models on different data sets, the effect of the proposed method on typical classification tasks and fine-grained classification tasks is verified. |
| Key words: deep neural networks generalization regularization weight-decay |
