rjxb软件学报Journal of Software1000-9825软件学报编辑部中国北京6420784962dae85843ce5f07a82b8aa1c202817484ac4dec4500b67e233596c2847f10.13328/j.cnki.jos.006420模式识别与人工智能PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE脑电情绪识别的深度学习研究综述Deep Learning for EEG-based Emotion Recognition: A Survey李锦瑶LIJin-Yao
Emotion is the external expression of affect, which has an influence on cognition, perception, and decision-making of people’s daily life. As one of the basic problems in the realization of overall computer intelligence, emotion recognition has been studied in depth and widely applied in fields of affective computing and human-computer interaction. Comparing with facial expression, speech and other physiological signals, using EEG to recognize emotion is attracting more attention for its higher temporal resolution, lower cost, better identification accuracy, and higher reliability. In recent years, more deep learning architectures are applied and have achieved better performance than traditional machine learning methods in this task. Deep learning for EEG-based emotion recognition is one of the research focuses and it remains many challenges to overcome. Considering that there exist few reviews literature to refer to, this study investigates the implementation of deep learning in EEG-based emotion recognition. Specifically, input formulation, deep learning architecture, experimental setting and results are surveyed. Besides, articles that evaluated their model on the widely used datasets, DEAP and SEED, perform qualitative and quantitative analysis are carefully screened from different aspects and a comparison is accomplished. Finally, the total work is summarized and the prospect of future work is given.
ReferencesCowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JGEmotion recognition in human-computer interaction2001181328010.1109/79.911197
Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 2001, 18(1): 32–80. [doi: 10.1109/79.911197]
Alarcão SM, Fonseca MJEmotions recognition using EEG signals: A survey201910337439310.1109/TAFFC.2017.2714671
Alarcão SM, Fonseca MJ. Emotions recognition using EEG signals: A survey. IEEE Transactions on Affective Computing, 2019, 10(3): 374–393. [doi: 10.1109/TAFFC.2017.2714671]
Zheng WL, Lu BLInvestigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks20157316217510.1109/TAMD.2015.2431497
Zheng WL, Lu BL. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 2015, 7(3): 162–175. [doi: 10.1109/TAMD.2015.2431497]
Piho L, Tjahjadi TA mutual information based adaptive windowing of informative EEG for emotion recognition202011472273510.1109/TAFFC.2018.2840973
Piho L, Tjahjadi T. A mutual information based adaptive windowing of informative EEG for emotion recognition. IEEE Transactions on Affective Computing, 2020, 11(4): 722–735. [doi: 10.1109/TAFFC.2018.2840973]
Van den Broek ELUbiquitous emotion-aware computing2013171536710.1007/s00779-011-0479-9
Van den Broek EL. Ubiquitous emotion-aware computing. Personal and Ubiquitous Computing, 2013, 17(1): 53–67. [doi: 10.1007/s00779-011-0479-9]
Russell JAA circumplex model of affect19803961161117810.1037/h0077714
Russell JA. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161–1178. [doi: 10.1037/h0077714]
Kleinsmith A, Bianchi-Berthouze NAffective body expression perception and recognition: A survey201341153310.1109/T-AFFC.2012.16
Kleinsmith A, Bianchi-Berthouze N. Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing, 2013, 4(1): 15–33. [doi: 10.1109/T-AFFC.2012.16]
Adolphs RRecognizing emotion from facial expressions: Psychological and neurological mechanisms200211216210.1177/1534582302001001003
Adolphs R. Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews, 2002, 1(1): 21–62. [doi: 10.1177/1534582302001001003]
Mathersul D, Williams LM, Hopkinson PJ, Kemp AHInvestigating models of affect: Relationships among EEG alpha asymmetry, depression, and anxiety20088456057210.1037/a0012811
Mathersul D, Williams LM, Hopkinson PJ, Kemp AH. Investigating models of affect: Relationships among EEG alpha asymmetry, depression, and anxiety. Emotion, 2008, 8(4): 560–572. [doi: 10.1037/a0012811]
Sammler D, Grigutsch M, Fritz T, Koelsch SMusic and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music200744229330410.1111/j.1469-8986.2007.00497.x
Sammler D, Grigutsch M, Fritz T, Koelsch S. Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology, 2007, 44(2): 293–304. [doi: 10.1111/j.1469-8986.2007.00497.x]
Liu YJ, Yu MJ, Zhao GZ, Song JJ, Ge Y, Shi YCReal-time movie-induced discrete emotion recognition from EEG signals20189455056210.1109/TAFFC.2017.2660485
Liu YJ, Yu MJ, Zhao GZ, Song JJ, Ge Y, Shi YC. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE Transactions on Affective Computing, 2018, 9(4): 550–562. [doi: 10.1109/TAFFC.2017.2660485]
Huang YJ, Wu CY, Wong AMK, Lin BSNovel active comb-shaped dry electrode for EEG measurement in hairy site201562125626310.1109/TBME.2014.2347318
Huang YJ, Wu CY, Wong AMK, Lin BS. Novel active comb-shaped dry electrode for EEG measurement in hairy site. IEEE Transactions on Biomedical Engineering, 2015, 62(1): 256–263. [doi: 10.1109/TBME.2014.2347318]
Paluš MNonlinearity in normal human EEG: Cycles, temporal asymmetry, nonstationarity and randomness, not chaos199675538939610.1007/s004220050304
Paluš M. Nonlinearity in normal human EEG: Cycles, temporal asymmetry, nonstationarity and randomness, not chaos. Biological Cybernetics, 1996, 75(5): 389–396. [doi: 10.1007/s004220050304]
Musha T, Terasaki Y, Haque HA, Ivamitsky GAFeature extraction from EEGs associated with emotions199711151910.1007/BF02471106
Musha T, Terasaki Y, Haque HA, Ivamitsky GA. Feature extraction from EEGs associated with emotions. Artificial Life and Robotics, 1997, 1(1): 15–19. [doi: 10.1007/BF02471106]
Kim J, André EEmotion recognition based on physiological changes in music listening200830122067208310.1109/TPAMI.2008.26
Kim J, André E. Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(12): 2067–2083. [doi: 10.1109/TPAMI.2008.26]
Lotfi E, Akbarzadeh-T MRPractical emotional neural networks201459617210.1016/j.neunet.2014.06.012
Lotfi E, Akbarzadeh-T MR. Practical emotional neural networks. Neural Networks, 2014, 59: 61–72. [doi: 10.1016/j.neunet.2014.06.012]
Song TF, Zheng WM, Song P, Cui ZEEG emotion recognition using dynamical graph convolutional neural networks202011353254110.1109/TAFFC.2018.2817622
Song TF, Zheng WM, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, 2020, 11(3): 532–541. [doi: 10.1109/TAFFC.2018.2817622]
Zhang GH, Yu MJ, Chen G, Han YH, Zhang D, Zhao GZ, Liu YJA review of EEG features for emotion recognition20194991097111810.1360/N112018-00337
Zhang GH, Yu MJ, Chen G, Han YH, Zhang D, Zhao GZ, Liu YJ. A review of EEG features for emotion recognition. Scientia Sinica Informationis, 2019, 49(9): 1097–1118 (in Chinese with English abstract). [doi: 10.1360/N112018-00337]
Zheng WL, Zhu JY, Lu BLIdentifying stable patterns over time for emotion recognition from EEG201910341742910.1109/TAFFC.2017.2712143
Zheng WL, Zhu JY, Lu BL. Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, 2019, 10(3): 417–429. [doi: 10.1109/TAFFC.2017.2712143]
Yang YM, Wu QMJ, Zheng WL, Lu BLEEG-based emotion recognition using hierarchical network with subnetwork nodes201810240841910.1109/TCDS.2017.2685338
Yang YM, Wu QMJ, Zheng WL, Lu BL. EEG-based emotion recognition using hierarchical network with subnetwork nodes. IEEE Transactions on Cognitive and Developmental Systems, 2018, 10(2): 408–419. [doi: 10.1109/TCDS.2017.2685338]
Wang ZM, Tong Y, Heng XPhase-locking value based graph convolutional neural networks for emotion recognition20197937119372210.1109/ACCESS.2019.2927768
Wang ZM, Tong Y, Heng X. Phase-locking value based graph convolutional neural networks for emotion recognition. IEEE Access, 2019, 7: 93711–93722. [doi: 10.1109/ACCESS.2019.2927768]
Alhagry S, Fahmy AA, El-Khoribi RAEmotion recognition based on EEG using LSTM recurrent neural network201781035535810.14569/IJACSA.2017.081046
Alhagry S, Fahmy AA, El-Khoribi RA. Emotion recognition based on EEG using LSTM recurrent neural network. International Journal of Advanced Computer Science and Applications, 2017, 8(10): 355–358. [doi: 10.14569/IJACSA.2017.081046]
Cimtay Y, Ekmekcioglu EInvestigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition2020207203410.3390/s20072034
Cimtay Y, Ekmekcioglu E. Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors, 2020, 20(7): 2034. [doi: 10.3390/s20072034]
Li YJ, Huang JJ, Zhou HY, Zhong NHuman emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks2017710106010.3390/app7101060
Li YJ, Huang JJ, Zhou HY, Zhong N. Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Applied Sciences, 2017, 7(10): 1060. [doi: 10.3390/app7101060]
Wang F, Wu SC, Zhang WW, Xu ZF, Zhang YH, Wu CD, Coleman SEmotion recognition with convolutional neural network and EEG-based EFDMs202014610750610.1016/j.neuropsychologia.2020.107506
Wang F, Wu SC, Zhang WW, Xu ZF, Zhang YH, Wu CD, Coleman S. Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia, 2020, 146: 107506. [doi: 10.1016/j.neuropsychologia.2020.107506]
Jirayucharoensak S, Pan-Ngum S, Israsena PEEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation2014201462789210.1155/2014/627892
Jirayucharoensak S, Pan-Ngum S, Israsena P. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. The Scientific World Journal, 2014, 2014: 627892. [doi: 10.1155/2014/627892]
Zhang T, Zheng WM, Cui Z, Zong Y, Li YSpatial–temporal recurrent neural network for emotion recognition201849383984710.1109/TCYB.2017.2788081
Zhang T, Zheng WM, Cui Z, Zong Y, Li Y. Spatial–temporal recurrent neural network for emotion recognition. IEEE Transactions on Cybernetics, 2018, 49(3): 839–847. [doi: 10.1109/TCYB.2017.2788081]
Hinton GE, Salakhutdinov RRReducing the dimensionality of data with neural networks2006313578650450710.1126/science.1127647
Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504–507. [doi: 10.1126/science.1127647]
Chai X, Wang QS, Zhao YP, Liu X, Bai O, Li YQUnsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition20167920521410.1016/j.compbiomed.2016.10.019
Chai X, Wang QS, Zhao YP, Liu X, Bai O, Li YQ. Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition. Computers in Biology and Medicine, 2016, 79: 205–214. [doi: 10.1016/j.compbiomed.2016.10.019]
Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PAStacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion20101133713408
Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 2010, 11: 3371–3408.
Hinton GE, Osindero S, Teh YWA fast learning algorithm for deep belief nets20061871527155410.1162/neco.2006.18.7.1527
Hinton GE, Osindero S, Teh YW. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7): 1527–1554. [doi: 10.1162/neco.2006.18.7.1527]
Chao H, Zhi HL, Dong L, Liu YLRecognition of emotions using multichannel EEG data and DBN-GC-based ensemble deep learning framework20182018975090410.1155/2018/9750904
Chao H, Zhi HL, Dong L, Liu YL. Recognition of emotions using multichannel EEG data and DBN-GC-based ensemble deep learning framework. Computational Intelligence and Neuroscience, 2018, 2018: 9750904. [doi: 10.1155/2018/9750904]
LeCun Y, Bottou L, Bengio Y, Haffner PGradient-based learning applied to document recognition199886112278232410.1109/5.726791
LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324. [doi: 10.1109/5.726791]
Liu Y, Ding YF, Li C, Cheng J, Song RC, Wan F, Chen XMulti-channel EEG-based emotion recognition via a multi-level features guided capsule network202012310392710.1016/j.compbiomed.2020.103927
Liu Y, Ding YF, Li C, Cheng J, Song RC, Wan F, Chen X. Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 2020, 123: 103927. [doi: 10.1016/j.compbiomed.2020.103927]
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation, 1997, 9(8): 1735–1780. [doi: 10.1162/neco.1997.9.8.1735]
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky VDomain-adversarial training of neural networks201617120962030
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17(1): 2096–2030.
Philippot PInducing and assessing differentiated emotion-feeling states in the laboratory19937217119310.1080/02699939308409183
Philippot P. Inducing and assessing differentiated emotion-feeling states in the laboratory. Cognition and Emotion, 1993, 7(2): 171–193. [doi: 10.1080/02699939308409183]
Gross JJ, Levenson RWEmotion elicitation using films1995918710810.1080/02699939508408966
Gross JJ, Levenson RW. Emotion elicitation using films. Cognition and Emotion, 1995, 9(1): 87–108. [doi: 10.1080/02699939508408966]
Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras IDEAP: A database for emotion analysis; using physiological signals201231183110.1109/T-AFFC.2011.15
Koelstra S, Muhl C, Soleymani M, Lee JS, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I. DEAP: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 2012, 3(1): 18–31. [doi: 10.1109/T-AFFC.2011.15]
Soleymani M, Lichtenauer J, Pun T, Pantic MA multimodal database for affect recognition and implicit tagging201231425510.1109/T-AFFC.2011.25
Soleymani M, Lichtenauer J, Pun T, Pantic M. A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing, 2012, 3(1): 42–55. [doi: 10.1109/T-AFFC.2011.25]
Katsigiannis S, Ramzan NDREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices20182219810710.1109/JBHI.2017.2688239
Katsigiannis S, Ramzan N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics, 2018, 22(1): 98–107. [doi: 10.1109/JBHI.2017.2688239]
Gao ZK, Wang XM, Yang YX, Li YL, Ma K, Chen GRA channel-fused dense convolutional network for EEG-based emotion recognition202113494595410.1109/TCDS.2020.2976112
Gao ZK, Wang XM, Yang YX, Li YL, Ma K, Chen GR. A channel-fused dense convolutional network for EEG-based emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, 2021, 13(4): 945–954. [doi: 10.1109/TCDS.2020.2976112]
Li Y, Wang L, Zheng WM, Zong Y, Qi L, Cui Z, Zhang T, Song TFA novel Bi-hemispheric discrepancy model for EEG emotion recognition202113235436710.1109/TCDS.2020.2999337
Li Y, Wang L, Zheng WM, Zong Y, Qi L, Cui Z, Zhang T, Song TF. A novel Bi-hemispheric discrepancy model for EEG emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, 2021, 13(2): 354–367. [doi: 10.1109/TCDS.2020.2999337]