| 本文已被：浏览 1763次 下载 3880次 |
码上扫一扫！ |
|
|
| 基于深度多任务学习的图像美感与情感联合预测研究 |
|
申朕1, 崔超然2, 董桂鑫2, 余俊3, 黄瑾1, 尹义龙1
|
|
1.山东大学 软件学院, 山东 济南 250101;2.山东财经大学 计算机科学与技术学院, 山东 济南 250014;3.Department of Computer Science and Engineering, Lehigh University, Bethlehem PA 18015, USA
|
|
| 摘要: |
| 图像美学评价和情感分析任务旨在使计算机可以辨认人类由受到图像视觉刺激而产生的审美和情感反应.现有研究通常将它们当作两个相互独立的任务.但是，人类的美感与情感反应并不是孤立出现的；相反，在心理认知层面上，两种感受的出现应是相互关联和相互影响的.受此启发，采用深度多任务学习方法在统一的框架下处理图像美学评价和情感分析任务，深入探索两个任务间的内在关联.具体来说，提出一种自适应特征交互模块将两个单任务的基干网络进行关联，以完成图像美学评价和情感分析任务的联合预测.该模块中引入了一种特征动态交互机制，可以根据任务间的特征依赖关系自适应地决定任务间需要进行特征交互的程度.在多任务网络结构的参数更新过程中，根据美学评价与情感分析任务的学习复杂度和收敛速度等差异，提出一种任务间梯度平衡策略，以保证各个任务可以在联合预测的框架下平衡学习.此外，构建了一个大规模的图像美学情感联合数据集UAE.据已有研究，该数据集是首个同时包含美感和情感标签的图像集合.本模型代码以及UAE数据集已经公布在https://github.com/zhenshen-mla/Aesthetic-Emotion-Dataset. |
| 关键词: 图像美学评价 图像情感分析 深度多任务学习 自适应特征交互模块 任务间梯度平衡策略 |
| DOI：10.13328/j.cnki.jos.006487 |
| 分类号:TP391 |
| 基金项目:国家自然科学基金（61701281，61876098）；国家重点研发计划（2018YFC0830100，2018YFC0830102）；山东省高等学校优势学科和人才团队培育计划 |
|
| Unified Image Aesthetic and Emotional Prediction Based on Deep Multi-task Learning |
|
SHEN Zhen1, CUI Chao-Ran2, DONG Gui-Xin2, YU Jun3, HUANG Jin1, YIN Yi-Long1
|
|
1.School of Software, Shandong University, Jinan 250101, China;2.School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan 250014, China;3.Department of Computer Science and Engineering, Lehigh University, Bethlehem PA 18015, USA
|
| Abstract: |
| Image aesthetic assessment and emotional analysis aim to enable computers to identify the aesthetic and emotional responses of human beings caused by visual stimulations, respectively. Existing research usually treats them as two independent tasks. However, people’s aesthetic and emotional responses do not appear in isolation. On the contrary, from the perspective of psychological cognition, the two responses are interrelated and mutually influenced. Therefore, this study follows the idea of deep multi-task learning to deal with image aesthetic assessment and emotional analysis under a unified framework and explore their relationship. Specifically, a novel adaptive feature interaction module is proposed to correlate the backbone networks of the two tasks and achieve a unified prediction. In addition, a dynamic feature interaction mechanism is introduced to adaptively determine the degree of feature interaction between the tasks according to the feature dependencies. As the multi-task network updates structural parameters, the study, based on the inconsistency in complexity and convergence speed between the two tasks, proposes a novel gradient balancing strategy to ensure that the network parameters of each task can be smoothly learned under the unified prediction framework. Furthermore, the study constructs a large-scale unified image aesthetic and emotional dataset–UAE. According to the study, UAE is the first image collection containing both aesthetic and emotional labels. Finally, the model and codes of the proposed method as well as the UAE dataset have been released at https://github.com/zhenshen-mla/Aesthetic-Emotion-Dataset. |
| Key words: aesthetic assessment emotion analysis deep multi-task learning adaptive feature interaction gradient balancing strategy |
