| 摘要: |
| 近几年，基于Transformer的预训练模型展现了强大的模态表征能力，促使了多模态的下游任务（如图像描述生成任务）正朝着完全端到端范式的趋势所转变，并且能够使得模型获得更好的性能以及更快的推理速度.然而，该技术所提取的网格型视觉特征中缺乏区域型的视觉信息，从而导致模型对对象内容的描述不精确.因此，预训练模型在图像描述生成任务上的适用性在很大程度上仍有待探索.针对这一问题，提出一种基于视觉区域聚合与双向协作学习的端到端图像描述生成方法（visual region aggregation and dual-level collaboration，VRADC）.为了学习到区域型的视觉信息，设计了一种视觉区域聚合模块，将有相似语义的网格特征聚合在一起形成紧凑的视觉区域表征.接着，双向协作模块利用交叉注意力机制从两种视觉特征中学习到更加有代表性的语义信息，进而指导模型生成更加细粒度的图像描述文本.基于MSCOCO和Flickr30k两个数据集的实验结果表明，所提的VRADC方法能够大幅度地提升图像描述生成的质量，实现了最先进的性能. |
| 关键词: 图像描述 端到端训练 预训练模型 视觉区域聚合 双向协作 |
| DOI：10.13328/j.cnki.jos.006773 |
| 分类号: |
| 基金项目:国家自然科技支撑计划（2022YFC2009900/2022YFC2009903）；国家自然科学基金（62122018，62020106008，61772116，61872064） |
|
| End-to-end Image Captioning via Visual Region Aggregation and Dual-level Collaboration |
|
SONG Jing-Kuan, ZENG Peng-Peng, GU Jia-Yang, ZHU Jin-Kuan, GAO Lian-Li
|
|
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
|
| Abstract: |
| In recent years, Transformer-based pre-trained models have demonstrated powerful capabilities of modality representation, which leads to a shift towards a fully end-to-end paradigm for multimodal downstream tasks, such as image captioning tasks, and enables better performance and faster inference speed of models. However, the grid visual features extracted with such pre-trained models lack regional visual information, which results in inaccurate descriptions of the object content. Thus, the applicability of pre-trained models in image captioning remains largely unexplored. Therefore, this study proposes a novel end-to-end image captioning method based on visual region aggregation and dual-level collaboration (VRADC). Specifically, to learn regional visual information, this study designs a visual region aggregation module that aggregates grid features with similar semantics to obtain a compact visual region representation. Next, the dual-level collaboration module uses the cross-attention mechanism to learn more representative semantic information from the two visual features, which guides the model to generate more fine-grained image captions. The experimental results on the MSCOCO dataset and Flickr30k dataset show that the proposed VRADC-based method can significantly improve the quality of image captioning and achieves state-of-the-art performance. |
| Key words: image captioning end-to-end training pre-trained model visual region aggregation dual-level collaboration |
