| 摘要: |
| 多模态信息抽取任务是指从非结构化或半结构化的多模态数据(包含文本和图像等)中提取结构化知识. 其研究内容主要包含多模态命名实体识别、多模态实体关系抽取和多模态事件抽取. 首先对多模态信息抽取任务进行分析, 然后对多模态命名实体识别、多模态实体关系抽取和多模态事件抽取这3个子任务的共同部分, 即多模态表示和融合模块进行归纳和总结. 随后梳理上述3个子任务的常用数据集和主流研究方法. 最后总结多模态信息抽取的研究趋势并分析该研究存在的问题和挑战, 为后续相关研究提供参考. |
| 关键词: 多模态信息抽取 多模态命名实体识别 多模态实体关系抽取 |
| DOI：10.13328/j.cnki.jos.007245 |
| 分类号: |
| 基金项目:国家自然科学基金(62276177, 61836007); 江苏高校优势学科建设工程项目 |
|
| Survey on Multimodal Information Extraction Research |
|
WANG Yong-Sheng, LI Pei-Feng, WANG Zhong-Qing, ZHU Qiao-Ming
|
|
School of Computer Science and Technology, Soochow University, Suzhou 215006, China
|
| Abstract: |
| Multimodal information extraction is a task to extract structured knowledge from unstructured or semi-structured multimodal data (such as text and images). It includes multimodal named entity recognition, multimodal relation extraction, and multimodal event extraction. This study analyzes multimodal information extraction tasks and summarizes the common part of the above three subtasks, i.e., a multimodal representation and fusion module. Moreover, it sorts out the commonly used datasets and mainstream research methods of the above three subtasks. Finally, it outlines research trends in multimodal information extraction and analyzes the existing problems and challenges in this field to provide a reference for future research. |
| Key words: multimodal information extraction (MIE) multimodal named entity recognition (MNER) multimodal entity relation extraction (MERE) |
