| 本文已被：浏览 3074次 下载 7081次 |
码上扫一扫！ |
|
|
| 面向视觉语言理解与生成的多模态预训练方法 |
|
刘天义1,2,3, 吴祖煊1,2,3, 陈静静1,2,3, 姜育刚1,2,3
|
|
1.复旦大学 计算机科学技术学院, 上海 200438;2.上海市智能信息处理重点实验室(复旦大学), 上海 200438;3.上海市智能视觉计算协同创新中心(复旦大学), 上海 200438
|
|
| 摘要: |
| 大多数现有的视觉语言预训练方法侧重于理解任务，并在训练时使用类似于BERT的损失函数（掩码语言建模和图像文本匹配）.尽管它们在许多理解类型的下游任务中表现良好，例如视觉问答、图像文本检索和视觉蕴涵，但它们不具备生成信息的能力.为了解决这个问题，提出了视觉语言理解和生成的统一多模态预训练（unified multimodal pre-training for vision-language understanding and generation，UniVL）.UniVL能够处理理解任务和生成任务，并扩展了现有的预训练范式，同时使用随机掩码和因果掩码，因果掩码即掩盖未来标记的三角形掩码，这样预训练的模型可以具有自回归生成的能力.将几种视觉语言理解任务规范为文本生成任务，并使用基于模版提示的方法对不同的下游任务进行微调.实验表明，在使用同一个模型时，理解任务和生成任务之间存在权衡，而提升这两个任务的可行方法是使用更多的数据.UniVL框架在理解任务和生成任务方面的性能与最近的视觉语言预训练方法相当.此外，实验还证明了基于模版提示的生成方法更有效，甚至在少数场景中它优于判别方法. |
| 关键词: 计算机视觉 多模态学习 预训练 |
| DOI：10.13328/j.cnki.jos.006770 |
| 分类号: |
| 基金项目:科技创新2030——“新一代人工智能”重大项目（2021ZD0112805）；国家自然科学基金青年基金（62102092） |
|
| Multimodal Pre-training Method for Vision-language Understanding and Generation |
|
LIU Tian-Yi1,2,3, WU Zu-Xuan1,2,3, CHEN Jing-Jing1,2,3, JIANG Yu-Gang1,2,3
|
|
1.School of Computer Science, Fudan University, Shanghai 200438, China;2.Shanghai Key Laboratory of Intelligent Information Processing (Fudan University), Shanghai 200438, China;3.Shanghai Collaborative Innovation Center of Intelligent Visual Computing (Fudan University), Shanghai 200438, China
|
| Abstract: |
| Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like loss functions (masked language modeling and image-text matching) during pre-training. Despite their good performance in the understanding of downstream tasks, such as visual question answering, image-text retrieval, and visual entailment, these methods cannot generate information. To tackle this problem, this study proposes unified multimodal pre-training for vision-language understanding and generation (UniVL). The proposed UniVL is capable of handling both understanding tasks and generation tasks. It expands existing pre-training paradigms and uses random masks and causal masks simultaneously, where causal masks are triangular masks that mask future tokens, and such pre-trained models can have autoregressive generation abilities. Moreover, several vision-language understanding tasks are turned into text generation tasks according to specifications, and the prompt-based method is employed for fine-tuning of different downstream tasks. The experiments show that there is a trade-off between understanding tasks and generation tasks when the same model is used, and a feasible way to improve both tasks is to use more data. The proposed UniVL framework attains comparable performance to recent vision-language pre-training methods in both understanding tasks and generation tasks. Moreover, the prompt-based generation method is more effective and even outperforms discriminative methods in few-shot scenarios. |
| Key words: computer vision multimodal learning pre-training |
