| 摘要: |
| 在大规模图像检索任务中, 图像哈希技术通常依赖大量人工标注数据来训练深度哈希模型, 但高昂的人工标注成本限制了其实际应用. 为缓解对人工标注的依赖, 现有研究尝试利用网络用户提供的文本作为弱监督信息, 引导模型从图像中挖掘和文本关联的语义信息. 然而, 用户标签中普遍存在噪声, 限制了这些方法的性能. 多模态预训练基础模型(如CLIP)具备较强的图像-文本对齐能力. 受此启发, 利用CLIP来优化用户标签, 并提出一种CLIP引导标签优化的弱监督哈希方法(CLIP-guided tag refinement hashing, CTRH). 该方法包含3个主要内容: 标签置换模块、标签赋权模块和标签平衡损失函数. 标签置换模块通过微调CLIP挖掘图像关联的潜在标签. 标签赋权模块利用优化后的文本和图像进行跨模态全局语义交互, 学习判别性的联合表示. 针对用户标签的分布不平衡问题, 设计了一种标签平衡损失, 通过动态加权增强模型对困难样本的表征学习. 在MirFlickr和NUS-WIDE两个通用数据集上与最先进的方法对比验证了所提方法的有效性. |
| 关键词: 图像检索 弱监督哈希 预训练多模态基础模型 标签优化 |
| DOI：10.13328/j.cnki.jos.007543 |
| 分类号:TP391 |
| 基金项目:国家自然科学基金(62425603, 62372233); 江苏省基础研究计划攀登项目(BK20240011) |
|
| Weakly Supervised Image Hashing via CLIP-guided Tag Refinement |
|
LI Ze-Chao1, JIN Lu1, WANG Hao-Hua1, TANG Jin-Hui2
|
|
1.School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China;2.Nanjing Forestry University, Nanjing 210037, China
|
| Abstract: |
| In large-scale image retrieval tasks, image hashing typically relies on a large amount of manually annotated data to train deep hashing models. However, the high cost of manual annotation limits its practical application. To alleviate this dependency, existing studies attempt to use texts provided by Web users as weak supervision to guide the model in mining semantic information associated with the texts from images. Nevertheless, the inherent noise in user tags often limits model performance. Multimodal pre-trained models such as CLIP exhibit strong image-text alignment capabilities. Inspired by this, this study utilizes CLIP to optimize user tags and proposes a weakly supervised hashing method called CLIP-guided tag refinement hashing (CTRH). The proposed method consists of three key components: a tag replacement module, a tag weighting module, and a tag-balanced loss function. The tag replacement module fine-tunes CLIP to mine potential image-relevant tags. The tag weighting module performs cross-modal global semantic interaction between the optimized text and images to learn discriminative joint representations. To address the imbalance of user tags, a tag-balanced loss is designed, which dynamically reweights hard samples to enhance the model’s representation learning. Experiments on two general datasets, MirFlickr and NUS-WIDE, verify the effectiveness of the proposed method compared to state-of-the-art approaches. |
| Key words: image retrieval weakly supervised hashing multimodal pre-trained foundation model tag refinement |
