Image hashing typically rely on large-scale manually annotated data to train deep hashing models. However, the high cost of manual annotation limits their practical application. To alleviate this dependency, recent studies have explored using user-provided textual tags as weak supervision to guide hash model capturing semantic information. Nevertheless, the inherent noise in user-generated tags often hinders model performance. Multimodal pre-trained foundation models, such as CLIP, exhibit strong image-text alignment capabilities. Inspired by this, we propose a CLIP-guided Tag Refinement Hashing (CTRH) framework that leverages CLIP to optimize noisy user tags for weakly supervised hashing. The proposed method consists of three key components: a tag replacement module, a tag weighting module, and a tag-balanced loss function. The tag replacement module fine-tunes CLIP to discover potential image-relevant tags. The tag weighting module performs cross-modal global semantic interaction among the refined text and images to learn discriminative joint representations. To address the tag imbalance problem, we design a tag-balanced loss that dynamically reweights training samples to enhance representation learning for hard instances. Extensive experiments conducted on two benchmark datasets, MirFlickr and NUS-WIDE, demonstrate that our method consistently outperforms state-of-the-art approaches, validating its effectiveness.