Label-specific features serve as an effective strategy for addressing multi-label classification tasks. By tailoring discriminative features to the individual preferences of each label, such features enhance the generalization capability of classification models. Existing methods typically focus on manipulating features to extract those relevant to label discrimination. Rather than following this conventional approach, this study explores a novel perspective based on feature invariance for label-specific feature learning. Specifically, invariance is injected into classifiers with respect to label-irrelevant features by intentionally manipulating these features for each class label. Accordingly, an invariance-based label-specific feature learning method, termed INVA, is proposed. INVA estimates the feature covariance matrix for each label to capture intra-class variation, thus identifying label-irrelevant features. Classifiers are then endowed with invariance to these features by solving a perturbation risk minimization problem. Furthermore, an upper bound of the perturbation risk is derived to enhance computational efficiency. Comprehensive experiments on standard multi-label benchmark datasets demonstrate the effectiveness of the proposed method.