In recent years, deep neural networks have continuously achieved breakthroughs in the classification task, but they will mistakenly give a wrong known class prediction when faced with unknown samples in the testing phase. The open set recognition is a possible way to solve the problem, which requires the model not only to classify the known classes, but also to distinguish the unknown samples accurately. Most of the existing methods are designed heuristically based on certain assumptions. Despite keeping the performance increasing, they have not analyzed the key factors that affect the task. This study analyzes the commonalities of existing methods by designing a new decision variable experiment and find that the ability of model to learn representations of known classes is an important factor. Then an open set recognition method is proposed based on model representation learning ability enhancement. Firstly, due to the powerful representation learning capabilities demonstrated by the contrastive learning and the label information contained in the open set recognition task, supervised contrastive learning is introduced to improve the modeling ability of the model to known classes. Secondly, considering that the correlation among the categories is the representation learning at the category level, and the hierarchical structure relationship among the categories is often presented, a multi-granularity inter-class correlation loss is designed by building the hierarchical structure in the label semantic space and measuring the multi-granularity inter-class correlation. The multi-granularity inter-class correlation loss constrains the model to learn the correlation among different known classes to further improve the representation learning ability of model. Finally, experimental results on multiple standard datasets verify the effectiveness of the proposed method on open set recognition tasks.