Graph-based index for high-dimensional vectors has become the mainstream solution for large-scale approximate nearest neighbor search (ANNS) due to its high efficiency. The search process over graph-based index typically consists of two stages: the first stage rapidly navigates from an entry point to a region near the query vector, while the second stage searches for the k nearest vectors within the localized region. However, due to the need to store a large number of adjacency relationships, graph-based indexes often incur high memory overhead. In practice, this leads to storing the index on external memory, where vector and graph data are loaded on demand during ANNS. This results in frequent I/O operations, which have become the primary bottleneck, accounting for over 90% of total query time. Existing systems exploit the fact that entry points and their nearby neighbors are frequently accessed, and adopt static caching strategies that preload these points and their multi-hop neighbors into memory to reduce I/O during the first stage. However, our analysis shows that the second stage contributes the majority of I/O cost, as it involves accessing a large number of graph vertices related to the query vector to ensure high recall. Since the accessed points in this stage varies dynamically with each query, static caching strategy fails to capture it effectively and thus becomes nearly ineffective. To address this issue, we propose GoVector, a hybrid caching strategy that integrates both static and dynamic components. Specifically, (1) the static cache preloads the entry point and its frequently accessed neighbors, while (2) the dynamic cache adaptively stores high-locality vertices encountered during the second stage of the search. Furthermore, to align with the similarity-driven search behavior of the second phase, we propose a vector-similarity-aware disk layout strategy that reorganizes the storage order of vertices, clustering similar vectors into the same or adjacent disk pages to enhance data locality. This dual-optimization approach greatly enhances cache hit rates and effectively reduces overall I/O overhead. Experiments on multiple public datasets demonstrate that GoVector achieves an average of 46% fewer I/O operations, 1.73× higher query throughput, and 42% lower latency compared to state-of-the-art disk-based graph indexing ANNS systems under 90% recall.