With the booming development and wide application of artificial intelligence, more and more deep learning-based Android malware detection methods and tools have been developed to detect newly emerged Android malware. However, deep learning models have been extensively proven to be vulnerable to adversarial attacks. Meanwhile, attacker shave started to propose adversarial attacks against Android malware detection methods to generate adversarial Android malware that can bypass detection. This study argues that the main reason current Android malware detection methods are vulnerable to such adversarial attacks is that these detectors are mostly built on single-modal features, which can be easily manipulated by attackers. Therefore, to improve the robustness of Android malware detection against adversarial attacks, the study proposes a robust Android malware detection method based on multi-modal fusion learning, namely RMDroid. RMDroid improves robustness in identifying adversarial malware without sacrificing accuracy in general Android malware detection. Specifically, RMDroid first extracts feature information from different modalities of Android apps and then uses the corresponding deep learning models to sufficiently learn feature vectors that characterize the deep semantics of each modality. Finally, an odd-one-out network is employed to reduce or even eliminate the influence of interfered modal features on the final malware prediction, thus improving robustness against adversarial attacks. The experimental results show that RMDroid achieves higher performance across five effectiveness metrics and one robustness metric compared to all baseline detection methods. In particular, given the same FPR, the TPR value of RMDroid is more than 10% higher than that of the best baseline detection method. In the case of the state-of-the-art adversarial attack of HRAT, RMDroid achieves over 96% in robustness, which is significantly higher than the robustness of both MaMaDroid and MalScan.