In recent years, deep reinforcement learning has been widely used in sequential decision making. The approach works well in many applications, especially in those scenarios with high-dimensional input and large state spaces. However, there are some limitations of these deep reinforcement learning methods, such as lack of interpretability, inefficient initial training, cold start, etc. In this paper, we propose a framework combining explicit knowledge reasoning with deep reinforcement learning, to alleviation the above problems. The framework successfully leverages high-level priori knowledge in the deep learning process via explicit knowledge representation, resulting in improvement of the training efficiency and the interpretability. The explicit knowledge is categorized into two kinds, namely, acceleration knowledge and safety knowledge. The former intervenes in the training, especially at the early stage, to speed up the learning process, while the latter keeps the agent from catastrophic actions to keep it safe. Our experiments in several domains with several baselines show that the proposed framework significantly improves the training efficiency and the interpretability, and the improvement is general for different reinforcement learning algorithms and different scenarios.