The persistent advance of deep learning algorithms and GPU computing power have promoted artificial intelligence in various fields including but not limited to compute vision, speech recognition, and natural language processing. Meanwhile, deep learning already began exploiting its usage in safety-critical areas exemplified by self-driving vehicles. Unfortunately, the successive severe traffic accidents in the past two years manifest that deep learning technology is still far from mature to fulfill safety-critical standards, and consequently the trustworthy artificial intelligence starts to attract a lot of research interests worldwide. This article conveys a state-of-the-art survey of the research on deep learning for real-time applications. It first introduces the main problems and challenges when deploying deep learning on the real-time embedded systems. Then, a detailed review covering various topics is provided, such as deep neural network lightweight design, GPU timing analysis and workload scheduling, shared resource management on the CPU+GPU SoC platform, deep neural network and network accelerator co-design. Finally, open issues and research directions are identified to conclude the survey.