The traditional matrix factorization method has a wide range of applications in prediction and recommendation tasks because of its high scalability and good performance. In the big data era, more and more contextual features can be obtained easily, while the traditional matrix factorization approach lacks effective use of context information. In this context, Factorization Machines (FM) is proposed and popular. To better grasp the development process of FM model and adapt FM approach to the real application, this paper reviews existing FM models and their optimization algorithms. First, it introduces the evolution process from traditional Matrix Factorization (MF) to FM model. Second, the paper summarizes the existing researches on FM method from the perspective of model accuracy and efficiency; Third, the paper presents the studies of four representative optimization algorithms, which are suitable for various FM models. Finally, the paper analyzes the challenges in the current FM model, proposes possible solutions for these problems, and discusses the future work.