In this paper, unascertained mathematical theory is applied to the study of software reliability modeling. It is used to study software fault process, describe software failure character and get software reliability parameters. Finally, a software reliability model based on unascertained theory is proposed in this paper. The new model changes the traditional modeling thought, and brakes away from the statistical distribution assumption about the variety of failure rate in the traditional software reliability modeling process. It has better applicability, which improves the incongruence of model application to some extent.