In order to protect the security of the execution environment of security-sensitive programs in computing devices, researchers have proposed the TEE technology, which provides a secure execution environment for security-sensitive programs that is isolated from the rich computing environment by isolating hardware and software. Side-channel attacks have evolved from traditionally requiring expensive equipment to now inferring confidential information using its access mode obtained basing only on microarchitecture states through software. The TEE architecture only provides an isolation mechanism and cannot resist this type of emerging software side-channel attacks. This paper thoroughly investigates the software side-channel attacks and corresponding countermeasures of the three TEE architectures of ARM TrustZone, Intel SGX and AMD SEV, and discusses the development trend of their attacks and defense mechanisms. First, we introduce the basic principles of ARM TrustZone, Intel SGX and AMD SEV, and elaborate on the definition and classification of software cache side-channel attacks, as well as the practical side-channel attack methods and steps. Second, from the perspective of processor instruction execution, we propose a TEE attack surface classification method, use this method to classify TEE software side-channel attacks, and explain the attacks combining software side-channel attacks and other attacks. Third, we discuss the threat model of TEE software side-channel attacks in detail. Finally, we comprehensively summarize the industry's countermeasures against TEE software side-channel attacks, and discuss some future research trends of TEE software side-channel attacks from two aspects of attack and defense.