Unsupervised learning-based action recognition methods have shown immense potential by leveraging vast amounts of unlabeled data, yielding competitive performance in action recognition tasks. To capture semantic information in action, many efforts have focused on multi-stream fusion techniques, which makes the models heavier and less flexible to train. However, existing single-stream methods struggle to provide rich semantic information through data augmentation. To address these challenges, we propose a novel joint-level semantic augmentation method based on a constructed motion manifold, which achieves significant performance gains with virtually no additional training costs. Our approach introduces random perturbations on the motion manifold constructed based on evolutionary metrics. We propose several semantic augmentation strategies, including directional semantic perturbations, magnitude semantic perturbations, and initial pose perturbations. To pay greater attention to samples with significant semantic differences, we introduce the Semantic Adaptive Weighted Loss (SAWL) based on motion manifold distance. SAWL encourages the model to pay more attention to these samples, leading to the learning of an embedding space with semantic invariance. Extensive experiments were conducted on three large-scale datasets, i.e., NTU-60, NTU-120, and PKU-MMD II. The results demonstrate that our single-stream approach, empowered by joint-level semantic augmentation, achieves state-of-the-art (SOTA) performance among single-stream methods and competitive performance among multi-stream methods.