Advances in Self-Supervised Learning: A Comprehensive Review of Contrastive and Generative Approaches

Le résumé fourni par la source

Self-supervised learning (SSL) has become a transformative paradigm in machine learning, which allows models to learn rich representations from unlabeled data without expensive human annotations. Recent Progress in SSL The recent progress of SSL is systematically reviewed, paying special attention to the two main methodologies: contrastive learning and generative masked modeling. We conduct a detailed study of popular strategies, such as SimCLR, MoCo, BYOL, MAE and BEiT in terms of underpinning mathematics, architectural novelties and empirical goodness on varied benchmarks. Our comparison results show the recent leading state-of-the-art SSL methods can now match/surpass the supervised learning baselines with ImageNet top-1 accuracies as 87.8% for MAE ViT-Huge and 77.1% for ReLICv2 ResNet50 which are both in line of them. We distill insights from more than 50 recent papers on applications in computer vision, natural language processing, medical imaging, and multimodal learning. We review the key trends such as removing negative pairs, analysis of mask effects on models and significance of data augmentation techniques. We close with a discussion of open issues and prospects for future work in terms of theoretical understanding, computational efficiency, application to domain areas, and model integration.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.