Abstract
Self-supervised representation learning (SSL) has rapidly become a compelling avenue of research in visual tasks. Its performance, comparable to that of supervised learning, combined with its ability to mimic human perception, makes it an attractive area of study. The inherent simplicity of SSL also makes it easily accessible. However, data augmentation and collapse avoidance remain significant challenges. To address these issues, we have explored the use of image overlay in combination with principal Convolutional Neural Network models and an efficient SimSiam network. Our investigation led to three main findings. Firstly, while image overlay did not perform as efficiently as existing optimized augmentation methods, it showed potential to enhance the effectiveness of SSL tasks. Secondly, our research underscored the critical roles of data volume and augmentation in preventing a model collapse in SSL. This stands in contrast to pre-trained supervised learning, which places more emphasis on input image resolution and model size. Finally, our results indicated stability in the loss function within SSL, hinting at its potential to refine the model training process and encouraging the exploration of innovative augmentation methods. These empirical insights could be instrumental in simplifying and democratising deep learning techniques, making them more accessible and appealing to a broader audience. These findings will stimulate further research in this domain and contribute to the ongoing evolution of deep learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Asano, Y., Rupprecht, C., Vedaldi, A.: Self-labelling via simultaneous clustering and representation learning. In: International Conference on Learning Representations (2020)
Caron, M., Bojanowski, P., Mairal, J., Joulin, A.: Unsupervised pre-training of image features on non-curated data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2959–2968 (2019)
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Adv. Neural. Inf. Process. Syst. 33, 9912–9924 (2020)
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)
Grill, J.B., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural. Inf. Process. Syst. 33, 21271–21284 (2020)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., Soudry, D.: Augment your batch: improving generalization through instance repetition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8129–8138 (2020)
Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H.: On feature decorrelation in self-supervised learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598–9608 (2021)
Huang, G., Sun, Yu., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 646–661. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_39
Jing, L., Vincent, P., LeCun, Y., Tian, Y.: Understanding dimensional collapse in contrastive self-supervised learning. In: International Conference on Learning Representations (2022)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada (2009)
Li, A.C., Efros, A.A., Pathak, D.: Understanding collapse in non-contrastive siamese representation learning. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXI, pp. 490–505. Springer (2022)
Purushwalkam, S., Gupta, A.: Demystifying contrastive self-supervised learning: invariances, augmentations and dataset biases. Adv. Neural. Inf. Process. Syst. 33, 3407–3418 (2020)
Reed, C.J., Metzger, S., Srinivas, A., Darrell, T., Keutzer, K.: Selfaugment: automatic augmentation policies for self-supervised learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2674–2683 (2021)
Tang, L., Xiang, X., Zhang, H., Gong, M., Ma, J.: Divfusion: darkness-free infrared and visible image fusion. Inf. Fusion 91, 477–493 (2023)
Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? Advances in neural information processing systems 27 (2014)
Zhang, C., Zhang, K., Zhang, C., Pham, T.X., Yoo, C.D., Kweon, I.S.: How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In: International Conference on Learning Representations (2022)
Zhou, T., Cheng, Q., Lu, H., Li, Q., Zhang, X., Qiu, S.: Deep learning methods for medical image fusion: a review. Computers in Biology and Medicine, p. 106959 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xiao, L., Li, W., Bai, Q., Nguyen, M. (2023). Exploring the Potential of Image Overlay in Self-supervised Learning: A Study on SimSiam Networks and Strategies for Preventing Model Collapse. In: Wu, S., Yang, W., Amin, M.B., Kang, BH., Xu, G. (eds) Knowledge Management and Acquisition for Intelligent Systems. PKAW 2023. Lecture Notes in Computer Science(), vol 14317. Springer, Singapore. https://doi.org/10.1007/978-981-99-7855-7_3
Download citation
DOI: https://doi.org/10.1007/978-981-99-7855-7_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-7854-0
Online ISBN: 978-981-99-7855-7
eBook Packages: Computer ScienceComputer Science (R0)