EEG-Based Contrastive Learning Models For Object Perception Using Multisensory Image-Audio Stimuli
Pages 39 - 47
Abstract
Multimedia sources such as images and audio commonly activate human senses to perceive objects, but limited research has explored the combined effect of these stimuli on predicting semantic object perception. In this study, we compare the performance of EEG signals elicited by image and audio stimuli in classifying semantic objects, revealing that image stimuli are more discriminative than audio stimuli. Building on this, we developed a contrastive learning model that integrates image and audio stimuli, further enhancing classification performance. Our research makes several key contributions: it compares classifier performance with uni-sensory versus multisensory stimuli, demonstrates improved performance with contrastive learning models using EEG data from both image and audio stimuli, and introduces a novel method to generate positive and negative pairs for contrastive learning models using cross-sensory EEG data. These findings enhance our understanding of how humans perceive multimedia sources and highlight the potential of multisensory integration in EEG-based classification.
References
[1]
Minkyu Ahn and Sung Chan Jun. 2015. Performance variation in motor imagery brain--computer interface: a brief review. Journal of neuroscience methods 243 (2015), 103--110.
[2]
Miguel Angrick, Maarten C Ottenhoff, Lorenz Diener, Darius Ivucic, Gabriel Ivucic, Sophocles Goulis, Jeremy Saal, Albert J Colon, Louis Wagner, Dean J Krusienski, et al. 2021. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Communications biology 4, 1 (2021), 1055.
[3]
Anahit Babayan, Miray Erbey, Deniz Kumral, Janis D Reinelt, Andrea MF Reiter, Josefin Röbbig, H Lina Schaare, Marie Uhlig, Alfred Anwander, Pierre-Louis Bazin, et al. 2019. A mind-brain-body dataset of MRI, EEG, cognition, emotion, and peripheral physiology in young and old adults. Scientific data 6, 1 (2019), 1--21.
[4]
Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. 2021. Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering 18, 4 (2021), 046020
[5]
Fabrizio Beverina, Giorgio Palmas, Stefano Silvoni, Francesco Piccione, Silvio Giove, et al. 2003. User adaptive BCIs: SSVEP and P300 based interfaces. Psych Nology J. 1, 4 (2003), 331--354.
[6]
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. 2023. Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[7]
Francesco Ferracuti, Alessandro Freddi, Sabrina Iarlori, Sauro Longhi, and Paolo Peretti. 2013. Auditory paradigm for a P300 BCI system using spatial hearing. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 871--876.
[8]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33 (2020), 21271--21284.
[9]
Mahyar Hamedi, Sh-Hussain Salleh, and Alias Mohd Noor. 2016. Electroen cephalographic motor imagery brain connectivity analysis for BCI: a review. Neural computation 28, 6 (2016), 999--1041.
[10]
Thi Kieu Khanh Ho and Narges Armanfard. 2023. Self-supervised learning for anomalous channel detection in EEG graphs: Application to seizure analysis. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 7866--7874.
[11]
PrannayKhosla,PiotrTeterwak,ChenWang,AaronSarna,YonglongTian,Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems 33 (2020), 18661 18673.
[12]
Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. 2018. EEGNet: a compact convolutional neural network for EEG-based brain--computer interfaces. Journal of neural engineering 15, 5 (2018), 056013.
[13]
Yang Li, Ji Chen, Fu Li, Boxun Fu, Hao Wu, Youshuo Ji, Yijin Zhou, Yi Niu, Guangming Shi, and Wenming Zheng. 2022. GMSS: Graph-based multi-task self-supervised learning for EEG emotion recognition. IEEE Transactions on Affective Computing 14, 3 (2022), 2512--2525.
[14]
Diego Lopez-Bernal, David Balderas, Pedro Ponce, and Arturo Molina. 2022. A state-of-the-art review of EEG-based imagined speech decoding. Frontiers in human neuroscience 16 (2022), 867281.
[15]
Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. 2020. Contrastive representation learning for electroencephalogram classification. In Machine Learning for Health. PMLR, 238--253.
[16]
David A Moses, Matthew K Leonard, Joseph G Makin, and Edward F Chang. 2019. Real-time decoding of question-and-answer speech dialogue using human cortical activity. Nature communications 10, 1 (2019), 3096.
[17]
Liang Ou, Thomas Do, Xuan-The Tran, Daniel Leong, Yu-Cheng Chang, Yu-Kai Wang, and Chin-Teng Lin. 2023. Improving CCA Algorithms on SSVEP Classifi cation with Reinforcement Learning Based Temporal Filtering. In Australasian Joint Conference on Artificial Intelligence. Springer, 376--386.
[18]
Jerrin Thomas Panachakel and Angarai Ganesan Ramakrishnan. 2021. Decoding covert speech from EEG-a comprehensive review. Frontiers in Neuroscience 15 (2021), 642251.
[19]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolu tional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5--9, 2015, proceedings, part III 18. Springer, 234--241.
[20]
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, MichaelTangermann,Frank Hutter, Wolfram Burgard, and Tonio Ball. 2017. Deep learning with convolutional neural networks for EEG decoding and visualization. Human brain mapping 38, 11 (2017), 5391--5420.
[21]
Martijn Schreuder, Benjamin Blankertz, and Michael Tangermann. 2010. A new auditory multi-class brain-computer interface paradigm: spatial hearing as an informative cue. PloS one 5, 4 (2010), e9813.
[22]
Alex Sherstinsky. 2020. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena 404 (2020), 132306.
[23]
Jiayao Sun, Jin Xie, and Huihui Zhou. 2021. EEG classification with transformer based models. In 2021 ieee 3rd global conference on life sciences and technologies (lifetech). IEEE, 92--93.
[24]
Xuan-The Tran, Thomas Do, Nikhil R Pal, Tzyy-Ping Jung, and Chin-Teng Lin. 2024. Multimodal fusion for anticipating human decision performance. Scientific Reports 14, 1 (2024), 13217.
[25]
Xuan-The Tran, Thomas Tien-Thong Do, and Chin-Teng Lin. 2023. Early Detec tion of Human Decision-Making in Concealed Object Visual Searching Tasks: An EEG-BiLSTM Study. In 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 1--4.
[26]
Laurens Van der Maaten andGeoffreyHinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).
[27]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[28]
Weining Weng, Yang Gu, Shuai Guo, Yuan Ma, Zhaohua Yang, Yuchen Liu, and Yiqiang Chen. 2024. Self-supervised Learning for Electroencephalogram: A Systematic Survey. arXiv preprint arXiv:2401.05446 (2024).
[29]
Holly Wilson, Mohammad Golbabaee, Michael J Proulx, Stephen Charles, and Ea monnO-Neill. 2023. EEG-based BCI dataset of semantic concepts for imagination and perception tasks. Scientific Data 10, 1 (2023), 386.
[30]
Chaoqi Yang, Cao Xiao, M Brandon Westover, Jimeng Sun, et al. 2023. Self supervised electroencephalogram representation learning for automatic sleep staging: model development and evaluation study. JMIR AI 2, 1 (2023), e46769.
[31]
Weishan Ye, Zhiguo Zhang, Min Zhang, Fei Teng, Li Zhang, Linling Li, Gan Huang, Jianhong Wang, Dong Ni, and Zhen Liang. 2023. Semi-supervised dual stream self-attentive adversarial graph contrastive learning for cross-subject eeg-based emotion recognition. arXiv preprint arXiv:2308.11635 (2023).
[32]
Danhua Zhu, Jordi Bieger, Gary Garcia Molina, and Ronald M Aarts. 2010. A sur vey of stimulation methods used in SSVEP-based BCIs. Computational intelligence and neuroscience 2010, 1 (2010), 702357
Index Terms
- EEG-Based Contrastive Learning Models For Object Perception Using Multisensory Image-Audio Stimuli
Recommendations
Multitasking, an EEG experiment: comparative analysis of cognitive workload during demanding stimuli presentation
AbstractThere are two perspectives regarding the human brain’s ability to multitask: concurrent multitasking where multiple tasks are performed simultaneously, and sequential multitasking with switching between tasks. This study investigates the human ...
Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli
The neural activity of speech sound processing (the N1 component of the auditory ERP) can be suppressed if a speech sound is accompanied by concordant lip movements. Here we demonstrate that this audiovisual interaction is neither speech specific nor ...
Eeg frequency tagging to dissociate the cortical responses to nociceptive and nonnociceptive stimuli
Whether the cortical processing of nociceptive input relies on the activity of nociceptive-specific neurons or whether it relies on the activity of neurons also involved in processing nonnociceptive sensory input remains a matter of debate. Here, we ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
67 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- GrapheneX
- DST Australian Autonomy Initiative agreement
- The Australian Research Council (ARC)
- Australian Cooperative Research Centres Projects (CRC-P) Round 11
- US Office of Naval Research Global under Cooperative
- The Australia Defence Innovation Hub
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 216Total Downloads
- Downloads (Last 12 months)216
- Downloads (Last 6 weeks)55
Reflects downloads up to 07 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in