research-article

Open access

Multimodal Classification of EEG During Physical Activity

Authors:

Hubert Cecotti,

Barry Giesbrecht,

Tobias HöllererAuthors Info & Claims

ICMI '19: 2019 International Conference on Multimodal Interaction

Pages 185 - 194

https://doi.org/10.1145/3340555.3353759

Published: 14 October 2019 Publication History

All formats PDF

Abstract

Brain Computer Interfaces (BCIs) typically utilize electroencephalography (EEG) to enable control of a computer through brain signals. However, EEG is susceptible to a large amount of noise, especially from muscle activity, making it difficult to use in ubiquitous computing environments where mobility and physicality are important features. In this work, we present a novel multimodal approach for classifying the P300 event related potential (ERP) component by coupling EEG signals with nonscalp electrodes (NSE) that measure ocular and muscle artifacts. We demonstrate the effectiveness of our approach on a new dataset where the P300 signal was evoked with participants on a stationary bike under three conditions of physical activity: rest, low-intensity, and high-intensity exercise. We show that intensity of physical activity impacts the performance of both our proposed model and existing state-of-the-art models. After incorporating signals from nonscalp electrodes our proposed model performs significantly better for the physical activity conditions. Our results suggest that the incorporation of additional modalities related to eye-movements and muscle activity may improve the efficacy of mobile EEG-based BCI systems, creating the potential for ubiquitous BCI.

References

[1]

Gregory D Abowd and Elizabeth D Mynatt. 2000. Charting past, present, and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 1(2000), 29–58.

Digital Library

[2]

Aurélien Appriou, Andrzej Cichocki, and Fabien Lotte. 2018. Towards robust neuroadaptive HCI: exploring modern machine learning methods to estimate mental workload from EEG signals. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW615.

Digital Library

[3]

P Aricò, G Borghini, G Di Flumeri, N Sciaraffa, and F Babiloni. 2018. Passive BCI beyond the lab: current trends and future directions. Physiological measurement 39, 8 (2018), 08TR02.

[4]

John Atkinson and Daniel Campos. 2016. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications 47 (2016), 35–41.

Digital Library

[5]

Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. 2012. Multiclass brain–computer interface classification by Riemannian geometry. IEEE Transactions on Biomedical Engineering 59, 4 (2012), 920–928.

[6]

Ali Bashashati, Mehrdad Fatourechi, Rabab K Ward, and Gary E Birch. 2007. A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals. Journal of Neural engineering 4, 2 (2007), R32.

[7]

G. Borg. 1970. Perceived exertion as an indicator of somatic stress.Scandinavian Journal of Rehabilitation Medicine 2, 2 (1970), 92–98. https://www.scopus.com/inward/record.uri?eid=2-s2.0-0014895192&partnerID=40&md5=fa8e27f6d6bf24a2f47dac587715f66c cited By 2796.

[8]

Raymond C Browning, Emily A Baker, Jessica A Herron, and Rodger Kram. 2006. Effects of obesity and sex on the energetic cost and preferred speed of walking. Journal of applied physiology 100, 2 (2006), 390–398.

[9]

Tom Bullock, Hubert Cecotti, and Barry Giesbrecht. 2015. Multiple stages of information processing are modulated during acute bouts of exercise. Neuroscience 307(2015), 138–150.

[10]

Tom Bullock, James C Elliott, John T Serences, and Barry Giesbrecht. 2017. Acute exercise modulates feature-selective responses in human cortex. Journal of cognitive neuroscience 29, 4 (2017), 605–618.

Digital Library

[11]

Hubert Cecotti, Miguel P Eckstein, and Barry Giesbrecht. 2014. Single-trial classification of event-related potentials in rapid serial visual presentation tasks using supervised spatial filtering. IEEE transactions on neural networks and learning systems 25, 11(2014), 2030–2042.

[12]

Hubert Cecotti and Axel Graeser. 2008. Convolutional neural network with embedded Fourier transform for EEG classification. In 2008 19th International Conference on Pattern Recognition. IEEE, 1–4.

[13]

Hubert Cecotti and Axel Graser. 2011. Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE transactions on pattern analysis and machine intelligence 33, 3(2011), 433–445.

Digital Library

[14]

H. Cecotti, A. Marathe, and A. Ries. 2015. Optimization of single-trial detection of event-related potentials through artificial trials. IEEE Trans. on Biomedical Engineering 62, 9 (2015), 2170–6.

[15]

Chih-Ming Chen, Jung-Ying Wang, and Chih-Ming Yu. 2017. Assessing the attention levels of students by using a novel attention aware system based on brainwave signals. British Journal of Educational Technology 48, 2 (2017), 348–369.

[16]

Guy Cheron, Géraldine Petit, Julian Cheron, Axelle Leroy, Anita Cebolla, Carlos Cevallos, Mathieu Petieau, Thomas Hoellinger, David Zarka, Anne-Marie Clarinval, 2016. Brain oscillations in sport: toward EEG biomarkers of performance. Frontiers in psychology 7 (2016), 246.

[17]

Luis F Ciria, Pandelis Perakakis, Antonio Luque-Casado, and Daniel Sanabria. 2018. Physical exercise increases overall brain oscillatory activity but does not influence inhibitory control in young adults. Neuroimage 181(2018), 203–210.

[18]

Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2015. Supporting novice to expert transitions in user interfaces. ACM Computing Surveys (CSUR) 47, 2 (2015), 31.

Digital Library

[19]

Gerhard Fischer. 2001. User modeling in human–computer interaction. User modeling and user-adapted interaction 11, 1-2 (2001), 65–86.

[20]

Nir Friedman, Tomer Fekete, Ya’akov Kobi Gal, and Oren Shriki. 2019. EEG-based Prediction of Cognitive Load in Intelligence Tests. Frontiers in Human Neuroscience 13 (2019), 191.

[21]

Fabien Grego, Jean-Marc Vallier, Maya Collardeau, Stéphane Bermon, Patricia Ferrari, Mirande Candito, Pascale Bayer, Marie-Noëlle Magnié, and Jeanick Brisswalter. 2004. Effects of long duration exercise on cognitive function, blood glucose, and counterregulatory hormones in male cyclists. Neuroscience letters 364, 2 (2004), 76–80.

[22]

Joseph T Gwin, Klaus Gramann, Scott Makeig, and Daniel P Ferris. 2010. Removal of movement artifact from high-density EEG recorded during walking and running. Journal of neurophysiology 103, 6 (2010), 3526–3534.

[23]

Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural computation 18, 7 (2006), 1527–1554.

Digital Library

[24]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017).

[25]

Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 3, 1 (2011), 18–31.

[26]

Laurens R Krol and Thorsten O Zander. 2017. Passive BCI-based Neuroadaptive Systems. In GBCIC.

[27]

Naveen Kumar and Jyoti Kumar. 2016. Measurement of cognitive load in HCI systems using EEG power spectrum: an experimental study. Procedia Computer Science 84 (2016), 70–78.

[28]

Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. 2018. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. Journal of neural engineering 15, 5 (2018), 056013.

[29]

Lun-De Liao, Chi-Yu Chen, I-Jan Wang, Sheng-Fu Chen, Shih-Yu Li, Bo-Wei Chen, Jyh-Yeong Chang, and Chin-Teng Lin. 2012. Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. Journal of neuroengineering and rehabilitation 9, 1(2012), 5.

[30]

Fu-Ren Lin and Chien-Min Kao. 2018. Mental effort detection using EEG data in E-learning contexts. Computers & Education 122 (2018), 63–79.

[31]

Ning-Han Liu, Cheng-Yu Chiang, and Hsuan-Chin Chu. 2013. Recognizing the degree of human attention using EEG signals from mobile sensors. Sensors 13, 8 (2013), 10273–10286.

[32]

Monika Lohani, Brennan R Payne, and David L Strayer. 2019. A review of psychophysiological measures to assess cognitive states in real-world driving. Frontiers in human neuroscience 13 (2019).

[33]

Fabien Lotte, Laurent Bougrain, Andrzej Cichocki, Maureen Clerc, Marco Congedo, Alain Rakotomamonjy, and Florian Yger. 2018. A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. Journal of neural engineering 15, 3 (2018), 031005.

[34]

Fabien Lotte, Marco Congedo, Anatole Lécuyer, Fabrice Lamarche, and Bruno Arnaldi. 2007. A review of classification algorithms for EEG-based brain–computer interfaces. Journal of neural engineering 4, 2 (2007), R1.

[35]

Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. 2012. Objective and Subjective Evaluation of Online Error Correction during P300-Based Spelling. Advances in Human-Computer Interaction 2012 (dec 2012), 1–13. https://doi.org/10.1155/2012/578295

[36]

Kevin Nathan and Jose L Contreras-Vidal. 2016. Negligible motion artifacts in scalp electroencephalography (EEG) during treadmill walking. Frontiers in human neuroscience 9 (2016), 708.

[37]

John Polich. 2007. Updating P300: an integrative theory of P3a and P3b. Clinical neurophysiology 118, 10 (2007), 2128–2148.

[38]

John Polich and JoséR. Criado. 2006. Neuropsychology and neuropharmacology of P3a and P3b.International journal of psychophysiology : official journal of the International Organization of Psychophysiology 60 2 (2006), 172–85.

[39]

H. Raza, S.M. Rathee, D.and Zhou, H. Cecotti, and G. Prasad. 2019. Covariate Shift Estimation based Adaptive Ensemble Learning for Handling Non-Stationarity in Motor Imagery related EEG-based Brain-Computer Interface. Neurocomputing 343(2019), 154–166.

Digital Library

[40]

Pedro Reis, Felix Kluge, Florian Gabsteiger, Vinzenz Tscharner, and Matthias Lochmann. 2014. Methodological aspects of EEG and Body dynamics measurements during motion.Frontiers in human neuroscience 8 (03 2014), 156. https://doi.org/10.3389/fnhum.2014.00156

[41]

Matthias Rötting, Thorsten Zander, Sandra Trösterer, and Jeronimo Dzaack. 2009. Implicit interaction in multimodal human-machine systems. In Industrial Engineering and Ergonomics. Springer, 523–536.

[42]

Ahmed Ben Said, Amr Mohamed, Tarek Elfouly, Khaled Harras, and Z Jane Wang. 2017. Multimodal deep learning approach for joint EEG-EMG data compression and classification. In 2017 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 1–6.

Digital Library

[43]

Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. 2017. Deep learning with convolutional neural networks for EEG decoding and visualization. Human brain mapping 38, 11 (2017), 5391–5420.

[44]

Andrew B Schwartz, X Tracy Cui, Douglas J Weber, and Daniel W Moran. 2006. Brain-controlled interfaces: movement restoration with neural prosthetics. Neuron 52, 1 (2006), 205–220.

[45]

Sergei L Shishkin, Yuri O Nuzhdin, Evgeny P Svirin, Alexander G Trofimov, Anastasia A Fedorova, Bogdan L Kozyrskiy, and Boris M Velichkovsky. 2016. EEG negativity in fixations used for gaze-based control: Toward converting intentions into actions with an eye-brain-computer interface. Frontiers in neuroscience 10 (2016), 528.

[46]

P. Simard, D. Steinkraus, and J.C. Platt. 2003. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. (Aug. 2003), 958–962.

[47]

Michael Tangermann, Klaus-Robert Müller, Ad Aertsen, Niels Birbaumer, Christoph Braun, Clemens Brunner, Robert Leeb, Carsten Mehring, Kai Miller, Gernot Mueller-Putz, Guido Nolte, Gert Pfurtscheller, Hubert Preissl, Gerwin Schalk, Alois Schlögl, Carmen Vidaurre, Stephan Waldert, and Benjamin Blankertz. 2012. Review of the BCI Competition IV. Frontiers in Neuroscience 6 (2012), 55. https://doi.org/10.3389/fnins.2012.00055

[48]

Jose Antonio Urigüen and Begoña Garcia-Zapirain. 2015. EEG artifact removal—state-of-the-art and guidelines. Journal of neural engineering 12, 3 (2015), 031001.

[49]

Jonathan R Wolpaw, Niels Birbaumer, Dennis J McFarland, Gert Pfurtscheller, and Theresa M Vaughan. 2002. Brain–computer interfaces for communication and control. Clinical neurophysiology 113, 6 (2002), 767–791.

[50]

Yasuo Yagi, Kerry L Coburn, Kristi M Estes, and James E Arruda. 1999. Effects of aerobic exercise and gender on visual and auditory P300, reaction time, and accuracy. European journal of applied physiology and occupational physiology 80, 5(1999), 402–408.

[51]

Zhong Yin and Jianhua Zhang. 2017. Cross-session classification of mental workload levels using EEG and an adaptive deep learning model. Biomedical Signal Processing and Control 33 (2017), 30–47.

[52]

Thorsten O Zander, Laurens R Krol, Niels P Birbaumer, and Klaus Gramann. 2016. Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences 113, 52(2016), 14898–14903.

Cited By

Bagheri SGood JAlavi H(2024)Visual and acoustic discomfort: A comparative study of impacts on individuals with and without ADHD using electroencephalogram (EEG)Building and Environment10.1016/j.buildenv.2024.111881264(111881)Online publication date: Oct-2024
https://doi.org/10.1016/j.buildenv.2024.111881
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Jeon JCai H(2023)Wearable EEG-based construction hazard identification in virtual and real environments: A comparative studySafety Science10.1016/j.ssci.2023.106213165(106213)Online publication date: Sep-2023
https://doi.org/10.1016/j.ssci.2023.106213
Show More Cited By

Recommendations

Machine learning-based EEG signals classification model for epileptic seizure detection
Abstract
The detection of epileptic seizures by classifying electroencephalography (EEG) signals into ictal and interictal classes is a demanding challenge, because it identifies the seizure and seizure-free states of an epileptic patient. In previous ...
Towards Robust Neuroadaptive HCI: Exploring Modern Machine Learning Methods to Estimate Mental Workload From EEG Signals
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems

Estimating mental workload from brain signals such as Electroencephalography (EEG) has proven very promising in multiple Human-Computer Interaction (HCI) applications, e.g., to design games or educational applications with adaptive difficulty, or to ...
Towards the detection of error-related potentials and its integration in the context of a P300 speller brain-computer interface

A P300 Speller is a brain-computer interface (BCI) that enables subjects to spell text on a computer screen by detecting P300 Event-Related Potentials in their electroencephalograms (EEG). This BCI application is of particular interest to disabled ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '19: 2019 International Conference on Multimodal Interaction

October 2019

601 pages

ISBN:9781450368605

DOI:10.1145/3340555

Editors:
Wen Gao
Peking University, China
,
Helen Mei Ling Meng
Chinese University of Hong Kong, China
,
Matthew Turk
Toyota Technological Institute at Chicago, USA
,
Susan R. Fussell
Cornell University, USA
,
Björn Schuller
Imperial College London / University of Augsburg, UK
,
Yale Song
Microsoft Research, USA
,
Kai Yu
Shanghai Jiao Tong University, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,327
Total Downloads

Downloads (Last 12 months)354
Downloads (Last 6 weeks)46

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bagheri SGood JAlavi H(2024)Visual and acoustic discomfort: A comparative study of impacts on individuals with and without ADHD using electroencephalogram (EEG)Building and Environment10.1016/j.buildenv.2024.111881264(111881)Online publication date: Oct-2024
https://doi.org/10.1016/j.buildenv.2024.111881
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Jeon JCai H(2023)Wearable EEG-based construction hazard identification in virtual and real environments: A comparative studySafety Science10.1016/j.ssci.2023.106213165(106213)Online publication date: Sep-2023
https://doi.org/10.1016/j.ssci.2023.106213
Liu YHöllerer TSra M(2022)SRI-EEG: State-Based Recurrent Imputation for EEG Artifact CorrectionFrontiers in Computational Neuroscience10.3389/fncom.2022.80338416Online publication date: 20-May-2022
https://doi.org/10.3389/fncom.2022.803384
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Ivaylov ILazarova MManolova A(2021)Multimodal Motor Imagery BCI Based on EEG and NIRS2021 56th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST)10.1109/ICEST52640.2021.9483551(73-76)Online publication date: 16-Jun-2021
https://doi.org/10.1109/ICEST52640.2021.9483551

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents