research-article

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines

Authors:

Erin Treacy Solovey,

Audrey Girouard,

Krysta Chauncey,

Leanne M. Hirshfield,

Angelo Sassaroli,

Sergio Fantini,

Robert J.K. JacobAuthors Info & Claims

UIST '09: Proceedings of the 22nd annual ACM symposium on User interface software and technology

Pages 157 - 166

https://doi.org/10.1145/1622176.1622207

Published: 04 October 2009 Publication History

Abstract

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

References

[1]

Bunce, S.C., Izzetoglu, M., Izzetoglu, K., Onaral, B. and Pourrezaei, K. Functional Near Infrared Spectroscopy: An Emerging Neuroimaging Modality. IEEE Engineering in Medicine and Biology Magazine, Special issue on Clinical Neuroengineering, 25 (4). 2006, 54--62.

[2]

Chen, D., Hart, J. and Vertegaal, R. Towards a Physiological Model of User Interruptability. in INTERACT'07, 2008, 439--451.

Digital Library

[3]

Chenier, F. and Sawan, M., A New Brain Imaging Device Based on fNIRS. in BIOCAS 2007, (2007), 1--4.

[4]

Coyle, S., Ward, T. and Markham, C., Physiological Noise in Near-infrared Spectroscopy: Implications for Optical Brain Computer Interfacing. in Proc. EMBS, (2004).

[5]

Devaraj, A., Izzetoglu, M., Izzetoglu, K. and Onaral, B. Motion Artifact Removal for fNIR Spectroscopy for Real World Application Areas. Proc. SPIE, 5588. 2004, 224--229.

[6]

Ehlis, A.-C., Bähne, C.G., Jacob, C.P., Herrmann, M.J. and Fallgatter, A.J. Reduced lateral prefrontal activation in adult patients with attention-deficit/hyperactivity disorder (ADHD) during a working memory task: A functional near-infrared spectroscopy (fNIRS) study. Journal of Psychiatric Research, 42 (13). 2008, 1060--1067.

[7]

Fink, A., Benedek, M., Grabner, R.H., Staudt, B. and Neubauer, A.C. Creativity meets neuroscience: Experimental tasks for the neuroscientific study of creative thinking. Methods, 42 (1). 2007, 68--76.

[8]

Folley, B.S. and Park, S. Verbal creativity and schizotypal personality in relation to prefrontal hemispheric laterality: A behavioral and near-infrared optical imaging study. Schizophrenia Research, 80 (2--3). 2005, 271--282.

[9]

Girouard, A., Solovey, E.T., Hirshfield, L.M., Chauncey, K., Sassaroli, A., Fantini, S. and Jacob, R.J.K., Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements. in INTERACT 2009, (2009), Springer.

Digital Library

[10]

Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P. and Rao, R.P.N. Feasibility and pragmatics of classifying working memory load with an electroencephalograph Proc. CHI'08, ACM, 2008.

Digital Library

[11]

Herrmann, M.J., Huter, T., Plichta, M.M., Ehlis, A.-C., Alpers, G.W., Mühlberger, A. and Fallgatter, A.J. Enhancement of activity of the primary visual cortex during processing of emotional stimuli as measured with event-related functional near-infrared spectroscopy and event-related potentials. Human Brain Mapping, 29 (1). 2008, 28--35.

[12]

Hirshfield, L., Chauncey, K., Gulotta, R., Girouard, A., Solovey, E.T., Jacob, R., Sassaroli, A. and Fantini, S., Combining Electroencephalograph and Near Infrared Spectroscopy to Explore Users' Mental Workload. in HCI International 2009 (2009).

Digital Library

[13]

Hirshfield, L.M., Solovey, E.T., Girouard, A., Kebinger, J., Sassaroli, A., Tong, Y., Fantini, S. and Jacob, R.J.K., Brain Measurement for Usability Testing and Adaptive Interfaces: An Example of Uncovering Syntactic Workload with Functional Near Infrared Spectroscopy. in Proc. CHI'09, (2009).

Digital Library

[14]

Hoshi, Y. Near-Infrared Spectroscopy for Studying Higher Cognition. in Neural Correlates of Thinking, 2009, 83--93.

[15]

Huppert, T. and Boas, D.A. (2005). Hemodynamic Evoked Response NIRS data analysis GUI Program User's Guide, 2009, from http://www.nmr.mgh.harvard.edu/PMI/resources/homer/HomER_UsersGuide_05-07-28.pdf

[16]

Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K. and Chance, B. Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks. IJHCI, 17 (2). 2004, 211--231.

[17]

Izzetoglu, M., Devaraj, A., Bunce, S. and Onaral, B. Motion Artifact Cancellation in NIR spectroscopy Using Wiener Filtering. IEEE Transactions on Biomedical Engineering, 52 (5). 2005, 934--938.

[18]

Lee, J.C. and Tan, D.S., Using a Low-Cost Electroencephalograph for Task Classification in HCI Research. in Proc. ACM Symposium on User Interface Software and Technology 2006, (2006).

Digital Library

[19]

Mappus, R.L., Venkatesh, G.R., Shastry, C., Israeli, A. and Jackson, M.M., An fNIR Based BMI for Letter Construction Using Continuous Control. in Proc. CHI 2009 Extended Abstracts, (2009).

Digital Library

[20]

Matthews, F., Pearlmutter, B.A., Ward, T.E., Soraghan, C. and Markham, C. Hemodynamics for Brain-Computer Interfaces. Signal Processing Magazine, IEEE, 25 (1). 2008, 87--94.

[21]

Millán, J.d.R., Renkens, F., Mouriño, J. and Gerstner, W. Brain-actuated interaction. Artif. Intell., 159 (1--2). 2004, 241--259.

Digital Library

[22]

Morioka, S., Yamada, M. and Komori, T. Frontal Lobe Activity during the Performance of Spatial Tasks: fNIRS Study. J Phys Ther Sci, 20. 2008, 135--139.

[23]

Ramnani, N. and Owen, A.M. Anterior prefrontal cortex: insights into function from anatomy and neuroimaging. Nat Rev Neurosci, 5 (3). 2004, 184.

[24]

Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K. and Birbaumer, N. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. NEUROIMAGE, 34 (4). 2007, 1416--1427.

[25]

Villringer, A. and Chance, B. Non-Invasive Optical Spectroscopy and Imaging of Human Brain Function. Trends in Neuroscience, 20. 1997, 435--442.

Cited By

De Santis AFantinelli S(2024)Digital Well-Being as a New Kind of Adaptation to the New Millennium Needs: A State-of-the-Art AnalysisElementa. Intersections between Philosophy, Epistemology and Empirical Perspectives10.7358/elementa-2023-0102-safa3:1-2Online publication date: 29-Jan-2024
https://doi.org/10.7358/elementa-2023-0102-safa
Ruotsalo TMäkelä KSpapé M(2024)Crowdsourcing Affective Annotations Via fNIRS-BCIIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327391615:1(297-308)Online publication date: Jan-2024
https://doi.org/10.1109/TAFFC.2023.3273916
Landowska AWilson MCraven MHarrington K(2024)Adaptative computerized cognitive training decreases mental workload during working memory precision task - A preliminary fNIRS studyInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103206184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103206
Show More Cited By

Index Terms

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Examining the Reliability of Using fNIRS in Realistic HCI Settings for Spatial and Verbal Tasks
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Recent efforts have shown that functional near-infrared spectroscopy (fNIRS) has potential value for brain sensing in HCI user studies. Research has shown that, although large head movement significantly affects fNIRS data, typical keyboard use, mouse ...
Measuring the effect of think aloud protocols on workload using fNIRS
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

The Think Aloud Protocol (TAP) is a verbalisation technique widely employed in HCI user studies to give insight into user experience, yet little work has explored the impact that TAPs have on participants during user studies. This paper utilises a brain ...
Using fNIRS brain sensing to evaluate information visualization interfaces
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We show how brain sensing can lend insight to the evaluation of visual interfaces and establish a role for fNIRS in visualization. Research suggests that the evaluation of visual design benefits by going beyond performance measures or questionnaires to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '09: Proceedings of the 22nd annual ACM symposium on User interface software and technology

October 2009

278 pages

ISBN:9781605587455

DOI:10.1145/1622176

General Chair:
Andrew Wilson
Microsoft Research, USA
,
Program Chair:
François Guimbretière
Cornell University, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '09

Sponsor:

UIST '09: The 22nd Annual ACM Symposium on User Interface Software and Technology

October 4 - 7, 2009

BC, Victoria, Canada

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

84
Total Citations
View Citations
1,292
Total Downloads

Downloads (Last 12 months)105
Downloads (Last 6 weeks)9

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

De Santis AFantinelli S(2024)Digital Well-Being as a New Kind of Adaptation to the New Millennium Needs: A State-of-the-Art AnalysisElementa. Intersections between Philosophy, Epistemology and Empirical Perspectives10.7358/elementa-2023-0102-safa3:1-2Online publication date: 29-Jan-2024
https://doi.org/10.7358/elementa-2023-0102-safa
Ruotsalo TMäkelä KSpapé M(2024)Crowdsourcing Affective Annotations Via fNIRS-BCIIEEE Transactions on Affective Computing10.1109/TAFFC.2023.327391615:1(297-308)Online publication date: Jan-2024
https://doi.org/10.1109/TAFFC.2023.3273916
Landowska AWilson MCraven MHarrington K(2024)Adaptative computerized cognitive training decreases mental workload during working memory precision task - A preliminary fNIRS studyInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103206184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103206
Chen MSolovey ESmith G(2023)Impact of BCI-Informed Visual Effect Adaptation in a Walking SimulatorProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582448(1-8)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3582448
Pöhlmann KMaior HFöcker JO'Hare LParke ALadowska ADickinson P(2023)I think I don’t feel sick: Exploring the Relationship Between Cognitive Demand and Cybersickness in Virtual Reality using fNIRSProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581063(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581063
Pečečnik K(2023)Situational awareness in the automotive domain - Effects on driving performance and driver's behavior2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)10.1109/ICECCME57830.2023.10253014(1-5)Online publication date: 19-Jul-2023
https://doi.org/10.1109/ICECCME57830.2023.10253014
Hirshfield LWickens CDoherty ESpencer CWilliams THayne L(2023)Toward Workload-Based Adaptive Automation: The Utility of fNIRS for Measuring Load in Multiple Resources in the BrainInternational Journal of Human–Computer Interaction10.1080/10447318.2023.226624240:22(7404-7430)Online publication date: 23-Oct-2023
https://doi.org/10.1080/10447318.2023.2266242
Howell-Munson AUnal DMowad TArrington CWalker ESolovey E(2023)Classification of Brain Signals Collected During a Rule Learning ParadigmArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky10.1007/978-3-031-36336-8_97(625-631)Online publication date: 30-Jun-2023
https://doi.org/10.1007/978-3-031-36336-8_97
Jiang WYu DWang CSarsenbayeva Zvan Berkel NGoncalves JKostakos V(2022)Near-infrared Imaging for Information Embedding and Extraction with Layered StructuresACM Transactions on Graphics10.1145/353342642:1(1-26)Online publication date: 12-Aug-2022
https://dl.acm.org/doi/10.1145/3533426
Howell-Munson ADoherty EGavriel PNicolas CNorton ANeamtu RYanco HWu YSolovey E(2022)Towards Brain Metrics for Improving Multi-Agent Adaptive Human-Robot Collaboration: A Preliminary StudyProceedings of the 1st Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3533406.3533419(1-10)Online publication date: 8-Jun-2022
https://dl.acm.org/doi/10.1145/3533406.3533419
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents