research-article

Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment

Authors:

Lars NybergAuthors Info & Claims

CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 869 - 878

https://doi.org/10.1145/1753326.1753454

Published: 10 April 2010 Publication History

Get Access

Abstract

The combination of virtual reality (VR) and brain measurements is a promising development of HCI, but the maturation of this paradigm requires more knowledge about how brain activity is influenced by parameters of VR applications. To this end we investigate the influence of two prominent VR parameters, 3d-motion and interactivity, while brain activity is measured for a mental rotation task, using functional MRI (fMRI). A mental rotation network of brain areas is identified, matching previous results. The addition of interactivity increases the activation in core areas of this network, with more profound effects in frontal and preparatory motor areas. The increases from 3d-motion are restricted to primarily visual areas. We relate these effects to emerging theories of cognition and potential applications for brain-computer interfaces (BCIs). Our results demonstrate one way to provoke increased activity in task-relevant areas, making it easier to detect and use for adaptation and development of HCI.

References

[1]

Aguirre, G.K., Detre, J.A., Alsop, D.C., and D'Esposito, M. The parahippocampus subserves topographical learning in man. Cereb. Cortex 6, 6 (1996), 823--829.

Crossref

Google Scholar

[2]

Backman, A. Colosseum3D-Authoring Framework for Virtual Environments. In Proc. EUROGRAPHICS Workshop IPT & EGVE Workshop, (2005), 225--226.

Digital Library

Google Scholar

[3]

Barsalou, L.W. Grounded Cognition. Annual Review of Psychology 59, 1 (2008), 617--645.

Crossref

Google Scholar

[4]

Beck, L., Wolter, M., Mungard, N., Kuhlen, T., and Sturm, W. Combining Virtual Reality and Functional Magnetic Resonance Imaging (fMRI): Problems and Solutions. In HCI and Usability for Medicine and Health Care. 2007, 335--348.

Digital Library

Google Scholar

[5]

Blanke, O. and Arzy, S. The Out-of-Body Experience: Disturbed Self-Processing at the Temporo-Parietal Junction. Neuroscientist 11, 1 (2005), 16--24.

Crossref

Google Scholar

[6]

Born, R.T. and Bradley, D.C. Structure and Function of Visual Area MT. Annual Review of Neuroscience 28, 1 (2005), 157--189.

Crossref

Google Scholar

[7]

Christou, G., Law, E.L., Green, W., and Hornbaek, K. Challenges in evaluating usability and user experience of reality-based interaction. In Proc. CHI 2009 Extended Abstracts, ACM (2009), 4811--4814.

Digital Library

Google Scholar

[8]

Cohen, M.S., Kosslyn, S.M., Breiter, H.C., et al. Changes in cortical activity during mental rotation A mapping study using functional MRI. Brain 119, 1 (1996), 89--100.

Crossref

Google Scholar

[9]

Constantinidis, C. and Wang, X. A neural circuit basis for spatial working memory. The Neuroscientist: A Review Journal Bringing Neurobiology, Neurology and Psychiatry 10, 6 (2004), 553--65.

Google Scholar

[10]

Curtis, C.E. and D'Esposito, M. Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences 7, 9 (2003), 415--423.

Crossref

Google Scholar

[11]

Cutrell, E. and Tan, D.S. BCI for passive input in HCI. In Proc. CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games, (2008).

Digital Library

Google Scholar

[12]

Decety, J. and Jeannerod, M. Mentally simulated movements in virtual reality: does Fitt's law hold in motor imagery? Behavioural Brain Research 72, 1-2 (1995), 127--134.

Crossref

Google Scholar

[13]

Dolan, R. Neuroimaging of Cognition: Past, Present, and Future. Neuron 60, 3 (2008), 496--502.

Crossref

Google Scholar

[14]

Ericsson, K.A., Prietula, M.J., and Cokely, E.T. The making of an expert. Harvard business review 85, 7/8 (2007), 114.

Google Scholar

[15]

Eriksson, J., Larsson, A., and Nyberg, L. Item-specific Training Reduces Prefrontal Cortical Involvement in Perceptual Awareness. Journal of Cognitive Neuroscience 20, 10 (2008), 1777--1787.

Digital Library

Google Scholar

[16]

Farrer, C. and Frith, C.D. Experiencing Oneself vs Another Person as Being the Cause of an Action: The Neural Correlates of the Experience of Agency. NeuroImage 15, 3 (2002), 596--603.

Crossref

Google Scholar

[17]

Fitts, P.M. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 6 (1954), 381--391.

Crossref

Google Scholar

[18]

Friston, K. and Stephan, K. Free-energy and the brain. Synthese 159, 3 (2007), 417--458.

Crossref

Google Scholar

[19]

Girouard, A., Hirshfield, L.M., Solovey, E., and Jacob, R.J.K. Using functional Near-Infrared Spectroscopy in HCI: Toward evaluation methods and adaptive interfaces. In Proc. CHI 2008 Workshop on Brain-Computer Interfaces for HCI and Games, (2008).

Google Scholar

[20]

Girouard, A. Adaptive brain--computer interface. In Proc. CHI 2009 Extended Abstracts, ACM Press (2009), 3097--3100.

Digital Library

Google Scholar

[21]

Hawkins, J. On Intelligence. Owl Books, 2005.

Digital Library

Google Scholar

[22]

Hawkins, J. and George, D. Hierarchical Temporal Memory. Numenta Inc., 2006.

Google Scholar

[23]

Hirshfield, L.M., Solovey, E.T., Girouard, A., et al. Using Brain Measurement to Evaluate Reality Based Interactions. In Proc. CHI 2009 Workshop on Challenges in the Evaluation of Usability and User Experience in Reality Based Interaction, (2009), 19.

Google Scholar

[24]

Hirshfield, L.M., Solovey, E.T., Girouard, A., et al. Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In Proc. CHI 2009, ACM Press (2009), 2185--2194.

Digital Library

Google Scholar

[25]

Jacob, R.J.K., Girouard, A., Hirshfield, L.M., et al. Reality-based interaction: a framework for post-WIMP interfaces. In Proc. CHI 2008, ACM Press (2008), 201--210.

Digital Library

Google Scholar

[26]

Jäncke, L. Virtual reality and the role of the prefrontal cortex in adults and children. Frontiers in Neuroscience 3, 1 (2009).

Crossref

Google Scholar

[27]

Kilner, J.M., Friston, K.J., and Frith, C.D. Predictive coding: an account of the mirror neuron system. Cognitive Processing 8, 3 (2007), 159--66.

Crossref

Google Scholar

[28]

Kosslyn, S.M., Ganis, G., and Thompson, W.L. Neural foundations of imagery. Nat Rev Neurosci 2, 9 (2001), 635--642.

Crossref

Google Scholar

[29]

Lécuyer, A., Lotte, F., Reilly, R.B., Leeb, R., Hirose, M., and Slater, M. Brain--Computer Interfaces, Virtual Reality, and Videogames. Computer 41, 10 (2008), 66--72.

Digital Library

Google Scholar

[30]

Minnery, B.S. and Fine, M.S. Neuroscience and the future of human-computer interaction. Interactions 16, 2 (2009), 70--75.

Digital Library

Google Scholar

[31]

Mourao-Miranda, J., Ecker, C., Sato, J.R., and Brammer, M. Dynamic Changes in the Mental Rotation Network Revealed by Pattern Recognition Analysis of fMRI Data. Journal of Cognitive Neuroscience 21, 5 (2009), 890--904.

Digital Library

Google Scholar

[32]

Parasuraman, R. and Wilson, G.F. Putting the brain to work: neuroergonomics past, present, and future. Human Factors 50, 3 (2008), 468--474.

Crossref

Google Scholar

[33]

Postma, A. and Barsalou, L.W. Spatial working memory and imagery: From eye movements to grounded cognition. Acta Psychologica 118, 2--3 (2009).

Google Scholar

[34]

Rizzo, A.A., Schultheis, M., Kerns, K.A., and Mateer, C. Analysis of assets for virtual reality applications in neuropsychology. Neuropsychological Rehabilitation 14, 1 (2004), 207--239.

Crossref

Google Scholar

[35]

Schacter, D.L., Addis, D.R., and Buckner, R.L. Remembering the past to imagine the future: the prospective brain. Nat Rev Neurosci 8, 9 (2007), 657--661.

Crossref

Google Scholar

[36]

Shepard, R.N. and Metzler, J. Mental Rotation of Three-Dimensional Objects. Science 171, 3972 (1971), 701--703.

Google Scholar

[37]

Sjölie, D., Bodin, K., Eriksson, J., and Janlert, L. Using brain imaging to assess interaction in immersive VR. In Proc. CHI 2009 Workshop on Challenges in the Evaluation of Usability and User Experience in Reality Based Interaction, (2009), 23.

Google Scholar

[38]

Sweetser, P. and Wyeth, P. GameFlow: a model for evaluating player enjoyment in games. Comput. Entertain. 3, 3 (2005), 3--3.

Digital Library

Google Scholar

[39]

Tagaris, G.A., Kim, S., Strupp, J.P., Andersen, P., U?urbil, K., and Georgopoulos, A.P. Mental Rotation Studied by Functional Magnetic Resonance Imaging at High Field (4 Tesla): Performance and Cortical Activation. Journal of Cognitive Neuroscience 9, 4 (1997), 419--432.

Digital Library

Google Scholar

Cited By

View all

Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Romero RCecotti H(2023)A Multimodal Document Viewer in Fully Immersive Virtual RealityImmersive Learning Research Network10.1007/978-3-031-47328-9_23(300-310)Online publication date: 31-Oct-2023
https://doi.org/10.1007/978-3-031-47328-9_23
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Show More Cited By

Index Terms

Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

Virtual reality with fMRI: a breakthrough cognitive treatment tool

The impact of virtual reality (VR) has been felt in a wide range of fields over the past 10 years. VR has been shown to be an effective treatment for anxiety, phobia, pain, posttraumatic stress disorder, stress inoculation training, and drug and alcohol ...
Effects of electrical brain stimulation on brain indices and presence experience in immersive, interactive virtual reality
Abstract
The subjective presence experience in virtual reality (VR) is associated with distinct brain activation patterns. Particularly, the dorsolateral prefrontal cortex (DLPFC) seems to play a central role. We investigated the effects of electric brain ...
Virtual reality with fMRI: a breakthrough cognitive treatment tool

The impact of virtual reality (VR) has been felt in a wide range of fields over the past 10 years. VR has been shown to be an effective treatment for anxiety, phobia, pain, posttraumatic stress disorder, stress inoculation training, and drug and alcohol ...

Reviews

Reviewer: Jeanine M. Meyer

Sj?lie et al. describe in this paper the examination of brain scans produced by functional magnetic resonance imaging (fMRI) on subjects solving puzzles. They discuss their work within the context of building virtual reality applications, prior studies using brain scanning, and theories of how the brain operates. While in the fMRI unit, each subject is fitted with glasses and shown a computer-generated three-dimensional (3D) scene. The task is to determine if two sets of connected blocks represent the same or different physical objects-that is, if one object can be rotated and translated to match the other. The puzzle is presented in three different ways: the still condition displays a static picture, the auto condition has the objects rotated by the application, and the interactive condition provides buttons for the subject to rotate the objects. For each of these three conditions, the puzzle is presented using standard and stereo projections. Puzzles are presented in random order and the time between tasks is used as the baseline. The scans reveal what part of the brain is active during each task. The pictures produced by the fMRI are encoded in such a way as to determine if and how scans made under different conditions and of different subjects are distinct. The analysis omits the cases when a subject made a mistake. Sj?lie et al. describe the encoding and the statistical analysis involved in making these determinations, but more explanation would have been helpful. The authors report that there were no significant effects for different conditions; this was confirmed by post-scan interviews-after all, mental rotation probably occurs for all three conditions. They do imply that the scans for all ten subjects were similar enough to make generalizations. They also note that their findings were consistent with prior work on 3D rotation tasks. The strongest effect appears to be the effect of interactivity. Anyone who builds virtual reality systems or is interested in concepts of cognitive theories such as presence, prediction, and mental workload will benefit from a careful study of this paper and by reading the many articles referenced. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2010

2690 pages

ISBN:9781605589299

DOI:10.1145/1753326

General Chair:
Elizabeth Mynatt
Georgia Institute of Technology
,
Program Chairs:
Geraldine Fitzpatrick
Vienna University of Technology
,
Scott Hudson
Carnegie Mellon University
,
Keith Edwards
Georgia Tech
,
Tom Rodden
University of Nottingham

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 April 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '10

Sponsor:

SIGCHI

CHI '10: CHI Conference on Human Factors in Computing Systems

April 10 - 15, 2010

Georgia, Atlanta, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
895
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Romero RCecotti H(2023)A Multimodal Document Viewer in Fully Immersive Virtual RealityImmersive Learning Research Network10.1007/978-3-031-47328-9_23(300-310)Online publication date: 31-Oct-2023
https://doi.org/10.1007/978-3-031-47328-9_23
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Ho SLiu PPalombo DHandy TKrebs C(2022)The role of spatial ability in mixed reality learning with the HoloLensAnatomical Sciences Education10.1002/ase.214615:6(1074-1085)Online publication date: 20-Jan-2022
https://doi.org/10.1002/ase.2146
Arroyo-Ferrer ANoreña DSerrano JRíos-Lago MRomero J(2021)Cognitive rehabilitation in a case of traumatic brain injury using EEG-based neurofeedback in comparison to conventional methodsJournal of Integrative Neuroscience10.31083/j.jin200204720:2Online publication date: 30-Jun-2021
https://doi.org/10.31083/j.jin2002047
Baceviciute SMottelson ATerkildsen TMakransky GBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Investigating Representation of Text and Audio in Educational VR using Learning Outcomes and EEGProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376872(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376872
Girouard AShaer OSolovey EPoor GJacob R(2019)The Reality of Reality-Based InteractionACM Transactions on Computer-Human Interaction10.1145/331961726:5(1-35)Online publication date: 12-Sep-2019
https://dl.acm.org/doi/10.1145/3319617
Hoe ZLee IChen CChang K(2019)Using an augmented reality-based training system to promote spatial visualization ability for the elderlyUniversal Access in the Information Society10.1007/s10209-017-0597-x18:2(327-342)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s10209-017-0597-x
Thanh Vi CHornbæk KSubramanian SGajos KMankoff JHarrison C(2017)Neuroanatomical Correlates of Perceived UsabilityProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology10.1145/3126594.3126657(519-532)Online publication date: 20-Oct-2017
https://dl.acm.org/doi/10.1145/3126594.3126657
Vourvopoulos ABadia SSeigneur JHernandez-Munoz JMcCullagh PSchmidt ATerada TWoo WMistry P(2016)Usability and Cost-effectiveness in Brain-Computer InteractionProceedings of the 7th Augmented Human International Conference 201610.1145/2875194.2875244(1-8)Online publication date: 25-Feb-2016
https://dl.acm.org/doi/10.1145/2875194.2875244
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Virtual reality with fMRI: a breakthrough cognitive treatment tool

Effects of electrical brain stimulation on brain indices and presence experience in immersive, interactive virtual reality

Virtual reality with fMRI: a breakthrough cognitive treatment tool

Reviews

Access critical reviews of Computing literature here