Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Why Are Other Teachers More Inclusive in Online Learning Than Us? Exploring Challenges Faced by Teachers of Blind and Visually Impaired Students: A Literature Review
Next Article in Special Issue
Building an Accessible and Flexible Multi-User Robotic Simulation Framework with Unity-MATLAB Bridge
Previous Article in Journal
Application of Deep Learning for Heart Attack Prediction with Explainable Artificial Intelligence
Previous Article in Special Issue
Metaverse Unveiled: From the Lens of Science to Common People Perspective
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Computers
Volume 13
Issue 10
10.3390/computers13100246
computers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments

by
Guillermo Bernal
1,*,
Hahrin Jung
1,
İsmail Emir Yassı
2,
Nelson Hidalgo
1,
Yodahe Alemu
1,
Tyler Barnes-Diana
3 and
Pattie Maes
1
1
Massachusetts Institute of Technology, Cambridge, MA 02139, USA
2
School of Medicine, Bursa Uludag University, 16059 Bursa, Turkey
3
Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI 02912, USA
*
Author to whom correspondence should be addressed.
Computers 2024, 13(10), 246; https://doi.org/10.3390/computers13100246
Submission received: 19 August 2024 / Revised: 16 September 2024 / Accepted: 18 September 2024 / Published: 26 September 2024
(This article belongs to the Special Issue Extended or Mixed Reality (AR + VR): Technology and Applications)
Browse Figures
Figure 1
<p><b>Left</b>: Plot of live physiological data stream from Galea headset (left screen), TETRIS VR environment (right screen), User wearing VR headset. <b>Right</b>: Outline of experiment procedure. For each of the two sessions, participants completed two pre-experiment questionnaires. Then, a headset fitting session was conducted. Once the headset was fitted, calibration data was collected (regular and eyes-closed). After a brief tutorial session, participants played either the intervention or control version of TETRIS. After completion of the game, the participant filled out two post-experiment questionnaires, and was debriefed on the session.</p> "> Figure 2
<p>Figures show the significant deviations from baseline for EEG and PPG signals before and after the helper event. (<b>a</b>) Frontal Theta power collapsed for electrodes Fp1, Fp2, Fz. (<b>b</b>) Parietal Theta power collapsed for electrodes POz, PO3, PO4. (<b>c</b>) Frontal Beta power collapsed for electrodes Fp1, Fp2, Fz. (<b>d</b>) Parietal Beta power collapsed for electrodes POz, PO3, PO4 (<b>e</b>) Frontal alpha power collapsed for electrodes Fp1, Fp2, Fz. (<b>f</b>) Parietal alpha power collapsed for electrodes POz, PO3, PO4. (<b>g</b>) Heart rate, in beats per minute, deviation from baseline. (<b>h</b>) RMSSD, a measure of HRV, deviation from baseline.</p> "> Figure 2 Cont.
<p>Figures show the significant deviations from baseline for EEG and PPG signals before and after the helper event. (<b>a</b>) Frontal Theta power collapsed for electrodes Fp1, Fp2, Fz. (<b>b</b>) Parietal Theta power collapsed for electrodes POz, PO3, PO4. (<b>c</b>) Frontal Beta power collapsed for electrodes Fp1, Fp2, Fz. (<b>d</b>) Parietal Beta power collapsed for electrodes POz, PO3, PO4 (<b>e</b>) Frontal alpha power collapsed for electrodes Fp1, Fp2, Fz. (<b>f</b>) Parietal alpha power collapsed for electrodes POz, PO3, PO4. (<b>g</b>) Heart rate, in beats per minute, deviation from baseline. (<b>h</b>) RMSSD, a measure of HRV, deviation from baseline.</p> "> Figure 3
<p>Estimated vs observed performance improvement—Estimated performance improvement is shown on the x-axis and observed performance improvement on the y-axis. Performance improvement is estimated as a linear combination of selected physiological signals that significantly deviate from baseline activity following the appearance of a helper during gameplay.</p> "> Figure 4
<p>Estimated vs observed questionnaire responses - For each of the four questionnaire responses regarding the helper, estimated vs observed questionnaire responses are displayed. Questions 1, 2 and 4 showed statistically significant (<span class="html-italic">p</span> &gt; 0.05) models.</p> ">
Versions Notes

Abstract

:
Mental workload, visuospatial processes and autonomic nervous system (ANS) activity are highly intertwined phenomena crucial for achieving optimal performance and improved mental health. Virtual reality (VR) serves as an effective tool for creating variety of controlled environments to better probe these features. This study investigates the relationship between mental and visuospatial workload, physiological arousal, and performance during a high-demand task in a VR environment. We utilized a modified version of the popular computer game TETRIS as the task, involving 25 participants, and employed a physiological computing VR headset that simultaneously records multimodal physiological data. Our findings indicate a broadband increase in EEG power just prior to a helper event, followed by a spike of visuospatial engagement (parietal alpha and beta 0-1-3 s) occurring concurrently with a decrease in mental workload (frontal theta 2–4 s), and subsequent decreases in visuospatial engagement (parietal theta at 14 s) and physiological arousal (HRV at 20 s). Regression analysis indicated that the subjective relief and helpfulness of the helper intervention was primarily driven by a decrease in physiological arousal and an increase in visuospatial engagement. These findings highlight the importance of multimodal physiological recording in rich environments, such as real world scenarios and VR, to understand the interplay between the various physiological responses involved in mental and visuospatial workload.

1. Introduction

Mental and visuospatial workloads are among the key drivers in human cognitive processes [1,2,3,4]. Studies in this field have particularly focused on the prefrontal and parietal cortices. The prefrontal cortex plays important roles in executive functions, such as the capacity for self-control and short and long-term planning, which are widely regarded as among the most crucial aspects of the human mind [5,6]. The parietal cortex, on the other hand, is regarded to be critically important in visuospatial information processing and working memory, and in the control of behavioral responses [7,8,9]. Interconnected systems of visuospatial working memory and attention are also critical for human interaction with rich stimulus environments. These systems are mediated by the frontoparietal network, though the degree to which they are dissociable is debated, with both working memory and attention is associated with activity in both the prefrontal cortex and the posterior parietal [10]. In addition, the ANS and the interaction between its two parts, the sympathetic and parasympathetic systems, are involved in top-down and bottom-up directions of mental workload [11,12]. Maintaining a healthy equilibrium in the operation of these systems can help people achieve optimal performance, better mental health, and improved well-being [13,14,15].
Studies focused on detecting and quantifying mental workload have increasingly used physiological measures, which have various advantages over using only self-reported measures, as the latter are not optimal in continuous assessment and are prone to retrospective reconstruction of the perceived mental workload [16]. Common physiological measures of workload and engagement include electroencephalography (EEG) recordings from the parietal and prefrontal regions of the brain [17,18] and measures of cardiac activity reflecting the state of ANS [19]. Although conflicting reports exist (for a recent review, see [20]), most studied EEG indices include increased frontal theta [21] and beta [22], increased parietal beta [23] and theta [24], and decreased parietal alpha [25,26] band powers. As a complementary modality to EEG, functional Near-Infrared Spectroscopy (fNIRS) studies also have shown similar findings, with increased prefrontal activation during elevated mental workload [27]. Cardiac indices for increased mental workload and physiological arousal include increased heart rate (HR) and decreased heart rate variability (HRV) [28,29]. As discussed in previous research, while these measures provide valuable insights on their own, using a multimodal approach can provide more robust representations of mental workload, as each signal modality represents different facets of mental workload [30,31].
While measuring physiological markers is critical in mental workload studies, the experimental environment and behavioral response elicited in participants are equally important. In behavioural sciences, researchers have studied virtual reality (VR) experiment environments as both an exciting focus of study [32,33] and a tool for designing realistic experiments [34,35]. VR is a valuable platform for studying participants’ behaviors as it is easy to create any number of environments. These environments can be designed to be safe, controllable, replicable, and accessible to participants from a variety of backgrounds and with a variety of needs [36,37]. Most importantly, VR is capable of efficiently generating realistic new environments that are a powerful tool for psychological researchers [38,39,40]. There have been important advancements in using VR as an experimental platform to study mental workload. Gupta et al. [41] tracked multiple signal modalities (EEG, HRV and Electrodermal Activity (EDA)) and subjective questionnaires to investigate the trust level towards an auditory virtual agent under different mental workload levels. In another study, Zhang et al. [42] used multimodal physiological data (EEG, EDA, Electromyography (EMG) and skin temperature) in a VR-based driving system to track the mental workload of users with Autism Spectrum Disorder, aiming to build a system that facilitates learning by adjusting task difficulty. Utilizing real-time EEG data in a VR environment, Dey et al. [43] presented an adaptive training system that can adjust task difficulty to an optimal challenging level to facilitate learning. The reader is further referred to the recent review by Hadjiaros et al. [44]. While these recent studies provide valuable insights in terms of mental workload corresponding to the prefrontal cortex activity, little attention has been given to visuospatial processing in the literature. Furthermore, the temporal dynamics of the central and autonomic nervous system activity in the context of mental workload in VR environments requires further research.
The study presented here aims to explore the relationship between visual-spatial cognitive demands, prefrontal cognitive workload, and the autonomic nervous system during a high-demand task in virtual reality. The popular computer game TETRIS was modified and used as the task in the experiment, which involved 25 participants. We used Galea [45], a physiological computing VR headset that simultaneously records physiological data (EEG, EDA, EMG, and PPG). We study the dynamics of the autonomic and central nervous systems, with a focus on the activity of visuospatial workload in the parietal region and the balance between the sympathetic and parasympathetic responses. To better probe these temporal characteristics, we have added a new feature to the classic game of TETRIS called “helper event” that introduces a swift change in task difficulty. Once the stack of pieces reaches 60% of the height of the playing field, a ball-shaped helper piece appears. It can be played like any other tetromino piece, and when placed, it clears the four rows of squares underneath it and sets the game level to 5, slowing down the speed at which the pieces fall. The findings of the study highlight the importance of cognitive and visuospatial workload in subjective experience and tension relief, as well as the significance of maintaining optimum mental workload for maximizing performance and balanced autonomic nervous system activity. This exploratory study contributes to the growing understanding of the relationship between cognitive load, visuospatial engagement, the autonomic nervous system, and performance.
The study hypothesizes a decrease in cognitive load, visuospatial engagement, and physiological arousal following the helper event, which suddenly makes the game easier. The study also hypothesizes that the physiological signals examined would be correlated with subjective questionnaire responses about the impact of the helper event.
Overall, this study aims to explore the relationship between the central and autonomic nervous system phenomena and cognitive state, particularly in the context of the helper clear event in a highly demanding scenario.

2. Methods

2.1. Participants

A total of 34 participants (15 male, 16 female, three non-binary; ages 18 to 44) participated in this study. Inclusion criteria were as follows: normal or corrected-to-normal vision; no known history of epileptic seizures, migraines, or vestibular dysfunction; no known history of claustrophobic events when using VR headsets; ability to set and follow instructions for 30 min. Participants had varying levels of familiarity with VR and the game TETRIS. The data was collected from 34 total participants, but 5 subjects were excluded based on performance criteria, 3 were excluded on the basis of channel data quality and 1 subject was excluded based on outlier timing. Ultimately, 25 subjects were available for data analysis.

2.2. Experimental Protocol

The experiment involved playing a modified version of the computer game TETRISTM in VR. The game implementation adhered to the original source code and was implemented in Unity3D (2020.3.11f1; Unity Software, Inc., San Fransico, CA, USA). The TETRIS game involves tetrominoes, which are geometric shapes made of four squares linked edge to edge in various arrangements, that descend one at a time vertically from the top of the computer screen. While a tetromino piece falls, the player can manipulate the piece (moving it left, right and rotating). The goal of TETRIS is to arrange the pieces so that complete horizontal rows are formed. When this is done, the squares in that row vanish, and players receive points.
In this experiment, participants used the provided controller to manipulate the pieces. The game also incremented in level every 10 s of the game. Each level would have a set speed at which the tetromino pieces fall, and this speed would increase logarithmically with the level of the game. This was done to produce an increase of difficulty throughout the game.
There were two sessions of the experiment: intervention and control. Both times, the TETRIS game followed the general rules described above. The intervention session had one additional helper condition. When the stack of pieces reach 60% of the height of the playing field, a ball shaped helper condition appears. This helper piece can be played like any other tetromino piece, and once it is placed, it immediately clears the four rows of squares underneath it and sets the game level to 5, decreasing the speed at which the pieces fall. To reduce possible confusion, researchers explained to each participant before beginning the intervention session that there would be a new piece appearing at some point during the game, that this place can be manipulated like any other tetromino, and that this piece will clear four rows once it is placed. The control condition did not include the helper intervention and had no other variations from the regular rules of TETRIS.
Each participant completed both the intervention and control conditions. The order in which each participant completed the intervention and control conditions was randomized to minimize the effects of novelty from the controllers and test environment on cognitive load from the experimental conditions.

2.3. Procedure

For both intervention and control sessions, the experiment had the following procedure as outlined in Figure 1, right diagram. First, participants filled out two pre-experiment questionnaires. Second, participants were met with a headset fitting and signal impedance check. Third, a 50-s calibration occurred to remove motion artifacts in subsequent data using Neuropype (see Data Preprocessing). Fourth, there was a second, 30-s-long calibration with eyes closed, during which baseline alpha signals were collected. Fifth, after calibration, participants played a tutorial of the TETRIS game for 30 s. This was done to help reduce the effects that new controllers and unfamiliarity with the game may have on mental workload. Sixth, participants played a game of TETRIS with either the intervention or the control condition. Seventh, after the game, participants were asked to fill out a post-experiment questionnaire. Lastly, participants were debriefed on the experiment and could provide feedback if needed.

2.4. Apparatus

The Galea headset was used to present VR content and to collect physiological data [45]. The VR content was presented on the Valve Index Headset (Valve Corporation, Bellevue, WA, USA). Participants used the Valvue Index Controllers. Electroencephalography (EEG) was collected with a sampling rate of 250 Hz from dry electrodes placed according to the international 10–20 system at Fp1, Fp2, Fz, CPz, POz, PO3, PO4, Oz, O1, and O2. Headset fitting sessions were conducted prior to the start of the experiment to ensure that impedance from EEG electrodes were under 10 uVrms. Photoplethysmography (PPG) was collected using a particle and proximity sensor on the forehead
The Galea headset generates a stream of physiological measurements. Simultaneously, Unity3D creates a stream on which event markers from game events are written. In particular, the moments of interest were (1) start and end of calibration, (2) start and end of tutorial, (3) start and end of game, (4) when stack of tetromino pieces reached 60% of height of playing field. Additional event markers were sent during the intervention condition when the helper piece appeared at the top of the screen and when the helper piece cleared rows. The streams from Galea and Unity3D were merged using Lab Streaming Layer (LSL) [46].

3. Data Preprocessing

3.1. EEG

Collected EEG data were preprocessed by Neuropype (Syntrogi Inc., La Jolla, CA, USA). The stream generated by the Galea headset was inputted to the Neuropype pipeline with deterministic timing option to ensure replicability of the offline data stream. Then, a 1–35 Hz finite impulse response (FIR) bandpass filter was applied. For artifact removal, data from the 50-s calibration was taken as a baseline, and artifacts were removed with a noise amplitude threshold of 4 standard deviations and a sliding window of 1 s. This threshold was determined for our settings by visually inspecting the resultant timeseries for noise, such as muscle artifacts, eye blinks and eye movements. Resultant time series was then exported for further analysis. Additionally, artifact regression node was utilized for EEG data with heart rate pulsation artifacts. Through this node, PPG channels were labeled as artifact channels, as they had similar signal waveforms with pulsation artifacts, and the heart beat artifacts were subtracted using recursive least squares method.

3.2. PPG

Collected PPG data were preprocessed using “peakdet”, a Python-based peak detection toolbox [47]. First, a 0.7–2.0 Hz bandpass filter was applied to the PPG channels, and an aggressive automatic peak detection was run on the data. Then, the detected peaks were manually inspected, so that faulty peaks are deselected and undetected peaks are selected.

4. Measurements

4.1. EEG

For each subject and for each of the target electrodes, power was calculated for each of the Alpha (8–13 Hz), Beta (15–30 Hz), and Theta (4–8 Hz) frequency bands for each second using 75% overlapping 4 s windows. The MATLAB ‘bandpower’ function was used to calculate the average power in the target frequency range (R2022a, The MathWorks Inc., Natick, MA, USA).

4.2. PPG

From the pre-processed PPG data, beats per minute (BPM) was calculated via a simple average for each second using 90% overlapping 10-s sliding windows. Heart rate variability (HRV) was measured using Root Mean Square of Successive Differences (RMSSD), again for each second using 90% overlapping 10-s sliding windows. RMSSD was utilized to account for the relatively short time window [48].

4.3. Baseline

The last s seconds of data from the beginning of the game to the first helper event (stack reached 60% of the height of the playing field) was taken, where s is the smallest number of available seconds across all subjects. From this data, baseline measurements of the means and standard deviation were taken for both power and PPG data for each subject and each electrode.

5. Analysis

5.1. Normalization

Given that power values are bounded (at 0) the alpha power data at any given second was highly non-normal, with a rightward skew. As such a log transform was performed to normalize the data. Thus for each data point for each subject the log of the alpha power was taken. This allows us to use statistics that assume normally distributed data for subsequent analysis.
x n o r m = l o g ( x )

5.2. Standardization

Given that the channel signal for each electrode/subject, as well as the corresponding power signals, vary based on factors beyond differences in underlying neural activity (e.g., signal impedance) the power signal was standardized for each electrode for each subject. Such standardization are often called “z-scores”.
z = ( x − x ¯ b a s e ) s b a s e
The standardized data was used for subsequent analysis. This effectively weights the contribution of each electrode for each subject’s power signal equally, preventing large variations due to signal quality across subjects/electrodes from distorting the group level data.

5.3. Time-Series Analysis

Power and PPG data were calculated, then examined over time at the group level. Specifically, once all subjects’ data were standardized and normalized, the data were centered over the game event of the helper piece dropping and clearing rows. This was done by simply aligning all time series for each helper event. Then, data were collapsed across electrodes (CAE) per region by simple averages of prefrontal and midline frontal (Fp1, Fp2, Fz), and parieto-occipital (PO3, PO4, POz) electrodes. Subsequent analysis was done for each second within the analysis window starting at the helper clear event to the end of the game, which varied for each subject. For each second in the analysis window, a group level one sample t-test was done to detect deviations from the baseline. A p-value cutoff of 0.05 was used to determine a significant event.

5.4. Linear Regression Models

A variety of regression models were examined in this work, both to predict the behavioral performance improvement associated with the helper clear event, as well as to predict questionnaire responses regarding the helper clear event collected after gameplay. Performance, in our setting, was operationalized as the gameplay time in a given block, namely baseline and post helper timeframes.
For each investigated question, linear models were iteratively created and tested. Possible predictors were selected from the significant group level deviations observed in the time series analysis. Of the group level deviations from baseline activity the deviation with the most significant (lowest p-value) correlation between the subject’s EEG/PPG data and the variable to be predicted was used as the first predictor variable for a linear model. Then the next most significant correlation between subject EEG/PPG data and the questionnaire was added to the model. This iteratively continued until the p-value of the corresponding linear model began to increase as additional predictors were added. This iterative approach, rather than an exploratory approach searching a broad combination of predictor variables, both provides a simple means of model comparison as well as helps control type II statistical error.

6. Results

6.1. Performance Improvement

Baseline performance did not deviate significantly from normality in either session (session 1: p = 0.8604; session 2: p = 0.7518), and no significant difference was observed in baseline performance between sessions (p = 0.5975). For the session containing post-helper-clear, performance did not deviate from normality (session 1: p = 0.1749). In contrast, post-helper-clear event performance did significantly deviate from normality when no helper occurred (session 2: p = 0.0056). Although there was no helper piece appearing, the event was marked anyways, using the criteria explained in Experimental Protocol. This deviation may reflect a low bound of possible time played scores. A rank sum test identified a significant difference in post-helper-clear event performance across groups (p = 2.1982 × 10−6), with better performance occurring in the group that received the helper clear.

6.2. Time-Series Analysis

This section presents the time series analysis results for both EEG power and PPG data. Table 1 highlights the significant events identified for each data stream and corresponding p-values. For all data streams, the time series analysis are graphically depicted in Figure 2 with the x-axis representing time and the y-axis representing normalized and standardized group data. Band power results are shown in a collection of prefrontal and midline frontal [Fp1 Fp2 Fz] and parieto-occipital [PO3 PO4 POz] electrodes. As described in the methods section, CAE power data in the corresponding region of interest is reported with respect to a baseline. Time series analysis for the PPG data is also shown below for both heart rate as well as heart rate variability measured in terms of RMSSD. Positive values mean increases in the corresponding measure relative to baseline and negative values indicate decreases relative to the baseline measurement. The dark blue line represents the estimated mean value, flanked by the the upper (yellow) and lower (light blue) bounds of a 95% confidence interval of the estimated mean. Significant events, as measured by single sample t-tests, are shown as vertical dotted lines in Figure 2 and reported in detail in the corresponding sections in Table 1.
Key events are summarized here, and detailed discussions can be found in the Discussion. The effects of the helper event (at t = 0) span multiple data streams over 20 s. Prior to the helper event, there was a notable increase in broadband EEG power, specifically in the frontal and parietal beta at 3 s, and in the frontal alpha at 4 s before the event, along with a positive trend across other power signals. Additionally, distinct changes were observed in parietal and frontal EEG activity, including a parietal beta increase from 1–3 s, a parietal alpha decrease from 0–1 s, and a frontal theta decrease from 2–4 s. Further post-event observations included a decrease in parietal theta at 14 s, an increase in HRV from 12–14 and at 20 s, and a decrease in heart rate at 7 s.

6.3. Predicting Performance Improvement

This section shows the results of a linear regression analysis predicting the magnitude of performance improvement using the significant deviations from baseline power and PPG activity that were observed in the time series analysis described above. A significant model ( F d f = 21 = 9.25; p = 0.00121) was obtained, with significant parietal predictors of performance improvement including parietal Beta activity at 3 s and parietal Theta activity at 5 s. The model has a high degree of predictive validity with an R 2 = 0.457, and an adjusted R 2 = 0.407.
y ^ = β 0 + β 1 x 1 + β 2 x 2
x 1 ≡ Parietal Beta at 3 s
x 2 ≡ Parietal Theta at 5 s
Table 2 shows the individual contributions of the estimated components.
Here we see that both parietal Beta and Theta power were significantly predictive of the performance improvement induced by the helper clear event, with the directionality of the effect of each predictor opposed. Low parietal Beta power and high parietal Theta power predicts the largest performance improvement. This directionality is inversely related to the overall effect of the deviations from baseline observed in both measures.
Figure 3 shows the observed performance improvement on the x-axis, and the predicted performance improvement, as calculated by our linear model, on the y-axis.
We conducted a post-experiment questionnaire to gather feedback from participants about the effectiveness of the helper condition in the game “the ball shaped piece in a game”, specifically its impact on the helper and helper clear event. We selected a subset of questions from the questionnaire and scored the responses using a standardized system of integers. We then created linear models based on a standardized equation to investigate the relationship between different variables and gain insights into the impact of the ball shaped piece on participants’ gameplay experience. These questions were: (1) Was the ball shaped piece helpful in the game? If yes, how helpful was it?; (2) Was the effect of the ball shaped piece relieving? If yes, how relieving was it?; (3) Did you notice that the game became slower immediately after the ball cleared some rows? If not, how much longer did it take for you to realize that the game was now slower?; (4) After [the] ball cleared some rows and slowed down the game, did you feel a tension relief in your body? If yes, to what degree did you feel this?
Questionnaire responses were scored, converting the selected options to integers. For questions 1, 2, & 4 responses were scored as: “Very” = 4, “Moderately” = 3, “Fairly” = 2, “Slightly” = 1, “Not at all” = 0. Question 3 was scored as: “I did not realize” = 4, “Within a couple moves” = 3, “Within one move” = 2, “Almost immediately” = 1, “Immediately” = 0.
For each investigated question, linear models based on the equation below were iteratively created and tested.
The results of the most statistically significant of the observed models are reported here for each of the four investigated questions. Figure 4 displays the observed and estimated questionnaire responses, as calculated by these linear models. Questions 1, 2 and 4 each had significant models, as measured both by the significance of the F-statistic corresponding to the linear model as well as the significance of the correlation coefficient between the estimated values using the linear model and the observed values. As a note, in the cases where the selected linear model used only a single predictor, the statistical significance of both the linear model and the correlation coefficient are equivalent. In the cases where multiple predictors are used the linear regression statistical results indicate the degree to which one or more of the predictors has a significant linear relationship with the questionnaire response. The correlation results are a more direct measure of the predictive validity of the model, indicating the extent and significance of the relationship between the predicted and observed values. The R 2 value indicates the percentage of variance in the observed data explained by the linear model of the predictor variables.

6.3.1. Question 1: Was the Ball Shaped Piece Helpful in the Game? If Yes, How Helpful Was It?

Our linear regression variables were x 1 ≡ PPG HRV at 20 s, and x 2 ≡ Parietal Beta at 3 s. Table 3 shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (3.65) was significant with a p-value (df = 22) = 0.0426. There was furthermore a significant correlation ( p = 0.011 ; R 2 = 0.2494 ) between the predicted and observed questionnaire response values.
Our findings indicate that the parietal Beta activity just following the helper clear event and the heart rate variability 20 s following the helper clear event are significantly related to subjective reports of how helpful the helper was. The correlation results indicate that 25% of the variance in subject questionnaire responses to question 1 is explained by a linear combination of the PPG HRV data 20 s after the helper clear event and parietal Beta activity 3 s after the helper clear event. While the individual predictor variables were not significant when individually tested (see table), the significance of the overall model indicates a significant effect. This may indicate a degree of correlation between the predictor variables, though a statistically significant correlation was not observed ( p > 0.05 ). Given that this model’s performance exceeded the performance of a model with only a single predictor variable, both terms likely contribute. The significant intercept term indicates, as expected, that the mean questionnaire responses differ from 0.

6.3.2. Question 2: “Was the Effect of the Ball Shaped Piece Relieving? If Yes, How Relieving Was It?”

In this case, our regression variables were x 1 ≡ Parietal Theta at 14 s, and x 2 ≡ Parietal Beta at 2 s. Table 3 below shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (5.39) was significant with a p-value (df = 22) = 0.0124. There was furthermore a significant correlation ( p = 0.0027 ; R 2 = 0.329 ).
Our findings indicate that the parietal Beta activity just following the helper, and parietal Theta 14 s following the helper, is significantly related to subjective reports of the relief experienced following the helper clear event. The correlation results indicate that 33% of the variance in the subject questionnaire responses can be explained by this linear model.

6.3.3. Question 3: Did You Notice That the Game Became Slower Immediately after the Ball Cleared Some Rows? If Not, How Much Longer Did It Take for You to Realize That the Game Was Now Slower?

Our regression variable in this case is x 1 ≡ Parietal Theta at 9 s. Table 3 shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (3.05) was not significant with a p-value (df = 22) = 0.0941. There was furthermore no significant correlation (p = 0.0953) between the predicted and observed questionnaire response values.
The intercept term, as expected, was significant. This indicates that the average questionnaire responses were greater than zero, though the insignificant finding of the linear model as a whole indicates there is no linear relationship between the investigated predictor variables and this particular questionnaire response.

6.3.4. Question 4: After the Ball Cleared Some Rows and Slowed Down the Game, Did You Feel a Tension Relief in Your Body? If Yes, to What Degree Did You Feel This?

The regression model variables were x 1 ≡ Frontal Beta at −3 s, and x 2 ≡ Parietal Beta at 3 s. Table 3 shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (5.36) was significant with a p-value (df = 22) = 0.0127. There was furthermore a significant correlation ( p = 0.0028 ; R 2 = 0.3276 ) between the predicted and observed questionnaire response values.
Our findings indicate that the frontal Beta activity just prior to the helper clear and parietal Beta activity just following are significantly related to subjective reports regarding the degree of tension relief subjects report following the helper clear event. The correlation results indicate that 33% of the variance in the subject questionnaire responses is explained by a linear combination of the Frontal Beta activity just prior and Parietal Beta just following the helper clear event.

7. Discussion

The present study investigated the impact of a helper event in VR Tetris gameplay on various cognitive and physiological measures. Following a helper intervention that cleared tiles, we observed group level decreases in mental and visuospatial workload (as measured by frontal and parietal oscillatory activity) and in physiological arousal (as measured by heart rate and heart rate variability). Furthermore, a significant benefit to performance was observed due to the helper clear event. Subsequent predictive analysis, however, indicates that performance improvement was not predicted by the significant decrease in frontal Theta, but rather by parietal activity commonly associated with visuospatial processes [49,50]. Although they were not specifically directed, processes such as visuospatial working memory and attention were constantly challenged during gameplay due to the nature of Tetris [51,52]. The present findings are consistent with a model of dissociable but interacting functions of the prefrontal and posterior parietal cortices. Our results suggest that performance in VR Tetris, a spatial task in a rich visual environment, is driven by parietal activity associated with visuospatial processing, rather than frontal activity. This dissociation may be due to the visuospatially rich quality of the task load [24]. This framework, and these results, highlights the importance of multiunit EEG recordings that cover a broad area of the scalp, thus networks of brain regions, when investigating VR environments using EEG. Frontal electrodes alone are not sufficient to capture the activity of the frontoparietal network that drives performance improvement in this task, and parietal electrodes may miss significant deviations in cognitive load.

7.1. Temporal Dynamics of Helper Response

The differences from baseline across data streams were largely focused on here in terms of those that contributed to substantial linear models of subjective questionnaire responses regarding the helper event, as well as those that provide specific insight in relation to our stated hypotheses.
The first event of note is an increase of both frontal and parietal beta power from baseline 3 s prior to the helper clear event, corresponding for most subjects to the time after the helper notification but before the clear event itself. This corresponds to an elevation across power signals. While significant deviations were observed in Frontal Alpha and Beta as well as Parietal Beta, there was a positive trend among all other power signals. This broadband increase in power from baseline may indicate a generalized increase in engagement, mental load, and fatigue [53], corresponding to an increase in mental and visuospatial workload [22,23] (frontal and parietal beta). This may reflect the high degree of task difficulty that subjects faced prior to the helper event occurring. The frontal beta at −3 s was a selected predictor variable with a positive coefficient in the linear model of questionnaire responses related to the subjective sensation of tension relief from the body, indicating the larger the positive deviation from baseline the more tension relief subjects reported. The tension relief may actually reflect the return to baseline rather than the elevated cognitive load itself. While this result differs from our hypothesis of a decrease in beta power following the helper clear event, the general pattern of a return to baseline from an elevated cognitive load preceding the helper clear event is consistent with the model underlying the hypothesis. In particular that the helper clear event itself reduces cognitive load.
The second event of note is a sustained increase in parietal beta from baseline 1–3 s following the helper clear. At 0–1 s there is additionally a significant decrease from parietal alpha. This is consistent with a spike in visual-spatial engagement following the helper clear [23,25,26]. This increase in engagement may reflect the subject’s re-evaluation of the game state. Subject’s parietal beta power in this period is one of the selected predictive variables for linear models for all three of significant linear models reported, regarding how helpful and relieving the helper was, as well as the degree to which subjects reported tension leaving their body. The subjective reporting of how helpful/relieving the helper was is significantly and robustly (across question wording) related to degree to which visuospatial engagement was devoted to re-evaluating the game state post helper clear event. At the same time, as predicted, was a large decrease in frontal theta power indicating a decrease in cognitive load [21]. While the decrease in frontal theta was significant, subjective reports of how helpful/relieving the helper intervention was were more driven by the broadband increase in power prior to intervention, as well as parietal activity and physiological arousal after the helper event occurred. While the directionality of the cognitive load finding is in line with our hypothesized result, that subjective reports of the helpfulness was driven more by parietal activity was a deviation from our hypothesis.
The third event of note is a decrease in parietal theta power at 14 s and a subsequent increase in heart rate variability 20 s following the helper clear event. There were additional significant increases from baseline in heart rate variability 12–14 s following the helper clear and a significant decrease in heart rate 7 s following the helper clear, implying a change in the ANS balance in favor of parasympathetic tone [28,29]. The decrease in parietal theta (a selected predictor in the linear model of how relieving the ball was) may indicate a corresponding decrease in sustained visuospatial engagement [24] in return reflecting a more relaxed approach to gameplay. This is consistent with the hypothesized and observed decrease in physiological arousal following the helper clear event as evidenced by the heart rate data. Furthermore the heart rate variability at 20 s was a selected predictor variable with a positive coefficient in the linear model of the question relating to the helpfulness of the helper clear. The effects of the helper clear event observed here spans multiple data streams across a time interval of over 20 s. The data streams include frontal and parietal EEG data as well as PPG heart rate variability data that in turn predict 25–33% of the variance observed in subjective questionnaire responses about how helpful or relieving the helper event was.

7.2. Limitations

Although present exploratory study provides insight into the dynamics of mental and visuospatial workload in VR environments, several limitations should be noted. Despite randomizing the order of intervention and control conditions and providing a tutorial session, individual variations in prior in experience with VR and Tetris might have still influenced the results. Future investigations could benefit from a more controlled selection process to address these individual differences more effectively. Additionally, although our task included a rich visuospatial environment, future studies could exploit the full, ambulatory potential of VR systems by utilizing mobile brain-body imaging. Furthermore, while we used a broad range of electrodes to cover the frontoparietal network, central electrodes near sensorimotor cortex might have been affected due to hand movements. It is previously reported that event-related desynchronization (ERD) may be observed in parietal electrodes due to volume conduction [24]. These effects could be isolated in future studies by utilizing source localization methods and increased number of electrodes [54]. Moreover, although fluctuations in EEG power bands and HR and HRV values have been extensively used and validated as a means to measure workload, further analysis methods such as event-related spectral perturbations (ERSPs), functional connectivity analyses, and emotion classification systems using spatio-temporal image encodings [55] could provide deeper insights. Additionally, incorporating physiological modalities such as EDA, EOG, EMG and skin temperature could be utilized to better probe the effects of mental workload. Finally, although Tetris was chosen for its well-defined cognitive and visuospatial demands, future research should explore a range of tasks with varying cognitive requirements to assess the generalizability of our results. Investigating different types of VR activities and real-world scenarios could also provide a more comprehensive understanding of how mental workload and visuospatial engagement varies across diverse contexts.

7.3. Conclusions

In this study we observed a broadband increase in EEG power just prior to the helper event followed by a spike of visuospatial engagement (parietal alpha and beta 0-1-3 s) that occurred concurrently with decrease in mental workload (frontal theta 2–4 s), in turn followed by a subsequent decrease in visuospatial engagement (parietal theta at 14 s) and decrease in physiological arousal (HRV at 20 s). While as expected there was a decrease in cognitive load in response the helper clear event, subjective relief and the subjective reports about how helpful the event was were more driven by physiological arousal and the parietal oscillatory response to the helper, commonly regarded as a proxy for visuospatial processes. Furthermore, our findings suggest the plausibility of the modulation of ANS balance using help interventions in VR tasks. By using online task load modifications similar to our helper piece, it may be possible to optimize workload in favor of overall mental health and well-being.

Author Contributions

G.B.: Writing—original draft, Visualization, Software, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. H.J.: Writing—original draft, Visualization, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. İ.E.Y.: Writing—original draft, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. N.H.: Writing—original draft, Visualization, Software, Methodology, Formal analysis, Data curation, Conceptualization. Y.A.: Software. T.B.-D.: Writing—review & editing, Methodology, Formal analysis, Data curation. P.M.: Writing—review & editing, Supervision, Resources, Project administration, Conceptualization. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with relevant guidelines outlined in local regulations and the Declaration of Helsinki. All procedures were approved by the Institutional Review Board of Massachusetts Institute of Technology (MIT) (ID: 2109000465).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data used and analyzed in this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology; Elsevier: Amsterdam, The Netherlands, 1988; Volume 52, pp. 139–183. [Google Scholar]
  2. Young, M.S.; Brookhuis, K.A.; Wickens, C.D.; Hancock, P.A. State of science: Mental workload in ergonomics. Ergonomics 2015, 58, 1–17. [Google Scholar] [CrossRef] [PubMed]
  3. Hancock, G.; Longo, L.; Young, M.; Hancock, P. Mental workload. In Handbook of Human Factors and Ergonomics; John Wiley & Sons: Hoboken, NJ, USA, 2021; pp. 203–226. [Google Scholar]
  4. Sweller, J. Cognitive load theory: Recent theoretical advances. In Cognitive Load Theory; Cambridge University Press: New York, NY, USA, 2010; pp. 29–47. [Google Scholar] [CrossRef]
  5. Friedman, N.P.; Robbins, T.W. The role of prefrontal cortex in cognitive control and executive function. Neuropsychopharmacology 2022, 47, 72–89. [Google Scholar] [CrossRef]
  6. Miller, E.K. The prefontral cortex and cognitive control. Nat. Rev. Neurosci. 2000, 1, 59–65. [Google Scholar] [CrossRef] [PubMed]
  7. Cohen, Y.E.; Andersen, R.A. A common reference frame for movement plans in the posterior parietal cortex. Nat. Rev. Neurosci. 2002, 3, 553–562. [Google Scholar] [CrossRef] [PubMed]
  8. Sack, A.T. Parietal cortex and spatial cognition. Behav. Brain Res. 2009, 202, 153–161. [Google Scholar] [CrossRef] [PubMed]
  9. Culham, J.C.; Kanwisher, N.G. Neuroimaging of cognitive functions in human parietal cortex. Curr. Opin. Neurobiol. 2001, 11, 157–163. [Google Scholar] [CrossRef]
  10. Stiles, J.; Reilly, J.S.; Levine, S.C.; Trauner, D.A.; Nass, R. Spatial Attention, Working Memory, and Executive Function. In Neural Plasticity and Cognitive Development: Insights from Children with Perinatal Brain Injury; Oxford University Press: Oxford, UK, 2012. [Google Scholar] [CrossRef]
  11. Thayer, J.F.; Lane, R.D. Claude Bernard and the heart–brain connection: Further elaboration of a model of neurovisceral integration. Neurosci. Biobehav. Rev. 2009, 33, 81–88. [Google Scholar] [CrossRef]
  12. Tanaka, M.; Mizuno, K.; Tajima, S.; Sasabe, T.; Watanabe, Y. Central nervous system fatigue alters autonomic nerve activity. Life Sci. 2009, 84, 235–239. [Google Scholar] [CrossRef]
  13. Hockey, G.R.J. Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework. Biol. Psychol. 1997, 45, 73–93. [Google Scholar] [CrossRef]
  14. Yerkes, R.M.; Dodson, J.D. The relation of strength of stimulus to rapidity of habit-formation. J. Comp. Neurol. Psychol. 1908, 18, 459–482. [Google Scholar] [CrossRef]
  15. Thayer, J.F.; Hansen, A.L.; Saus-Rose, E.; Johnsen, B.H. Heart rate variability, prefrontal neural function, and cognitive performance: The neurovisceral integration perspective on self-regulation, adaptation, and health. Ann. Behav. Med. 2009, 37, 141–153. [Google Scholar] [CrossRef] [PubMed]
  16. Vanneste, P.; Raes, A.; Morton, J.; Bombeke, K.; Van Acker, B.B.; Larmuseau, C.; Depaepe, F.; Van den Noortgate, W. Towards measuring cognitive load through multimodal physiological data. Cogn. Technol. Work. 2021, 23, 567–585. [Google Scholar] [CrossRef]
  17. Fernandez Rojas, R.; Debie, E.; Fidock, J.; Barlow, M.; Kasmarik, K.; Anavatti, S.; Garratt, M.; Abbass, H. Electroencephalographic workload indicators during teleoperation of an unmanned aerial vehicle shepherding a swarm of unmanned ground vehicles in contested environments. Front. Neurosci. 2020, 14, 40. [Google Scholar] [CrossRef]
  18. Antonenko, P.; Paas, F.; Grabner, R.; Van Gog, T. Using electroencephalography to measure cognitive load. Educ. Psychol. Rev. 2010, 22, 425–438. [Google Scholar] [CrossRef]
  19. Malik, M.; Bigger, J.T.; Camm, A.J.; Kleiger, R.E.; Malliani, A.; Moss, A.J.; Schwartz, P.J. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. Eur. Heart J. 1996, 17, 354–381. [Google Scholar] [CrossRef]
  20. Chikhi, S.; Matton, N.; Blanchet, S. EEG power spectral measures of cognitive workload: A meta-analysis. Psychophysiology 2022, 59, e14009. [Google Scholar] [CrossRef]
  21. So, W.K.Y.; Wong, S.W.H.; Mak, J.N.; Chan, R.H.M. An evaluation of mental workload with frontal EEG. PLoS ONE 2017, 12, e0174949. [Google Scholar] [CrossRef]
  22. Spitzer, B.; Haegens, S. Beyond the Status Quo: A Role for Beta Oscillations in Endogenous Content (Re)Activation. eNeuro 2017, 4, ENEURO.0170-17.2017. [Google Scholar] [CrossRef]
  23. Mapelli, I.; Özkurt, T.E. Brain Oscillatory Correlates of Visual Short-Term Memory Errors. Front. Hum. Neurosci. 2019, 13, 33. [Google Scholar] [CrossRef]
  24. Natalizio, A.; Sieghartsleitner, S.; Schreiner, L.; Walchshofer, M.; Esposito, A.; Scharinger, J.; Pretl, H.; Arpaia, P.; Parvis, M.; Solé-Casals, J.; et al. Real-time estimation of EEG-based engagement in different tasks. J. Neural Eng. 2024, 21, 016014. [Google Scholar] [CrossRef]
  25. Zhu, Y.; Wang, Q.; Zhang, L. Study of EEG characteristics while solving scientific problems with different mental effort. Sci. Rep. 2021, 11, 23783. [Google Scholar] [CrossRef] [PubMed]
  26. Puma, S.; Matton, N.; Paubel, P.V.; Raufaste, É.; El-Yagoubi, R. Using theta and alpha band power to assess cognitive workload in multitasking environments. Int. J. Psychophysiol. 2018, 123, 111–120. [Google Scholar] [CrossRef] [PubMed]
  27. Chen, A.; Hao, S.; Han, Y.; Fang, Y.; Miao, Y. Exploring the effects of different BCI-based attention training games on the brain: A functional near-infrared spectroscopy study. Neurosci. Lett. 2024, 818, 137567. [Google Scholar] [CrossRef] [PubMed]
  28. Charles, R.L.; Nixon, J. Measuring mental workload using physiological measures: A systematic review. Appl. Ergon. 2019, 74, 221–232. [Google Scholar] [CrossRef]
  29. Solhjoo, S.; Haigney, M.C.; McBee, E.; van Merrienboer, J.J.; Schuwirth, L.; Artino, A.R.; Battista, A.; Ratcliffe, T.A.; Lee, H.D.; Durning, S.J. Heart rate and heart rate variability correlate with clinical reasoning performance and self-reported measures of cognitive load. Sci. Rep. 2019, 9, 1–9. [Google Scholar] [CrossRef]
  30. Chen, F.; Zhou, J.; Wang, Y.; Yu, K.; Arshad, S.Z.; Khawaji, A.; Conway, D. Robust Multimodal Cognitive Load Measurement; Springer: Berlin/Heidelberg, Germany, 2016. [Google Scholar]
  31. Kramer, A.F. Physiological metrics of mental workload: A review of recent progress. In Multiple Task Performance; CRC Press: Boca Raton, FL, USA, 2020; pp. 279–328. [Google Scholar]
  32. Radianti, J.; Majchrzak, T.A.; Fromm, J.; Wohlgenannt, I. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comput. Educ. 2020, 147, 103778. [Google Scholar] [CrossRef]
  33. Cipresso, P.; Giglioli, I.A.C.; Raya, M.A.; Riva, G. The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Front. Psychol. 2018, 9, 2086. [Google Scholar] [CrossRef]
  34. Armengol-Urpi, A.; Sarma, S.E. Sublime: A hands-free virtual reality menu navigation system using a high-frequency SSVEP-based brain-computer interface. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, Tokyo, Japan, 28 November–1 December 2018; pp. 1–8. [Google Scholar]
  35. Castelvecchi, D. Low-cost headsets boost virtual reality’s lab appeal. Nature 2016, 533, 153–154. [Google Scholar] [CrossRef]
  36. Strickland, D. Virtual reality for the treatment of autism. In Virtual Reality in Neuro-Psycho-Physiology; IOS Press: Amsterdam, The Netherlands, 1997; pp. 81–86. [Google Scholar]
  37. Strickland, D.; Hodges, L.; North, M.; Weghorst, S. Overcoming phobias by virtual exposure. Commun. ACM 1997, 40, 34–39. [Google Scholar] [CrossRef]
  38. Zheng, W.L.; Lu, B.L. A multimodal approach to estimating vigilance using EEG and forehead EOG. J. Neural Eng. 2017, 14, 026017. [Google Scholar] [CrossRef]
  39. Jones, C.R.; Pickles, A.; Falcaro, M.; Marsden, A.J.; Happé, F.; Scott, S.K.; Sauter, D.; Tregay, J.; Phillips, R.J.; Baird, G.; et al. A multimodal approach to emotion recognition ability in autism spectrum disorders. J. Child Psychol. Psychiatry 2011, 52, 275–285. [Google Scholar] [CrossRef] [PubMed]
  40. Bernal, G.; Yang, T.; Jain, A.; Maes, P. PhysioHMD: A conformable, modular toolkit for collecting physiological data from head-mounted displays. In Proceedings of the 2018 ACM International Symposium on Wearable Computers, Singapore, 8–12 October 2018; pp. 160–167. [Google Scholar]
  41. Gupta, K.; Hajika, R.; Pai, Y.S.; Duenser, A.; Lochner, M.; Billinghurst, M. Measuring human trust in a virtual assistant using physiological sensing in virtual reality. In Proceedings of the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Atlanta, GA, USA, 22–26 March 2020; pp. 756–765. [Google Scholar]
  42. Zhang, L.; Wade, J.; Bian, D.; Fan, J.; Swanson, A.; Weitlauf, A.; Warren, Z.; Sarkar, N. Cognitive load measurement in a virtual reality-based driving system for autism intervention. IEEE Trans. Affect. Comput. 2017, 8, 176–189. [Google Scholar] [CrossRef] [PubMed]
  43. Dey, A.; Chatburn, A.; Billinghurst, M. Exploration of an EEG-based cognitively adaptive training system in virtual reality. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 220–226. [Google Scholar]
  44. Hadjiaros, M.; Neokleous, K.; Shimi, A.; Avraamides, M.N.; Pattichis, C.S. Virtual Reality Cognitive Gaming Based on Brain Computer Interfacing: A Narrative Review. IEEE Access 2023, 11, 18399–18416. [Google Scholar] [CrossRef]
  45. Bernal, G.; Hidalgo, N.; Russomanno, C.; Maes, P. Galea: A physiological sensing system for behavioral research in Virtual Environments. In Proceedings of the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Christchurch, New Zealand, 12–16 March 2022; pp. 66–76. [Google Scholar]
  46. Lab Streaming Layer (LSL). 2020. Available online: https://github.com/sccn/labstreaminglayer (accessed on 1 January 2024).
  47. Markello, R.; DuPre, E. Physiopy/Peakdet: A Toolbox for Physiological Peak Detection Analyses. 2020. Available online: https://zenodo.org/records/7244954 (accessed on 1 January 2024).
  48. Munoz, M.L.; van Roon, A.; Riese, H.; Thio, C.; Oostenbroek, E.; Westrik, I.; de Geus, E.J.C.; Gansevoort, R.; Lefrandt, J.; Nolte, I.M.; et al. Validity of (Ultra-)Short Recordings for Heart Rate Variability Measurements. PLoS ONE 2015, 10, e0138921. [Google Scholar] [CrossRef]
  49. Di Dona, G.; Ronconi, L. Beta oscillations in vision: A (preconscious) neural mechanism for the dorsal visual stream? Front. Psychol. 2023, 14, 1296483. [Google Scholar] [CrossRef]
  50. Pisella, L. Visual perception is dependent on visuospatial working memory and thus on the posterior parietal cortex. Ann. Phys. Rehabil. Med. 2017, 60, 141–147. [Google Scholar] [CrossRef]
  51. Agren, T.; Hoppe, J.M.; Singh, L.; Holmes, E.A.; Rosén, J. The neural basis of Tetris gameplay: Implicating the role of visuospatial processing. Curr. Psychol. 2023, 42, 8156–8163. [Google Scholar] [CrossRef]
  52. Lau-Zhu, A.; Holmes, E.A.; Butterfield, S.; Holmes, J. Selective Association Between Tetris Game Play and Visuospatial Working Memory: A Preliminary Investigation. Appl. Cogn. Psychol. 2017, 31, 438–445. [Google Scholar] [CrossRef]
  53. Hamann, A.; Carstengerdes, N. Assessing the development of mental fatigue during simulated flights with concurrent EEG-fNIRS measurement. Sci. Rep. 2023, 13, 4738. [Google Scholar] [CrossRef]
  54. Xie, W.; Richards, J. Cortical Source Localization in EEG Frequency Analysis 2022. In The Oxford Handbook of EEG Frequency; Oxford Academic: Oxford, UK, 2022. [Google Scholar] [CrossRef]
  55. Avola, D.; Cinque, L.; Mambro, A.D.; Fagioli, A.; Marini, M.R.; Pannone, D.; Fanini, B.; Foresti, G.L. Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition. Int. J. Neural Syst. 2024, 34, 2450024. [Google Scholar] [CrossRef]
Computers 13 00246 g001
Figure 1. Left: Plot of live physiological data stream from Galea headset (left screen), TETRIS VR environment (right screen), User wearing VR headset. Right: Outline of experiment procedure. For each of the two sessions, participants completed two pre-experiment questionnaires. Then, a headset fitting session was conducted. Once the headset was fitted, calibration data was collected (regular and eyes-closed). After a brief tutorial session, participants played either the intervention or control version of TETRIS. After completion of the game, the participant filled out two post-experiment questionnaires, and was debriefed on the session.
Figure 1. Left: Plot of live physiological data stream from Galea headset (left screen), TETRIS VR environment (right screen), User wearing VR headset. Right: Outline of experiment procedure. For each of the two sessions, participants completed two pre-experiment questionnaires. Then, a headset fitting session was conducted. Once the headset was fitted, calibration data was collected (regular and eyes-closed). After a brief tutorial session, participants played either the intervention or control version of TETRIS. After completion of the game, the participant filled out two post-experiment questionnaires, and was debriefed on the session.
Computers 13 00246 g001
Computers 13 00246 g002aComputers 13 00246 g002b
Figure 2. Figures show the significant deviations from baseline for EEG and PPG signals before and after the helper event. (a) Frontal Theta power collapsed for electrodes Fp1, Fp2, Fz. (b) Parietal Theta power collapsed for electrodes POz, PO3, PO4. (c) Frontal Beta power collapsed for electrodes Fp1, Fp2, Fz. (d) Parietal Beta power collapsed for electrodes POz, PO3, PO4 (e) Frontal alpha power collapsed for electrodes Fp1, Fp2, Fz. (f) Parietal alpha power collapsed for electrodes POz, PO3, PO4. (g) Heart rate, in beats per minute, deviation from baseline. (h) RMSSD, a measure of HRV, deviation from baseline.
Figure 2. Figures show the significant deviations from baseline for EEG and PPG signals before and after the helper event. (a) Frontal Theta power collapsed for electrodes Fp1, Fp2, Fz. (b) Parietal Theta power collapsed for electrodes POz, PO3, PO4. (c) Frontal Beta power collapsed for electrodes Fp1, Fp2, Fz. (d) Parietal Beta power collapsed for electrodes POz, PO3, PO4 (e) Frontal alpha power collapsed for electrodes Fp1, Fp2, Fz. (f) Parietal alpha power collapsed for electrodes POz, PO3, PO4. (g) Heart rate, in beats per minute, deviation from baseline. (h) RMSSD, a measure of HRV, deviation from baseline.
Computers 13 00246 g002aComputers 13 00246 g002b
Computers 13 00246 g003
Figure 3. Estimated vs observed performance improvement—Estimated performance improvement is shown on the x-axis and observed performance improvement on the y-axis. Performance improvement is estimated as a linear combination of selected physiological signals that significantly deviate from baseline activity following the appearance of a helper during gameplay.
Figure 3. Estimated vs observed performance improvement—Estimated performance improvement is shown on the x-axis and observed performance improvement on the y-axis. Performance improvement is estimated as a linear combination of selected physiological signals that significantly deviate from baseline activity following the appearance of a helper during gameplay.
Computers 13 00246 g003
Computers 13 00246 g004
Figure 4. Estimated vs observed questionnaire responses - For each of the four questionnaire responses regarding the helper, estimated vs observed questionnaire responses are displayed. Questions 1, 2 and 4 showed statistically significant (p > 0.05) models.
Figure 4. Estimated vs observed questionnaire responses - For each of the four questionnaire responses regarding the helper, estimated vs observed questionnaire responses are displayed. Questions 1, 2 and 4 showed statistically significant (p > 0.05) models.
Computers 13 00246 g004
Table 1. Table showing the statistically significant events around the helper condition (t = 0 s) for frontal Theta power (FTP), parietal Theta power (PTP), frontal alpha power (FAP), parietal alpha power (PAP), frontal Beta power (FBP), and parietal Beta power (PBP) based on EEG power spectrum collapsed across electrodes (CAE). We also report the significant events of PPG changes including beats per minute (-BPM) and root-mean square of successive differences (RMSSD).
Table 1. Table showing the statistically significant events around the helper condition (t = 0 s) for frontal Theta power (FTP), parietal Theta power (PTP), frontal alpha power (FAP), parietal alpha power (PAP), frontal Beta power (FBP), and parietal Beta power (PBP) based on EEG power spectrum collapsed across electrodes (CAE). We also report the significant events of PPG changes including beats per minute (-BPM) and root-mean square of successive differences (RMSSD).
FTP-CAE FAP-CAE FBP-CAE
Time of event−7234163550 −445152 −4−30
Estimated Mean−0.2277−0.3511−0.5505−0.4547−0.3108−0.246−0.3552 0.4499−0.2924−0.2932−0.3696 0.5170.44080.3164
p-value0.0290.01081.4708 × 10−61.8957 × 10−40.03790.03750.0424 0.03320.03140.04220.0031 0.01810.00650.044
PTP-CAE PAP-CAE
Time of event−71569141638 013646
Estimated Mean−0.3068−0.3854−0.594−0.5093−0.4159−0.3427−0.3812−0.2841 −0.3995−0.39770.39770.4674
p-value0.04470.00563.8388 × 10−40.00610.00760.04650.04020.0325 0.01570.01620.01980.0496
PBP-CAE PPG-BPM PPG-RMSSD
Time of event−3−2123 −5−107 −212131420
Estimated Mean0.49170.45330.29630.40250.4699 −0.3912−0.5192−0.5427−0.4698 0.53530.49360.50470.52020.5547
p-value0.01460.04870.02030.03170.0191 0.02210.02760.02350.0232 0.0250.04110.04020.04760.0447
Table 2. Performance improvement model coefficients.
Table 2. Performance improvement model coefficients.
Coefficient Estimatep Value
β 0 158.121.7066 × 10−8
β 1 −49.8430.0029947
β 2 58.0480.0045931
Table 3. Individual contributions of the estimated components for question 1, 2, 3, and 4.
Table 3. Individual contributions of the estimated components for question 1, 2, 3, and 4.
Coeff. est.SEtstatp ValueCoeff. est.SEtstatp Value
Q1Q 2
β 0 1.67650.250316.69779.8599 × 10−72.51090.2026612.3892.1483 × 10−11
β 1 0.629150.283742.21730.037241−0.482110.22578−2.13530.044115
β 2 0.438760.224251.95660.0632110.407240.208941.94910.06415
Q 3Q 4
β 0 1.64210.35604.61240.0000121.67650.25036.69779.86 × 10−7
β 1 −0.76360.43776−1.74610.09410.629150.283742.21730.0372
β 2 ----0.438760.22431.95660.0632
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Bernal, G.; Jung, H.; Yassı, İ.E.; Hidalgo, N.; Alemu, Y.; Barnes-Diana, T.; Maes, P. Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments. Computers 2024, 13, 246. https://doi.org/10.3390/computers13100246

AMA Style

Bernal G, Jung H, Yassı İE, Hidalgo N, Alemu Y, Barnes-Diana T, Maes P. Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments. Computers. 2024; 13(10):246. https://doi.org/10.3390/computers13100246

Chicago/Turabian Style

Bernal, Guillermo, Hahrin Jung, İsmail Emir Yassı, Nelson Hidalgo, Yodahe Alemu, Tyler Barnes-Diana, and Pattie Maes. 2024. "Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments" Computers 13, no. 10: 246. https://doi.org/10.3390/computers13100246

APA Style

Bernal, G., Jung, H., Yassı, İ. E., Hidalgo, N., Alemu, Y., Barnes-Diana, T., & Maes, P. (2024). Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments. Computers, 13(10), 246. https://doi.org/10.3390/computers13100246

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop