Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

VIMSNet: an effective network for visually induced motion sickness detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Visually induced motion sickness (VIMS) often occurs when people are exposed to virtual reality (VR) environments. It is necessary to develop a method that can detect the VR user’s VIMS level effectively according to physiological signals. In this study, subjects were visually stimulated by a VR vehicle-driving simulator (VDS) to induce VIMS, and an EEG device with only four electrodes was used to collect EEG data. The data were simply preprocessed and then inputted into our well-designed classifier (VIMSNet). Two important modules are presented in VIMSNet for high-performance classification: a parallel-feature extraction (PFE) module is designed to extract features with different scales from EEG data, and a feature attention (FA) module is designed to emphasize useful features while suppressing less useful ones. Experimental results show that our proposed VIMSNet outperforms the state-of-the-art (SOTA) classifiers in various classification tasks: for per-subject binary classification, the proposed classifier achieved an average accuracy and kappa of 0.967 and 0.930, respectively; for multiple-subject binary classification, it yielded an accuracy of 0.953 and a kappa of 0.903; for multiple-subject four-level classification, the accuracy and kappa were 0.925 and 0.874, respectively. Our study also indicates that it is feasible to accurately detect VIMS using an EEG device with a small number of electrodes. Source code is available at https://github.com/threedteam/VIMSNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Fernandes, A.S., Feiner, S.K.: Combating VR sickness through subtle dynamic field-of-view modification. In: 2016 IEEE Symposium on 3D User Interfaces (3DUI), pp. 201–210. IEEE, Greenville, SC, USA (2016)

  2. Chardonnet, J.R., Mirzaei, M.A., Merienne, F.: Features of the postural sway signal as indicators to estimate and predict visually induced motion sickness in virtual reality. Int. J. Hum. Comput. Interact. 33(10), 771–785 (2017)

    Article  Google Scholar 

  3. Fan, J., Wade, J.W., Key, A.P., Warren, Z.E., Sarkar, N.: EEG-based affect and workload recognition in a virtual driving environment for ASD intervention. IEEE Trans. Biomed. Eng. 65(1), 43–51 (2018)

    Article  Google Scholar 

  4. Hettinger, L.J., Riccio, G.E.: Visually induced motion sickness in virtual environments. Presence Teleoper. Virtual Environ. 1(3), 306–310 (1992)

    Article  Google Scholar 

  5. Kennedy, R.S., Drexler, J., Kennedy, R.C.: Research in visually induced motion sickness. Appl. Ergon. 41(4), 494–503 (2010)

    Article  Google Scholar 

  6. Ohyama, S., Nishiike, S., Watanabe, H., Matsuoka, K., Akizuki, H., Takeda, N., Harada, T.: Autonomic responses during motion sickness induced by virtual reality. Auris Nasus Larynx 34(3), 303–306 (2007)

    Article  Google Scholar 

  7. Miyazaki, J., Yamamoto, H., Ichimura, Y., Yamashiro, H., Murase, T., Yamamoto, T., Umeda, M., Higuchi, T.: Inter-hemispheric desynchronization of the human MT+ during visually induced motion sickness. Exp. Brain Res. 233(8), 2421–2431 (2015)

    Article  Google Scholar 

  8. Classen, S., Bewernitz, M., Shechtman, O.: Driving simulator sickness: an evidence-based review of the literature. Am. J. Occup. Ther. 65(2), 179–188 (2011)

    Article  Google Scholar 

  9. Ko, L.W., Lee, H.C., Tsai, S.F., Shih, T.C., Lin, C.T.: EEG-based motion sickness classification system with genetic feature selection. In: 2013 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), pp. 158–164. IEEE, Singapore (2013)

  10. Li, Y., Liu, A., Ding, L.: Machine learning assessment of visually induced motion sickness levels based on multiple biosignals. Biomed. Signal Process. Control 49, 202–211 (2019). https://doi.org/10.1016/j.bspc.2018.12.007

    Article  Google Scholar 

  11. Wei, C.S., Ko, L.W., Chuang, S.W., Jung, T.P., Lin, C.T.: Genetic feature selection in EEG-based motion sickness estimation. In: The 2011 International Joint Conference on Neural Networks, pp. 365–369. IEEE, San Jose, CA, USA (2011)

  12. Lin, C.T., Tsai, S.F., Ko, L.W.: EEG-based learning system for online motion sickness level estimation in a dynamic vehicle environment. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1689–1700 (2013)

    Article  Google Scholar 

  13. Liu, R., Xu, M., Zhang, Y., Peli, E., Hwang, A.D.: A pilot study on electroencephalogram-based evaluation of visually induced motion sickness. J. Imaging Sci. Technol. 64(2), 20501-1-20501–10 (2020)

    Article  Google Scholar 

  14. Ahin Sadk, E., Saraolu, H.M., Canbaz Kabay, S., Tosun, M., Keskinkl, C., Akda, G.: Investigation of the effect of rosemary odor on mental workload using EEG: an artificial intelligence approach. SIViP (2021). https://doi.org/10.1007/s11760-021-01992-5

    Article  Google Scholar 

  15. Ouyang, C.S., Yang, R.C., Chiang, C.T., Wu, R.C., Lin, L.C.: EEG autoregressive modeling analysis: a diagnostic tool for patients with epilepsy without epileptiform discharges. Clin. Neurophysiol. 131(8), 1902–1908 (2020)

    Article  Google Scholar 

  16. Liao, C.Y., Tai, S.K., Chen, R.C., Hendry, H.: Using EEG and deep learning to predict motion sickness under wearing a virtual reality device. IEEE Access 8, 126784–126796 (2020). https://doi.org/10.1109/ACCESS.2020.3008165

  17. Jeong, D., Yoo, S., Yun, J.: Cybersickness analysis with EEG using deep learning algorithms. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 827–835. IEEE, Osaka, Japan (2019)

  18. Liu, R., Eli, P., Hwang, A.D.: Measuring visually induced motion sickness using wearable devices. In: IS and T International Symposium on Electronic Imaging Science and Technology, 2017, Human Vision and Electronic Imaging, pp. 218–223. Society for Imaging Science and Technology, Burlingame, CA, USA (2017)

  19. Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)

    Article  Google Scholar 

  20. Behrang, K., Heiko, H.: Validating an efficient method to quantify motion sickness. Hum. Factors 53(4), 415–426 (2011)

    Article  Google Scholar 

  21. Hussain, W., Sadiq, M.T., Siuly, S., Rehman, A.U.: Epileptic seizure detection using 1 D-convolutional long short-term memory neural networks. Appl. Acoust. 177,(2021). https://doi.org/10.1016/j.apacoust.2021.107941

  22. Jun, T.J., Kang, S.J., Lee, J.G., Kweon, J., Na, W., Kang, D., Kim, D., Kim, D., Kim, Y.H.: Automated detection of vulnerable plaque in intravascular ultrasound images. Med. Biol. Eng. Comput. 57, 863–876 (2019). https://doi.org/10.1007/s11517-018-1925-x

    Article  Google Scholar 

  23. Farsi, L., Siuly, S., Kabir, E., Wang, H.: Classification of alcoholic EEG signals using a deep learning method. IEEE Sens. J. 21(3), 3552–3560 (2021)

  24. Hu, J., Shen, L., Samuel, A., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)

    Article  Google Scholar 

  25. Hwang, A.D., Deng, H., Gao, Z., Peli, E.: Quantifying visually induced motion sickness (VIMS) during stereoscopic 3D viewing using temporal VIMS rating. J. Imaging Sci. Technol. 61(6), 60405-1-60405–9 (2017)

  26. Wei, C.S., Chuang, S.W., Wang, W.R., Ko, L.W., Lin, C.T.: Implementation of a motion sickness evaluation system based on EEG spectrum analysis. In: 2011 IEEE International Symposium of Circuits and Systems (ISCAS), pp. 1081–1084. IEEE, Rio de Janeiro, Brazil (2011)

  27. Emami, A., Kunii, N., Matsuo, T., Shinozaki, T., Kawai, K., Takahashi, H.: Seizure detection by convolutional neural network-based analysis of scalp electroencephalography plot images. NeuroImage-Clin. 22, 101684 (2019). https://doi.org/10.1016/j.nicl.2019.101684

    Article  Google Scholar 

  28. Li, Yh., Wang, Ny., Shi, Jp., Hou, Xd., Liu, Jy.: Adaptive batch normalization for practical domain adaptation. Pattern Recognit. 80, 109–117 (2018). https://doi.org/10.1016/j.patcog.2018.03.005

    Article  Google Scholar 

  29. Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., Muller, P.A.: Deep learning for time series classification: a review. Data Min. Knowl. Discov. 33, 917–963 (2019). https://doi.org/10.1007/s10618-019-00619-1

    Article  MathSciNet  MATH  Google Scholar 

  30. Craik, A., He, Y., Contreras-Vidal, J.L.: Deep learning for electroencephalogram (EEG) classification tasks: a review. J. Neural Eng. 16(3),(2019)

Download references

Acknowledgements

This work was supported in part by the Fundamental Research Funds for the Central Universities (Grant Number 2019CDYGZD004); the National Institutes of Health P30 core grant (Grant Number P30EY003790); the Chongqing Foundation and Advanced Research Project (Grant Number cstc2019jcyj-msxmX0622); the Sichuan Science and Technology Program (Grant Number 2019YFSY0026); and the Open Fund Project of the Key Laboratory of Chongqing University Cancer Hospital.

Author information

Authors and Affiliations

  1. College of Computer Science, Chongqing University, Chongqing, 400044, China

    Ran Liu, Shanshan Cui, Yang Zhao & Xi Chen

  2. Schepens Eye Research Institute, Massachusetts Eye and Ear, Department of Ophthalmology, Harvard Medical School, Boston, MA, USA

    Ran Liu & Alex D. Hwang

  3. Chongqing University Cancer Hospital, Chongqing, 400030, China

    Lin Yi

Authors
  1. Ran Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shanshan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lin Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alex D. Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, R., Cui, S., Zhao, Y. et al. VIMSNet: an effective network for visually induced motion sickness detection. SIViP 16, 2029–2036 (2022). https://doi.org/10.1007/s11760-022-02164-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02164-9

Keywords

Search

Navigation