research-article

HoloLearn: Wearable Mixed Reality for People with Neurodevelopmental Disorders (NDD)

Authors:

Beatrice Aruanno,

Franca Garzotto,

Emanuele Torelli,

Francesco VonaAuthors Info & Claims

ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

Pages 40 - 51

https://doi.org/10.1145/3234695.3236351

Published: 08 October 2018 Publication History

Abstract

Our research explores the potential of wearable Mixed Reality (MR) for people with Neuro-Developmental Disorders (NDD). The paper presents HoloLearn, a MR application designed in cooperation with NDD experts and implemented using HoloLens technology. The goal of HoloLearn is to help people with NDD learn how to perform simple everyday tasks in domestic environments and improve autonomy. An original feature of the system is the presence of a virtual assistant devoted to capture the user's attention and to give her/him hints during task execution in the MR environment. We performed an exploratory study involving 20 subjects with NDD to investigate the acceptability and usability of HoloLearn and its potential as a therapeutic tool. HoloLearn was well-accepted by the participants and the activities in the MR space were perceived as enjoyable, despite some usability problems associated to HoloLens interaction mechanism. More extensive and long term empirical research is needed to validate these early results, but our study suggests that HoloLearn could be adopted as a complement to more traditional interventions. Our work, and the lessons we learned, may help designers and developers of future MR applications devoted to people with NDD and to other people with similar needs.

Supplementary Material

suppl.mov (fp065.mp4)

Supplemental video

Download
45.83 MB

References

[1]

Adobe. 2018. Mixamo. Retrieved March 20, 2018 from: https://www.mixamo.com/

[2]

American Psychiatric Association. 2013. Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub. 33--41.

[3]

Beatrice Aruanno, Franca Garzotto, and Mario Covarrubias Rodriguez. 2017. HoloLens-based Mixed Reality Experiences for Subjects with Alzheimer's Disease. In Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (CHItaly '17), Article 15, 9 pages.

Digital Library

[4]

Miguel Bernardes, Fernando Barros, Marco Simoes, and Miguel Castelo-Branco. 2015. A serious game with virtual reality for travel training with Autism Spectrum Disorder. In 2015 International Conference on Virtual Rehabilitation (ICVR), 127--128.

[5]

LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. 2016. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4872--4883.

Digital Library

[6]

Evren Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Rajiv Dubey. 2016. Locomotion in Virtual Reality for Individuals with Autism Spectrum Disorder. In Proceedings of the 2016 Symposium on Spatial User Interaction (SUI '16), 33--42.

Digital Library

[7]

Lal Bozgeyikli, Evren Bozgeyikli, Andrew Raij, Redwan Alqasemi, Srinivas Katkoori, and Rajiv Dubey. 2017. Vocational Rehabilitation of Individuals with Autism Spectrum Disorder with Virtual Reality. ACM Transactions on Accessible Computing (TACCESS). 10, 2 (April 2017), Article 5, 25 pages.

Digital Library

[8]

Lal Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Redwan Alqasemi. 2017. A Survey on Virtual Reality for Training Individuals with Autism Spectrum Disorder: Design Considerations. IEEE Transactions on Learning Technologies. 11, 2 (April-June 2018), 133--151.

[9]

Brain Power. 2018. "Empower Me" by Brain Power, Retrieved February 16, 2018 from: http://www.brain-power.com/autism/

[10]

Ross Brown, Laurianne Sitbon, Lauren Fell, Stewart Koplick, Chris Beaumont, and Margot Brereton. 2016. Design insights into embedding virtual reality content into life skills training for people with intellectual disability. In Proceedings of the 28th Australian Conference on Computer-Human Interaction (OzCHI '16). ACM, New York, NY, USA, 581--585.

Digital Library

[11]

Vanessa Camilleri, Matthew Montebello, and Alexiei Dingli. 2017. Walking in small shoes: Investigating the power of VR on empathising with children's difficulties. In 2017 23rd International Conference on Virtual System & Multimedia (VSMM), 1--6.

[12]

Xavier Casas, Gerardo Herrera, Inmaculada Coma, Marcos Fernández. 2012. A Kinect-based Augmented Reality System for Individuals with Autism Spectrum Disorders. In GRAPP/IVAPP. 440--446.

[13]

Adam Dachis. 2017. What's the Difference Between AR, VR, and MR? (11 September 2017). Retrieved February 18, 2018 from: https://next.reality.news/news/whats-difference-between-ar-vr-and-mr-0171163/

[14]

Dan Ehninger, Weidong Li, Kevin Fox, Michael P. Stryker, & Alcino J. Silva. 2008. Reversing neurodevelopmental disorders in adults. Neuron, 60, 6 (December 2008), 950--960.

[15]

Mariano Etchart and Alessandro Caprarelli. 2018. A wearable immersive web-virtual reality approach to remote neurodevelopmental disorder therapy. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (AVI '18), Article 61, 3 pages.

Digital Library

[16]

Franca Garzotto and Matteo Forfori. 2006. FaTe2: storytelling edutainment experiences in 2D and 3D collaborative spaces. In Proceedings of the 2006 conference on Interaction design and children (IDC '06), 113--116.

Digital Library

[17]

Franca Garzotto, Mirko Gelsomini, Daniele Occhiuto, Vito Matarazzo, and Nicolò Messina. 2017. Wearable Immersive Virtual Reality for Children with Disability: a Case Study. In Proceedings of the 2017 Conference on Interaction Design and Children (IDC '17), 478--483.

Digital Library

[18]

Franca Garzotto, Nicolò Messina, Vito Matarazzo, Lukasz Moskwa, Gianluigi Oliva, and Riccardo Facchini. 2018. Empowering Interventions for Persons with Neurodevelopmental Disorders through Wearable Virtual Reality and Bio-sensors. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18), Paper LBW618, 6 pages.

Digital Library

[19]

Franca Garzotto, Matteo Valoriani, and Laura Bartoli. (2014). Touchless motion-based interaction for therapy of autistic children. In Virtual, Augmented Reality and Serious Games for Healthcare 1. Intelligent Systems Reference Library, Vol 68. Springer, Berlin, Heidelberg. 471--494.

[20]

Franca Garzotto, Mirko Gelsomini, Francesco Clasadonte, Daniele Montesano, and Daniele Occhiuto. 2016. Wearable Immersive Storytelling for Disabled Children. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16), 196--203.

Digital Library

[21]

Mirko Gelsomini, Franca Garzotto, Daniele Montesano, Daniele Occhiuto. 2016. Wildcard: A wearable virtual reality storytelling tool for children with intellectual developmental disability. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 5188--5191.

[22]

Mirko Gelsomini, Franca Garzotto, Vito Matarazzo, Nicolò Messina, and Daniele Occhiuto. 2017. Creating Social Stories as Wearable Hyper-Immersive Virtual Reality Experiences for Children with Neurodevelopmental Disorders. In Proceedings of the 2017 Conference on Interaction Design and Children (IDC '17), 431--437.

Digital Library

[23]

Gerardo Herrera, Rita Jordan, and Lucia Vera. 2006. Abstract concept and imagination teaching through virtual reality in people with autism spectrum disorders. Technology and Disability, 18, 4 (December 2006), 173--180.

[24]

Gerardo Herrera, Francisco Alcantud, Rita Jordan, Amparo Blanquer, Gabriel Labajo, Cristina De Pablo. 2008. Development of symbolic play through the use of virtual reality tools in children with autistic spectrum disorders: Two case studies. Autism, 12, 2 (March 2008), 143--157.

[25]

Naomi Josman, Hadass Milika Ben-Chaim, Shula Friedrich, and Patrice L. Weiss. (2008). Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism. International Journal on Disability and Human Development, 7, 1 (May 2011), 49--56.

[26]

Runpeng Liu, Joseph P. Salisbury, Arysha Vahabzadeh, & Ned T. Sahin. 2017. Feasibility of an autism-focused augmented reality smartglasses system for social communication and behavioral coaching. Frontiers in pediatrics, 5, (June 2017).

[27]

Microsoft. 2017. Microsoft HoloLens. Retrieved March 20, 2018 from: https://www.microsoft.com/en-us/hololens/

[28]

Microsoft. 2018. Microsoft Holographic Gaze. Retrieved April 11, 2018 from: https://docs.microsoft.com/en-us/windows/mixed-reality/gaze

[29]

Microsoft. 2018. Microsoft Holographic Gestures. Retrieved April 11, 2018 from: https://docs.microsoft.com/en-us/windows/mixed-reality/gestures

[30]

Microsoft. 2018. Microsoft Holographic Voice Input. Retrieved April 11, 2018 from: https://docs.microsoft.com/en-us/windows/mixed-reality/voice_input

[31]

Microsoft. 2018. Microsoft Holographic Spatial Mapping. Retrieved April 11, 2018 from: https://docs.microsoft.com/en-us/windows/mixed-reality/spatial-mapping

[32]

Microsoft. 2018. Microsoft Holographic Spatial Sound. Retrieved April 11, 2018 from: https://docs.microsoft.com/en-us/windows/mixed-reality/holograms-220

[33]

Microsoft. 2017. MixedRealityToolkit-Unity. Retrieved April 7, 2018 from: https://github.com/Microsoft/MixedRealityToolkit-Unity

[34]

Microsoft. 2017. Use the HoloLens clicker. Retrieved April 12, 2018 from: https://support.microsoft.com/it-it/help/12646/hololens-use-the-hololens-clicker

[35]

Microsoft. 2018. Visual Studio. Retrieved April 2, 2018 from: https://www.visualstudio.com/

[36]

Nigel Newbutt, Connie Sung, Hung Jen Kuo, Michael J. Leahy and Boyang Tong. 2016. Brief Report: A Pilot Study of the Use of a Virtual Reality Headset in Autism Populations. Journal of Autism and Developmental Disorders, 46, 6 (September 2016), 3166--3176.

[37]

Sarah Parsons & Peter Mitchell. 2002. The potential of virtual reality in social skills training for people with autistic spectrum disorders. Journal of Intellectual Disability Research, 46, 5 (April 2014), 430--443.

[38]

Dorothy Strickland. 1997. Virtual reality for the treatment of autism. Studies in health technology and informatics. 81--86.

[39]

Andrea Tartaro. 2006. Storytelling with a virtual peer as an intervention for children with autism. SIGACCESS Access. Comput. 84 (January 2006), 42--44.

Digital Library

[40]

Unity. 2018. Retrieved April 3, 2018 from: https://unity3d.com/

[41]

Unity. 2018. Unity User Manual. Retrieved April 3, 2018 from: https://docs.unity3d.com/Manual/index.html

[42]

U.S. Environmental Protection Agency. 2013. America's Children and the Environment, Third Edition. (January 2013). 233--253. Retrieved February 25, 2018 from: https://www.epa.gov/sites/production/files/2015-06/documents/ace3_2013.pdf

[43]

Simon Wallace, Sarah Parsons, Alice Westbury, Katie White, Kathy White, and Anthony Bailey. 2010. Sense of presence and atypical social judgments in immersive virtual environments Responses of adolescents with Autism Spectrum Disorders. Autism, 14, 3 (May 2010), 199--213.

Cited By

Lyu YLiu DAn PTong XZhang HKatsuragawa KZhao J(2024)EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR ActivitiesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997388:4(1-36)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699738
Leharanger MRodriguez Martinez EBalédent OVandromme L(2023)Familiarization with Mixed Reality for Individuals with Autism Spectrum Disorder: An Eye Tracking StudySensors10.3390/s2314630423:14(6304)Online publication date: 11-Jul-2023
https://doi.org/10.3390/s23146304
Bauer VBouchara TDuris OLabossière CClément MBourdot P(2023)Head-mounted augmented reality to support reassurance and social interaction for autistic children with severe learning disabilitiesFrontiers in Virtual Reality10.3389/frvir.2023.11060614Online publication date: 28-Jun-2023
https://doi.org/10.3389/frvir.2023.1106061
Show More Cited By

Index Terms

HoloLearn: Wearable Mixed Reality for People with Neurodevelopmental Disorders (NDD)
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
2. Social and professional topics
  1. User characteristics
    1. People with disabilities

Recommendations

HoloLens-based Mixed Reality Experiences for Subjects with Alzheimer's Disease
CHItaly '17: Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter

HoloLens technology enables mixed reality experiences that integrate holographic objects into the real world in which the mixed reality head-mounted device is used. Since HoloLens' launch in March 2016, some mixed-reality applications of this technology ...
Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

We present a mobile system that enhances mixed reality experiences and games with force feedback by means of electrical muscle stimulation (EMS). The benefit of our approach is that it adds physical forces while keeping the users' hands free to interact ...
Lifelogging in Mixed Reality
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Lifelogging is a growing social phenomenon where personal devices capture records of one’s life for future recalling and reminiscing. At the same time, mixed reality worlds are increasingly available due to advances in head-mounted display technology. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

October 2018

508 pages

ISBN:9781450356503

DOI:10.1145/3234695

General Chair:
Faustina Hwang
University of Reading, UK
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
David Flatla
University of Guelph, Canada

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASSETS '18

Sponsor:

SIGACCESS

ASSETS '18: The 20th International ACM SIGACCESS Conference on Computers and Accessibility

October 22 - 24, 2018

Galway, Ireland

Acceptance Rates

ASSETS '18 Paper Acceptance Rate 28 of 108 submissions, 26%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
732
Total Downloads

Downloads (Last 12 months)69
Downloads (Last 6 weeks)8

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lyu YLiu DAn PTong XZhang HKatsuragawa KZhao J(2024)EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR ActivitiesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997388:4(1-36)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699738
Leharanger MRodriguez Martinez EBalédent OVandromme L(2023)Familiarization with Mixed Reality for Individuals with Autism Spectrum Disorder: An Eye Tracking StudySensors10.3390/s2314630423:14(6304)Online publication date: 11-Jul-2023
https://doi.org/10.3390/s23146304
Bauer VBouchara TDuris OLabossière CClément MBourdot P(2023)Head-mounted augmented reality to support reassurance and social interaction for autistic children with severe learning disabilitiesFrontiers in Virtual Reality10.3389/frvir.2023.11060614Online publication date: 28-Jun-2023
https://doi.org/10.3389/frvir.2023.1106061
Vatavu R(2023)Leveraging Sensorimotor Realities for Assistive Technology Design Bridging Smart Environments and Virtual WorldsProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594834(247-253)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594834
de Carvalho ABraz Cdos Santos SFerreira RPrates R(2023)Serious Games for Children with Autism Spectrum Disorder: A Systematic Literature ReviewInternational Journal of Human–Computer Interaction10.1080/10447318.2023.219405140:14(3655-3682)Online publication date: 11-Apr-2023
https://doi.org/10.1080/10447318.2023.2194051
Delimitros MStergiouli AIatraki GMikropoulos TKoutromanos G(2022)A model for the design of immersive learning enactments for students with disabilitiesProceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion10.1145/3563137.3563145(141-145)Online publication date: 31-Aug-2022
https://dl.acm.org/doi/10.1145/3563137.3563145
de Carvalho ABraz CPrates RSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)How are games for autistic children being evaluated?Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3559127(1-13)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3559127
Colley MKränzle TRukzio E(2022)Accessibility-Related Publication Distribution in HCI Based on a Meta-AnalysisExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519701(1-28)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519701
Bauer VBouchara TBourdot P(2022)Extended Reality Guidelines for Supporting Autism Interventions Based on Stakeholders’ NeedsJournal of Autism and Developmental Disorders10.1007/s10803-022-05447-953:5(2078-2111)Online publication date: 4-Mar-2022
https://doi.org/10.1007/s10803-022-05447-9
Bauer VBouchara TDuris OLabossière CClément MBourdot P(2022)Evaluating the Acceptability and Usability of a Head-Mounted Augmented Reality Approach for Autistic Children with High Support NeedsVirtual Reality and Mixed Reality10.1007/978-3-031-16234-3_4(53-72)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-16234-3_4
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents