Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Automated Behavior Labeling During Team-Based Activities Involving Neurodiverse and Neurotypical Partners Using Multimodal Data

  • Conference paper
  • First Online:
Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges (ICPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13643))

Included in the following conference series:

  • 790 Accesses

Abstract

The employment setting for autistic individuals in the USA is grim. Based on reports, individuals with ASD struggle to secure and retain employment due to challenges in communicating and collaborating with others in workplace settings which is often attributed to their social skills deficit. Current programs that support collaborative skills development in vocational settings rely on manual evaluation and feedback by human observers, which can be resource straining and receptive to bias. Using a collaborative virtual environment (CVE) allows neurodiverse individuals to develop teamwork skills by working together with a neurotypical partner in a shared virtual space. An effective CVE system can provide real-time prompts by recognizing the user’s behavior to promote teamwork. As such, it is crucial to be able to automatically label both users’ behaviors. In this paper, we propose using K-means clustering to automate behavior labeling in a workplace CVE. The results show that K-means clustering enables high accuracy in predicting the user’s behavior, therefore, confirming that it can be used in future studies to support real-time prompts to encourage teamwork in a CVE.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We are using both identity-first and people-first language to respect both views by interchangeably using the term ‘autistic individuals and ‘individuals with ASD’. [32].

References

  1. Neurodiversity hiring: Global diversity and inclusion at microsoft. https://www.crosoft.com/en-us/diversity/inside-microsoft/cross-disability/neurodiversityhiring

  2. Specialsterne: Assessment. https://www.specialisterneni.com/about-us/assessment/

  3. Unity website. https://unity3d.com/unity

  4. Afsar, P., Cortez, P., Santos, H.: Automatic visual detection of human behavior: a review from 2000 to 2014. Expert Syst. Appl. 42(20), 6935–6956 (2015). https://doi.org/10.1016/j.eswa.2015.05.023. https://www.sciencedirect.com/science/article/pii/S0957417415003516

  5. Agran, M., Hughes, C., Thoma, C., Scott, L.: Employment social skills: What skills are really valued? Career Develop. Trans. Except. Individ. 39, 111–120 (2014). https://doi.org/10.1177/2165143414546741

  6. Almaguer, E., Yasmin, S.: A haptic virtual kitchen for the cognitive empowerment of children with autism spectrum disorder. In: Stephanidis, C., Antona, M. (eds.) HCII 2019. CCIS, vol. 1088, pp. 137–142. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30712-7_18

    Chapter  Google Scholar 

  7. Amat, A., et al.: Collaborative virtual environment to encourage teamwork in autistic adults in workplace settings, pp. 339–348 (2021). https://doi.org/10.1007/978-3-030-78092-0_22

  8. Apostolellis, P., Bowman, D.: C-olive: Group co-located interaction in VEs for contextual learning, pp. 129–130 (2014). https://doi.org/10.1109/VR.2014.6802085

  9. Ashton, M.C., Lee, K.: The HEXACO-60: a short measure of the major dimensions of personality. J. Pers. Assess. 91(4), 340–345 (2009). https://doi.org/10.1080/00223890902935878

    Article  Google Scholar 

  10. Basden, B., Basden, D., Bryner, S., Thomas, R.r.: Developing methods for understanding social behavior in a 3D virtual learning environment. J. Exp. Psychol. Learn Mem. Cogn. 23(5), 1176–91 (1997). https://doi.org/10.1037//0278-7393.23.5.1176

  11. Bekele, E., Zheng, Z., Swanson, A.R., Crittendon, J.A., Warren, Z., Sarkar, N.: Understanding how adolescents with autism respond to facial expressions in virtual reality environments. IEEE Trans. Visual Comput. Graphics 19, 711–720 (2013)

    Article  Google Scholar 

  12. Bernard-Opitz, Vand Ross, K., Tuttas, M.: Computer assisted instruction for autistic children. Ann. Acad. Med. Singap. 19(5), 611–616 (1990)

    Google Scholar 

  13. Bernard-Opitz, V., Sriram, N., Nakhoda, S.: Enhancing social problem solving in children with autism and normal children through computer-assisted instruction. J. Autism Develop. Disord. 31, 377–84 (2001). https://doi.org/10.1023/A:1010660502130

  14. Bian, D., et al.: A novel virtual reality driving environment for autism intervention, pp. 474–483 (2013). https://doi.org/10.1007/978-3-642-39191-0_52

  15. Bird, G., Cook, R.: Mixed emotions: the contribution of alexithymia to the emotional symptoms of autism. Transl. Psych. 3, e285 (2013). https://doi.org/10.1038/tp.2013.61

  16. Breland, J., Shiratuddin, M.F.: A study on collaborative design in a virtual environment. Int. J. Learn. 16, 385–398 (2009). https://doi.org/10.18848/1447-9494/CGP/v16i03/46179

  17. Checa, D., Bustillo, A.: A review of immersive virtual reality serious games to enhance learning and training. Multimedia Tools Appl. 79(9), 5501–5527 (2019). https://doi.org/10.1007/s11042-019-08348-9

    Article  Google Scholar 

  18. Chen, W.: Multitouch tabletop technology for people with autism spectrum disorder: a review of the literature. Procedia Comput. Sci. 14, 198–207 (2012). https://doi.org/10.1016/j.procs.2012.10.023

  19. Constantino, J., Gruber, C.: Social responsiveness scale (srs). Western Psychological Services Los Angeles (2005)

    Google Scholar 

  20. Creed, P., Macintyre, S.: The relative effects of deprivation of the latent and manifest benefits of employment on the well-being of unemployed people. J. Occup. Health Psychol. 6, 324–31 (2001). https://doi.org/10.1037/1076-8998.6.4.324

  21. Deiglmayr, A., Spada, H., Rummel, N.: A rating scheme for assessing the quality of computer-supported collaboration processes. Int. J. Comput.-Support. Collaborative Learn. 2(1), 63–86 (2007). https://doi.org/10.1007/s11412-006-9005-x

  22. Duchenne, O., Laptev, I., Sivic, J., Bach, F., Ponce, J.: Automatic annotation of human actions in video, pp. 1491–1498 (2009). https://doi.org/10.1109/ICCV.2009.5459279

  23. D’Mello, S., Olney, A., Person, N.: Mining collaborative patterns in tutorial dialogues. J. Educ. Data Min. 2, 1–37 (2010)

    Google Scholar 

  24. Echeverria, V., Martinez-Maldonado, R., Buckingham Shum, S.: Towards collaboration translucence: giving meaning to multimodal group data (2019). https://doi.org/10.1145/3290605.3300269

  25. Fredriksson, T., Issa Mattos, D., Bosch, J., Olsson, H.: Data Labeling: an Empirical Investigation into Industrial Challenges and Mitigation Strategies, pp. 202–216 (2020). https://doi.org/10.1007/978-3-030-64148-1_13

  26. Hanmin, Y., Hao, L., Qianting, S.: An improved semi-supervised k-means clustering algorithm, pp. 41–44 (2016). https://doi.org/10.1109/ITNEC.2016.7560315

  27. Hayashi, Y.: Detecting collaborative learning through emotions: an investigation using facial expression recognition, pp. 89–98 (2019). https://doi.org/10.1007/978-3-030-22244-4_12

  28. Hendricks, D.: A short review on the current understanding of autism spectrum disorders. J. Vocat. Rehabil. 32(2), 125–134 (2010)

    Article  Google Scholar 

  29. Hoffman, G., Breazeal, C.: Collaboration in human-robot teams. Collection of Technical Papers - AIAA 1st Intelligent Systems Technical Conference 2 (09 2004). https://doi.org/10.2514/6.2004-6434

  30. Hong, J.H., Ramos Rojas, J., Dey, A.: Toward personalized activity recognition systems with a semipopulation approach. IEEE Transactions on Human-Machine Systems, pp. 1–12 (2015). https://doi.org/10.1109/THMS.2015.2489688

  31. Inc., T.M.: Matlab (2018)

    Google Scholar 

  32. Kenny, L., Hattersley, C., Molins, B., Buckley, C., Povey, C., Pellicano, E.: Which terms should be used to describe autism? perspectives from the UK autism community. Autism 20(4), 442–62 (2016). https://doi.org/10.1177/1362361315588200

    Article  Google Scholar 

  33. Kulman, R., Slobuski, T., Seitsinger, R.: Teaching 21st Century, Executive-Functioning, and Creativity Skills with Popular Video Games and Apps, pp. 159–174. ETC Press, Pittsburgh, PA, USA (2014). https://doi.org/10.5555/2811147.2811157

  34. Kusumaningrum, R., Farikhin, F.: An automatic labeling of K-means clusters based on chi-square value. J. Phys. Conf. Ser. 801, 012071 (2017). https://doi.org/10.1088/1742-6596/801/1/012071

  35. Litchfield, P., Cooper, C., Hancock, C., Watt, P.: Work and wellbeing in the 21st century \(\dagger \). Int. J. Environ. Res. Public Health 13, 1065 (2016). https://doi.org/10.3390/ijerph13111065

  36. Lord, C., et al.: The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J. Autism Develop. Disorders 30, 205–223 (2000). https://doi.org/10.1023/A:1005592401947

  37. M. Khawam, A., DiDona, T., S. Hern, B.: Effectiveness of teamwork in the workplace. Int. J. Sci. Basic Appl. Res. (IJSBAR) 32(3), 267–286 (2017). https://gssrr.org/index.php/JournalOfBasicAndApplied/article/view/7134

  38. Martinez-Maldonado, R., Gasevic, D., Echeverria, V., Nieto, G., Swiecki, Z., Buckingham Shum, S.: What do you mean by collaboration analytics? a conceptual model. J. Learn. Analyt. 8, 126–153 (2021). https://doi.org/10.18608/jla.2021.7227

  39. Mihoub, A., Lefebvre, G.: Wearables and social signal processing for smarter public presentations. ACM Trans. Interact. Intell. Syst. 9, 9 (2018). https://doi.org/10.1145/3234507

  40. Müller, P., et al.: Multimediate: multi-modal group behaviour analysis for artificial mediation, pp. 4878–4882 (2021). https://doi.org/10.1145/3474085.3479219

  41. Norris, M.W., Spicer, K., Byrd, T.: Virtual reality: the new pathway for effective safety training. Professional Saf. 64(06), 36–39 (2019)

    Google Scholar 

  42. Okada, S., et al.: Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets, pp. 169–176 (2016). https://doi.org/10.1145/2993148.2993154

  43. Palliya Guruge, C., Oviatt, S., Haghighi, P., Pritchard, E.: Advances in multimodal behavioral analytics for early dementia diagnosis: a review, pp. 328–340 (2021). https://doi.org/10.1145/3462244.3479933

  44. Park, H.R., et al.: A short review on the current understanding of autism spectrum disorders. Exper. Neurobiol. 25, 1 (2016). https://doi.org/10.5607/en.2016.25.1.1

  45. Parsons, S., Mitchell, P.: The potential of virtual reality in social skills training for people with autistic spectrum disorders. J. Intell. Disability Res. JIDR 46, 430–443 (2002). https://doi.org/10.1046/j.1365-2788.2002.00425.x

  46. Putnam, C., Chong, L.: Software and technologies designed for people with autism: what do users want? abstract, pp. 3–10 (2008). https://doi.org/10.1145/1414471.1414475

  47. Saiano, M., et al.: Natural interfaces and virtual environments for the acquisition of street crossing and path following skills in adults with autism spectrum disorders: a feasibility study. J. Neuroeng. Rehabil. 12, 17 (2015). https://doi.org/10.1186/s12984-015-0010-z

  48. Salah, A., Gevers, T., Sebe, N., Vinciarelli, A.: Challenges of human behavior understanding, vol. 6219, pp. 1–12 (2010). https://doi.org/10.1007/978-3-642-14715-9_1

  49. Salah, A., Pantic, M., Vinciarelli, A.: Recent developments in social signal processing, pp. 380–385 (2011). https://doi.org/10.1109/ICSMC.2011.6083695

  50. Sanyal, S., Hisam, M.: The impact of teamwork on work performance of employees: a study of faculty members in Dhofar university 20 (2018). https://doi.org/10.9790/487X-2003011522

  51. Schmidt, M., Laffey, J., Schmidt, C., Wang, X., Stichter, J.: Developing methods for understanding social behavior in a 3D virtual learning environment. Comput. Human Behav. 28, 405–413 (2012). https://doi.org/10.1016/j.chb.2011.10.011

  52. Scott, M., Falkmer, M., Girlder, S., Falkmer, T.: Viewpoints on factors for successful employment for adults with autism spectrum disorder. PLoS One 6, 0139281 (2015). https://doi.org/10.1371/journal.pone.0139281

  53. Stauch, T.A., Plavnick, J.B.: Teaching vocational and social skills to adolescents with Autism using video modeling. Educ. Treat. Child. 43(2), 137–151 (2020). https://doi.org/10.1007/s43494-020-00020-4

    Article  Google Scholar 

  54. Sung, C., Connor, A., Chen, J., Lin, C.C., Kuo, H.J., Chun, J.: Development, feasibility, and preliminary efficacy of an employment-related social skills intervention for young adults with high-functioning autism. Autism 23, 136236131880134 (2018). https://doi.org/10.1177/1362361318801345

  55. Taylor, J., Seltzer, M.: Employment and post-secondary educational activities for young adults with autism spectrum disorders during the transition to adulthood. J. Autism Dev. Disord. 41(5), 566–74 (2011). https://doi.org/10.1007/s10803-010-1070-3

    Article  Google Scholar 

  56. Vajda, S., Rangoni, Y., Cecotti, H.: Semi-automatic ground truth generation using unsupervised clustering and limited manual labeling: application to handwritten character recognition. Pattern Recogn. Lett. 58, 23–28 (2015). https://doi.org/10.1016/j.patrec.2015.02.001. https://www.sciencedirect.com/science/article/pii/S0167865515000380

  57. Valencia, K., Rusu, C., Quiñones, D., Jamet, E.: The impact of technology on people with autism spectrum disorder: systematic literature review. Sensors 19(20) (2019). https://doi.org/10.3390/s19204485

  58. Walsh, E., Holloway, J., Lydon, H.: An evaluation of a social skills intervention for adults with autism spectrum disorder and intellectual disabilities preparing for employment in ireland: a pilot study. J. Autism Dev. Disord. 48(5), 1727–1741 (2017). https://doi.org/10.1007/s10803-017-3441-5

    Article  Google Scholar 

  59. Williams, C., Wright, B., Callaghan, G., Coughlan, B.: Do children with autism learn to read more readily by computer assisted instruction or traditional book methods?: A pilot study. Autism : Int. Jo. Res. Pract. 6, 71–91 (2002). https://doi.org/10.1177/1362361302006001006

  60. Zhang, L., Warren, Z., Swanson, A., Weitlauf, A., Sarkar, N.: Understanding performance and verbal-communication of children with ASD in a collaborative virtual environment. J. Autism Dev. Disord. 48(8), 2779–2789 (2018). https://doi.org/10.1007/s10803-018-3544-7

    Article  Google Scholar 

  61. Zhao, Z., Chen, Y., Liu, J., Shen, Z., Liu, M.: Cross-people mobile-phone based activity recognition, pp. 2545–2550 (2011). https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-423

  62. Zheng, Z., et al.: Impact of robot-mediated interaction system on joint attention skills for children with autism. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), pp. 1–8 (2013)

    Google Scholar 

Download references

Acknowledgments

We are grateful for the support provided by NSF grants 1936970 and 2033413 as well as NSF NRT grant DGE 19-22697 for this research. We would also like to thank the Vanderbilt Treatment and Research Institute for Autism Spectrum Disorders (TRIAD) team; Amy Weitlauf and Amy Swanson for their expert advice on interventions for autistic individuals and recruitments for the study. The authors are solely responsible for the contents and opinions expressed in this manuscript.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Engineering, Vanderbilt University, Nashville, TN, USA

    Abigale Plunk, Ashwaq Zaini Amat, D. Mitchell Wilkes & Nilanjan Sarkar

  2. Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, USA

    Nilanjan Sarkar

Authors
  1. Abigale Plunk
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashwaq Zaini Amat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Mitchell Wilkes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nilanjan Sarkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abigale Plunk .

Editor information

Editors and Affiliations

  1. York University, Toronto, ON, Canada

    Jean-Jacques Rousseau

  2. Ontario Tech University, Oshawa, ON, Canada

    Bill Kapralos

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Plunk, A., Amat, A.Z., Wilkes, D.M., Sarkar, N. (2023). Automated Behavior Labeling During Team-Based Activities Involving Neurodiverse and Neurotypical Partners Using Multimodal Data. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13643. Springer, Cham. https://doi.org/10.1007/978-3-031-37660-3_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37660-3_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37659-7

  • Online ISBN: 978-3-031-37660-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation