Sign Language-Based versus Touch-Based Input for Deaf Users with Interactive Personal Assistants in Simulated Kitchen Environments
Authors: 
Paige S Devries, Nina Tran, Keith Delk, Melanie Miga, Richard Carlisle Taulbee, Pranav Pidathala, 
Abraham Glasser, 
Raja Kushalnagar, Christian VoglerAuthors Info & Claims
Article No.: 290, Pages 1 - 9
Abstract
In this study, we assess the usability of interactive personal assistants (IPAs), such as Amazon Alexa, in a simulated kitchen smart home environment, with deaf and hard of hearing users. Participants engage in activities in a way that causes their hands to get dirty. With these dirty hands, they are tasked with two different input methods for IPAs: American Sign Language (ASL) in a Wizard-of-Oz design, and smart home apps with a touchscreen. Usability ratings show that participants significantly preferred ASL over touch-based apps with dirty hands, although not to a larger extent than in comparable previous work with clean hands. Participants also expressed significant enthusiasm for ASL-based IPA interaction in Netpromoter scores and in questions about their overall preferences. Preliminary observations further suggest that having dirty hands may affect the way people sign, which may pose challenges for building IPAs that natively support sign language input.
Supplemental Material
MP4 File
Talk Video
- Download
- 52.83 MB
- Transcript
References
[1]
Aashaka Desai, Lauren Berger, Fyodor O. Minakov, Vanessa Milan, Chinmay Singh, Kriston Pumphrey, Richard E. Ladner, Hal Daumé, Alex X. Lu, Naomi Caselli and Danielle Bragg. 2023. ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition. arXiv:2304.05934v2 [cs.CV]. https://doi.org/10.48550/arXiv.2304.05934
[2]
Amazon. [n.d.]. What is Tap to Alexa? https://www.amazon.com/b?ie=UTF8&node=21213735011
[3]
Amazon. [n.d.]. Alexa App. https://www.amazon.com/Alexa-App/b?ie=UTF8&node=18354642011
[4]
Apple. [n.d.]. Photo Booth. https://apps.apple.com/us/app/photo-booth/id1208226939
[5]
Apple. [n.d.] Improved Speech Recognition for People Who Stutter. Retrieved from https://machinelearning.apple.com/research/speech-recognition
[6]
Aaron Bangor, Philip Kortum and James Miller. 2009. Determining what Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies 4(3), 114–123.
[7]
Larwan Berke, Matt Huenerfauth and Kasmira Patel. 2019. Design and Psychometric Evaluation of American Sign Language Translations of Usability Questionnaires. ACM Transactions on Accessible Computing (TACCESS), 12(2), 1-43.
[8]
Jeffrey P. Bigham, Raja Kushalnagar, Ting-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage. 2017. On how deaf people might use speech to control devices." In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 383-384.
[9]
Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '19). Association for Computing Machinery, New York, NY, USA, 16–31. https://doi.org/10.1145/3308561.3353774
[10]
John Brooke. 2013. SUS: A Retrospective. Journal of Usability Studies 8(2), 29–40.
[11]
Raymond Fok, Harmanpreet Kaur, Skanda Palani, Martez E. Mott, and Walter S. Lasecki. 2018. Towards More Robust Speech Interactions for Deaf and Hard of Hearing Users. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18). Association for Computing Machinery, New York, NY, USA, 57–67. https://doi.org/10.1145/3234695.3236343
[12]
Abraham Glasser, Kesavan Kushalnagar, and Raja Kushalnagar. 2017. Deaf, hard of hearing, and hearing perspectives on using automatic speech recognition in conversation. Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility.
[13]
Abraham Glasser. 2019. Automatic Speech Recognition Services: Deaf and Hard-of-Hearing Usability. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). Association for Computing Machinery, New York, NY, USA, Paper SRC06, 1–6. https://doi.org/10.1145/3290607.3308461
[14]
Abraham Glasser, Vaishnavi Mande, and Matt Huenerfauth. 2021. Understanding Deaf and Hard-of-Hearing Users’ Interest in Sign-Language Interaction with Personal-Assistant Devices. In Proceedings of the 18th International Web for All Conference (W4A ‘21), April 19-20, 2021, Ljubljana, Slovenia. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3430263.3452428
[15]
Abraham Glasser, Matthew Watkins, Kira Hart, Sooyeon Lee, and Matt Huenerfauth. 2022. Analyzing Deaf and Hard-of-Hearing Users’ Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 306, 1–12. https://doi.org/10.1145/3491102.3501987
[16]
Abraham Glasser. 2023. Empirical Investigations and Dataset Collection for American Sign Language-Aware Personal Assistants. Available from ProQuest Dissertations & Theses Global. (2796329430).
[17]
Google. 2023. Project Euphonia. Retrieved from https://sites.research.google/euphonia/about/
[18]
Linda Gottermeier, and Raja Kushalnagar. 2016. User evaluation of automatic speech recognition systems for deaf-hearing interactions at school and work. Audiology Today 28.2 (2016): 20-34.
[19]
Francois Grosjean. 2010. Bilingualism, biculturalism, and deafness. International Journal of Bilingual Education and Bilingualism, 13(2), pp.133-145.
[20]
Larry Hardesty. 2021. Voiceitt Extends The Voice Revolution To People With Nonstandard Speech. (June 2021). Retrieved January 4, 2024 from http://ccrma.stanford.edu/∼jos/bayes/bayes.html https://www.amazon.science/latest-news/voiceitt-extends-the-voice-revolution-to-people-with-nonstandard-speech
[21]
Charlene A. Johnson. 2010. Articulation of Deaf and Hearing Spaces Using Deaf Space Design Guidelines: A Community Based Participatory Research with the Albuquerque Sign Language Academy. https://digitalrepository.unm.edu/arch_etds/18/
[22]
Valerie K. Jones. 2022. Why people use virtual assistants: Understanding engagement with Alexa. Journal of Brand Strategy, 11(1), pp.80-101.
[23]
Bret Kinsella and Ava Mutchler. 2018. Smart Speaker Consumer Adoption Report. Voicebot.ai. https://voicebot.ai/wp-content/uploads/2018/10/voicebot-smart-speaker-consumer-adoption-report.pdf
[24]
Kashyap Kompella. 2023. Recognizing Atypical Speech Is ASR's Achilles’ Heel. Speech Technology, vol. 28, no. 3, p. 8.
[25]
Franklin Mingzhe Li, Jamie Dorst, Peter Cederberg, and Patrick Carrington. 2021. Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '21). Association for Computing Machinery, New York, NY, USA, Article 30, 1–11. https://doi.org/10.1145/3441852.3471215
[26]
Irene Lopatovska, Katrina Rink, Ian Knight, Kieran Raines, Kevin Cosenza, Harriet Williams, Perachya Sorsche, David Hirsch, Qi Li, and Adriana Martinez. 2018. Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science, 51(4), 984–997. https://doi.org/10.1177/0961000618759414
[27]
Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017. Comparing Social Robot, Screen and Voice Interfaces for Smart-Home Control. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 580–592. https://dl.acm.org/doi/10.1145/3025453.3025786
[28]
Vaishnavi Mande, Abraham Glasser, Becca Dingman, and Matt Huenerfauth. 2021. Deaf Users’ Preferences Among Wake-Up Approaches during Sign-Language Interaction with Personal Assistant Devices. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA '21). Association for Computing Machinery, New York, NY, USA, Article 370, 1–6. https://doi.org/10.1145/3411763.3451592
[29]
Campbell McDermid. 2018. Learning to Interpret: Working from English into American Sign Language. RIT Press.
[30]
Ross E. Mitchell, Travas A. Young, Bellamie Bachleda, and Michael A. Karchmer. 2005. How Many People Use ASL in the United States? Why Estimates Need Updating. In Sign language Studies, Volume 6, Number 3, 2006. 37 pages.
[31]
Philips. [n.d.]. The Philips Hue app. https://www.philips-hue.com/en-us/explore-hue/apps/bridge
[32]
RealVNC. [n.d.] VNC Viewer. https://www.realvnc.com/en/connect/download/viewer/
[33]
Jason Rodolitz, Evan Gambill, Brittany Willis, Christian Vogler, and Raja Kushalnagar. 2019. Accessibility of Voice-Activated Agents for People who are Deaf or Hard of Hearing. Journal on Technology and Persons with Disabilities 7 (2019), 144–156. http://hdl.handle.net/10211.3/210397
[34]
Jeff Sauro and James R. Lewis. 2011. When designing usability questionnaires, does it hurt to be positive? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). Association for Computing Machinery, New York, NY, USA, 2215–2224. https://doi.org/10.1145/1978942.1979266
[35]
Jeff Sauro. 2012. Predicting Net Promoter Scores from System Usability Scale Scores. MeasuringU. https://measuringu.com/nps-sus/ (last accessed: January 25, 2024)
[36]
Julia Schwarz, Charles Claudius Marais, Tommer Leyvand, Scott E. Hudson, and Jennifer Mankoff. 2014. Combining body pose, gaze, and gesture to determine intention to interact in vision-based interfaces. In CHI ’14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 26-May 1, 2014, Toronto, Canada, 9 pages. https://doi.org/10.1145/2556288.2556989
[37]
Nina Tran, Paige DeVries, Matthew Seita, Raja Kushalnagar, Abraham Glasser and Christian Vogler. 2024. Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI 2024). https://doi.org/10.1145/3613904.3642094
[38]
Gabriella Wojtanowski, Colleen Gilmore, Barbra Seravalli, Kristen Fargas, Christian Vogler, and Raja Kushalnagar. 2020. “Alexa, Can You See Me?” Making Individual Personal Assistants for the Home Accessible to Deaf Consumers. The Journal on Technology and Persons with Disabilities (2020), 130.
[39]
Robert F Woolson. 2007. Wilcoxon signed‐rank test. Wiley encyclopedia of clinical trials, 1-3.
[40]
World Federation of the Deaf. [n.d.]. Our work. http://wfdeaf.org/our-work/
Index Terms
- Sign Language-Based versus Touch-Based Input for Deaf Users with Interactive Personal Assistants in Simulated Kitchen Environments
Recommendations
Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing SystemsWith the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and ...
Deaf Users’ Preferences Among Wake-Up Approaches during Sign-Language Interaction with Personal Assistant Devices
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsPersonal-assistant devices like Amazon Alexa and Google Assistant are increasingly popular among consumers. Users activate these systems through some type of wake-up approach, e.g. using a wake-word “Alexa” or “Ok, Google.” Voice-based interaction poses ...
Analyzing Deaf and Hard-of-Hearing Users’ Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsAs voice-based personal assistant technologies proliferate, e.g., smart speakers in homes, and more generally as voice-control of technology becomes increasingly ubiquitous, new accessibility barriers are emerging for many Deaf and Hard of Hearing (DHH) ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Data Availability
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 148Total Downloads
- Downloads (Last 12 months)148
- Downloads (Last 6 weeks)44
Reflects downloads up to 26 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format