research-article

Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS

Authors:

Meredith Ringel MorrisAuthors Info & Claims

DIS '17: Proceedings of the 2017 Conference on Designing Interactive Systems

Pages 129 - 134

https://doi.org/10.1145/3064663.3064762

Published: 10 June 2017 Publication History

Abstract

Communicating at a natural speed is a significant challenge for users of augmentative and alternative communication (AAC) devices, especially when input is provided by eye gaze, as is common for people with ALS and similar conditions. One way to improve AAC throughput is by drawing on contextual information from the outside world. Toward this goal, we present SceneTalk, a prototype gaze-based AAC system that uses computer vision to identify objects in the user's field of view and suggests words and phrases related to the current scene. We conducted a formative evaluation of SceneTalk with six people with ALS, in which we evaluated their preference for user interface modes and output preferences. Participants agreed that integrating contextual awareness into their AAC device could be helpful across a diverse range of situations.

References

[1]

Meghan Allen, Joanna McGrenere, and Barbara Purves. 2007. The design and field evaluation of PhotoTalk: A digital image communication application for people. In Proceedings of ASSETS 2007, 187--194.

Digital Library

[2]

ALS Association. Epidemiology of ALS and Suspected Clusters. ALSA.org. Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html

[3]

Amazon Mechanical Turk. Retrieved January 1, 2017 from http://www.mturk.com/

[4]

David Beukelman, Susan Fager, and Amy Nordness. 2011. Communication support for people with ALS. Neurology Research International 2011.

[5]

David Beukelman, Pat Mirenda, Kathryn Garrett, and Janice Light. 2012. Augmentative and Alternative Communication: Supporting Children and Adults with Complex Communication Needs, Fourth Edition. Paul H. Brookes Publishing Co, Baltimore.

[6]

Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of UIST 2010, 333--342.

Digital Library

[7]

Carrie Demmans Epp, Justin Djordjevic, Shimu Wu, Karyn Moffatt, and Ronald M. Baecker. 2012. Towards providing just-in-time vocabulary support for assistive and augmentative communication. In Proceedings of IUI 2012, 33--36.

Digital Library

[8]

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K. Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From captions to visual concepts and back. In Proceedings of CVPR 2015, 1473--1482.

[9]

Anna Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish Kulkarni, Shaun Kane, and Meredith Ringel Morris. 2017. Toward everyday gaze input: Accuracy and precision of eye tracking and implications for design. In Proceedings of CHI 2017, to appear.

Digital Library

[10]

Alexander Fiannaca, Ann Paradiso, Mira Shah, and Meredith Ringel Morris. 2017. AACRobat: Using mobile devices to lower communication barriers and provide autonomy with gaze-based AAC. In Proceedings of CSCW 2017, 683--695.

Digital Library

[11]

D. Jeffery Higginbotham, Ann M. Bisantz, Michelle Sunm, Kim Adams, and Fen Yik. 2009. The effect of context priming and task type on augmentative communication performance. Augmentative and Alternative Communication 25, 1: 19--31.

[12]

Shaun K. Kane, Barbara Linam-Church, Kyle Althoff, and Denise McCall. 2012. What we talk about: Designing a context-aware communication tool for people with aphasia. In Proceedings of ASSETS 2012, 49--56.

Digital Library

[13]

Shaun K. Kane, Meredith Ringel Morris, Ann Paradiso, and Jon Campbell. 2017. "At times avuncular and cantankerous, with the reflexes of a mongoose": Understanding self-expression through augmentative and alternative communication devices. In Proceedings of CSCW 2017, 1166--1179.

Digital Library

[14]

Per Ola Kristensson and Keith Vertanen. 2012. The potential of dwell-free eye-typing for fast assistive gaze communication. In Proceedings of ETRA 2012, 241--244.

Digital Library

[15]

Andrew Kurauchi, Wenxin Feng, Ajjen Joshi, Carlos Morimoto, and Margrit Betke. 2016. EyeSwipe: Dwell-free text entry using gaze paths. In Proceedings of CHI 2016, 1952--1956.

Digital Library

[16]

I. Scott MacKenzie and Xuang Zhang. 2008. Eye typing using word and letter prediction and a fixation algorithm. In Proceedings of ETRA 2008, 55--58.

Digital Library

[17]

Päivi Majaranta, Ulla-Kaija Ahola, and Oleg Špakov. 2009. Fast gaze typing with an adjustable dwell time. In Proceedings of CHI 2009, 357--360.

Digital Library

[18]

Päivi Majaranta and Kari-Jouko Räihä. 2002. Twenty years of eye typing: systems and design issues. In Proceedings of ETRA 2002, 15--22.

Digital Library

[19]

Microsoft Cognitive Services. Retrieved January 1, 2017 from http://www.microsoft.com/cognitiveservices

[20]

Martez Mott, Shane Williams, Jacob O. Wobbrock, and Meredith Ringel Morris. 2017. Improving dwellbased gaze typing with dynamic, cascading dwell times. In Proceedings of CHI 2017, to appear.

Digital Library

[21]

Joan Murphy. 2004. "I prefer contact this close": Perceptions of AAC by people with motor neurone disease and their communication partners. Augmentative and Alternative Communication 20, 4: 259--271.

[22]

Diogo Pedrosa, Maria Da Graça Pimentel, Amy Wright, and Khai N. Truong. 2015. Filteryedping: Design challenges and user performance of dwell-free eye typing. ACM Transactions on Accessible Computing 6, 1: 1--37.

Digital Library

[23]

Keith Trnka, Debra Yarrington, John McCaw, Kathleen F. McCoy, and Christopher Pennington. 2007. The effects of word prediction on communication rate for AAC. In Proceedings of HLT 2007, 173--176.

Digital Library

[24]

Outi Tuisku, Päivi Majaranta, Poika Isokoski, and Kari-Jouko Räihä. 2008. Now Dasher! Dash away!: Longitudinal study of fast text entry by eye gaze. In Proceedings of ETRA 2008, 19--26.

Digital Library

[25]

David J. Ward and David JC MacKay. 2002. Fast hands-free writing by gaze direction. Nature 418.6900: 838.

[26]

Kristin Williams, Karyn Moffatt, Denise McCall, and Leah Findlater. 2015. Designing conversation cues on a head-worn display to support persons with aphasia. In Proceedings of CHI 2015, 231--240.

Digital Library

[27]

Shaomei Wu, Jeffrey Wieland, Omar Farivar, and Jill Schiller. 2017. Automatic alt-text: Computer-generated image descriptions for blind users on a social network service. In Proceedings of CSCW 2017, 1180--1192.

Digital Library

[28]

Kathryn M. Yorkston and David R. Beukelman. 1981. Communication efficiency of dysarthric speakers as measured by sentence intelligibility and speaking rate. Journal of Speech and Hearing Disorders 46, 3: 296--301.

Cited By

Villa SWeiss YLu MZiarko MSchmidt ANiess J(2024)Envisioning Futures: How the Modality of AI Recommendations Impacts Conversation Flow in AR-enhanced DialogueProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685731(182-193)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685731
Curtis HLau YNeate T(2024)Breaking Badge: Augmenting Communication with Wearable AAC Smartbadges and DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642327(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642327
Cai SVenugopalan STomanek KKane SMorris MCave RMacdonald RCampbell JCasey BKornman EVance DBeavers J(2023)SpeakFaster Observer: Long-Term Instrumentation of Eye-Gaze Typing for Measuring AAC CommunicationExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3573870(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3573870
Show More Cited By

Index Terms

Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS
1. Social and professional topics

Recommendations

An augmentative and alternative communication tool for children and adolescents with cerebral palsy
Medical teamwork, collaboration and patient-centred care

People with disabilities face many accessibility problems in their daily lives. One such group of disabled people, children who suffer from cerebral palsy, faces specific challenges, which arise due to motor dysfunction and communication disorders. In ...
Towards a Methodology to Support Augmentative and Alternative Communication by means of Personalized Gestural Interaction
IHC '18: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems

Augmentative and Alternative Communication (AAC) involves the use of non-verbal modes as a complement or substitute for spoken language, supporting communicative abilities of people, especially people with speech limitations. Computing systems have been ...
Personalized interactive gesture recognition assistive technology
IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems

Computing systems have the potential to contribute with interactive and low-cost solutions to support Augmentative and Alternative Communication (AAC) by applying different technologies to address different users' characteristics and needs. Computer ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '17: Proceedings of the 2017 Conference on Designing Interactive Systems

June 2017

1444 pages

ISBN:9781450349222

DOI:10.1145/3064663

General Chair:
Oli Mival
Edinburgh Napier University, UK
,
Program Chairs:
Michael Smyth
Edinburgh Napier University, UK
,
Peter Dalsgaard
Aarhus University, Denmark

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

DIS '17

Sponsor:

SIGCHI

DIS '17: Designing Interactive Systems Conference 2017

June 10 - 14, 2017

Edinburgh, United Kingdom

Acceptance Rates

DIS '17 Paper Acceptance Rate 107 of 487 submissions, 22%;

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
374
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Villa SWeiss YLu MZiarko MSchmidt ANiess J(2024)Envisioning Futures: How the Modality of AI Recommendations Impacts Conversation Flow in AR-enhanced DialogueProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685731(182-193)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685731
Curtis HLau YNeate T(2024)Breaking Badge: Augmenting Communication with Wearable AAC Smartbadges and DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642327(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642327
Cai SVenugopalan STomanek KKane SMorris MCave RMacdonald RCampbell JCasey BKornman EVance DBeavers J(2023)SpeakFaster Observer: Long-Term Instrumentation of Eye-Gaze Typing for Measuring AAC CommunicationExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3573870(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3573870
Valencia SCave RKallarackal KSeaver KTerry MKane S(2023)“The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC UsersProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581560(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581560
Ivanyi BTjemsland TTsalidis de Zabala CToth LDyrholm MNaylor SParadiso ALamb DChudge JAdjorlu ASerafin S(2023)DuoRhythmo: Design and remote user experience evaluation (UXE) of a collaborative accessible digital musical interface (CADMI) for people with ALS (PALS)Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581285(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581285
Udgata SKumar G(2023)Internet of Things-Based Digital Life to Provide Assisted Living for Visually Challenged PersonsNovel Developments in Futuristic AI-based Technologies10.1007/978-981-99-3076-0_1(1-18)Online publication date: 20-Jul-2023
https://doi.org/10.1007/978-981-99-3076-0_1
Curtis HNeate TVazquez Gonzalez C(2022)State of the Art in AAC: A Systematic Review and TaxonomyProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544810(1-22)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544810
Beneteau EParadiso APratt W(2021)Telehealth experiences of providers and patients who use augmentative and alternative communicationJournal of the American Medical Informatics Association10.1093/jamia/ocab27329:3(481-488)Online publication date: 13-Dec-2021
https://doi.org/10.1093/jamia/ocab273
Kane SGuo AMorris M(2020)Sense and AccessibilityProceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3373625.3416990(1-14)Online publication date: 26-Oct-2020
https://dl.acm.org/doi/10.1145/3373625.3416990
Beneteau EBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Who Are You Asking?: Qualitative Methods for Involving AAC Users as Primary Research ParticipantsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376883(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376883
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents