research-article

Speech Interaction with Personal Assistive Robots Supporting Aging at Home for Individuals with Alzheimer’s Disease

Authors:

Alex MihailidisAuthors Info & Claims

ACM Transactions on Accessible Computing (TACCESS), Volume 7, Issue 2

Article No.: 6, Pages 1 - 22

https://doi.org/10.1145/2744206

Published: 22 May 2015 Publication History

Abstract

Increases in the prevalence of dementia and Alzheimer’s disease (AD) are a growing challenge in many nations where healthcare infrastructures are ill-prepared for the upcoming demand for personal caregiving. To help individuals with AD live at home for longer, we are developing a mobile robot, called ED, intended to assist with activities of daily living through visual monitoring and verbal prompts in cases of difficulty. In a series of experiments, we study speech-based interactions between ED and each of 10 older adults with AD as the latter complete daily tasks in a simulated home environment. Traditional automatic speech recognition is evaluated in this environment, along with rates of verbal behaviors that indicate confusion or trouble with the conversation. Analysis reveals that speech recognition remains a challenge in this setup, especially during household tasks with individuals with AD. Across the verbal behaviors that indicate confusion, older adults with AD are very likely to simply ignore the robot, which accounts for over 40% of all such behaviors when interacting with the robot. This work provides a baseline assessment of the types of technical and communicative challenges that will need to be overcome for robots to be used effectively in the home for speech-based assistance with daily living.

References

[1]

American Psychiatric Association. 2000. Delirium, dementia, and amnestic and other cognitive disorders. In Diagnostic and Statistical Manual of Mental Disorders, Text Revision (DSM-IV-TR) (4th ed.). American Psychiatric Association, Arlington, VA.

[2]

A. Baba, S. Yoshizawa, M. Yamada, A. Lee, and K. Shikano. 2004. Acoustic models of the elderly for large-vocabulary continuous speech recognition. Electronics and Communications in Japan (Part II: Electronics) 87, 7 (2004), 49--57.

[3]

A. Bauer, D. Wollherr, and M. Buss. 2008. Human-robot collaboration: A survey. International Journal of Humanoid Robotics 5 (2008), 47--66.

[4]

M. Begum, R. Wang, R. Huq, and A. Mihailidis. 2013. Performance of daily activities by older adults with dementia: The role of an assistive robot. In Proceedings of the IEEE International Conference on Rehabilitation Robotics.

[5]

A. J. Bharucha, V. Anand, J. Forlizzi, M. A. Dew, C. F. Reynolds III, S. Stevens, and H. Wactlar. 2009. Intelligent assistive technology applications to dementia care: Current capabilities, limitations, and future challenges. American Journal of Geriatric Psychiatry 17, 2 (February 2009), 88--104.

[6]

F. Boller and J. Becker. 1983. DementiaBank database. (1983).

[7]

J. L. Burke and R. R. Murphy. 1999. Situation Awareness, Team Communication, and Task Performance in Robot-Assisted Technical Search: Bujold Goes to Bridgeport. CMPSCI Technical Report CRASAR-TR2004-23. University of South Florida.

[8]

B. Davis and M. Maclagan. 2009. Examining pauses in Alzheimer’s discourse. American Journal of Alzheimer’s Disease and Other Dementias 24, 2 (2009), 141--154.

[9]

Y. Ephraim and D. Malah. 1985. Speech enhancement using a minimum mean-square error log-spectral amplitude estimator. IEEE Transactions on Acoustics, Speech and Signal Processing 33, 2 (April 1985), 443--445.

[10]

J. Erkelens, J. Jensen, and R. Heusdens. 2007. A data-driven approach to optimizing spectral speech enhancement methods for various error criteria. Speech Communication 49 (2007), 530--541.

Digital Library

[11]

M. F. Folstein, S. E. Folstein, T. White, and M. A. Messer. 2001. Mini-Mental State Examination User’s Guide. Psychological Assessment Resources, Odessa, FL.

[12]

K. Fraser, F. Rudzicz, and E. Rochon. 2013. Using text and acoustic features to diagnose progressive aphasia and its subtypes. In Proceedings of Interspeech 2013.

[13]

A. Freedy, E. de Visser, G. Weltman, and N. Coeyman. 2007. Measurement of trust in human-robot collaboration. In Proceedings of International Conference on Collaborative Technologies and Systems. 17--24.

[14]

J. Gast, A. Bannat, T. Rehrl, F. Wallhoff, G. Rigoll, C. Wendt, S. Schmidt, M. Popp, and B. Farber. 2009. Real-time framework for multimodal human-robot interaction. In Proceedings of the IEEE International Conference on Human-System Interaction. 276--283.

Digital Library

[15]

S. Gauthier, M. Panisset, J. Nalbantoglu, and J. Poirier. 1997. Alzheimer’s disease: Current knowledge, management and research. Canadian Medical Association Journal 157 (1997), 1047--1052.

[16]

R. Goldfarb and M. J. S. Pietro. 2004. Support systems: Older adults with neurogenic communication disorders. Journal of Ambulatory Care Management 27, 4 (2004), 356--365.

[17]

H. Goodglass and E. Kaplan. 1983. The Assessment of Aphasia and Related Disorders (2nd ed.). Lea and Febiger, Philadelphia, PA.

[18]

M. A. Goodrich and A. C. Schultz. 2007. Human-robot interaction: A survey. Foundations and Trends in Human-Computer Interaction 1 (2007), 203--275.

Digital Library

[19]

D. Graff and C. Cieri. 2003. English Gigaword. Linguistic Data Consortium. (2003).

[20]

P. Green and L. Wei-Haas. 1985. The rapid development of user interfaces: Experience with the Wizard of Oz method. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting.

[21]

S. A. Green, M. Billinghurst, X. Chen, and J. G. Chase. 2008. Human-robot collaboration: A literature review and augmented reality approach in design. International Journal Advanced Robotic Systems 5 (2008), 1--18.

[22]

C. Guinn and A. Habash. 2012. Technical Report FS-12-01. Association for the Advancement of Artificial Intelligence.

[23]

T. Hopper. 2001. Indirect interventions to facilitate communication in Alzheimers disease. Seminars in Speech and Language 22, 4 (2001), 305--315.

[24]

R. Jarvis, O. Gupta, S. Effendi, and Z. Li. 2009. Multimodal robot/human interaction for assisted living. International Journal on Advances in Intelligent Systems 2, 2 & 3 (2009), 288--302.

[25]

S. Klemmer, B. Hartmann, and L. Takayama. 2006. How bodies matter: Five themes for interaction design. In Proceedings of the Conference on Designing Interactive Systems. 140--149.

Digital Library

[26]

T. Lee and A. Mihaildis. 2005. An intelligent emergency response system: Preliminary development and testing of automated fall detection. Journal of Telemedicine and Telecare 11 (2005), 194--198.

[27]

M. Little, P. McSharry, I. Moroz, and S. Roberts. 2006. Nonlinear, biophysically-informed speech pathology detection. In Proceedings of ICASSP 2006. 1080--1083.

[28]

E. Lucet. 2012. Social Mobiserv Kompai robot to assist people. In euRobotics Workshop on Robots in Healthcare and Welfare.

[29]

I. B. Mauss and M. D. Robinson. 2009. Measure of emotion: A review. Cognition and Emotion 23, 2 (2009), 209.

[30]

D. McColl and G. Nejat. 2013. Meal-time with a socially assistive robot and older adults at a long-term care facility. Journal of Human-Robot Interaction 2, 1 (2013), 152--171.

Digital Library

[31]

A. Mihailidis, J. N. Boger, T. Craig, and J. Hoey. 2008. The COACH prompting system to assist older adults with dementia through handwashing: An efficacy study. BMC Geriatrics 8, 28 (2008).

[32]

M. Mouad, L. Adouane, P. Schmitt, D. Khadraoui, B. Gâteau, and P. Martinet. 2010. Multi-agents based system to coordinate mobile teamworking robots. In Proceedings of the 4th Companion Robotics Institute. Brussels.

[33]

E. D. Mynatt, A.-S. Melenhorst, A. D. Fisk, and W. A. Rogers. 2004. Aware technologies for aging in place: Understanding user needs and attitudes. IEEE Pervasive Computing 3 (2004), 36--41.

Digital Library

[34]

G. Nejat and M. Ficocelli. 2008. Can I be of assistance&quest; The intelligence behind an assistive robot. In Proceedings of the IEEE International Conference on Robotics and Automation. 3564--3569.

[35]

P. Olivier, A. Monk, G. Xu, and J. Hoey. 2009. Ambient kitchen: Designing situation services using a high fidelity prototyping environment. In Proceedings of the ACM 2nd International Conference on Pervasive Technologies Related to Assistive Environments.

Digital Library

[36]

J. B. Orange, R. B. Lubinsky, and D. J. Higginbotham. 1996. Conversational repair by individuals with dementia of the Alzheimer’s type. Journal of Speech and Hearing Research 39 (August 1996), 881--895.

[37]

J. B. Orange, K. M. van Gennep, L. Miller, and A. M. Johnson. 1998. Resolution of communication breakdown in dementia of the Alzheimer’s type: A longitudinal study. Journal of Applied Communication Research 26 (1998), 120--138.

[38]

S. V. Pakhomov, G. E. Smith, D. Chacon, Y. Feliciano, N. Graff-Radford, R. Caselli, and D. S. Knopman. 2010. Computerized analysis of speech and language to identify psycholinguistic correlates of frontotemporal lobar degeneration. Cognitive and Behavioral Neurology 23 (2010), 165--177.

[39]

K. Park, S. J. Lee, H.-Y. Jung, and Y. Lee. 2009. Human-robot interface using robust speech recognition and user localization based on noise separation device. In The 18th IEEE International Symposium on Robot and Human Interactive Communication, 2009 (RO-MAN’09). 328--333.

[40]

T. Pellegrini, I. Trancoso, A. Hämäläinen, A. Calado, M. S. Dias, and D. Braga. 2012. Impact of age in ASR for the elderly: Preliminary experiments in European Portuguese. In Advances in Speech and Language Technologies for Iberian Languages. Springer, 139--147. http://link.springer.com/chapter/10.1007/978-3-642-35292-8&lowbar;15

[41]

H. Peng, F. Long, and C. Ding. 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 8 (2005), 1226--1238.

Digital Library

[42]

M. E. Pollack. 2006. Autominder: A case study of assistive technology for elders with cognitive impairment. Generations 30 (2006), 67--69.

[43]

C. Pope and B. H. Davis. 2011. Finding a balance: The Carolinas Conversation Collection. Corpus Linguistics and Linguistic Theory 7, 1 (2011).

[44]

J. Reilly, J. Troche, and M. Grossman. 2011. Language processing in dementia. In The Handbook of Alzheimer’s Disease and Other Dementias, A. E. Budson and N. W. Kowall (Eds.). Wiley-Blackwell.

[45]

B. Roark, M. Mitchell, J.-P. Hosom, K. Hollingshead, and J. Kaye. 2011. Spoken language derived measures for detecting mild cognitive impairment. IEEE Transactions on Audio, Speech, and Language Processing 19, 7 (2011), 2081--2090.

Digital Library

[46]

E. Rochon, G. S. Waters, and D. Caplan. 2000. The relationship between measures of working memory and sentence comprehension in patients with Alzheimer’s disease. Journal of Speech, Language, and Hearing Research 43 (2000), 395--413.

[47]

F. Rudzicz. 2006. Clavius: Bi-directional parsing for generic multimodal interaction. In Proceedings of the Joint Meeting of the International Conference on Computational Linguistics and the Association for Computational Linguistics.

Digital Library

[48]

F. Rudzicz, L. C. Currie, A. Danks, T. Mehta, and S. Zhao. 2014. Automatically identifying trouble-indicating speech behaviors in Alzheimer’s disease. In Proceedings of ACM ASSETS’14.

Digital Library

[49]

F. Rudzicz, R. Wilson, A. Mihailidis, E. Rochon, and C. Leonard. 2012. Communication strategies for a computerized caregiver for individuals with Alzheimer’s disease. In Proceedings of the 3rd Workshop on Speech and Language Processing for Assistive Technologies (SLPAT’12) at the 13th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL’12).

Digital Library

[50]

P. Saini, B. de Ruyter, P. Markopoulos, and A. van Breemen. 2005. Benefits of social intelligence in home dialogue systems. In Proceedings of INTERACT 2005. 510--521.

Digital Library

[51]

E. A. Schegloff, G. Jefferson, and H. Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language 53, 2 (1977), 361--382.

[52]

H. Schlosberg. 1954. 3-dimensions of emotion. Psychological Review 6, 2 (1954), 81--88.

[53]

A. Serna, H. Pigot, and V. Rialle. 2007. Modeling the progression of Alzheimer’s disease for cognitive assistance in smart homes. User Modelling and User-Adapted Interaction 17 (2007), 415--438.

Digital Library

[54]

T. Shibata. 2004. An overview of human interactive robots for psychological enrichment. Proceedings of the IEEE 92, 11 (2004), 1749--1758.

[55]

J.f A. Small, E. S. Andersen, and D. Kempler. 1997. Effects of working memory capacity on understanding rate-altered speech. Aging, Neuropsychology, and Cognition 4, 2 (1997), 126--139.

[56]

J.f A. Small, G. Gutman, S. Makela, and B. Hillhouse. 2003. Effectiveness of communication strategies used by caregivers of persons with Alzheimer’s disease during activities of daily living. Journal of Speech, Language, and Hearing Research 46, 2 (2003), 353--367.

[57]

M. Snover, B. Dorr, and R. Schwartz. 2004. A lexically-driven algorithm for disfluency detection. In Proceedings of HLT-NAACL 2004: Short Papers. 157--160.

Digital Library

[58]

A. Tapus and M. Chetouani. 2010. ROBADOM: The impact of a domestic robot on the psychological and cognitive state of the elderly with mild cognitive impairment. In Proceedings of the International Symposium on Quality of Life Technology Intelligent Systems for Better Living. Las Vegas, NV.

[59]

A. Tapus, J. Fasola, and M. J. Matarić. 2008. Cognitive assistance and physical therapy for dementia patients. In ICRA Workshop on Social Interaction with Intelligent Indoor Robots.

[60]

C. K. Tomoeda, K. A. Bayles, D. R. Boone, A. W. Kaszniak, and T. J. Slauson. 1990. Speech rate and syntactic complexity effects on the auditory comprehension of alzheimer patients. Journal of Communication Disorders 23, 2 (1990), 151--161.

[61]

M. Vacher, B. Lecouteux, and F. Portet. 2012. Recognition of voice commands by multisource ASR and noise cancellation in a smart home environment. In Proceedings of the 20th European Signal Processing Conference (EUSIPCO’12). 1663--1667.

[62]

K. Vertanen. 2006. Baseline WSJ Acoustic Models for HTK and Sphinx: Training Recipes and Recognition Experiments. Technical Report. Cavendish Laboratory, University of Cambridge.

[63]

J. G. Victores, F. R. Ca nadillas, S. Morante, A. Jardón, and C. Balaguer. 2013. Assistive robot multi-modal interaction with augmented 3D vision and dialogue. Advances in Intelligent Systems and Computing 252 (2013), 209--217.

[64]

R. Vipperla. 2011. Automatic Speech Recognition for Ageing Voices Ravichander Vipperla. PhD. University of Edinburgh.

[65]

R. Vipperla, M. Wolters, K. Georgila, and S. Renals. 2009. Speech input from older users in smart environments: Challenges and perspectives. In Universal Access in Human-Computer Interaction: Intelligent and Ubiquitous Interaction Environments, Constantine Stephanidis (Ed.). Lecture Notes in Computer Science, Vol. 5615. Springer, Berlin, 117--126.

Digital Library

[66]

C. M. Watson. 1999. An analysis of trouble and repair in the natural conversations of people with dementia of the Alzheimer’s type. Aphasiology 13, 3 (1999), 195--218.

[67]

R. Wilson, E. Rochon, A. Mihailidis, and C. Leonard. 2012. Examining success of communication strategies used by formal caregivers assisting individuals with Alzheimer’s disease during an activity of daily living. Journal of Speech, Language, and Hearing Research 55 (April 2012), 328--341.

[68]

S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. (A.) Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland. 2006. The HTK Book (version 3.4).

[69]

S. Zhao, F. Rudzicz, L. G. Carvalho, C. Márquez-Chin, and S. Livingstone. 2014. Automatic detection of expressed emotion in Parkinson’s disease. In Proceedings of ICASSP 2014.

Cited By

Addlesee AEshghi A(2024)You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarificationFrontiers in Dementia10.3389/frdem.2024.13430523Online publication date: 12-Mar-2024
https://doi.org/10.3389/frdem.2024.1343052
Xygkou AAng CSiriaraya PKopecki JCovaci AKanjo EShe W(2024)MindTalker: Navigating the Complexities of AI-Enhanced Social Engagement for People with Early-Stage DementiaProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642538(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642538
Lima MGrollman DBroadbent EJu WSoh HWilliams T(2024)Home Integration of Conversational Robots to Enhance Ageing and Dementia CareCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638378(115-117)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638378
Show More Cited By

Index Terms

Speech Interaction with Personal Assistive Robots Supporting Aging at Home for Individuals with Alzheimer’s Disease
1. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Expert systems

Recommendations

A Smart Home Based on Multi-heterogeneous Robots and Sensor Networks for Elderly Care
Intelligent Robotics and Applications
Abstract
To tackle behavioral assistance for elderly care in daily life, our laboratory rollout seven homecare robots: walking support robot, gait rehabilitation robot, intelligent wheelchair robot, excretory support robot, personal care robot, intelligent ...
Modeling the progression of Alzheimer's disease for cognitive assistance in smart homes

Smart homes provide support to cognitively impaired people (such as those suffering from Alzheimer's disease) so that they can remain at home in an autonomous and safe way. Models of this impaired population should benefit the cognitive assistance's ...
Automatic screening of mild cognitive impairment and Alzheimer’s disease by means of posterior-thresholding hesitation representation
Abstract
Dementia is a chronic or progressive clinical syndrome, characterized by the deterioration of problem-solving skills, memory and language. In Mild Cognitive Impairment (MCI), which is often considered to be the prodromal stage of ...
Highlights
- Automatic diagnosis of dementia using a non-invasive way: the speech of the patient.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Accessible Computing

ACM Transactions on Accessible Computing Volume 7, Issue 2

Special Issue on Speech and Language Processing for AT (Part 3)

July 2015

83 pages

ISSN:1936-7228

EISSN:1936-7236

DOI:10.1145/2785580

Editors:
Matt Huenerfauth
Rochester Institute of Technology, USA
,
Kathleen F. McCoy
University of Delaware, USA

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2015

Accepted: 01 March 2015

Revised: 01 December 2014

Received: 01 March 2014

Published in TACCESS Volume 7, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
1,150
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)10

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Addlesee AEshghi A(2024)You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarificationFrontiers in Dementia10.3389/frdem.2024.13430523Online publication date: 12-Mar-2024
https://doi.org/10.3389/frdem.2024.1343052
Xygkou AAng CSiriaraya PKopecki JCovaci AKanjo EShe W(2024)MindTalker: Navigating the Complexities of AI-Enhanced Social Engagement for People with Early-Stage DementiaProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642538(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642538
Lima MGrollman DBroadbent EJu WSoh HWilliams T(2024)Home Integration of Conversational Robots to Enhance Ageing and Dementia CareCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638378(115-117)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638378
Yang XTang MUshaw G(2024)A Serious Game Prototype of Combined Cognitive Training for Alzheimer’s Patients2024 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE International Conference on Robotics, Automation and Mechatronics (RAM)10.1109/CIS-RAM61939.2024.10673387(537-542)Online publication date: 8-Aug-2024
https://doi.org/10.1109/CIS-RAM61939.2024.10673387
Karami VYaffe MGore GMoon AAbbasgholizadeh Rahimi S(2024)Socially Assistive Robots for patients with Alzheimer's Disease: A scoping reviewArchives of Gerontology and Geriatrics10.1016/j.archger.2024.105409123(105409)Online publication date: Aug-2024
https://doi.org/10.1016/j.archger.2024.105409
Jiménez SFavela JCervantes LRamírez-Noriega AJuárez-Ramírez R(2024)Evaluating smart speakers as assistants for informal caregivers of people with dementiaUniversal Access in the Information Society10.1007/s10209-024-01091-5Online publication date: 14-Feb-2024
https://doi.org/10.1007/s10209-024-01091-5
Tucci CAmaro IDella Greca ATortora G(2024)RetroMind and the Image of Memories: A Preliminary Study of a Support Tool for Reminiscence TherapyArtificial Intelligence in HCI10.1007/978-3-031-60615-1_31(456-468)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60615-1_31
Astorga MCruz-Sandoval DFavela J(2023)A Social Robot to Assist in Addressing Disruptive Eating Behaviors by People with DementiaRobotics10.3390/robotics1201002912:1(29)Online publication date: 19-Feb-2023
https://doi.org/10.3390/robotics12010029
Balakrishnan BRajesh VTharunraj RKumar M(2023)Voice based Autonomous Pill Assistant for Alzheimer Patients2023 International Conference on Inventive Computation Technologies (ICICT)10.1109/ICICT57646.2023.10133937(736-740)Online publication date: 26-Apr-2023
https://doi.org/10.1109/ICICT57646.2023.10133937
Gagnon-Roy MBier NGiroux SCouture MPigot HLe Dorze GGosselin NZarshenas SHendryckx CBottari C(2023)COOK technology to support meal preparation following a severe traumatic brain injury: a usability mixed-methods single-case study in a real-world environmentDisability and Rehabilitation: Assistive Technology10.1080/17483107.2023.226432619:5(2113-2130)Online publication date: 13-Oct-2023
https://doi.org/10.1080/17483107.2023.2264326
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents