Understanding How Sensory Changes Experienced by Individuals with a Range of Age-Related Cognitive Changes Can Affect Technology Use

There are nearly 1 billion adults over the age of 60 worldwide [140], with 6–12% of these adults diagnosed with mild cognitive impairment [104]. In addition, there are nearly 50 million people worldwide living with a diagnosis of dementia and a projected 10 million new cases each year [141]. With consideration for the number of people living with age-related cognitive changes, researchers are conducting studies to report on the assistive technologies currently available for use by people with dementia and mild cognitive impairment. One study projected technology trends for people who will be living with dementia in the future [142]. In terms of technologies that are currently available, Gibson et al. surveyed the landscape of available assistive technologies, dividing the resulting types of technologies into three categories: devices used “by,” “on,” and “with” people with dementia [42]. Lorenz et al. note how technologies are designed for use by people with mild cognitive impairment or those in the earlier stages of dementia and shift to targeting family carers and healthcare professionals as the condition progresses [69]. Despite this use, little research has focused on the usability of assistive technology that supports everyday life—how people can understand, use, and learn to use these technologies [74].

Some research has gone in depth on barriers to everyday technology use. Many studies discuss the importance of recognizing individuality for successful technology introduction, such as abilities, habits, attitudes, and environments [101]. Some research includes accessibility and usability issues in their accounts of barriers. While sensory barriers are sometimes mentioned, they have not been examined thoroughly. For example, in an analysis of the difficulties people with early-stage dementia have using everyday technology, Nygård and Starkhammar refer to weak vision in a section on the design of artefacts as one contextual condition that can be a hindrance to technology use or appropriate force (e.g., holding a remote volume button too long) as an example of communication difficulties in the use of technology [91]. In the present article, we directly study the sensory changes people with age-related cognitive changes experience, with a focus on how these changes effect everyday technology use.

Changing cognition has often been centered in technology design for people with dementia and mild cognitive impairment. One vein of work seeks to use technology to screen for age-related cognitive changes [49, 108, 119], prevent cognitive decline [79, 130], or improve cognition [6, 8, 125] using varying kinds of technologies. For example, Alves et al. developed a web application, Scrapbook, to support psychologists in performing cognitive therapy as well as reminiscence with people with dementia [6]. MemHolo, a mixed reality system, was developed as a cognitive training tool to practice short-term and spatial memory for people with Alzheimer's disease [8]. One project, MobiAssist, targets both physical and cognitive capabilities of people with dementia through exergames [123–126]. Virtual reality has been used as a cognitive training tool for people with mild cognitive impairments to simulate typical daily activities [129]. Other immersive technologies like PENCODER rely heavily on experiences and objects that are familiar to people with mild cognitive impairment and have positively influenced upward trends in self-esteem among participants [110].

Other research seeks to accommodate cognitive changes people experience in dementia. Much research has been done developing technologies to assist in navigation in case the person with dementia walks to places that the caregiver is not comfortable with [133, 134] or if the person with dementia forgets where they are and cannot get home [55]. Other technologies are designed to assist with cognitive barriers to task completion. These technologies often aim to stimulate thinking through step-by-step directions [20] and are targeted to support specific tasks such as hand washing [76], flushing the toilet [29], and cooking [26, 93]. Researchers used an augmented reality system called GhostHands to build upon similar types of task prompting tools, with the platform offering multi-sensory prompts that involved primary visual and auditory cues [136]. CoasterChat's designers, Diks et al., also found that the instructions accompanying the app pilot were most well received when they combined visual movements and auditory instruction [25]. This approach has also been taken in commercially available systems, such as MapHabit, which pairs visual with audio or written instructions to support everyday task completion [20, 72]. Muñoz et al. found that their touchscreen app called A Better Visit enabled interactions between caregiver–person-with-dementia dyads by curating content that specifically supported “pre-existing interests” [87]. This application created multi-sensory experiences in apps like Tic-Tac-Tango—which played familiar ballroom music and replaced typical X's and O's with visually stimulating dance moves—and sought to “support personhood” through personalization and familiar experiences [87]. Like MapHabit, both systems use multimodal prompts—the designers explain this design decision was made to ensure “interfaces that are highly adaptable to the cognitive support needs and any perception disabilities” [72].

The above examples that utilize multimodal prompts indicate that even when designing for cognitive support (rather than, for example, recreation), sensory abilities matter for the design of usable and accessible systems. Our past work introduces nuances to the relationship between sensory stimulation and cognition, such as that indiscriminate use of multi-modal stimulation can actually create anxiety and disrupt use and that people with dementia vary in their ideal mode of sensory input for comprehension [28]. The present article takes a systematic approach to understanding sensory changes people with dementia experience as they affect technology use.

Human-Computer Interaction (HCI) technologies for people with dementia, particularly in later stages, and people with mild cognitive impairment, often recognize sensory abilities as ways to engage people with dementia. Design decisions that leverage sensory experiences are supported by research: a critical dementia perspective recognizes the importance of multisensory experiences and that physical and embodied experiences are ways of knowing [66]. The experience-centered design perspective attends to aesthetics and touch as both providing insights into the lives of people with dementia as well as creating opportunities for absorbing experiences [82]. Froehlich et al. posit that media has such an important role for people with dementia that there should be a new category of assistive technologies that involve media creation and consumption for well-being [39].

Researchers have used a variety of technologies to engage sensory abilities, often including multimedia. For example, the CIRCA project used touchscreens to present music, videos, and pictures to stimulate long-term memories to prompt conversations [4, 45]. More recent research may reflect an understanding of the importance of embodiment in dementia, adding haptics [60], gesture-based interaction [63, 111], and tangible objects to support multimedia engagement to support reminiscence [59]. Guan et al. also found that technology could act as a “communication modality” when adapted from embodied cues, noting that embodied cuing can be more successful than abstract verbal language when encouraging people with dementia to mimic activities or behaviors [46]. Further, embodied cues that stimulated several senses—such as a caregiver visually depicting the activity while humming—were particularly effective [46]. Another example leveraging embodied interaction alongside sensory modalities is in the Closer To Nature Installation, which used a display of a farm scene, old-fashioned water pump, and sensor-enabled, touch-responsive goat [36].

Audio has received particular focus. Researchers have investigated conversational assistances, such as through Google home and Amazon Alexa, to facilitate conversations between people with mild cognitive impairment and their care-partners [143]. Other researchers have focused auditory interactions due to the documented benefits of music therapy for people with dementia [128]. Researchers in HCI have designed technologies to facilitate musical creativity [83, 102, 103] and to give people with dementia more control over playing music [109]. When developing their application A Better Visit, Muñoz et al. found that embedding musical elements throughout the cooperative games encouraged engagement and concentration among people with dementia, especially if that music was familiar to them [87]. Houben et al. point out that music only represents a subset of the sounds that people perceive, and in contrast to experiences with music, experiences of everyday sounds for people with dementia are significantly under studied [57]. The authors explored the reactions of people with early and moderate dementia to generic, everyday sounds, finding not only that this provided opportunities for meaningful interactions but also that soundscapes could feel chaotic and overwhelming [57]. Muñoz et al. found similar success when using sounds to facilitate accurate guessing in the game Reveal, using animal sounds to help users recognize what animal was represented in a picture before they could see it fully [87].

We recruited from four groups: people who self-reported subjective cognitive decline, people diagnosed with mild cognitive impairment, people diagnosed with mild to moderate dementia, and healthcare professionals with experience working with people across the spectrum of dementia. We include individuals with subjective cognitive decline and mild cognitive impairment in this study as a way to compare and contrast the effects of sensory changes on technology use to those diagnosed with dementia. Data from interviews with healthcare professionals were included to provide perspectives on technology use for a broader population of people with dementia, given the specifics of the sample of people with dementia (who were largely regular technology users) and to expand our understanding to those in the later stages of dementia.

We recruited people with dementia and mild cognitive impairment through large dementia advocacy organizations and snowball sampling. Initial contacts were made at a large dementia organization conference in 2019. These participants then shared our study information with other people they knew in their online support groups as well as on Twitter. We recruited a smaller number of people who experience subjective cognitive decline through personal contacts and snowball sampling. We recruited healthcare professionals through various professional societies Facebook pages (e.g., Therapeutic Recreation Directory) and snowball sampling.

For participants with cognitive impairments to qualify for the study, they first had to complete a screener call with the researchers where they were asked questions to determine their eligibility for the study as well as their capacity to consent. Capacity to consent was determined using the UC Davis Alzheimer's Disease Center procedures [121]. All participants gave informed consent before participating in the study.

To determine eligibility, we chose to rely on self-reporting a diagnosis of dementia, mild cognitive impairment, or experience with subjective cognitive decline rather than formal staging of the severity of cognitive impairment by a clinician (e.g., instruments such as the Mini-Mental State Exam or Montreal Cognitive Assessment). Relying on self-reported diagnosis is in alignment with accepted approaches in HCI that recognize that “It may be hard to get detailed information about the medical status of someone with a disability… because of the sensitivity involved in sharing personal health data” [67]. When participants with dementia did not know their stage, we asked them to self-assess their stage of dementia at the time of the study based on the categories we drew from the US Center of Disease Control and Prevention and National Institute of Aging's function-based stage classification [88, 89, 118]. These categories included subjective cognitive decline, mild cognitive impairment, mild, moderate, and advanced dementia, as well as vignettes of functional changes associated with each of these categories (see References [88, 89, 118]). Each category was read to participants where they indicated which best fit their functional abilities. Function-based stage classifications is a particularly relevant measure to technology use, as it indicates one's ability to carry out daily activities and tasks. Though we include the self-assessed stage in the table, throughout our findings we focus more on the self-reported diagnosis and not on the self-assessment of stage, as these self-assessments may not align with a clinical diagnosis of stage.

All participants also had to self-report regular use of technology. We chose to include regular technology use as a criteria so that we could gain insights into experiences with technology—while recognizing that these participants may not be representative of the broader population of people with dementia (although there is a trend toward greater technology use by people with dementia: Research has found that 54.14% of people with mild cognitive impairment or dementia reported using their smartphones and tablets almost every day [48]). Inclusion criteria for healthcare professionals were that they had at least 3 years of experience working with people with dementia. All procedures were approved by the University Institutional Review Board.

Interviews were semi-structured, lasting approximately 1 hour. Nearly all of the interviews were conducted remotely via video-conferencing. Though several of the interviews were conducted in-person (prior to COVID-19 quarantine restrictions) when participants were located within a reasonable distance from our university. Each interview was audio/video recorded with the consent of participants, resulting in 40 hours and 40 minutes of recording used for analysis.

For the interview protocols, questions were structured to be very general concerning participants’ technology use. The general nature of the questions and the semi-structured nature of the interview allowed us to ask further probing questions depending on the answers given by participants. This helped us to obtain data on a variety of ways technology was used to address people with dementia's unique sensory needs. In the first portion of the interviews, participants with dementia, mild cognitive impairment, and subjective cognitive decline were asked questions concerning their current use of technology, and healthcare professionals were asked about their professional strategies to engage people with dementia, with us probing for more detail on the strategies that involved technology. Examples of questions in this section included, “Have the changes you've experienced with dementia affected your technology use? If so, can you please explain in what ways your technology use has changed?” To healthcare professionals, we asked questions such as, “Can you tell me about strategies you use to make an activity accessible to people with dementia?” and “How have you modified technologies so that people with dementia can still engage with them?” With all participants, we asked in detail about their motivations for using technology and the challenges that technologies were used to address. We probed deeply into the answers where sensory changes were discussed, asking about how these changes were experienced, and how technologies were used to address them. Next, we asked all participants questions concerning their ideas for future technologies. The interviews with participants with dementia, mild cognitive impairment, and subjective cognitive decline concluded with a discussion of technologies that participants no longer used and why. Interviews with healthcare professionals concluded by giving them the option to review a website database of assistive technologies in terms of usability and usefulness to people with dementia. Please see supplementary materials for further detail. All participants received a $20 Amazon gift card after completing the interview.

This is an extended version of our accepted ASSETS paper [28]. As such, our analysis included the 30 participants from the previous version of this work (11 people with mild to moderate dementia and 19 healthcare professionals). In this extended version, we include an additional 11 participants. Four of these additional participants are individuals living with dementia, 2 are individuals living with mild cognitive impairment, and 5 are people who self-report subjective cognitive decline. In the findings section these participants are denoted with pseudonyms, with additional context of either their type of dementia or subjective cognitive decline. We use pseudonyms as a way to avoid dehumanization [10].

The average age of participants with dementia was 62.5 years old (range 57–73). The average age of participants with mild cognitive impairment was 63 years old (range 55–71). The average age of participants with subjective cognitive decline was 75.8 years old (range 71–84). All participants with dementia, mild cognitive impairment, or subjective cognitive decline identified as Caucasian with the exception of Everly, who identified as African-American, and Margaret, who declined to self-identify her ethnicity. See Table 1 for more detail on participants age, gender, country of residency, type of dementia or cognitive impairment, and stage of dementia. Participants with dementia included individuals who identified as dementia advocates and others who were a part of advocacy organizations’ peer-support groups but were not participating in active roles within these organizations.

We interviewed 19 healthcare professionals, with an average of 15 years of experience working with people with dementia (ranging from 3 to 46 years). All participants identified as female and Caucasian, with the exception of Esther, who identified as multi-ethnic, and Tiffany, who identified as Asian. All healthcare professionals were interviewed separately with the exception of two who were interviewed together to accommodate their time constraints. Healthcare professionals worked with a range of cognitive abilities from people with mild cognitive impairment through people with advanced dementia. Their average age was 47.8 years old (range 26–70). Throughout the findings, healthcare professional quotes are denoted with Pr# rather than pseudonyms as a way to distinguish their quotes from quotes by participants with dementia, mild cognitive impairment, or subjective cognitive decline. To provide further context for healthcare professionals’ quotes, we distinguish the type of healthcare professional when introducing their quote in the findings. See Table 2 for a summary of each healthcare professional's age, country of practice, occupation, and years of experience.

For this extended version of our original paper [28], we took a thematic analysis approach [18]. This involved a secondary analysis of the data that were used for the ASSETS paper [28] and the additional data described above. We initially familiarize ourselves with the data by open coding six transcripts. These six transcripts included two transcripts from interviews with people with dementia, one transcript from an interview with a participant with mild cognitive impairment, and three transcripts from interviews with healthcare professionals. Our initial open codes from these six transcripts included codes such as “switching visual to audio,” “dexterity affected,” and “when visual is hard.” This open coding process resulted in our interest in further understanding two aspects of the data: (a) the sensory changes people with age-related cognitive changes experience affecting technology use and (b) accommodation strategies that are used when changes in sensory abilities occur. We designed a codebook that was organized by sense, with open codes that seemed relevant to those senses. We then went back through all interview transcripts, including the six transcripts used for open coding, and categorized the data using this codebook, adding to it when new codes emerged. Codes reflecting accommodation strategies were added, such as “speech slowing,” “mirroring,” “embodied cueing,” and “audio still too hard.” By the 20th transcript, we reached data saturation, as no new codes were added. All transcripts were then focus coded using the established codebook. Following focused coding, through a collaborative and iterative process of considering the fit of each code with different sensory categories, each code was finalized in terms of where it was grouped within specific sensory categories, and the higher-level sensory categories were determined: visual, auditory, dexterity, proprioception, and smell. For example, we determined that the accommodation strategy of “embodied cueing” should be included under the sense “proprioception.” We coded all instances, including where participants did not talk about technology directly. In our findings, we include instances that do not refer to technology use only when they reveal future opportunities for technical interventions to engage the changing sensory abilities of people with age-related cognitive changes. When quoting participants in the findings, we have removed false starts and filler words (e.g., “like”) only when they affected readability.

The larger aim of our analysis was to explore how sensory changes experienced by people with a range of age-related cognitive changes effect technology use. To reach this aim, when writing the article we were intentional to include quotes in each section from all applicable participants in each participant group who described related sensory changes. We then structured our findings sections in a way that demonstrates the sensory changes experienced by participants from those experienced by all participant groups to those experienced by participants in the most advanced cognitive changes. Visual and auditory changes are included as the first and second findings section as these changes were described by all participants across the range of age-related cognitive changes. One participant with subjective cognitive decline described dexterity changes but these were largely described by people with dementia or healthcare professionals who worked with people with dementia, which is why it is the third findings section. Proprioception is described in the fourth findings section, as these changes were only described by people with dementia and healthcare professionals who worked with people with dementia. Finally, olfactory (smell) changes were only described by healthcare professionals in relation to their clients in the most advanced stages of dementia. These examples and organization of our findings serve to go beyond description of the data to illustrate sensory changes experienced by participants across the spectrum of age-related cognitive changes—which we discuss in further detail in the discussion.

One limitation of this study is the reliance on self-reporting rather than requiring clinically confirmed diagnosis of dementia, diagnosis of mild cognitive impairment, or experience with subjective cognitive decline. We did not formally assess participants’ cognitive abilities. And not all participants knew what stage of dementia they were in, so some self-assessed. Future work should consider partnering with clinicians to conduct these formal assessments of cognitive ability.

The average age of participants with dementia was 62.5 years. Dementia diagnosed under the age of 65 is considered early onset [1], representing 9% of diagnoses [139]. This relatively younger group of participants may be overrepresented in our research due to the hesitance of the general population to self-identify as a person living with dementia due to stigma [10, 116] that can lead to unwillingness to discuss experiences with researchers [112]. As many of our participants were active in various dementia advocacy organizations, these participants appear to be a part of the rise of the “young, active person with dementia” involved in publicly sharing information about their condition with researchers [21]. In addition, the recruitment requirement that participants had to use technology regularly may also have led to a relatively younger group of participants [11, 92]. Finally, recruiting participants with dementia using Twitter may have resulted in a sample of younger participants with dementia who are also dementia advocates, as has been found in previous work [117].

The limited racial diversity of participants is another limitation of our study. All but one participant with dementia, mild cognitive impairment, or subjective cognitive decline identified as Caucasian and one participant chose not to identify. Research shows a higher prevalence of dementia in the African American and Latinx communities in the United States [7], which was not represented in our participant pool. Additionally, the healthcare professionals who participated in our study were also primarily Caucasian. Researchers have suggested several barriers in research recruitment of different ethnic groups such as lack of trust in research due to a history of ethical issues [24, 34], institutional barriers to education [73], and stigma consciousness [73]. There is a need for further work to ensure that research includes more diverse demographics of people with dementia and healthcare professionals. Finally, our findings come primarily from participants residing in the United States, the United Kingdom, or Canada. Our findings are certainly influenced by these geographic and cultural settings.

Given our study utilized a single interview, the scope of the data collected is limited to the perceptions and accounts of participants that they shared within our 1-hour interview. Future work should consider using an ethnographic or longitudinal method to understand how sensory changes fluctuate and affect technology use as people progress with dementia. Further, most interviews were conducted remotely, which limited data collection to participants’ verbal explanation, which can be arduous and suboptimal as an approach for some individuals. Future work should consider including observations and diary studies to further document sensory changes and their effect on technology use.

Participants described visual changes they experienced due to age-related cognitive changes that interacted with their technology use, including changes in visual acuity, color perception, ability to select an object out of a busy environment, ability to see the big picture, and surface dyslexia. Changes in vision have been associated with mild cognitive impairment and Alzheimer's disease in clinical research [2, 14, 47, 68, 106].

Participants described auditory changes they experienced that interacted with their technology use, including fluctuating sensitivity to sound, changing speech and language patterns, and perceiving and identifying different sounds. Clinical researchers have identified hearing loss as a possible indicator and risk factor for age-related cognitive changes and dementia [70].

People with age-related cognitive changes can experience changes in dexterity or fine motor ability that are unique to the experiences of “normal aging” [38, 96]. Dyspraxia, a condition that partially limits motor function [41], was described by participants. For example, David and Annette, living with vascular dementia and Alzheimer's disease respectively, explained their difficulty “getting dressed” due to dexterity changes: “I'm no longer able to do zip up buttons and fiddly garments” [Annette]. Healthcare professionals in our study even use dexterity and tactile skills as determinants of progression of dementia, with Pr3 using a “box and blocks” test that involves sorting blocks quickly. Though changes in dexterity due to aging are not unique to those living with dementia. As Peter, living with subjective cognitive decline, describes how precise or accurate movements were increasingly difficult to accomplish: “I sometimes hit something which I'm not expecting to.” In activities composed of “slight” movements, like unloading the dishwasher and “taking out the glass or the cup” he sometimes “hit[s] the edge of the countertop” because of disruptions in his dexterity and tactile abilities.

Participants described a variety of dexterity changes that made using technology challenging. This included difficulty with mice, trackpads, and keyboards while using their computer. When using a mouse was difficult, Pr6, an Activities Service Supervisor, suggests “a laptop would be better for them to use,” because they could use the trackpad. But, some people, such as Phillip, prefer desktops with a mouse, because “I have trouble with laptops. I have trouble coordinating the mouse [through the trackpad] using my finger.” Still others described difficulties with computer keyboards due to “massive pains in my hands” because of a medication, which makes typing on a keyboard difficult [Arthur] or other comorbidities which caused “a tremor” [Pr13].

Touchscreens require some precise motor movements or selection between options that can be close together, which were challenging for some participants. Frank found sliding a button on his phone to answer a call difficult due to the precision and sustained movement required. When using his phone, Phillip often “send[s] them the text by accident rather than call them, you know, because I hit the wrong thing.” Peter experienced something similar, explaining how he “minimize[s] composing” emails and surfing online using his phone, because he hits “two keys at same time” or “the wrong key.” Because of the challenges he encounters with precisely typing on his phone, he chooses to only surf or send emails on his computer where he can utilize the keyboard. Phillip, living with Alzheimer's disease and semantic dementia, explained the interplay of memory and dexterity required to operate touchscreens and how this was impacted by his changing abilities: “And I don't remember how to do the right amount of pressure and feel to do the swipe.” In contrast, Ted, living with subjective cognitive decline, said “I've learned how hard and long to press a button to make it actually work most the time.”

In response to the mismatch between dexterity abilities and what was required to operate the touch-screen devices successfully, participants switched to larger devices (Arthur and Phillip), and envisioned solutions like a single press physical button in a consistent location for their phone (Phillip) or auditory input to answer the phone rather than swiping (“then I haven't got to touch anything” Frank). As Sharon, living with vascular microangiopathy, noted her desire for a computer to “alternate between modes of inquiry” by “use[ing] a linguistic and touch interface” where one could visually use icons, tactilely press on the screen, and have an audio interface, so that someone wouldn't have to rely solely on interact with the device through touch.

Participants described changes in proprioception or their ability to move their bodies, which affected their daily lives, including fluctuations in motor memory, and changes in balance.

Clinicians have documented the unique changes in olfaction (smell) that people with age-related cognitive changes experience [5, 84, 100]. Regardless, healthcare professionals in our study point to smell as one of the remaining ways to engage the senses of people with even in “the very end-stage dementia” [Pr2]. As Pr2, an occupational therapist, described “the only thing I could do with them is, like, we had this sensory bin, and we have different scents and aromas that they could smell.” Specifically, one of her clients, “who had really just looked like she was not engaged in anything. She suddenly would smell something, and she would look up and make eye contact and smile” [Pr2]. Similarly, Pr6, an Activities Service Supervisor, uses “sensory stimulation” through “scents or lotions, hand massages, that kind of thing” to engage with “residents who are unable to verbalize.” Pr7 and Pr8, an Activities Director and Lifestyle Director, respectively, utilize flower arranging activities for the tactile and scent engagement but also as a way to “bring back memories… You know a flower for happy occasions. Even though it could have been sad occasions, it's something that will trigger, mostly a good memory” [Pr8]. In other instances, Pr8 described, “put[ting] on a pot of soup in a crockpot and just the aroma of the food would help increase their appetite” and provide a stimulating experience. Aromatherapy was used by some healthcare professionals as an engaging sensory experience, because “when they breathe it in, they don't have to do anything your body knows what to do. It just reacts.” This instinctive reaction to aromatherapy was used by Pr11, a Memory Care Activities Coordinator, when “someone who, let's just say they're in hospice care, they're at their end stage. One of the things we can have Google Assistant to play music, to set some aromatherapy for them to set the scene with the lights… So, that would help advance the quality of life for that individual going through that last journey of life.” Healthcare professionals were the only group of participants to describe changes in olfactory ability. All instances were described in relation to provide engaging or enjoyable experiences for those in the very end stages of dementia.

Participants described changes to vision like lowered visual acuity, color perception, and surface dyslexia, which affected reading, texting, and finding documents, resembling those described in the clinical literature [98, 105]. For visual changes, participants benefited from changing font size, as well as from existing accessibility features that have been designed for people with visual impairments, such as utilizing voice to text dictation or voice recording messages (as noted in our previous work [28]). Participants also emphasized how changes in color perception made it important for technology to include adjustable color, intensity of brightness, and color contrast on their devices [28]. Some participants used the attention-drawing properties of color, along with conventions of meaning assigned to different colors, to accommodate cognitive changes. For example, they color coded tasks of a digital calendar, using red to signal urgency [Helen]. We can also learn from the barriers and workarounds participants described, such as difficulty selecting something from a cluster of items, needing everything for a task to be visible at one time, as well as from their experiences with surface dyslexia. Designers may attempt tactics such as leveraging visuospatial organization and split screens to keep all documents for a task visible at one time, as well as simplifying text phrases, using images or short video tutorials instead of or in addition to words, and utilizing auditory read-aloud settings either independently or to “triangulate” auditory information with written information.

Though auditory interaction was described by many participants as their preferred mode, some participants described experiencing fluctuating sensitivity to sound—as described in the clinical literature [71]—meaning in some instances they needed increased volume and in others they needed to utilize noise canceling headphones to block out all sound. Other participants described difficulty perceiving and identifying sounds. Clinical research shows people can experience this in the form of “word deafness” [61, 64, 94] or in the form of difficulty distinguishing musical tones [43, 56], which can affect the way audible notifications on devices are perceived by users with more significant age-related cognitive changes, such as dementia. This work also has implications for the design of voice enabled smart speakers, as some people with age-related cognitive changes may need smart speakers to speak “slower” and “pronounce better” if they are experiencing “word deafness.” Additionally, participants describe changes in speech and language that affect their use of voice-enabled smart-speakers. For example, some participants described developing a stutter and experiencing slowed speech. To accommodate these changes, participants describe wanting voice-enabled smart speakers to be able to adjust based on the auditory and speech needs of the user. Though the nuanced needs of users with dementia and age-related cognitive changes in voice interactions with technology is a topic beginning to be investigated [19], further work is needed in this area.

Dexterity changes, as described in the clinical literature [38, 41, 96], affected touchscreen use, as these devices require precise motor movements for selection between options and the use of the right amount of pressure, which was challenging for some of our participants. These findings provide a more nuanced understanding that advances previous work [35, 62, 120, 135] on the effects of dexterity changes people experience on the accessibility of touchscreen devices and how they imagine future accommodations. To accommodate their dexterity changes, some participants switched to larger devices. Others envisioned solutions like a single press physical button in a consistent location for their phone or utilizing auditory input to answer the phone rather than swiping. Previous work with people with dementia has shown the effectiveness of auto-snap on touchscreen devices as one accommodation for dexterity changes [135]. Further work is needed to expand our understanding of future technical adaptations to address dexterity changes for people experiencing age-related cognitive changes.

Similarly to Guan et al.’s work [46], our findings point to the usefulness of miming actions for people with dementia to mirror when people are no longer able to understand verbal communication. This finding supports Bouvier, Hinz, and Schmidt's concept of a “virtual coach in a mirror” for people with dementia [16]. This finding also provides further justification for extending research efforts in Augmented Reality (as in References [8, 137]) and XBOX Kinect (as in Reference [32]), as our findings demonstrate the potential of these tools to provide accessible sensory prompting when verbal interactions become less accessible. Additionally, with changes in proprioception, participants described the importance of motor movement to trigger procedural memory. This finding points to the potential of using technology to prompt motor movement. For example, future work should consider testing the use of haptic gloves [78] to prompt motor movements for people with dementia.

With the changes in balance that people with dementia experience [38], this opens up new opportunities for voice-enabled smart speakers to facilitate everyday tasks such as turning on and off devices without having to stand up, which could put someone in jeopardy of falling and hurting themselves. However, this brings up debates over agency vs. safety in dementia technology design [27, 54, 55, 65, 113], which are of critical importance for technology designers in this space to consider.

Our findings also indicate using physical objects as embodied cues to spark motor memory for people with dementia. This finding links efforts to designing accessible interfaces for dementia with the extensive research designing for embodied interactions with people with dementia [12, 31, 66, 81, 131, 132]. For example, one participant described using a Nerf tennis racket attachment to their Wii remote to engage with a client. This person primarily uses a wheelchair due to changes in physical abilities. However, when the client touched what looked and felt like a tennis racket, they were able to stand up and play Wii Tennis, because they were prompted with a personally meaningful embodied cue. This finding implies the potential of including embodied cues in virtual environments (as in Reference [63]) but specifically to facilitate exercise and movement for people with dementia and mild cognitive impairment (as in References [33, 77, 123, 124, 126]). This finding also points to the potential for designing Internet-of-Things environments, providing embodied cues through a sense of touch with familiar objects that are interconnected with technical environments for use by people with dementia.

Embodied cues were also used to assist with focus and comprehension for people in the mild to moderate stages of dementia, such as described by Helen's use of Post-it notes as “tactile reminder[s]” for assistance with navigation. In particular, the permanence and continuous presence of physical objects appear to be beneficial (as noted in our previous work [28]). Uhlig et al.'s concept of designing digitally augmented everyday reminder objects to “break down and communicate complex information via sensory input and output” [122] serves as inspiration for future work that aims to provide embodied reminders for people with dementia. At the same time, it is key to consider how embodied cues are only useful if the object held meaning or was a part of daily routines for someone before they developed dementia.

Smell also changes as a result of dementia [5, 40, 84, 100]. Participants described using aromatherapy, the smell of slow cooking food, and sensory bins to engage with people in the most severe stages of dementia. In these instances, smell actually facilitated reminiscence and positive feelings. This finding points to the potential of incorporating smell into multi-sensory environmental therapy for people with dementia [23] and including the use of smell in combination with virtual reality [9, 95] to support reminiscence and enjoyment for people with dementia.

In this section, we provide preliminary evidence for accessible sensory interactions with technology for a range of age-related cognitive changes, as described by participants in our study. Our intention is to support HCI researchers and technology designers in investigating a continuum of accessible sensory interactions for people with age-related cognitive changes and thus be better equipped to design adjustable, multi-modal devices to adapt to a range of abilities and changes in abilities. Further, we intend to help researchers and technology designers distinguish between sensory changes people with different age-related cognitive changes may experience, which can assist in designing for more targeted groups of end-users with age-related cognitive changes. These distinctions will also assist technology designer and developers in making technologies adaptable with the progression of age-related cognitive changes, which is particularly useful when considering the often unique changes in ability each individual with dementia can experience.

Participants with subjective cognitive decline, mild cognitive impairment, and mild to moderate dementia described how visual changes they experienced affected their interactions with technology. For example, many participants described having to use voice-to-text due to changes in visual acuity or auditorily listening to text-based information be read aloud to accommodate surface dyslexia or mental fatigue. In addition, healthcare professionals noted how verbal communication and interaction is best for people in the very mild stages of dementia. Participants across the range of age-related cognitive changes also described their increased difficulty with dexterity, specifically with fine-motor skills required for using touchscreen devices and computers. To accommodate changes in dexterity participants described their desire for a single button system or always using voice to interact with their devices. Based on these findings, we posit that verbal interactions with technology may be more accessible for some people with more mild age-related cognitive changes, such as subjective cognitive decline, mild cognitive impairment, and mild dementia.

Participants in our study described changes in speech and language patterns, such as developing a stutter, slowed speech, and word deafness, which affected people's ability to interact with others and technologies through verbal communication. When these verbal and auditory changes occurred, participants noted the use of visual stimulations as the next best mode of interaction. As one healthcare professional noted, her interactions with clients start with primarily verbal interactions, and as her clients progress she uses more visual interactions [Pr2]. But not all forms of visual stimulation were considered equal. As several participants described, written information was more difficult for people to understand than visual images (e.g., Bill, Sharon, Pr17). Healthcare professionals also described how color perception changes for people shifting from the mild to moderate stages of dementia [Pr3]. Though providing visual prompts through miming and asking people to mirror their actions was one strategy that healthcare professionals described when verbal or auditory interaction was less accessible.

When visual prompting became less accessible, people with dementia and healthcare professionals described using embodied cues (e.g., Helen's sticky notes) or for those who were in the more advanced stages of dementia interacting using hand-over-hand prompting (Pr8’s coloring exercise). Specifically, healthcare professionals described the use of more tactile or physical interactions in the end stages of the condition rather than visual or verbal, such as in the example Pr12 gave of the woman who was a life-long tennis player. This is an especially accessible form of interaction for those in the later stages of dementia, as these embodied cues may spark procedural or motor memories that are preserved much longer into the progression of dementia. Though, when designing technical interactions triggered by embodied cues for those in the later stages of dementia, it is important to consider changes in balance that people may experience that could make certain activities, such as playing Wii tennis, difficult and potentially hazardous.

Interacting using one's sense of smell was only described when participants referred to people who were in the very last stages of dementia who were approaching end of life. For example, one healthcare professional even associated sensory prompting through aromatherapy with hospice care for people with dementia in their last days of life. Therefore, based on these findings, we posit that technical interventions that include interactions with users’ sense of smell may be particularly useful for interacting with those in the latest stages of dementia, although clinical research has shown odor identification can be more difficult for people with Alzheimer's disease, vascular dementia [5, 84, 100] and mild cognitive impairment [40]. Future work is needed to investigate the olfactory abilities that remain and the potential of technical interactions that utilize the sense of smell for people with various types of dementia as well as across the spectrum of age-related cognitive changes.

Although our findings provide some evidence for accessible sensory interactions for people with different age-related cognition changes, our findings also demonstrate instances where participants’ visual, auditory, and proprioception abilities fluctuate day to day or even throughout a day. Thus, accessible sensory interaction may fluctuate within a day or even within an hour. With this understanding of the fluctuations of the condition and the wide variety of ability changes, we urge future researchers to conduct longitudinal studies with people with various age-related cognitive changes to better understand how sensory abilities change over time and how these sensory changes affect technology use. Future work in this area should also consider systematically testing different sensory modes of interactions with technology to better understand how to provide accessible interactions with technology for those experiencing age-related cognitive changes.

It is particularly important for future research to investigate the varying effects on the sensory ability changes people experience with different types of dementia [38, 43, 56, 97, 99]. While we provide an overview of these sensory changes, this article does not comprehensively break down sensory changes by different types of dementia (e.g., dementia with Lewy bodies, vascular dementia, and Alzheimer's disease). One direction for future work in HCI is linking specific accessibility features and building accessibility profiles for each of the different types of dementia. Though we recognize dementia is complex and even those with the same type of dementia may experience differences in sensory changes, this mapping would help technology designers and developers to make adaptable and customizable accessibility options to accommodate the spectrum of sensory changes with dementia.