Lights, Camera, Access: A Closeup on Audiovisual Media Accessibility and Aphasia

Given the significant role of audiovisual media in everyday life, access to it is paramount. The importance of access to audiovisual media is embodied through the European Accessibility Act [81], Section 508 of the Rehabilitation Act [16], and the Web Content Accessibility Guidelines (WCAG 2.1) [35], which all require video to be accessible to people with disabilities, with – at a minimum – non-redundant synchronised audio descriptions [64]. Consequently, researchers have explored ways to support viewing through accessibility interventions, such as audio descriptions and subtitles. Previous research has investigated these accessibility interventions with a wide range of devices and different viewing contexts, from more conventional methods of viewing, such as television [15, 25] and mobile devices [24, 86], to web-based viewing patterns [14, 83], and, more recently, novel interactive viewing patterns, including mixed reality [50], augmented reality [65], and virtual reality [54]. Moreover, with increasing consumption of new forms of audiovisual media through social media – such as short-form video formats like TikTok [13, 70] – researchers have explored novel accessibility interventions that are better suited to those viewing contexts [88].

While accessibility interventions are deployed on a wide range of devices, the interventions often explore existing interventions that were originally created for different contexts, rather than interventions that are designed to suit the device and viewing context [59]. Many of these interventions were initially designed for ‘conventional’ television viewing. For instance, subtitles and closed captions designed for the DHH communities and audio descriptions designed for the BVI community [44] were borne out of television standards and the capabilities of the time. These interventions have been inherited by new viewing contexts, for example, the implementation of subtitles in 360° immersive videos [31, 63], or extending the existing interventions with new features, such as dynamic subtitle placement [11] or enhancing audio description through sound design [46]. Additionally, novel technologies have been utilised to either improve the production of these interventions – such as incorporating machine learning methods in creating subtitles [94] and audio descriptions [87] – or improving their capabilities – such as using eye tracking to better place subtitles [39]. While these interventions can support a wide range of communities with disabilities [32, 45] and are used by people without disabilities, they often fail to consider the complex, variable nature of disability [84]. People with CCNs – such as people living with aphasia [37] – can find such interventions unsuitable, as they may experience challenges with reading or language comprehension. Instead, prior research has explored alternative approaches to supporting viewing, including strategic group viewing for people with aphasia that involved introducing the content prior to viewing and group discussions supported by a speech and language therapist (SLT) [8], or by using scaffolding techniques and outside support to support virtual reality viewing for people with intellectual disability [78]. Alternative interventions can also be adapted to the viewing device, such as work by Guinness et al. [24] that utilises mobile robots to guide BVI viewers using smart tablet devices. To further our understanding on how to render audiovisual media more accessible for people with CCNs necessitates their direct involvement in research.

Accessibility researchers have called for more direct involvement of people living with disabilities in research [48], especially when it comes to the design and development of assistive technologies [5, 52]. This can be achieved through participatory design (PD), methods that view people with disabilities as active experts on their own disabilities with agency [21, 75], and have been used within Human-Computer Interaction research to collaborate with disabled individuals. However, a systematic review by Mack et al. [48] finds that only \(10\%\) of papers in the field reporting a user study used PD methods, suggesting that these communities are rarely engaged. Prior PD research has involved a wide range of communities, including deaf-blind individuals [4], older adults [41], and people with dementia [40]. Traditional PD methods, however, are often inaccessible for certain communities, such as people with autism spectrum disorder [18] and people with aphasia [34]. Aphasia is a language impairment caused by damage to the language areas of the brain, commonly because of a stroke, head injury, or because of degenerative damage, such as that caused by dementia [51]. Aphasia does not impact the person’s intelligence, ability to recall events, have opinions, or problem solve [82]. Oftentimes, aphasia can occur with other impairments because of damage to other areas of the brain, such as motor or cognitive impairments [3, 51]. These challenges with communication can make it difficult to recruit and directly engage people with aphasia, especially when it comes to providing informed consent or when working cooperatively in group environments [91]. Additionally, traditional PD methods often heavily rely on language-based processes, which can be cognitively demanding and result in fatigue [91]. To address these barrier, researchers have explored tangible and non-verbal design languages to facilitate access to PD and empower participants to easily express their thoughts and ideas [73]. This also includes special considerations when it comes to running such sessions with people with aphasia, such as having short and direct tasks introduced verbally, probing participants for feedback rather than relying on think-aloud, and preparing materials in an accessible manner, such as using text, verbal communication and images (e.g., visual rating scales) [23, 71]. Moreover, the use of tangible design languages – non-verbal and physically manipulable design presentations – can help support participants, such as using images to facilitate communication or tangible artefacts to represent concepts and ideas [91], or using co-designed personas to help people with aphasia engage with PD and offer the participants a different outlook on their health condition than their own [55].

The survey was hosted on Qualtrics ¹ and got approval from the King’s College London ethics board. It was anonymous and requested no personal information from the respondents other than their age, gender identity, and experience with audiovisual media. We followed several mitigation strategies in order to improve the accessibility of the survey for people with aphasia. For instance, in line with nationally recommended guidelines on preparing documents and materials for people with aphasia [26], and from feedback from SLTs and fellow researchers working with people with aphasia, the survey was broken up into thematic blocks that grouped no more than six related questions. Additionally, the questions were accompanied by representative images, such as a picture of a person reading a book for the question on reading ability.

The survey started with informed consent in which participants were first asked to read information about the research and survey, followed by true or false questions about the research to test attention. We also presented screening questions to ensure the participants fall within our eligibility criteria – being an adult who has had aphasia for at least six months. Participants were then asked demographic questions, including their age, gender identity, and their self-evaluated reading, writing, speaking, and understanding abilities on a five point Likert scale going from finding that aspect of language “very hard” to “very easy”. Answers were mostly presented as multiple choice to reduce barriers related to text entry with open answer fields, with images accompanying each option. Participants could provide examples or elaborate further using open-text field responses if they wanted to, but these responses were optional. The next section asked questions about the respondents viewing context, including the devices they used, how they interacted with said devices, how much audiovisual media they consumed, the types of media they consumed, and what activities, if any, they did while consuming the media. Finally, respondents were asked about their viewing experience both prior to their aphasia and with their aphasia. This included their general viewing experience, as well as how they experienced different viewing aspects, such as understanding dialogue, following the narrative or storyline, reading on-screen text, and tracking who the speaker was. These questions had a five-point Likert scale ranging from “I found/find it very hard” to “I found/find it very easy” and were accompanied by smiley face icons. An optional open text entry field was provided to enter additional details or examples.

The survey was shared with four aphasia support groups and charities in the UK, the US, and Australia through their social media pages (e.g., Facebook) and other online resources (e.g., blogs). The decision to distribute the survey through these existing online support groups was to allow us to reach a wide range of people living with aphasia who already participated in online communities, and were more comfortable with the use of technology. In total, we received responses from 41 people living with aphasia, 21 of whom identify as female and 20 as male. The respondents’ ages ranged from 30 to 90, with the average age being 59.44 (SD = 14.62). Even though respondents could be assisted by another person, such as a friend or family member, only 4 reported such support. Participants were asked to evaluate self-reported language abilities on a five-point Likert scale, with 1 representing finding that aspect of language to be “very hard” and 5 representing finding it “very easy”. The average Likert score for reading was 2.78 (SD = 1.26), writing was 2.10 (SD = 1.00), speaking was 2.20 (SD = 1.12), and understanding speech was 3.41 (SD = 1.20). We did not classify these participants in terms of receptive and expressive aphasia, as doing so would require extensive assessments from an SLT. To thank respondents for their time they could choose to participate in a prize raffle for a $50 and two $25 Amazon vouchers.

We analysed the resulting data in two ways. First, we conducted a statistical analysis of the viewing experience data, including the respondents’ overall viewing experience and their experiences with different viewing aspects. We looked at the difference in viewing experiences across multiple viewing aspects for respondents prior to their aphasia and after, conducting a Wilcoxon Paired Signed-Rank test [47]. The Wilcoxon Paired Signed-Rank test does not provide a standardised effect size; therefore, the effect size is an estimate following recommendations from Rosenthal et al. [72]. Additionally, we explored how language ability affects viewing experience by comparing respondents’ self-reported language abilities and the scores of the different viewing aspects, conducting a Pearson Correlation test [27]. Second, we looked at the respondents’ qualitative answers, focusing on the barriers they face when accessing audiovisual media. These were categorised based on whether the response related to viewing before aphasia or with aphasia, as well as by the affected viewing aspect.

We now discuss both the quantitative (Likert) and qualitative results (typed comments) of the survey, reflecting on overall findings in terms of limitations and the implications for the field.

This online survey represents the first study to explore whether aphasia affects audiovisual media consumption. It is an important contribution because previous papers on aphasia – or any population with CCNs – relied on anecdotal evidence. That is, previously, it was assumed that people with communication face challenges experiencing digital content, but without concrete evidence borne out of data related to audiovisual media consumption. While the survey has limitations (see section 5.4), likely receiving responses from a positively biased sample in favour of people with milder forms of aphasia, the results indicate that aphasia has a significant negative impact on viewing experience, affecting multiple different viewing aspects. People with more severe forms of aphasia, therefore, might well experience more barriers when accessing audiovisual media. Additionally, the data we collected allows us to highlight the specific aspects of language that have the highest impact on the viewing experience, notably understanding spoken language, indicating the direction future research on the topic should prioritise, which we elaborate on in the discussion. To supplement the quantitative results of the online survey we ran focus group sessions, providing us with insightful qualitative responses.

Both focus groups followed the same procedure. They were held a week apart and lasted about two and a half hours each. The sessions were video and audio recorded, with participants being given three options on how they wanted to be presented in the final stored videos and video stills: not to be shown at all, to be shown with their face blurred, to be fully visible. The sessions were divided into three main sections: informed consent and demographics, an open discussion of audiovisual media viewing experiences, and a video viewing and critiquing activity, as can be seen in the bottom left image of Figure 3. The session was introduced by a researcher, explaining what this research was about and going through the information sheet and consent form. This, along with all other aspects of the sessions, was supported by the use of appropriate communication strategies, SLTs, and tangible communication aids [91]. These communication aids consisted of paper-based visual aids, such as a tangible Likert scale participants could point to, and pen and paper, to allow participants to use an additional non-verbal communication method. Following the informed consent, participants filled out a short questionnaire asking demographic questions, such as age and gender identity, their self-perceived language abilities, and questions about their viewing context, such as the devices used, and the type of content consumed.

We started with an icebreaker asking participants to discuss their favourite films. This was followed by an open discussion of barriers participants face when accessing audiovisual media, as well as what facilitates their viewing and what methods they use to overcome the barriers. Participants then engaged in a video viewing and critiquing activity, allowing them to reflect on various aspects of the video clips. We selected 11 video clips that represent a wide range of different types of content, including various broadcast formats (e.g., films, documentaries, news broadcasts) and genres (see Table 3), and different levels of audiovisual media complexity, using a complexity heuristics, such as intense speech (e.g., multiple people talking) and novelty (e.g., new actions happening or a scene change occurring) [56]. The video clips were relatively short, lasting between 36 and 96 seconds (average = 63.4, SD = 18.2). These choices, while not exhaustive, allowed us to elicit critical reflection to a wide range of common viewing situations participants face. Clips were played on a large screen and projected in front of all participants for the video viewing (see Figure 3) and were introduced prior to their viewing by a researcher. Once the video clip was finished, ample time was given to discuss any aspects of the video clip that participants found challenging, along with characteristics that they found facilitated their viewing. The discussion was assisted by the SLTs who made sure every participant could express their thoughts by asking probing questions (e.g., asking a quiet participant if they found a video clip challenging) and supporting their responses.

Participants for the focus group sessions were recruited through Dyscover, an aphasia charity in the South East of England (UK) that offers support sessions and activities to people living with aphasia. We approached the charity about running the session and they recruited participants from people that attend their support sessions. Prior to signing up for the sessions, participants were given information about the research and the plan for the sessions and had the opportunity to ask questions. In total, we recruited 10 participants, with 5 participants per session – see Table 4. We also included 3 SLTs with experience supporting people with aphasia in the sessions, with at least 2 SLTs per session. Participants age ranged from 52 to 71, with the average age being 58.90 (SD = 5.28), and participants had aphasia for between 3 and 16 years (average = 7.90, SD = 3.69). All participants were fluent in English prior to their stroke. To allow participants to express themselves more easily and fairly in their respective groups, we divided the participants by the severity of their expressive aphasia – a weakness in their ability to express themselves with relative strength in understanding, as opposed to receptive aphasia – with participants in Session 2 having more severe expressive aphasia. The SLTs at Dyscover assessed the participant’s expressive aphasia. Participants were compensated 20 GBP for their time in the form of an Amazon voucher. In line with recommendations from Mack et al. [48] and others to operate in a familiar space for participants, the two sessions took place a week apart at Dyscover. Additionally, we followed an accessible consenting procedure with the help of SLTs and tangible communication aids, and offered sufficient time for each activity, as well as a break halfway through, along with any pauses participants required. The location was separated from the other support group attendees. The participants were seated at a large table – see Figure 3 – and a projector was available for the video viewing and critiquing activity. None of the participants used aided augmentative and alternative communication (AAC) other than the tangible communication aids we provided.

The video recordings from both sessions were transcribed by the first author using NVivo 14. This transcription included both verbal and non-verbal communication, since many people with aphasia can often find it challenging to communicate verbally [91], such as P6 who had limited verbal communication abilities and relied on non-verbal communication. Therefore, the transcript included instances when participants used pen and paper, such as in the bottom right image in Figure 3, as well as physical communication, an example of which can be seen in Figure 4. This transcript was then analysed using thematic analysis by the same researcher, as recommended by Braun and Clarke [6], to identify perspectives on the accessibility barriers and facilitators participants faced when accessing audiovisual media, and classify them. The coding was done inductively based on the barriers and facilitators that participants explicitly mentioned in their reflections. Following the initial analysis, the themes and sub-themes were discussed with the other researchers. In total, the transcript contained 185 relevant references categorised into 26 codes. The results were compiled into a taxonomy of barriers and facilitators based on key viewing aspects, broken down by their causes and effects, as can be seen in Figure 5.

We now present the qualitative results from the focus group session, in which participants discussed the barriers they faced, as well as the comments from the speech and language therapists, synthesising the findings from both the reflections from everyday life and the critique of the clips. The transcript for these sessions had 185 references across 26 base codes. From the thematic analysis of the discussions, four main themes emerged: Understanding Speech, Cognitive Load, On-Screen Text and Subtitles, and Following Narrative (see Figure 5).

The viewers’ ability to understand language is integral to their viewing experience. Indeed, with the importance of language in audiovisual media to convey information and meaning, people living with aphasia can find it prohibitively challenging to access some types of audiovisual media, requiring additional cognitive effort simply to understand what is being said, and to triangulate the various stimuli to make sense of the material. Moreover, these difficulties with understanding dialogue and narration interacted substantially with other aspects of viewing, notably with understanding and following the narrative or storyline being presented. For instance, failing to understand what a character said could make the viewer miss a vital piece of information, leading to confusion and a loss of understanding. It is important to note that a loss of understanding at one point leads to more challenges with subsequent parts of the content. This might be thought of as a cascading failure; once one challenge is experienced, the probability of experiencing other challenges increases, and users must then work harder to recoup lost information, leading to further cognitive effort and challenges. Critically – the barriers that audiovisual media presents are highly contingent on changes in the content over time. Generally, the high pace of content affected all aspects of viewing negatively, as presented in the taxonomy in Figure 5. The four main themes we evaluated around the barriers people living with aphasia face are caused, in part, by high pace and a limited window of time to assimilate information. This affected many different aspects of the content, such as fast-paced dialogue, rapid narrative shifts (e.g., fast-paced montages, high-intensity action scenes), or limited reading time, and ultimately can lead to cascading failures. Additionally, if a user experiences a barrier, recovery becomes more challenging because the user does not control the rate of the interaction. This contrasts with prior research on usability testing of social networking sites for people with aphasia, where users control the pace by the very nature of the activity [71]. Participants in the focus groups repeatedly voiced a preference for and greater understanding of audiovisual media that presented information in a clearly segmented manner with pauses between pieces of information, such as the two documentary-style video clips we presented (VC1 and VC5). When information was not presented in such a way, participants expressed appropriating the rewind and pause features to manually slow down the pace, allowing them to personalise the viewing experience to their own needs and improve understanding [43, 74]. Doing so, however, requires additional active effort and can be frustrating, reducing the enjoyment of the audiovisual media and introducing further cognitive effort [9, 93].

Our results highlight the complex, interacting nature of barriers in multimodal (audio/visual) content – e.g., when barriers are present in one modality, they impact others, such as the aforementioned example of pace and timing, as well as challenges with keeping track and understanding speech. For instance, the additional effort viewers with aphasia have to put into understanding what is being said, especially when the speech is hindered by poor clarity or distracting external noise (e.g., loud background music), can lead to missing visual or narrative elements, such as registering a joke was made or missing a key piece of on-screen text. Some of the barriers reported by the survey respondents and the focus group participants were not exclusive to audiovisual media and were analogous to those reported in the literature from ‘real world’ communication contexts – e.g., communicating in noisy public environments [61] or participating in communal activities [28, 29]. Like in these ‘real world’ contexts, supplementary information in the form of contextual auditory or visual cues that helped guide the viewer even if they missed some information (e.g., the presence of a laugh track after a joke) alleviated these issues. Offering additional information in the form of audio or visual cues has been shown to facilitate understanding within other contexts involving people with aphasia, such as using text-to-speech while reading [36] or navigating a smartphone [22]. Indeed, presenting information in a multimodal manner is reflected in guidelines and recommendations when preparing materials for people with aphasia [26], ensuring that people with variable symptoms and severities of aphasia can find at least one of these communication modalities accessible.

Facilitators included material intrinsic to the audiovisual media – for example, clear and simple language in speech. Facilitators also included support outside of the audiovisual media itself, and indeed non-technology enabled support, such as having a friend or family member facilitating the viewing experience. The taxonomy we present in Figure 5 includes series of elements present in audiovisual media and how these positively impact the understanding of the media, including facilitators that were used to address several barriers (e.g., control over pace, outside assistance). We find, however, that many of these facilitators are not always available to the viewer or require additional – and challenging – viewer interaction, such as pausing and rewinding, which can reduce the enjoyment of audiovisual media by removing its passive viewing element [20]. This is especially prominent when viewers find elements of dialogue to be challenging, at times requiring them to listen to the same words multiple times in a process similar to re-reading text to improve understanding [33]. These challenges around understanding dialogue and narration in the audiovisual media are alleviated through multiple approaches, by having information being presented in different ways (e.g., speech, text, and images), improving its clarity (e.g., slowing down speech, using clear language), or changing its structure (e.g., repetition, segmentation of ideas), among others.

We find that experiencing barriers in understanding the dialogue or narration, as is the case for many communities with CCNs for whom audiovisual media introduces multiple significant barriers [77, 78], can impede other viewing aspects. Existing technical interventions tend to address and facilitate a single viewing aspect, such as subtitles presenting audio information in a textual manner [20, 53] or audio descriptions describing visual information with spoken words [46, 86]. These interventions are often considered exemplars of universal design for audiovisual media access [10, 62], however, have been shown to introduce challenges for people with disabilities [7], and the general population more broadly [38]. While much work has been done to make audiovisual media more accessible through such technical interventions, the scope of much past research is limited to a small range of interventions and communities living with disabilities, as highlighted by a 2023 systematic review by Nevsky et al. [59]. Technical interventions that address barriers faced by people living with aphasia have not been widely explored, and the interventions that have received more attention are unlikely to be suitable for people with aphasia and other communities living with CCNs. This is because the highly variable and complex nature of these communication impairments require approaches that are more variable, rather than the “one size fits all” approach many existing technical interventions use.

Future research that aims to facilitate the viewing experience for people living with aphasia will need to consider more variable approaches that can address the access barriers of the viewers, regardless of how the barriers present. This is reflected in the way our focus group participants adapted their viewing to facilitate access, using multiple coping methods and by interacting with the audiovisual media directly and using outside assistance (e.g., family members), often using a combination of these methods. One approach that merits attention is offering the viewer greater levels of personalisation, allowing them to shape the content to their own access needs [32, 85, 88]. Prior research by Putnam et al. [68] revealed that people with CCNs, such as people with autism spectrum disorder, have voiced their desire for personalisation and customisation to bridge accessibility barriers. For instance, Ward et al. [89, 90] explored the use of audio personalisation to allow hard-of-hearing viewers to alter the sound levels of different audio tracks based on their narrative importance, which can be used to isolate important dialogue or sound effects, improving the viewing experience. Similar techniques could be used to address many of the barriers people living with aphasia face with distracting background noise, altering audio playback to what they personally need. This approach might work by leveraging technologies such as Object-Based Media (OBM), a method of creating and experiencing audiovisual media in which various content is rendered at runtime, unique to each individual, and supported by underlying metadata [2] – e.g., about specific user needs or the technology being used. While OBM allows every viewer to have their own accessible personalised version of a piece of media [30], for the most part, this technology has been used in implementing existing accessibility interventions, such as subtitles [20, 30] and audio descriptions [53]. OBM principles and technologies might allow researchers to explore novel technical accessibility interventions that address the specific needs of the viewer, including barriers to understanding speech – such as by allowing the viewer to change the newsreader – or offering the possibility to slow down or stagger the content delivery; giving more time to process information and reducing the cognitive load of viewing. Future research could also explore accessibility interventions that utilise foundational models, working across varying input modalities (e.g., speech, text, images, video), such as large language models (LLMs). For instance, LLMs could enable automated summaries of whole storylines or individual scenes to support following complex narratives or allow viewers to ask questions to an LLM-powered chatbot about the content they are consuming. These approaches might be analogous to prior work where SLTs supported users with aphasia in narrative comprehension by providing summaries of audiovisual media to support access [8].

Our research highlights the need for future research into such personalisable interventions to collaborate with and involve multiple stakeholders, including the end users, but also the content creators whose ‘creative vision’ might be affected by the content, in the design process [5, 48]. It is also important to note that we should be careful not to perpetuate existing social biases through our interventions that affect content. For instance, participants’ understanding of voices was often affected by accents. David Attenborough’s ‘Received Pronunciation’ might have been considered positively due to the region in which we run the research (i.e. familiarity), and/or affected by the existing cultural bias in the UK which places such accents as consistently highest ranked by the public in terms of ‘prestige’ and ‘pleasantness’ [76]. If we wish to consider interventions which affect voice – e.g. allowing for the selection of a different newsreader – we must consider these biases and norms, and how they shape society.

With the variable nature of aphasia, PD approaches will allow the creation of interventions that address the real needs of such communities [75]. This is reflected by the different barriers the small number of participants in our focus groups faced when accessing audiovisual media, suggesting that they would require different interventions to facilitate viewing. There are also important insights into the types of technologies these communities want, as well as considerations about how these technologies will be used by them, as people living with disabilities are very diverse, including their technology literacy [66] or physical abilities [17]. For instance, many people with aphasia may struggle with certain types of inputs, such as touch screen keyboards [69], because of physical impairments caused by their stroke, such as Hemiplegia or paralysis. Moreover, appropriate design techniques have to be used, such as, in the case of aphasia, the use of tangible design languages in co-design and the involvement of experts, such as SLTs [91].

Running an online survey with people living with aphasia introduces some challenges and limitations. First, finding participants who can complete such a survey can be difficult. The people who participate in online communities are likely to be more comfortable with reading, writing, and using technology, meaning we might have missed the voices of people living with more severe aphasia. Second, designing an online survey that is accessible to people living with aphasia requires special considerations about the types of questions asked, the way those questions are presented, and the overall duration of the survey. The focus groups were successful in exploring access barriers and facilitators in audiovisual media, and the video viewing and critiquing activity offered possibilities to reflect on challenges in a wide range of content. The short duration of the video clips during the critiquing activity, however, did not require the participants to focus on them for long time periods, therefore we could not explore the impact of fatigue. Although we did capture some information about this in the reflective workshops, stimuli of longer duration would support us in understanding this more completely. Finally, while for our workshop we sought a diverse sample in terms of aphasic difficulties, our sample size was ‘WEIRD’ (Western, Educated, Industrialised, Rich, and Democratic) [42] compared to the world population.