Abstract
Closed sign language interpreting makes media accessible to deaf and hard-of-hearing viewers who use sign language as their primary mode of communication. Analogous to subtitles, this feature allows to toggle sign language interpretation on and off, and customize its appearance in conjunction with videos. This paper provides information on designing closed interpreting in a media player through a pair of mixed-method studies. The first study assesses the usability of technical sign language interpreting features, while the second one assesses how users interact with the content. Results indicate above-average usability for the technical features. Additionally, preliminary results suggest that the optimal configuration of the SLI depends on the type of content viewed and that user preferences vary. Overall, the customizability of features and placement will be important in closed-interpreting implementations.
You have full access to this open access chapter, Download conference paper PDF
Keywords
1 Introduction
The availability of Sign Language in media varies world-wide, primarily due to differences in accessible communication services, such as Sign Language Interpreters (SLI) [1]. This variation is largely influenced by each country’s recognition of sign language (SL) and the legal requirements to provide SLI in media. Approximately 2.8% of the U.S. population—including Deaf or hard-of-hearing people (DHH), family members, educators, and interpreters—are SL users. Within this group, 100,000 to 200,000 people are Deaf primary users of American Sign Language (ASL) [2].
The inclusion of SL in media offers benefits that subtitles cannot, such as real-time representation of intonation, emotions, and other auditory information. Additionally, literacy levels within the Deaf community vary widely [2, 3]. Many people who are proficient in reading closed captions still prefer SL as their native or primary language. Access to information in SL is recognized as a human right under Article 21 of the United Nations Convention on the Rights of Persons with Disabilities [4]. The W3C Web Content Accessibility Guidelines (WCAG) require sign language media for Level AAA compliance, as per Success Criterion 1.2.6 [5]. Additionally, the World Association of SLI and the World Federation of the Deaf guidelines require it for emergency broadcasts [6]. The use of SLI in media is not a recent innovation but rather a technology that has existed for over 70 years and remains under-investigated [7].
A milestone in the U.S. was reached when the National Association of the Deaf successfully sued the White House to ensure the provision of SLI during all briefings and emergency communication [8, 9]. However, this requirement has not been universally applied across all media outlets, and enforcement mechanisms are lacking to ensure widespread content availability. In some instances, media outlets retransmitted press briefings but cropped out the SLI, highlighting the challenges in consistently applying W3C Level AAA standards across different media. More work on state-of-the art technology is needed to ensure universal provision of SLI in video programming.
Our goal is to ensure that interpreted content is accessible and can be toggled on and off, like subtitles for the DHH. Toward this goal, we cover a pair of usability studies on a functional proof of concept implementation named “Closed Interpreting Accessibility” (CIA). They build upon previous research conducted in Europe [7, 10], Asia [11], and the U.S. [12], and focus on how specific features and customization options available in the CIA prototype can enhance accessibility.
This paper is structured around two research questions:
-
1.
RQ1: Which technical features are most useful in supporting SL in visual media?
-
2.
RQ2: How do viewers of the Deaf community interact with and process information under different viewing contexts?
2 Related Work
Bosch et al. [7] investigated placement and viewing behavior on TV, including aspects such as eye gaze patterns, attention shifts, and visual strategies. They found that users focus more on SLI positioned close to the main screen area, rather than in a picture-in-picture (PIP) format and prefer a medium size for the interpreter’s video. A study in Asia focused on users choosing settings such as color, placement, and on/off features [11]. This research revealed a preference among deaf users for medium-sized interpreter videos with no background color, positioned outside the video window, depending on the content type. Additionally, Kushalnagar et al. [12] found that an interpreter window size of at least one-third of the full screen is optimal for visual processing. Debevc et al. [10] and Kushalnagar et al. [12] also noted a preference among deaf users for access to customized controls and a transparent background for the SLI’s video.
From these studies, a clear pattern emerges. Deaf users prefer medium-sized sign language interpretation with a transparent background and, crucially, the ability to customize these settings according to individual preferences and needs. The concept of CIA is fairly new and has not been investigated much in the U.S. yet. This project aims to learn more about the available features of sign language interpreter placement, background, size, and positioning should be applied in practice. Our prototype made modifications to the open-source WCAG-compliant AblePlayer [13], which added functionality to drag, and resize the SLI, as well as change background color and transparency.
3 Method
Two studies were conducted. The first focused on user evaluation and usability of the proposed new features with the proof-of-concept platform, where users can adjust the SLI, and the second focused on analyzing eye gaze and attention behavior.
3.1 Study #1: Technical Features: Evaluation and Usability
Ten DHH and hearing individuals participated in the study. All were proficient in sign language, and five also used speech. Each received a $25 gift card. This study employed a mixed-methods approach under controlled conditions. The experimental design had two components: (i) video viewing with user customization, with descriptive observation, and (ii) evaluation surveys. The videos were presented on a mockup news website using AblePlayer. We added SL support through a drop-down menu for selecting a SLI with different color backgrounds (Blue, Green, and White) and transparency levels from 25% to 100%. Users could drag and reposition the SLI.
Four videos were presented to each participant: an Oscar ceremony clip, a news segment on gas prices, an interview, and an animal documentary, each ranging from 45 to 75 s. Two Certified Deaf Interpreters (CDI), one African American and one White, provided interpretation. The combination of four videos and two interpreters was randomized on a per-participant basis. Observational data were collected using the Think Aloud Protocol (TAP) [14], allowing participants to express their thoughts, feelings, and decisions in real-time while being recorded. An ASL version of the System Usability Scale (ASL-SUS) was used to evaluate the system’s usability, along with a 7-point Likert-style adjective scale about the overall user-friendliness of the technology [15]. This scale included 10 questions, each rated on a 5-point Likert scale. Additionally, a question about the overall user-friendliness of the technology was asked with a 7-point Likert scale ranging from Worst Imaginable to Best Imaginable.
3.2 Study #2: Interaction and Processing by Users
Nine DHH individuals, aged 18 to 44, participated in the study: five males, three females, and one nonbinary individual. Three were Black, three White, two Asian, and one Hispanic. Five participants preferred both English and ASL, while the rest preferred ASL. Each received a $25 gift card. This study utilized a mixed-methods approach under controlled conditions. The participants observed stimuli on a 27′′ monitor, positioned 42′′ away at eye level, and a mounted webcam recorded eye movements. The Gorilla Experiment Builder [16], sampled the video every 50 ms to generate eye tracking data. Due to technical difficulties, eye-tracking data was analyzed for only one participant, rather than all participants.
The stimuli had two 60-s videos depicting two scenarios: passive (interview) and active (basketball game). For each scenario, the SLI was presented in three sizes (regular, large, and super) and two positions with respect to the video (interior, exterior): see Fig. 1 for examples from the active scenario. We excluded Interior/Super because it did not fit inside the video. Participants watched 10 videos in random order to cover all video types, sizes and positioning combinations. The SL content was prerecorded by a CDI as in Study #1. Participants completed a brief survey and provided feedback after each video. A 5-point Likert scale assessed (i) comprehension, (ii) visibility, and (iii) cohesion, followed by an open-ended question.
4 Results
4.1 Study #1: Technical Features: Evaluation and Usability
Upon aggregating the scores of all ten participants, the mean System Usability Scale (SUS) score 82.2, which indicates good overall usability and effective system design. Notably, two participants recorded lower scores of 65 and 55, respectively. Despite these variations, the mean score remains significantly above the average SUS benchmark of 68, denoting a high level of usability. On the adjective scale, reactions were positive. 33.3% rated user-friendliness as ‘Good’ (score 5), while a larger segment, 50%, perceived it as ‘Excellent’ (score 6). Finally, 16.7% of users rated it as the ‘Best Imaginable’ (score 7). Participants liked the ability to reposition the interpreter within the window video: “The draggable signer window is fantastic…” Customizable backgrounds had a mixed reception, especially the transparency feature. Two-thirds of the participants felt that this feature was not helpful because the variability of foreground color can interfere with the viewing experience. One participant commented: “It (transparency) looks cool at first, but when I enter Fullscreen mode and activate it with 50% opacity with a busy background, it is difficult to view”.
This an experimental study where the platform was built on AblePlayer, and it was not fully tested beforehand for its UIUX. Therefore, there were some glitches, which were resolved during the study. Notably, the intuitiveness of the preference settings is complex and not friendly to inexperienced computer users. Some participants require more time and guidance to take advantage of the features offered. Ultimately, most participants agreed that the CIA was a useful platform to support the accessibility of SL in the media. Few respondents indicated they would still prefer PIP or auto-generated live captions for immediate accessibility.
4.2 Study #2: Interaction and Processing by Users
In the passive scenario, participants predominantly focused on the right side of the screen where the interviewer and interpreter were positioned. The heatmap (Fig. 2) provides insight into a participant’s gaze activity, highlighting attention on both the interviewer and, notably, the interpreter. Participants slightly preferred the exterior interpreter scenarios (Fig. 3); however, paired t-tests showed no significant differences. Participants commented that exterior placement facilitated optimal visibility of both the interviewer and the guest without any visual obstruction: “I prefer an interpreter on the exterior from outside of the video because it is easy to see clearly on the side of the video than ‘in the way’ of the video screen”. We observed that larger interpreter sizes, particularly the Exterior/Super setting, obscured the background video or introduced visual distractions.
In the active scenario of a basketball game, data from one participant showed more variability in their eye gaze patterns. Their attention was not solely confined to the interpreter but engaged with the entire screen. This variation can largely be attributed to the moving and unpredictable nature of the on-screen action, primarily centered around the movement of the basketball (Fig. 4). No consistent preferences as to placement emerged (Fig. 5), although one participant commented on a preference for the interior interface: “Less strain on my eyes and still want to see the actions and the interpreter at the same time”. Others noted that larger interpreter sizes obstructed the content in the corner of the screen, interfering at critical moments, such as a winning shot. A participant stated: “A large-sized interpreter outside the video is much better and clearer”.
5 Discussion
Participants liked the usability of the CIA prototype. In retrospect, both studies have addressed key research questions in terms of proving improved accessibility in the media with SL. The SL-specific features presented in the first study are all useful, especially with respect to the sizing, draggable interpreter placement, and background options. That way, the outlets could add the accessibility feature without compromising the primary content delivery, such as adding a PIP that cannot be turned on and off. To enhance user experience, it is essential to include a comprehensive set of options that balance a wide variety of choices with an intuitive preference-adjustment interface. These findings are consistent with the specific elements explored in the study conducted by Bosch et al. [7], Debvec et al. [10], and Yi et al. [11].
Results from the second study show that participant preferences with respect to interpreter placement and sizes vary, and limited data on the eye gaze suggests that placement may depend on the type of video shown; however, additional analysis of eye gaze is needed to confirm this. Furthermore, an interpreter occupying approximately one-third of the screen size, referred to as ‘Regular’, was preferred for both types of content. This finding aligns with the work of Kushalnagar et al. [12]. To enhance user experience, a comprehensive set of options must be included, balancing the variety of choices with an intuitive interface for adjusting preferences. This should encompass the option for broader adoption of this technology in mainstream media.
These studies collectively reveal a clear pattern and alignment with findings from other research in the field. Ultimately, this study’s primary goal is to enhance user experience, so personalization of the interpreter layout should be prioritized to align with individual viewing patterns of the diverse user’s preferences. This includes fully accessible preferred language options by the DHH community while encouraging the media outlets to include SLI in their programming without compromising the aesthetic and viewing screen landscape. Moreover, considering the integration of other devices, like Smart TVs and tablets with gesture commands, is essential due to their unique functionalities and the potential impact on user interaction. To achieve comprehensive and inclusive results, future studies should adopt a systematic research design incorporating diverse community members’ perspectives.
References
WFD. The Legal Recognition of National Sign Languages (2020). https://wfdeaf.org/news/the-legal-recognition-of-national-sign-languages/. Accessed 01 Feb 2024
Mitchell, R.E., Young, T.A.: How many people use sign language? A national health survey-based estimate. J. Deaf Stud. Deaf Educ. 28(1), 1–6 (2023). https://doi.org/10.1093/deafed/enac031
Garberoglio, C.L., Cawthon, S.W., Bond, M.: Assessing English literacy as a predictor of postschool outcomes in the lives of deaf individuals. J. Deaf Stud. Deaf Educ. 19(1), 50–67 (2014). https://doi.org/10.1093/deafed/ent038
Batterbury, S.C.E.: Language justice for sign language peoples: the UN convention on the rights of persons with disabilities. Lang. Policy 11(3), 253–272 (2012). https://doi.org/10.1007/s10993-012-9245-8
W3C. Understanding Success Criterion 1.2.6: Sign Language (Prerecorded) | WAI | W3C. https://www.w3.org/WAI/WCAG21/Understanding/sign-language-prerecorded.html. Accessed 30 Jan 2024
WFD-WASLI. Guidelines on Access to Information in National Sign Languages During Emergency Broadcasts (2021). https://wfdeaf.org/wp-content/uploads/2021/01/WFD-WASLI-Guidelines-on-Access-to-Information-in-SL-During-Emergency-Broadcasts_Final.pdf
Bosch-Baliarda, M., Soler-Vilageliu, O., Orero, P.: Sign language interpreting on TV: a reception study of visual screen exploration in deaf signing users. MonTI Monogr. Trad. E Interpret. 12, 108–143 (2020). https://doi.org/10.6035/MonTI.2020.12.04
NAD - Historic Win: White House Ordered to Provide Sign Language Interpreters for COVID-19 Briefings. https://www.nad.org/2020/09/23/historic-win-white-house-ordered-to-provide-sign-language-interpreters-for-covid-19-briefings/. Accessed 30 Jan 2024
NAD - Position Statement on Accessible Emergency Management for Deaf and Hard of Hearing People. https://www.nad.org/about-us/position-statements/position-statement-on-accessible-emergency-management-for-deaf-and-hard-of-hearing-people/. Accessed 30 Jan 2024
Debevc, M., Kosec, P., Holzinger, A.: Improving multimodal web accessibility for deaf people: sign language interpreter module. Multimed. Tools Appl. 54(1), 181–199 (2011). https://doi.org/10.1007/s11042-010-0529-8
Yi, J.H., Kim, S., Noh, Y.-G., Ok, S., Hong, J.-H.: Design proposal for sign language services in TV broadcasting from the perspective of people who are deaf or hard of hearing. Appl. Sci. 11(23), 11211 (2021). https://doi.org/10.3390/app112311211
Kushalnagar, R., Seita, M., Glasser, A.: Closed ASL interpreting for online videos. In: Proceedings of the 14th International Web for All Conference, Perth, Western Australia, Australia, pp. 1–4. ACM (2017). https://doi.org/10.1145/3058555.3058578
Able Player. Able Player. https://ableplayer.github.io/ableplayer/. Accessed 30 Jan 2024
Nielsen, J., Clemmensen, T., Yssing, C.: Getting access to what goes on in people’s heads? Reflections on the think-aloud technique. In: Proceedings of the Second Nordic Conference on Human-Computer Interaction, NordiCHI 2002, pp. 101–110. Association for Computing Machinery, New York (2002). https://doi.org/10.1145/572020.572033
Berke, L., Huenerfauth, M., Patel, K.: Design and psychometric evaluation of american sign language translations of usability questionnaires. ACM Trans. Access. Comput. 12(2), 1–43 (2019). https://doi.org/10.1145/3314205
Anwyl-Irvine, A.L., Massonnié, J., Flitton, A., Kirkham, N., Evershed, J.K.: Gorilla in our midst: an online behavioral experiment builder. Behav. Res. Methodshav. Res. Methods 52(1), 388–407 (2020). https://doi.org/10.3758/s13428-019-01237-x
Acknowledgments
This study was supported by the National Science Foundation (NSF) through the Research Experiences for Undergraduates (REU) Site Grants (Award Numbers 1757836 and 2150429), Dr. Raja Kushalnagar, PI, and Dr. Christian Vogler, Co-PI. This study is approved by the Gallaudet University Institutional Review Board; IRB-FY22-118.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
The authors have no conflicts of interest to declare.
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2024 The Author(s)
About this paper
Cite this paper
Boudreault, P. et al. (2024). Closed Sign Language Interpreting: A Usability Study. In: Miesenberger, K., Peňáz, P., Kobayashi, M. (eds) Computers Helping People with Special Needs. ICCHP 2024. Lecture Notes in Computer Science, vol 14751. Springer, Cham. https://doi.org/10.1007/978-3-031-62849-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-62849-8_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-62848-1
Online ISBN: 978-3-031-62849-8
eBook Packages: Computer ScienceComputer Science (R0)