research-article

Public Access

Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy Levels

Authors:

Jessica Trussell,

Matt HuenerfauthAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 279, Pages 1 - 12

https://doi.org/10.1145/3411764.3445038

Published: 07 May 2021 Publication History

All formats PDF

Abstract

Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.

Supplementary Material

VTT File (3411764.3445038_videopreviewcaptions.vtt)

Download
.81 KB

Supplementary Material (3411764.3445038_supplementalmaterials.zip)

Download
7.05 KB

MP4 File (3411764.3445038_videopreview.mp4)

Preview video

Download
1.36 MB

References

[1]

J. Albertini and C. Mayer. 2011. Using Miscue Analysis to Assess Comprehension in Deaf College Readers. Journal of Deaf Studies and Deaf Education 16, 1 (2011), 35–46. https://doi.org/10.1093/deafed/enq017

[2]

Oliver Alonzo, Lisa Elliot, Becca Dingman, and Matt Huenerfauth. 2020. Reading Experiences and Interest in Reading-Assistance Tools Among Deaf and Hard-of-Hearing Computing Professionals. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, 13. https://doi.org/10.1145/3373625.3416992

Digital Library

[3]

Oliver Alonzo, Matthew Seita, Abraham Glasser, and Matt Huenerfauth. 2020. Automatic Text Simplification Tools for Deaf and Hard of Hearing Adults: Benefits of Lexical Simplification and Providing Users with Autonomy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376563

Digital Library

[4]

Mahmoud Azab, Chris Hokamp, and Rada Mihalcea. 2015. Using Word Semantics To Assist English as a Second Language Learners. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. Association for Computational Linguistics, Denver, Colorado, 116–120. https://doi.org/10.3115/v1/N15-3024

[5]

Larwan Berke, Sushant Kafle, and Matt Huenerfauth. 2018. Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 91, 12 pages. https://doi.org/10.1145/3173574.3173665

Digital Library

[6]

Jan De Belder and Marie-Francine Moens. 2010. Text simplification for children. In Prroceedings of the SIGIR workshop on accessible search systems. ACM; New York, 19–26.

[7]

Joshua R De Leeuw. 2015. jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior research methods 47, 1 (2015), 1–12.

[8]

Siobhan Devlin and Gary Unthank. 2006. Helping Aphasic People Process Online Information. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Portland, Oregon, USA) (Assets ’06). ACM, New York, NY, USA, 225–226. https://doi.org/10.1145/1168987.1169027

Digital Library

[9]

Jack Downey. 2010. Careers in Software: Is There Life after Programming?. In Proceedings of the 2010 Special Interest Group on Management Information System’s 48th Annual Conference on Computer Personnel Research on Computer Personnel Research (Vancouver, BC, Canada) (SIGMIS-CPR ’10). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/1796900.1796912

Digital Library

[10]

Mary C Dyson and Mark Haselgrove. 2001. The Influence of Reading Speed and Line Length on the Effectiveness of Reading from Screen. Int. J. Hum.-Comput. Stud. 54, 4 (April 2001), 585–612. https://doi.org/10.1006/ijhc.2001.0458

Digital Library

[11]

Susan R Easterbrooks and Sharon Baker. 2002. Language Learning in Children Who Are Deaf and Hard of Hearing: Multiple Pathways.ERIC.

[12]

Matt Huenerfauth. 2008. Evaluation of a Psycholinguistically Motivated Timing Model for Animations of American Sign Language. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada) (Assets ’08). Association for Computing Machinery, New York, NY, USA, 129–136. https://doi.org/10.1145/1414471.1414496

Digital Library

[13]

Matt Huenerfauth, Lijun Feng, and Noémie Elhadad. 2009. Comparing Evaluation Techniques for Text Readability Software for Adults with Intellectual Disabilities. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, Pennsylvania, USA) (Assets ’09). Association for Computing Machinery, New York, NY, USA, 3–10. https://doi.org/10.1145/1639642.1639646

Digital Library

[14]

Matt Huenerfauth and Hernisa Kacorri. 2015. Best practices for conducting evaluations of sign language animation. In 30th Annual International Technology and Persons with Disabilities Conference Scientific/Research Proceedings. California State University, Northridge.

[15]

Kentaro Inui, Atsushi Fujita, Tetsuro Takahashi, Ryu Iida, and Tomoya Iwakura. 2003. Text Simplification for Reading Assistance: A Project Note. In Proceedings of the Second International Workshop on Paraphrasing - Volume 16 (Sapporo, Japan) (PARAPHRASE ’03). Association for Computational Linguistics, Stroudsburg, PA, USA, 9–16. https://doi.org/10.3115/1118984.1118986

Digital Library

[16]

Marcel Adam Just and Patricia A. Carpenter. 1984. Using Eye Fixations to Study Reading Comprehension. Erlbaum, 151–182.

[17]

Lynda J Katz and Franklin C Brown. 2019. Aptitude and achievement testing. In Handbook of Psychological Assessment. Elsevier, 143–168.

[18]

Poorna Kushalnagar, Scott Smith, Melinda Hopper, Claire Ryan, Micah Rinkevich, and Raja Kushalnagar. 2018. Making cancer health text on the Internet easier to read for deaf people who use American Sign Language. Journal of Cancer Education 33, 1 (2018), 134–140.

[19]

Walter S. Lasecki, Luz Rello, and Jeffrey P. Bigham. 2015. Measuring Text Simplification with the Crowd. In Proceedings of the 12th Web for All Conference(Florence, Italy) (W4A ’15). Association for Computing Machinery, New York, NY, USA, Article 4, 9 pages. https://doi.org/10.1145/2745555.2746658

Digital Library

[20]

Qisheng Li, Meredith Ringel Morris, Adam Fourney, Kevin Larson, and Katherina Reinecke. 2019. The Impact of Web Browser Reader Views on Reading Speed and User Experience. In CHI 2019. ACM. https://www.microsoft.com/en-us/research/publication/the-impact-of-web-browser-reader-views-on-reading-speed-and-user-experience/

[21]

Angrosh Annayappan Mandya, Tadashi Nomoto, and Advaith Siddharthan. 2014. Lexico-syntactic text simplification and compression with typed dependencies. In 25th International Conference on Computational Linguistics.

[22]

Marc Marschark, John A. Albertini, and Harry G. Lang. 2002. Educating deaf students: from research to practice. Oxford University Press.

[23]

Carolyn Morrison, Marc Marschark, Thomastine Sarchet, Carol M. Convertino, Georgianna Borgna, and Richard Dirmyer. 2013. Deaf students’ metacognitive awareness during language comprehension. European Journal of Special Needs Education 28, 1 (2013), 78–90. https://doi.org/10.1080/08856257.2012.749610

[24]

Carol Padden, Tom Humphries, and Carol Padden. 2009. Inside deaf culture. Harvard University Press.

[25]

Gustavo Paetzold and Lucia Specia. 2016. Understanding the Lexical Simplification Needs of Non-Native Speakers of English. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 717–727. https://www.aclweb.org/anthology/C16-1069

[26]

S. J. Parault and H. M. Williams. 2010. Reading Motivation, Reading Amount, and Text Comprehension in Deaf and Hearing Adults. Journal of Deaf Studies and Deaf Education 15, 2 (2010), 120–135. https://doi.org/10.1093/deafed/enp031

[27]

LeAdelle Phelps and Barbara Jane Branyan. 1990. Academic achievement and nonverbal intelligence in public school hearing-impaired children. Psychology in the Schools 27, 3 (1990), 210–217.

[28]

Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify or Help?: Text Simplification Strategies for People with Dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (Rio de Janeiro, Brazil) (W4A ’13). ACM, New York, NY, USA, Article 15, 10 pages. https://doi.org/10.1145/2461121.2461126

Digital Library

[29]

Luz Rello, Roberto Carlini, Ricardo Baeza-Yates, and Jeffrey P. Bigham. 2015. A Plug-in to Aid Online Reading in Spanish. In Proceedings of the 12th Web for All Conference(Florence, Italy) (W4A ’15). Association for Computing Machinery, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/2745555.2746661

Digital Library

[30]

Luz Rello, Martin Pielot, and Mari-Carmen Marcos. 2016. Make It Big!: The Effect of Font Size and Line Spacing on Online Readability. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 3637–3648. https://doi.org/10.1145/2858036.2858204

Digital Library

[31]

Luz Rello, Horacio Saggion, Ricardo Baeza-Yates, and Eduardo Graells. 2012. Graphical schemes may improve readability but not understandability for people with dyslexia. In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations. 25–32.

[32]

Naomi B Robbins, Richard M Heiberger, 2011. Plotting Likert and other rating scales. In Proceedings of the 2011 Joint Statistical Meeting. 1058–1066.

[33]

Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015. Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish. ACM Trans. Access. Comput. 6, 4, Article 14 (May 2015), 36 pages. https://doi.org/10.1145/2738046

Digital Library

[34]

Tenace Setor, Damien Joseph, and Shirish C. Srivastava. 2015. Professional Obsolescence in IT: The Relationships between the Threat of Professional Obsolescence, Coping and Psychological Strain. In Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research (Newport Beach, California, USA) (SIGMIS-CPR ’15). Association for Computing Machinery, New York, NY, USA, 117–122. https://doi.org/10.1145/2751957.2751962

Digital Library

[35]

Matthew Shardlow. 2014. Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). European Language Resources Association (ELRA), Reykjavik, Iceland, 1583–1590. http://www.lrec-conf.org/proceedings/lrec2014/pdf/479_Paper.pdf

[36]

Matthew Shardlow. 2014. A survey of automated text simplification. International Journal of Advanced Computer Science and Applications 4, 1(2014), 58–70.

[37]

Advaith Siddharthan. 2014. A survey of research on text simplification. ITL - International Journal of Applied Linguistics 165, 2(2014), 259–298. https://doi.org/10.1075/itl.165.2.06sid

[38]

Barbara K Strassman. 1997. Metacognition and reading in children who are deaf: A review of the research. Journal of Deaf Studies and Deaf Education(1997), 140–149.

[39]

C. B. Traxler. 2000. The Stanford Achievement Test, 9th Edition: National Norming and Performance Standards for Deaf and Hard-of-Hearing Students. Journal of Deaf Studies and Deaf Education 5, 4 (Jan 2000), 337–348. https://doi.org/10.1093/deafed/5.4.337

[40]

Shari Trewin, Diogo Marques, and Tiago Guerreiro. 2015. Usage of Subjective Scales in Accessibility Research. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). ACM, New York, NY, USA, 59–67. https://doi.org/10.1145/2700648.2809867

Digital Library

[41]

Dawn Walton, Georgianna Borgna, Marc Marschark, Kathryn Crowe, and Jessica Trussell. 2019. I am not unskilled and unaware: deaf and hearing learners’ self-assessments of linguistic and nonlinguistic skills. European Journal of Special Needs Education 34, 1 (2019), 20–34. https://doi.org/10.1080/08856257.2018.1435010

[42]

Willian Massami Watanabe, Arnaldo Candido Junior, Vinícius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluísio. 2009. Facilita: Reading Assistance for Low-literacy Readers. In Proceedings of the 27th ACM International Conference on Design of Communication (Bloomington, Indiana, USA) (SIGDOC ’09). ACM, New York, NY, USA, 29–36. https://doi.org/10.1145/1621995.1622002

Digital Library

[43]

Gary S. Wilkinson and Gary J. Robertson. 2006. Wide Range Achievement Test 4 professional manual. Psychological Assessment Resources, Inc.

[44]

Kristian Woodsend and Mirella Lapata. 2011. Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Edinburgh, Scotland, UK., 409–420. https://www.aclweb.org/anthology/D11-1038

[45]

Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in Current Text Simplification Research: New Data Can Help. Transactions of the Association for Computational Linguistics 3 (2015), 283–297. https://doi.org/10.1162/tacl_a_00139

[46]

Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing Statistical Machine Translation for Text Simplification. Transactions of the Association for Computational Linguistics 4 (2016), 401–415. https://cocoxu.github.io/publications/tacl2016-smt-simplification.pdf

[47]

Peixuan Yan. 2018. Effects of Metacognition on English Reading Outcomes for d/Deaf and Hard of Hearing Students. Ph.D. Dissertation. The Ohio State University.

[48]

Chen-Hsiang Yu and Robert C. Miller. 2010. Enhancing Web Page Readability for Non-native Readers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). ACM, New York, NY, USA, 2523–2532. https://doi.org/10.1145/1753326.1753709

Digital Library

Cited By

Alonzo OLee SAmin AMaddela MXu WHuenerfauth M(2024)Design and Evaluation of an Automatic Text Simplification Prototype with Deaf and Hard-of-hearing ReadersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675645(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675645
Säuberli AHolzknecht FHaller PDeilen SSchiffl LHansen-Schirra SEbling S(2024)Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual DisabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642570(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642570
Imashev AOralbayeva NSandygulova A(2024)An exploratory user study towards developing a unified, comprehensive assessment apparatus for deaf signers, specifically tailored for signing avatars evaluation: challenges, findings, and recommendationsMultimedia Tools and Applications10.1007/s11042-024-20365-xOnline publication date: 29-Oct-2024
https://doi.org/10.1007/s11042-024-20365-x
Show More Cited By

Index terms have been assigned to the content through auto-classification.

Recommendations

Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While ...
Reading Experiences and Interest in Reading-Assistance Tools Among Deaf and Hard-of-Hearing Computing Professionals
ASSETS '20: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility

Automatic Text Simplification (ATS) software replaces text with simpler alternatives. While some prior research has explored its use as a reading assistance technology, including some empirical findings suggesting benefits for deploying this technology ...
Reading-Assistance Tools Among Deaf and Hard-of-Hearing Computing Professionals in the U.S.: Their Reading Experiences, Interests and Perceptions of Social Accessibility
Automatic Text Simplification (ATS) software aims at automatically rewrite complex text to make it simpler to read. Prior research has explored the use of ATS as a reading assistance technology, identifying benefits from providing these technologies to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
708
Total Downloads

Downloads (Last 12 months)185
Downloads (Last 6 weeks)34

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alonzo OLee SAmin AMaddela MXu WHuenerfauth M(2024)Design and Evaluation of an Automatic Text Simplification Prototype with Deaf and Hard-of-hearing ReadersProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675645(1-18)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675645
Säuberli AHolzknecht FHaller PDeilen SSchiffl LHansen-Schirra SEbling S(2024)Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual DisabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642570(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642570
Imashev AOralbayeva NSandygulova A(2024)An exploratory user study towards developing a unified, comprehensive assessment apparatus for deaf signers, specifically tailored for signing avatars evaluation: challenges, findings, and recommendationsMultimedia Tools and Applications10.1007/s11042-024-20365-xOnline publication date: 29-Oct-2024
https://doi.org/10.1007/s11042-024-20365-x
Säuberli AHansen-Schirra SHolzknecht FGutermuth SDeilen SSchiffl LEbling S(2023)Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile applicationFrontiers in Communication10.3389/fcomm.2023.11756258Online publication date: 3-Aug-2023
https://doi.org/10.3389/fcomm.2023.1175625
Kim JAhn SHong J(2023)Visible Nuances: A Caption System to Visualize Paralinguistic Speech Cues for Deaf and Hard-of-Hearing IndividualsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581130(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581130
Kim JLee CKim JHong J(2023)Interactive description to enhance accessibility and experience of deaf and hard-of-hearing individuals in museumsUniversal Access in the Information Society10.1007/s10209-023-00983-223:2(913-926)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1007/s10209-023-00983-2
Glasser AMinakov FBragg D(2022)ASL Wiki: An Exploratory Interface for Crowdsourcing ASL TranslationsProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544827(1-13)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544827
Alonzo OTrussell JWatkins MLee SHuenerfauth M(2022)Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy LevelsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517566(1-10)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517566
Tseng CJung SElglaly YLiu YLudi SMerkle LDoyle MSheard JSoh LDorn B(2022)Exploration on Integrating Accessibility into an AI CourseProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 110.1145/3478431.3499399(864-870)Online publication date: 22-Feb-2022
https://dl.acm.org/doi/10.1145/3478431.3499399

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten