research-article

Types of problems encountered by automated tool accessibility assessments, expert inspections and user testing: a systematic literature mapping

Authors:

Carlos Alberto Silva,

Arthur F. B. A. de Oliveira,

Delvani Antônio Mateus,

Heitor Augustus Xavier Costa,

André Pimenta FreireAuthors Info & Claims

IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems

Article No.: 52, Pages 1 - 11

https://doi.org/10.1145/3357155.3358479

Published: 22 October 2019 Publication History

Abstract

The use of websites has become essential for the development of daily activities. However, not all people can have full access to web-based services and content due to many websites being inaccessible to disabled people, such as people with visual disabilities. In this context, even though developers may demonstrate effort to attempt to create more accessible content, there is limited information about the characteristics of different web accessibility evaluation methods. Thus, the aim of the present study was to perform a meta-analysis of 38 types of accessibility problems on websites extracted from 19 studies in the literature. The studies performed automated assessments using tools, expert-based inspections and user tests involving disabled people. The results confirm other considerations made in the literature, showing that automated assessment methods have significant limitations in the adequate coverage of accessibility problems, covering less than 40% of the types of problems encountered. This way, the paper concludes by showing a consolidation of data from the literature to reinforce claims that effective web accessibility evaluations should count on expert-based inspections and user evaluations involving disabled users.

References

[1]

Ali Abdolrahmani, and Ravi Ruber. 2016. Should I trust it when I cannot see it? Credibility assessment for blind web users. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, 191--199.

Digital Library

[2]

ShadiAbou-Zahra. 2008. Web accessibility evaluation. In: Web accessibility. Springer, London, 2008, p. 79--106.

[3]

Amaia Aizpurua, Myriam Arrue, Simon Harper, and Markel Vigo. 2014. Are users the gold standard for accessibility evaluation? In Proceedings of the 11th International Web for All Conference.

Digital Library

[4]

Amaia Aizpurua, Myriam Arrue, and Markel Vigo. 2015. Prejudices, memories, expectations and confidence influence experienced accessibility on the Web. Computers in Human Behavior 51, PA: 152--160.

[5]

Chieko Asakawa, and Barbara Leporini. 2009. Screen readers. Chapter 28. In Universal Access Handbook. Org. Constantine Stephanidis.

[6]

Raj Kumar Bhardwaj. 2017. A comprehensive digital environment for visually impaired students: user's perspectives. Library Hi Tech 35, 4: 542--557.

[7]

Giorgio Brajnik. 2008. Beyond conformance: the role of accessibility evaluation methods. In Proceedings of the 9th International Conference on Web Information Systems Engineering, 63--80.

Digital Library

[8]

Giorgio Brajnik, Yeliz Yesilada, and Simon Harper. 2011. The expertise effect on web accessibility evaluation methods. Human-Computer Interaction 26, 3: 246--283.

[9]

Diego Fernández Buenano, Tania Acosta, and Sergio Luján-Mora. 2018. The use of la to evaluate the performance of students with visual disabilities when applying accessibility criteria in online courses. In Proceedings of the 1st Latin American Workshop on Learning Analytics.

[10]

Michael Crystian Nepomuceno Carvalho, Felipe Silva Dias, Aline Grazielle Silva Reis, and André Pimenta Freire. 2018. Accessibility and usability problems encountered on websites and applications in mobile devices by blind and normal-vision users. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, 2022--2029.

Digital Library

[11]

Yujeong Cho, Geunseong Jung, and Jaehhyuk Cha. 2018. Proposal and evaluation of webpage menu interface for people with low vision. International Journal of Advanced Science and Technology 119: 33--52.

[12]

Mexhid Ferati, Bahtijar Vogel, Arianit Kurti, Bujar Raufi, and David Salvador Astals. 2014. Web accessibility for visually impaired people: requirements and design issues. In Proceedings of the 2nd International Workshop, Usability- and Accessibility-Focused Requirements Engineering.

[13]

André Pimenta Freire. 2012. Disabled people and the Web: user-based measurement of accessibility. Ph.D. Dissertation. University of York.

[14]

Marcio Maestrelo Funes, Renata P. M. Fortes, and Rudinei Goularte. 2017. Gestural interaction for accessibility of web videos: a case study analysis with blind and visually impaired users. In Proceedings of the 23rd Brazillian Symposium on Multimedia and the Web, 501--504.

Digital Library

[15]

Prathik Gadde, and Davide Bolchini. 2014. From screen reading to aural glancing: towards instant access to key page sections. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers and Accessibility, 67--74.

Digital Library

[16]

Thea van der Geest, Hans van der Meij, and Carolina van Puffelen. 2014. Self-assessed and actual Internet skills of people with visual impairments. Universal Access in the Information Society 13, 2: 161--174.

[17]

Ramiro Gonçalves, Tânia Rocha, José Martins, Frederico Branco, and Manuel Au-Yong-Oliveira. 2018. Evaluation of e-commerce websites accessibility and usability: an e-commerce platform analysis with the inclusion of blind users. Universal Access in the Information Society 17, 3: 567--583. https://doi.org/10.1007%2Fs10209-017-0557-5

Digital Library

[18]

Governo Eletrônico. 2014. eMAG - Modelo de Acessibilidade em Governo Eletrônico. Retrieved May 29, 2019 from http://emag.governoeletronico.gov.br/

[19]

Chandra Harrison, and Helen Petrie. 2007. Severity of usability and accessibility problems in eCommerce and eGovernment websites. People and Computer XX-Engage. Springer. London, p 255--262.

[20]

Mohammed Saleh Hassouna, Noraidah Sahari Ashaari, and Amirah Ismail. 2017. University website accessibility for totally blind users. Journal of Information and Communication Technology 16, 1: 63--80.

[21]

Abid Ismail, K. S. Kuppusamy, and Ab Shakoor Nengroo. 2018. Multi-tool accessibility assessment of government department websites: a case-study with JKGAD. Disability and Rehabilitation: Assistive Technology 13, 6: 504--516.

[22]

ISO. International Organization for Standardization. 2008. ISO 9241-171 - Ergonomics of human-system interaction - Part 171: Guidance on software accessibility.

[23]

ISO. International Organization for Standardization. 2018. ISO 9241-11 - Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.

[24]

Melody Y. Ivory. 2003. Automated web site evaluation: researchers' and practioners' perspectives. Springer Science & Business Media.

[25]

Paul Jaeger. 2006. Assessing section 508 compliance on federal e-government web sites: a multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly 23, 2: 169--190.

[26]

Jonathan Lazar. 2006. Web usability: a user-centered design approach. MA: Pearson Addison Wesley, Boston.

[27]

Janaína R. Loureiro, Maria Istela Cagnin, and Débora M. B. Paiva. 2014. Web accessibility in social networking services. In Proceedings of the 14th International Conference on Computational Science and Its Applications, 586--601.

[28]

Janaína R. Loureiro, Maria Istela Cagnin, and Débora M. B. Paiva. 2015. Analysis of web accessibility in social networking services through blind users' perspective and an accessible prototype. In Proceedings of the 15th International Conference on Computational Science and Its Applications, 117--131. https://www.

[29]

Rita Kosztyánné Mátrai. 2018. How to make an electronic library accessible. Electronic Library 36, 4: 620--632.

[30]

Rosa Navarrete, and Sergio Luján-Mora. 2016. OER-based learning and people with disabilities. In Proceedings of the International Conference on Interactive Collaborative and Blended Learning, 25--34.

[31]

Tiago C. Nogueira, Deller J. Ferreira, Sérgio T. Carvalho, and Luciana O. Berreta. 2017. Evaluating Responsive Web Design's Impact on Blind Users. IEEE Multimedia 24, 2: 86--95.

Digital Library

[32]

Letícia Seixas Pereira, and Dominique Archambault. 2017. Web Widgets Barriers for Visually Impaired Users. Studies in Health Technology and Informatics 242: 836--842.

[33]

Letícia Seixas Pereira, and Dominique Archambault. 2018. Correlating navigation barriers on web 2.0 with accessibility guidelines. In Proceedings of the 16th International Conference on Computers Helping People with Special Needs. https://doi.org/10.1007%2F978-3-319-94277-3_3

Digital Library

[34]

Helen Petrie, and Nigel Bevan. 2009. The evaluation of accessibility, usability, and user experience.

[35]

Jill A. Power. 2018. EBSCO information services usability study on accessibility. Reference Services Review 46, 3: 449--459.

[36]

Christopher Power, André Pimenta Freire, Helen Petrie, and David Swallow. 2012. Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the 30th ACM Conference on Human Factors in Computing Systems, 433--442.

Digital Library

[37]

Costin Pribeanu, Paul Fogarassy-Neszly, and Aure lPâtru. 2014. Municipal web sites accessibility and usability for blind users: preliminary results from a pilot study. Universal Access in the Information Society 13, 6: 339--349. https://doi.org/10.1007%2Fs10209-013-0315-2

Digital Library

[38]

Laura A. C. Rodrigues, and Soraia S. Prietch. 2018. Analysis, redesign and validation of accessibility resources applied to an official electronic journal for the promotion of equal access to public acts. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems.

Digital Library

[39]

Shrirang Sahasrabudhe, and Rahul Singh. 2016. Accessibility problems of blind mHealth users, a pilot study. In Proceedings of the 22nd Americas Conference on Information Systems: Surfing the IT Innovation Wave.

[40]

US Government. Section 508 guideline. Retrieved May 29, 2019 from https://www.section508.gov/.

[41]

Markel Vigo, Justin Brown, and Vivienne Conway. 2013. Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility.

Digital Library

[42]

W3C. 2016. Web accessibility evaluation tools list. Retrieved May 01, 2019 from https://www.w3.org/WAI/ER/tools/.

[43]

W3C. 2019. Cartilha de acessibilidade na web. Retrieved February 21, 2019 from http://www.w3c.br/pub/Materiais/PublicacoesW3C/cartilha-w3cbr-acessibilidade-web-fasciculo-I.html.

[44]

W3C. 2019. Web Content Accessibility Guidelines (WCAG). Retrieved May 29, 2019 from https://www.w3.org/WAI/standards-guidelines/wcag/

[45]

Brian Wentz, Dung Pham, Erin Feaser, Dylan Smith, James Smith, and Allison Wilson. 2018. Documenting the accessibility of 100 US bank and finance websites. Universal Access in the Information Society: 1--10. https://doi.org/10.1007%2Fs10209-018-0616-6

[46]

Yong Jeong Yi. 2018. Web accessibility of healthcare Web sites of Korean government and public agencies: a user test for persons with visual impairment. Universal Access in the Information Society: 1--16. https://doi.org/10.1007%2Fs10209-018-0625-5

[47]

Emilene Zitkus, Aline C. Brigatto, Ana Lya M. Ferrari, Gabriel H. C. Bonfim, Idinei F. P. Carvalho Filho, Thaís D. Reis, Fausto O. Medola, and Luis C. Paschoarelli. 2016. Accessibility and usability of websites intended for people with disabilities: a preliminary study. In Proceedings of the 18th International Conference on Human-Computer Interaction, 678--688.

Cited By

Mateus DDa Silva FRigatto SDa Silva TParreira Júnior PSouza MFreire A(2024)The Legal Handling of Digital Accessibility: a Comparison of Evaluation and Policy Approaches in Federal-Level Cases in Brazil and the United StatesJournal of the Brazilian Computer Society10.5753/jbcs.2024.325930:1(363-379)Online publication date: 26-Sep-2024
https://doi.org/10.5753/jbcs.2024.3259
Laitano MOrtiz-Núñez RStable-Rodríguez Y(2024)User participation in digital accessibility evaluations: Reviewing methods and implicationsEducation for Information10.3233/EFI-240014(1-16)Online publication date: 12-Sep-2024
https://doi.org/10.3233/EFI-240014
Palmer ZOswal S(2024)Constructing Websites with Generative AI Tools: The Accessibility of Their Workflows and Products for Users With DisabilitiesJournal of Business and Technical Communication10.1177/10506519241280644Online publication date: 28-Sep-2024
https://doi.org/10.1177/10506519241280644
Show More Cited By

Index Terms

Types of problems encountered by automated tool accessibility assessments, expert inspections and user testing: a systematic literature mapping
1. Human-centered computing
  1. Accessibility
    1. Accessibility design and evaluation methods

Recommendations

The extent of mobile accessibility coverage in WCAG 2.1: sufficiency of success criteria and appropriateness of relevant conformance levels pertaining to accessibility problems encountered by users who are visually impaired
Abstract
People who are blind or visually impaired exhibit different interaction and access needs on mobile touchscreen devices. This is rooted in the reliance on non-visual access for using electronic devices. In comparison to regular computers, non-...
Analysis and integration of web accessibility metrics
W4A '10: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A)

"Analysis and Integration of Web Accessibility Metrics" was presented as a grade thesis. It included the development of "OceanAcc", an evaluation and analysis tool that performs a semi-automatic evaluation process. With little user intervention, can ...
Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated Tools
IHC '23: Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems

Mobile apps should be usable and accessible to everyone, including visually impaired people. Accessibility evaluation techniques are essential to identify accessibility problems and help to avoid problems that hinder access by people with disabilities. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

IHC '19: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems

October 2019

679 pages

ISBN:9781450369718

DOI:10.1145/3357155

General Chairs:
Rodrigo Laiola Guimaraes
Universidade Federal do Espirito Santo (UFES)
,
Celso Alberto Saibel Santos
Universidade Federal do Espirito Santo (UFES)
,
Program Chairs:
Vagner Figueredo de Santana
IBM Research
,
Artur Kronbauer
Universidade Salvador (UNIFACS)

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SBC: Brazilian Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conselho Nacional de Desenvolvimento Científico e Tecnológico

Conference

IHC '19

Sponsor:

SBC

IHC '19: XVIII Brazilian Symposium on Human Factors in Computing Systems

October 22 - 25, 2019

Espírito Santo, Vitória, Brazil

Acceptance Rates

IHC '19 Paper Acceptance Rate 56 of 165 submissions, 34%;

Overall Acceptance Rate 331 of 973 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
252
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)3

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mateus DDa Silva FRigatto SDa Silva TParreira Júnior PSouza MFreire A(2024)The Legal Handling of Digital Accessibility: a Comparison of Evaluation and Policy Approaches in Federal-Level Cases in Brazil and the United StatesJournal of the Brazilian Computer Society10.5753/jbcs.2024.325930:1(363-379)Online publication date: 26-Sep-2024
https://doi.org/10.5753/jbcs.2024.3259
Laitano MOrtiz-Núñez RStable-Rodríguez Y(2024)User participation in digital accessibility evaluations: Reviewing methods and implicationsEducation for Information10.3233/EFI-240014(1-16)Online publication date: 12-Sep-2024
https://doi.org/10.3233/EFI-240014
Palmer ZOswal S(2024)Constructing Websites with Generative AI Tools: The Accessibility of Their Workflows and Products for Users With DisabilitiesJournal of Business and Technical Communication10.1177/10506519241280644Online publication date: 28-Sep-2024
https://doi.org/10.1177/10506519241280644
Pool J(2023)Accessibility MetatestingProceedings of the 20th International Web for All Conference10.1145/3587281.3587282(1-4)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3587281.3587282
Mateus DSilva FSilva TFreire ASantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Evaluation methods in legal procedures concerning digital accessibility in BrazilProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3559130(1-12)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3559130
Mateus DSilva CEler MFreire ASantos CBento Villela MGasparini IUchôa Conte T(2020)Accessibility of mobile applicationsProceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems10.1145/3424953.3426633(1-10)Online publication date: 26-Oct-2020
https://dl.acm.org/doi/10.1145/3424953.3426633

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents