Article

Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAs

Authors:

Obianuju Okafor,

Wajdi Aljedaani,

Stephanie LudiAuthors Info & Claims

HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging: 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings

Pages 479 - 500

https://doi.org/10.1007/978-3-031-17902-0_34

Published: 26 June 2022 Publication History

Abstract

Accessibility is a required quality for websites today, and several tools exist to test for this quality. These tools are highly advantageous, but sadly they also have some limitations. A particular set of challenges they face is in the evaluation of Rich Internet Applications (RIAs). In this paper, we carry out an experiment to compare and analyze different accessibility testing tools as they evaluate 10 educational websites. We judged these tools based on their error detection, guideline coverage, speed, similarity to one another, and their relative performance when evaluating RIAs. The experiment findings revealed the strength and limitations of each tool. The results of this experiment also exposed that there are many guidelines and success criteria that accessibility testing tools are not able to cover, and that some evaluation tools are similar to each other in terms of the results they produce. Lastly, this experiment highlights a discrepancy in the behavior of the tools when evaluating RIAs compared to when evaluating static websites, although some more than others. This experiment has some limitations which we presented. As a future work, we intend to work with an expert to determine the accuracy of the results produced from the experiment. We also intend to delve deeper into the limitations of these tools and come up with possible solutions.

References

[1]

Al-Ahmad, A., Ahmaro, I.Y., Mustafa, M.: Comparison between web accessibility evaluation tools, January 2013. https://www.researchgate.net/publication/279181298

[2]

Alsaeedi A Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks Information 2020 11 1 40

[3]

Baazeem, I.S., Al-Khalifa, H.S.: Advancements in web accessibility evaluation methods: how far are we? In: Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services. iiWAS 2015, 11–13 December 2015. ACM Press, New York.

[4]

Brajnik, G., Lomuscio, R.: Samba: a semi-automatic method for measuring barriers of accessibility. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility, Assets 2007, October 15–17 2007, pp. 43–50. ACM Press, New York.

[5]

Campoverde-Molina M, Lujan-Mora S, and Garcia LV Empirical studies on web accessibility of educational websites: a systematic literature review IEEE Access 2020 8 91676-91700

[6]

Fortes, R.P., Antonelli, H.L., de Lima Salgado, A.: Accessibility and usability evaluation of rich internet applications. In: Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web, Webmedia 2016, 08–11 November 2016, pp. 7–8. ACM Press, New York.

[7]

Human Network Contributor: Websites fail to comply with accessibility for people with disabilities. November 2020. https://isemag.com/2020/11/telecom-98-percent-of-websites-fail-to-comply-with-accessibility-requirements-for-people-with-disabilities/

[8]

Investis Digital: Accessibility testing - manual or automated? January 2020. https://www.investisdigital.com/blog/web-design-and-development/accessibility-testing-manual-or-automated. Accessed 23 Oct 2021

[9]

Ismail, A., Kuppusamy, K.: Web accessibility investigation and identification of major issues of higher education websites with statistical measures: a case study of college websites. J. King Saud Univ. Comput. Inf. Sci. (2019)

[10]

Kumar, A.: 5 free must have web accessibility testing tools - leader in offshore accessibility testing — section 508 compliance — wcag conformance — barrierbreak, October 2021. https://www.barrierbreak.com/5-free-must-have-web-accessibility-testing-tools/

[11]

Law Office of Lainey Feingold: Digital accessibility laws around the globe, May 2013. https://www.lflegal.com/2013/05/gaad-legal/. Accessed 23 Oct 2021

[12]

Linaje M, Lozano-Tello A, Perez-Toledano MA, Preciado JC, Rodriguez-Echeverria R, and Sanchez-Figueroa F Providing ria user interfaces with accessibility properties J. Symb. Comput. 2011 46 207-217

[13]

Lopes, R., Gomes, D., Carriço, L.: Web not for all: A large scale study of web accessibility. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). W4A 2010. Association for Computing Machinery, New York (2010).

[14]

Mankoff, J., Fait, H., Tran, T.: Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In: In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2005, pp. 41–50. ACM Press, New York (2005)

[15]

Misfud, J.: 8 free web-based website accessibility evaluation tools - usability geek. https://usabilitygeek.com/10-free-web-based-web-site-accessibility-evaluation-tools/

[16]

Mucha, J.M.: Combination of automatic and manual testing for web accessibility. Master’s thesis, Grimstad, Norway (2018). Accessed 05 Aug 2019

[17]

Neova Tech Solutions: 7 web accessibility testing tools you should know, April 2020. https://www.neovasolutions.com/2020/04/08/7-web-accessibility-testing-tools-you-should-know/. Accessed 03 Mar 2022

[18]

Smith, S.: Website accessibility standards should be higher to help disabled people - vox, May 2019. https://www.vox.com/the-goods/2019/2/5/18210912/websites-ada-compliance-lawsuits. Accessed 04 Mar 2022

[19]

Software Testing Help: Top 20 accessibility testing tools for web applications, Feb 2022. https://www.softwaretestinghelp.com/accessibility-testing-tools/. Accessed 03 Mar 2022

[20]

Timbi-Sisalima, C., Amor, C.I.M., Tortosa, S.O., Hilera, J.R., Aguado-Delgado, J.: Comparative analysis of online web accessibility evaluation tools. In: Proceedings of the 25th International Conference on Information Systems Development, ISD 2016, pp. 562–573. University of Economics, ACM Press, New York (2016)

[21]

Usablenet: Quick guide to manual accessibility testing and why it’s important, June 2018. https://blog.usablenet.com/quick-guide-to-manual-accessibility-testing-and-why-its-important. Accessed 02 Mar 2022

[22]

Usablenet: Automated wcag testing is not enough for web accessibility ada compliance [blog], July 2020. https://blog.usablenet.com/automated-wcag-testing-is-not-enough-for-web-accessibility-ada-compliance. Accessed 02 Mar 2022

[23]

Vigo M and Brajnik G Automatic web accessibility metrics: where we are and where we can go Interact. Comput. 2011 23 137-155

[24]

Vigo, M., Brown, J., Conway, V.: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, w4A 2013, pp. 1–10. ACM Press, New York (2013).

[25]

W3C: Web accessibility evaluation tools list. https://www.w3.org/WAI/ER/tools/

[26]

W3C: Web content accessibility guidelines (wcag) 2.0, December 2008. https://www.w3.org/TR/WCAG20/. Accessed 04 Mar 2022

[27]

W3C: Web content accessibility guidelines (wcag) 2.1, June 2018. https://www.w3.org/TR/WCAG21/. Accessed 04 2022

[28]

W3C Web Accessibility Initiative (WAI): Wcag 2 overview, July 2005. https://www.w3.org/WAI/standards-guidelines/wcag/. Accessed 04 Mar 2022

[29]

Watanabe, W.M., Fortes, R.P.M., Dias, A.L.: Using acceptance tests to validate accessibility requirements in ria. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2012, 16–17 April 2012. ACM Press, New York.

[30]

World Wide Web Consortium: Wai-aria, January 2016. https://www.w3.org/WAI/standards-guidelines/aria/. Accessed 24 Aug 2019

[31]

World Wide Web Consortium: Accessible rich internet applications (wai-aria) 1.1. https://www.w3.org/TR/wai-aria/. Accessed 05 Sept 2019

Cited By

Aljedaani WChimpiri UGaddam DShaik VKarasala YEler M(2024)Beyond Sight: Empowering Visually Impaired Users with Audible GraphsProceedings of the 21st International Web for All Conference10.1145/3677846.3677864(39-41)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677864
Aljedaani WRudhravaram SChintham AHabib AEler M(2024)A11yPDF: Bridging the Gap to Inclusive PDFsProceedings of the 21st International Web for All Conference10.1145/3677846.3677859(37-38)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677859
Alghamdi AAljedaani WEler MLudi S(2024)Accessibility Guidelines and Standards: Analyzing Stack Overflow PostsProceedings of the 21st International Web for All Conference10.1145/3677846.3677857(118-122)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677857

Index Terms

Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAs

Index terms have been assigned to the content through auto-classification.

Recommendations

The usability of accessibility evaluation tools
UAHCI'07: Proceedings of the 4th international conference on Universal access in human-computer interaction: applications and services

A range of accessibility evaluation tools are available to support Web developers in producing accessible Web resources. It is important that these tools support web developers very well, so that the developers can concentrate on the tasks of Web ...
Comparing accessibility evaluation tools: a method for tool effectiveness

This paper claims that effectiveness of automatic tools for evaluating web site accessibility has to be itself evaluated, given the increasingly important role that these tools play. The paper presents a comparison method for a pair of tools that takes ...
Interdependent components of web accessibility
W4A '05: Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A)

Increasingly, the Web is providing unprecedented access to information and interaction for people with disabilities. However, the Web will not be equally accessible, allowing people with disabilities to access and contribute to the Web, until:• ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging: 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings

Jun 2022

631 pages

ISBN:978-3-031-17901-3

DOI:10.1007/978-3-031-17902-0

Editors:
Vincent G. Duffy
Purdue University West Lafayette, West Lafayette, IN, USA
,
Qin Gao
Tsinghua University, Beijing, China
,
Jia Zhou
Chongqing University, Chongqing, China
,
Margherita Antona
Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece
,
Constantine Stephanidis
University of Crete and Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

© Springer Nature Switzerland AG 2022.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 26 June 2022

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aljedaani WChimpiri UGaddam DShaik VKarasala YEler M(2024)Beyond Sight: Empowering Visually Impaired Users with Audible GraphsProceedings of the 21st International Web for All Conference10.1145/3677846.3677864(39-41)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677864
Aljedaani WRudhravaram SChintham AHabib AEler M(2024)A11yPDF: Bridging the Gap to Inclusive PDFsProceedings of the 21st International Web for All Conference10.1145/3677846.3677859(37-38)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677859
Alghamdi AAljedaani WEler MLudi S(2024)Accessibility Guidelines and Standards: Analyzing Stack Overflow PostsProceedings of the 21st International Web for All Conference10.1145/3677846.3677857(118-122)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677857

View Options

View options

Figures

Tables

Media

View Table of Conten