article

Automatic web accessibility metrics: Where we are and where we can go

Authors:

Giorgio BrajnikAuthors Info & Claims

Interacting with Computers, Volume 23, Issue 2

Pages 137 - 155

https://doi.org/10.1016/j.intcom.2011.01.001

Published: 01 March 2011 Publication History

Abstract

The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.

References

[1]

Use of guidelines to automatically verify Web accessibility. Universal Access in the Information Society. v3 i1. 71-79.

Digital Library

[2]

Abou-Zahra, S., 2010. Conformance Evaluation of Web Sites for Accessibility. <http://www.w3.org/WAI/eval/conformance> (accessed June 2010).

[3]

Alonso, F., Fuertes, J.L., González, A.L., Martínez, L., 2010. On the testability of WCAG 2.0 for beginners. In: International Cross-Disciplinary Conference on Web Accessibility, W4A'10. ACM Press (Article 9).

Digital Library

[4]

Arrue, M., Vigo, M., Abascal, J., 2005. Quantitative metrics for web accessibility evaluation. ICWE 2005 Workshop on Web Metrics and Measurement.

[5]

Bailey, J., Burd, E., 2005. Tree-map visualisation for web accessibility. In: Computer Software and Applications Conference, COMPSAC'05. IEEE Press, pp. 275-280.

Digital Library

[6]

Bailey, J., Burd, E., 2007. Towards more mature web maintenance practices for accessibility. IEEE Workshop on Web Site Evolution, WSE'07. IEEE Press, pp. 81-87.

Digital Library

[7]

Comparing accessibility evaluation tools: a method for tool effectiveness. Universal Access in the Information Society. v3 i3-4. 252-263.

Digital Library

[8]

Brajnik, G., Lomuscio, R., 2007. SAMBA: a semi-automatic method for measuring barriers of accessibility. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'07. ACM Press, pp. 43-49.

Digital Library

[9]

Brajnik, G., Mulas, A., Pitton, C., 2007. Effects of sampling methods on web accessibility evaluations. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'07. ACM Press, pp. 59-66.

Digital Library

[10]

Brajnik, G., 2008. Beyond Conformance: the role of Accessibility Evaluation Methods. 2nd International Workshop on Web Usability and Accessibility, IWWUA'08. LNCS 5176. Springer, pp. 63-80.

Digital Library

[11]

Brajnik, G., 2009. Validity and reliability of web accessibility guidelines. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'09. ACM Press, pp. 131-138.

Digital Library

[12]

Brajnik, G., Yesilada, Y., Harper, S., 2010. Testability and validity of WCAG 2.0: the expertise effect. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'10. ACM Press, 43-50.

Digital Library

[13]

Brusilovsky, P., 2007. Adaptive Navigation Support. The Adaptive Web. LNCS 4321. Springer, pp. 263-290.

Digital Library

[14]

Bühler, C., Heck, H., Perlick, O., Nietzio, A., Ullveit-Moe, N., 2006. Interpreting results from large scale automatic evaluation of web accessibility. In: Computers Helping People with Special Needs, ICCHP'06. LNCS 4061. Springer, pp. 184-191.

Digital Library

[15]

Caldwell, B., Cooper, M., Guarino Reid, L., Vanderheiden, G., 2008. Web Content Accessibility Guidelines 2.0. W3C Accessibility Initiative. <http://www.w3.org/TR/WCAG20/> (accessed June 2010).

[16]

Casado-Martínez, C., Martínez-Normand, L., Goodwin-Olsen, M., 2009. Is it possible to predict the manual web accessibility result using the automatic result? Universal Access in HCI, Part III, HCII'09. LNCS 5616. Springer, pp. 645-653.

Digital Library

[17]

Cassidy, B., Cockton, G., Bloor, C., Coventry, L., 2005. Capability, Acceptability and Aspiration for: collecting accessibility data with prototypes. In: Proceedings of BCS Conference on HCI'05, vol. 2, pp. 138-143.

[18]

Chisholm, W., Vanderheiden, G., Jacobs, I., 1999. Web Content Accessibility Guidelines 1.0. W3C Web Accessibility Initiative. <http://www.w3.org/TR/WAI-WEBCONTENT/> (accessed June 2010).

[19]

Craig, J., Cooper, M., 2009. Accessible Rich Internet Applications (WAI-ARIA) 1.0. W3C Working Draft. <http://www.w3.org/TR/wai-aria/> (accessed June 2010).

[20]

Dujmovic, J.J., 1996. A method for evaluation and selection of complex hardware and software systems. In: International Computer Measurement Group Conference, Computer Measurement Group, pp. 368-378.

[21]

Fukuda, K., Saito, S., Takagi, H., Asakawa, C., 2005. Proposing new metrics to evaluate Web usability for the blind. In: Extended Abstracts of Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'05. ACM Press, pp. 1387-1390.

Digital Library

[22]

Goble, C., Harper, S., Stevens, R., 2000. The travails of visually impaired web travellers. In: ACM Conference on Hypertext and Hypermedia, Hypertext'00. ACM Press, pp. 1-10.

Digital Library

[23]

González, J., Macías, M., Rodríguez, R., Sánchez, F., 2003. Accessibility Metrics of Web Pages for Blind End-Users. Web Engineering. LNCS 2722. Springer, pp. 374-383.

Digital Library

[24]

Hackett, S., Parmanto, B., Zeng, X., 2004. Accessibility of internet web sites through time. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'04. ACM Press, pp. 32-39.

Digital Library

[25]

Augmenting the mobility of profoundly blind Web travellers. New Review of Hypermedia and Multimedia. v11 i1. 103-128.

Digital Library

[26]

A study of the evaluator effect in usability testing. Human-Computer Interaction. v23 i3. 251-277.

[27]

ISO/IEC 9126-1, 2001. Software Engineering - Software Product Quality - Part 1: Quality Model. International Standardization Organization.

[28]

Ivory, M., Yu, S., Gronemyer, K., 2004. Search result exploration: a preliminary study of blind and sighted users' decision making and performance. In: Extended Abstracts of Conference on Human Factors in Computing Systems, CHI'04. ACM Press, pp. 1453-1456.

Digital Library

[29]

Kobsa, A., 1999. Adapting Web Information for the Disabled and the Elderly. WebNet'99, pp. 32-37.

[30]

Beyond web content accessibility guidelines: design of enhanced text user interfaces for blind internet users. International Journal of Human-Computer Studies. v66 i4. 257-270.

Digital Library

[31]

Lopes, R., Carriço, L., 2008. The impact of accessibility assessment in macro scale universal usability studies of the web. In: International Cross-Disciplinary Conference on Web accessibility, W4A'08. ACM Press, pp. 5-14.

Digital Library

[32]

Mankoff, J., Fait, H., Tran, T., 2005. Is Your Web Page Accessible? A comparative study of methods for assessing web page accessibility for the blind. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'05. ACM Press, pp. 41-50.

Digital Library

[33]

Evaluating and designing web site quality. IEEE Multimedia. v10 i1. 34-43.

Digital Library

[34]

Mirri, S., Muratori, L.A., Roccetti, M., Salomoni, P., 2009. Metrics for accessibility on the Vamola project. In: International Cross-Disciplinary Conference on Web accessibility, W4A'09. ACM Press, pp. 142-145.

Digital Library

[35]

Homepage Usability: 50 Websites Deconstructed. New Riders Publishing.

Digital Library

[36]

. Handbook of Human Perception and Performance, 1986.Wiley.

[37]

Measuring web application quality with WebQEM. IEEE Multimedia. v9 i4. 20-29.

Digital Library

[38]

Olsina, L., Sassano, R., Mich, L., 2008. Specifying quality requirements for the web 2.0 applications. International Workshop on Web-Oriented Software Technologies, IWWOST'08, pp. 50-56.

[39]

Metric for web accessibility evaluation. Journal of the American Society for Information Science and Technology. v56 i13. 1394-1404.

Digital Library

[40]

Petrie, H., Kheir, O., 2007. Relationship between accessibility and usability of web sites. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'07. ACM Press, pp. 397-406.

Digital Library

[41]

Sirithumgul, P., Suchato, A., Punyabukkana, P., 2009. Quantitative evaluation for web accessibility with respect to disabled groups. In: International Cross-Disciplinary Conference on Web accessibility, W4A'09. ACM Press, pp. 136-141.

Digital Library

[42]

Shelly, C., Barta, M., 2010. Application of traditional software testing methodologies to web accessibility. In: International Cross-Disciplinary Conference on Web accessibility, W4A'10. ACM Press (Article 11).

Digital Library

[43]

Maximum Accessibility: Making Your Web Site More Usable for Everyone. Addison-Wesley.

Digital Library

[44]

Sullivan, T., Matson, R., 2000. Barriers to use: usability and content accessibility on the web's most popular sites. In: ACM Conference on Universal Usability, CUU'00. ACM Press, pp. 139-144.

Digital Library

[45]

Adaptive techniques for universal access. User modeling and user-adapted interaction. The Journal of Personalization Research. v11 i1-2. 159-179.

Digital Library

[46]

Takagi, H., Asakawa, C., Fukuda, K., Maeda, J., 2004. Accessibility designer: visualizing usability for the blind. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'04. ACM Press, pp. 177-184.

Digital Library

[47]

Takagi, H., Kawanaka, S., Kobayashi, M., Sato, D., Asakawa, C., 2009. Collaborative web accessibility improvement: challenges and possibilities. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'09. ACM Press, pp. 195-202.

Digital Library

[48]

Trochim, W.M.J., 2006. The Research Methods Knowledge Base. <http://www.socialresearchmethods.net/kb/> (accessed June 2010).

[49]

Velleman, E., Meerbeld, C., Strobbe, C., Koch, J., Velasco, C.A., Snaprud, M., Nietzio, A., 2007. D-WAB4, Unified Web Evaluation Methodology (UWEM 1.2 Core). <http://www.wabcluster.org/uwem1_2/> (accessed June 2010).

[50]

Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J., 2007. Quantitative metrics for measuring web accessibility. In: International Cross-Disciplinary Conference on Web accessibility, W4A'07. ACM Press, pp. 99-107.

Digital Library

[51]

Vigo, M., Kobsa, A., Arrue, M., Abascal, J., 2007b. User-Tailored web accessibility evaluations. In: ACM Conference on Hypertext and Hypermedia, Hypertext'07. ACM Press, pp. 95-104.

Digital Library

[52]

Vigo, M., Leporini, B., Paternò, F., 2009b. Enriching web information scent for blind users. In: ACM SIGACCESS Conference on Computers and Accessibility, ASSETS'09. ACM press, pp. 123-130.

Digital Library

[53]

Tool independence for the web accessibility quantitative metric. Disability and Rehabilitation: Assistive Technology. v4 i4. 248-263.

[54]

Watanabe, T., 2007. Experimental evaluation of usability and accessibility of heading elements. In: International Cross-Disciplinary Conference on Web accessibility, W4A'07. ACM Press, pp. 157-164.

Digital Library

[55]

Woolrych, A., Cockton, G., Hindmarch, M., 2004. Falsification testing for usability inspection method assessment. In: Dearden, A., Watts, L., (Eds.) Proceedings of HCI 2004. Research Press International, vol. 2, pp. 137-140.

Cited By

Alhadreti O(2024)Investigating the accessibility of banking websites in Saudi ArabiaUniversal Access in the Information Society10.1007/s10209-023-01008-823:4(1795-1810)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-023-01008-8
Alhadreti O(2024)Accessibility, performance and engagement evaluation of Saudi higher education websites: a comparative study of state and private institutionsUniversal Access in the Information Society10.1007/s10209-023-00971-623:4(1671-1688)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-023-00971-6
Martins BDuarte C(2024)Large-scale study of web accessibility metricsUniversal Access in the Information Society10.1007/s10209-022-00956-x23:1(411-434)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10209-022-00956-x
Show More Cited By

Recommendations

Quantitative metrics for measuring web accessibility
W4A '07: Proceedings of the 2007 international cross-disciplinary conference on Web accessibility (W4A)

This paper raises the need for quantitative accessibility measurement and proposes three different application scenarios where quantitative accessibility metrics are useful: Quality Assurance within Web Engineering, Information Retrieval and ...
Large-scale study of web accessibility metrics
Abstract
Evaluating the accessibility of web resources is usually done by checking the conformance of the resource against a standard or set of guidelines (e.g., the WCAG 2.1). The result of the evaluation will indicate what guidelines are respected (or ...
Analysis and integration of web accessibility metrics
W4A '10: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A)

"Analysis and Integration of Web Accessibility Metrics" was presented as a grade thesis. It included the development of "OceanAcc", an evaluation and analysis tool that performs a semi-automatic evaluation process. With little user intervention, can ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Interacting with Computers

Interacting with Computers Volume 23, Issue 2

March, 2011

82 pages

ISSN:0953-5438

EISSN:1873-7951

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2011.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 March 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alhadreti O(2024)Investigating the accessibility of banking websites in Saudi ArabiaUniversal Access in the Information Society10.1007/s10209-023-01008-823:4(1795-1810)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-023-01008-8
Alhadreti O(2024)Accessibility, performance and engagement evaluation of Saudi higher education websites: a comparative study of state and private institutionsUniversal Access in the Information Society10.1007/s10209-023-00971-623:4(1671-1688)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1007/s10209-023-00971-6
Martins BDuarte C(2024)Large-scale study of web accessibility metricsUniversal Access in the Information Society10.1007/s10209-022-00956-x23:1(411-434)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10209-022-00956-x
Inal YMishra DTorkildsby A(2022)An Analysis of Web Content Accessibility of Municipality Websites for People with Disabilities in NorwayNordic Human-Computer Interaction Conference10.1145/3546155.3547272(1-12)Online publication date: 8-Oct-2022
https://dl.acm.org/doi/10.1145/3546155.3547272
Joyner SRiegelhuth AGarrity KKim YKim N(2022)Visualization Accessibility in the Wild: Challenges Faced by Visualization DesignersProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517630(1-19)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517630
Okafor OAljedaani WLudi S(2022)Comparative Analysis of Accessibility Testing Tools and Their Limitations in RIAsHCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging10.1007/978-3-031-17902-0_34(479-500)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-17902-0_34
Lazar JTurner MVelarde LWesterwiller J(2022)Evaluating the Potential Implementation of New Metrics in Automated Web Accessibility Testing Software for ManagersUniversal Access in Human-Computer Interaction. Novel Design Approaches and Technologies10.1007/978-3-031-05028-2_2(22-39)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-05028-2_2
Aydin AKo YUckun URamakrishnan IAshok VDemartini GZuccon GCulpepper JHuang ZTong H(2021)Non-Visual Accessibility Assessment of VideosProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482457(58-67)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482457
Khan AKhusro S(2021)An insight into smartphone-based assistive solutions for visually impaired and blind people: issues, challenges and opportunitiesUniversal Access in the Information Society10.1007/s10209-020-00733-820:2(265-298)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s10209-020-00733-8
AlMeraj ZBoujarwah FAlhuwail DQadri R(2020)Evaluating the accessibility of higher education institution websites in the State of Kuwait: empirical evidenceUniversal Access in the Information Society10.1007/s10209-020-00717-820:1(121-138)Online publication date: 15-Apr-2020
https://dl.acm.org/doi/10.1007/s10209-020-00717-8
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents