Keywords

1 Introduction

Accessibility represents nowadays an essential aspect to be considered in the development of Web applications [1]. The companies are adopting technologies as one of the main means of dissemination and visibility of the information, which should be accessible to all people, easy to use, accurate and safe [2]. This assertion can also be applied to the web domain, where the technological advance has allowed companies to provide online services through online applications [3]. This digital transformation has led to the identification of four categories of change [4]: (1) the expectations of the customer’s experience, (2) the innovation in products and services, (3) the disappearance of barriers between industries and regulation, and (4) the accompaniment by society to the new technologies. Therefore, one of the main concerns of the big enterprises is the development of tools that allow customers to get their goals with satisfaction.

One of the global goals of companies is to provide electronic services that allow all people to access them, regardless of their auditory, movement, visual and cognitive abilities [5]. To achieve this, several factors must be considered in the design of interfaces. The impact of not taking into account web accessibility standards in software products leads customers with visual disabilities to stop using the services due to their inaccessibility [6]. Therefore, it is essential to perform a continuous accessibility assessment of the components of a system to verify if all the interfaces can be accessed independently of the disability of a specific user [7]. The way to determine if a software product meets the necessary compliance of the existing guidelines that would allow it to be accessible, is through evaluation methods [8]. The accessibility is as essential, that companies can win a legal problem if customers can not use the services provided through the interfaces that present accessibility problems [6]. Finally, within the social values adopted by companies, equality, inclusion and social commitment are considered, and in this sense, their web applications must make reflect the values of the organization and help to improve its image further [9]. Therefore, accessibility can be considered as an innovation for the relationship between the organization and all its stakeholders, which becomes a substantial competitive advantage for companies [6].

Given the importance of accessibility in software products, in this paper we describe the process of a systematic literature review that was performed with the purpose of identifying the main accessibility assessment methods that are currently used by the scientific community. The paper is structured as follows. In Sect. 2 we present the essential concepts and definitions that were used to develop the research work. In Sect. 3, we describe the protocol that guided the entire process of systematic review. Likewise, the findings and results are discussed. Finally, in Sect. 4, the conclusions and future works are established.

2 Background

2.1 Web Accessibility

Web accessibility can be defined as a universal access to the Web, which does not depend on the hardware or software used, nor the language, culture or physical or mental abilities of users [10].

The goal of web accessibility is to ensure that the information or services delivered through web sites are available and can be used by the widest possible audience [11].

2.2 WCAG Web Accessibility Guidelines

The WCAG guidelines (Web Content Accessibility Guidelines) are aimed at those who design or develop content for the Web. They consist of specific recommendations, written in a generic way. The objective is to make the content presented in an accessible way. The Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C) elaborated annexed guides that exploit each point and detail the steps to follow to implement and comply with them [12].

2.3 WCAG 2.0 Guide

These guidelines provide guidance and examples to meet the guidelines using specific technologies. For this, they have adequacy levels A, AA, AAA that will be applied differently to each one of their criteria. The WCAG 2.0 guidelines are composed of 4 principles, 12 guidelines and 61 compliance criteria, each of which will have different levels of compliance (A, AA, AAA), in addition to a set of sufficient techniques and advisory techniques [11]. The principles are the following:

  1. 1.

    Principle 1: Perceptible: The information and the components of the user interface must be presented to the users so that they can perceive them. It is composed of 4 guidelines and 22 compliance criteria.

  2. 2.

    Principle 2: Operable: The components of the user interface and navigation must be operable. It consists of 4 guidelines and 20 compliance criteria.

  3. 3.

    Principle 3: Understandable: Information and user interface management must be understandable. It consists of 3 guidelines and 17 compliance criteria.

  4. 4.

    Principle 4: Robust: The content must be robust enough to rely on its interpretation by a wide variety of user agents, including assistive technologies. It consists of 1 guideline and 2 compliance criteria.

3 Systematic Mapping Review

A systematic literature review is a method to analyze, evaluate and interpret all relevant studies to a specific research question. In spite of the systematic literature review is frequently used in Medicine, there are proposals to use this methodology in the field of Software Engineering. Kitchenham and Charters [13] establish a set of steps to achieve relevant and rigorous systematic studies for software engineering topics. The steps of this methodology are presented in the subsequent sections.

3.1 Research Questions

The purpose of this systematic review is to explain the current methods and techniques that are used to evaluate web accessibility. Besides, we identified the domains where this evaluation where taken and which disabilities are considered in the evaluation. The main objective of this research is to review the literature and situate it into the current scenario. In this way, we formulated the following research questions:

  • RQ1: Which methods and techniques are reported in the literature for the evaluation of accessibility in web applications?

  • RQ2: Which domains are the most evaluated?

  • RQ3: Which disabilities are considered for the evaluation of web accessibility?

After we have established the research questions, we defined the general concepts based on PICOC. Since our research is not intended to compare interventions, the “comparison” criterion was not considered. Table 1 contains the concepts.

Table 1. Definition of the general concepts using PICOC
Full size table

3.2 Search String

The search strategy was based on the general concepts. To obtain more relevant studies some synonymous were selected. We only considered relevant studies, whose publication date was since 2015, in order to analyze the current state of art.

  • C1: (“web application*” OR “website*” OR “web site*” OR “web page*”)

  • C2: (“web accessibility”)

  • C3: (“method*” OR “technique*”)

  • C4: (“evaluation” OR “verification” OR “validation”)

The resulting string was:

(“web application*” OR “website*” OR “web page*”) AND (“web accessibility”) AND (“method*” OR “technique*”) AND (“evaluation” OR “verification” OR “validation”).

3.3 Search Process

To perform the search process we use four recognized databases: IEEExplore, ACM Digital Library, SCOPUS and SpringerLink. Grey literature was excluded since it is not peer reviewed.

3.4 Selection of Primary Studies

Each study that was retrieved from the automated search in the databases, was examined by the authors in order to determine its inclusion in this study. The process of evaluation involved a review of the entire document: title, abstract, introduction, background, state of the art, methodology, study case, results and conclusions. Furthermore, we established some inclusion criteria to determine the inclusion of the study.

  • Studies that present a web accessibility evaluation a specific domain explaining the method used.

  • Studies that present a comparison between two or more web accessibility evaluation methods.

  • Studies that present the assessment of the accessibility of a domain focused on a disability.

On the other hand, we established the exclusion criteria:

  • Studies that present evaluation of other factors such as usability, user experience, etc.

  • Studies where assistive technology for the disabled is presented.

  • Articles where algorithms for the development of web pages for the disabled are presented.

3.5 Data Extraction

The information we extracted form the selected studies include the following:

  1. (a)

    Paper ID

  2. (b)

    Paper Title

  3. (c)

    Author(s)

  4. (d)

    Year of publication

  5. (e)

    Database in which the study was found.

The automated search for our systematic mapping was performed on December 20th, 2018. We obtained 343 studies from the four consulted databases. After the application of the inclusion and exclusion criteria, 21 of these papers were selected for the analysis. Table 2 shows the results of the search process.

Table 2. Summary of search results
Full size table

3.6 Data Analysis and Results

The list of the selected articles are presented in Table 3.

Table 3. Selected studies
Full size table

The purpose of the first question was to find the methods or techniques used for the evaluation of web accessibility. Table 4 shows the methods found and the frequency with which they appear in the articles selected for this research. Then we proceed to explain each method more in depth.

Table 4. Methods for evaluating web accessibility
Full size table

Automatic Tools: When using automated tools, a methodology followed to evaluate the domains covered by the studies S1, S2, S4 and S6. Figure 1 shows the steps.

Fig. 1.
figure 1

The seven phases to evaluated web accessibility using automatic tools

Full size image
  1. (a)

    Define the evaluation scope: In this phase it is necessary to identify the pages involved in the evaluation. Besides, we must establish the adequacy level (A, AA, AAA), which depends on the domain the study will be evaluated.

  2. (b)

    Explore the target website: The purpose of this phase is to determine the principal features that are essential for the domain. This include style, designs, structures, functions and processes. In this phase, the evaluator needs to check for broken links.

  3. (c)

    Select a sample: In this phase, the main page of the website is identified and the future pages that follow the navigability of the website.

  4. (d)

    Audit sample selected: Each page is tested by the automatic tool in order to meet the adequacy level established in phase 1. The evaluator also must verify if the data is introduced correctly and the notifications and interactions are verified and evaluated.

  5. (e)

    Report the findings: The results are documented. Generally, most of the automatic tools present reports explained the problems they found and the broken guidelines. In some cases, they present a solution to modify.

  6. (f)

    Analyze the results: The problems that were identified in the previous phase are discussed in order to decide if these problems will be corrected.

  7. (g)

    Correct errors: Developers and designer can follow the suggestions of the automatic tools or they can adjust their pages as they consider.

Expert Evaluation: Although the evaluation of experts has not been very approached in the literature, articles S3, S6 refer to web accessibility experts must make an evaluation on the web pages contrasting what is indicated in the guidelines such as WCAG 2.0 and if these are complying depending on the domain. When a methodology is not specified, as in the case of automated tools, the experts consider it important that before specifying the evaluation, the characteristics of the domain to be worked are specified. In the studies found, experts evaluate the page through interaction. When they find an error, this is recorded, as these same classify the severity of this.

User Testing: Manual tests are an accessibility evaluation method. The main advantage lies in the fact that specific accessibility problems and violations can be found directly, because the equation can be designed according to the purpose sought. Within the criteria to consider, you can use international standards or you can establish your own criteria. There is a lot of reference to the team of evaluators that will design the tests. Regarding the participants that are used for the tests, S3 indicates that they must be people who interact frequently with the domain to be evaluated. On the other hand, the specific tasks in user tests are focused on complying with the processes that are embodied in the web page.

Regarding the domains that have been addressed, it can be seen that the education sector has been the most studied, followed by the government sector. Some studies do focus on the importance of the domain and the reason for the study, however, in other studies the study focuses more on the methodology that selects a test domain. Table 5 specifies the domains that have been addressed.

Table 5. Domains evaluated
Full size table

Finally, the disabilities that have been addressed in the literature are presented. Most studies specify what disability is being addressed. This is because the selected method and the work structure used do not emphasize a specific disability. In the case of automated tools, they do not detect accessibility problems according to the type of disability. The S7 study does emphasize having participants who suffer from different disabilities, which does analyze the interaction between the person and the web page in order to detect problems or difficulties in fulfilling the tasks. Table 6 shows the disabilities addressed in the literature.

Table 6. Disabilities worked in the studies
Full size table

4 Conclusions and Future Works

Some evaluation methods have been proposed to determine the level of web accessibility in web applications. Although automated tools are the most used in studies, they do not always lead to the identification of all web accessibility problems that exist. Expert testing can be the most effective method to ensure compliance with some standards, while user testing seems to work efficiently to verify how people with disabilities could perform certain tasks.

Following a predefined protocol, we identified 343 studies, of which 20 were selected. This work allowed us to determine that: (1) automatic tools, (2) expert evaluation and (3) user tests are the most used techniques according to the literature. In addition, in this study, we have determined the domains in which web accessibility assessments have been carried out, the most frequent being (1) education, (2) government, (3) health, (4) entertainment. Finally, the disabilities in which they have worked were determined, being the most frequent result that the majority of studies do not focus the evaluation on any disability.

As future work, comparisons can be made between accessibility assessment methods to complement each other. On the other hand, you can contrast the accessibility guidelines that several methods follow to contrast them with the needs that each disability needs when interacting with web applications.