Open AccessArticle

Developing Usability Guidelines for mHealth Applications (UGmHA)

Eman Nasr

^*,

Wafaa Alsaggaf

and

Doaa Sinnari

Information Technology Department, Faculty of Computing and Information Technology, King AbdulAziz University, Jeddah 21589, Saudi Arabia

Author to whom correspondence should be addressed.

Multimodal Technol. Interact. 2023, 7(3), 26; https://doi.org/10.3390/mti7030026

Submission received: 31 January 2023 / Revised: 20 February 2023 / Accepted: 24 February 2023 / Published: 28 February 2023

(This article belongs to the Special Issue Feature Papers in Multimodal Technologies and Interaction—Edition 2023)

Download

Browse Figures

Versions Notes

Abstract

Mobile health (mHealth) is a branch of electronic health (eHealth) technology that provides healthcare services using smartphones and wearable devices. However, most mHealth applications were developed without applying mHealth specialized usability guidelines. Although many researchers have used various guidelines to design and evaluate mHealth applications, these guidelines have certain limitations. First, some of them are general guidelines. Second, others are specified for mHealth applications; however, they only cover a few features of mHealth applications. Third, some of them did not consider accessibility needs for the elderly and people with special needs. Therefore, this paper proposes a new set of usability guidelines for mHealth applications (UGmHA) based on Quinones et al.’s formal methodology, which consists of seven stages starting from the Exploratory stage and ending with the Refining stage. What distinguishes these proposed guidelines is that they are easy to follow, consider the feature of accessibility for the elderly and people with special needs and cover different features of mHealth applications. In order to validate UGmHA, an experiment was conducted on two applications in Saudi Arabia using UGmHA versus other well-known usability guidelines to discover usability issues. The experimental results show that the UGmHA discovered more usability issues than did the other guidelines.

Keywords:

usability; usability guidelines; mHealth applications; usability guidelines for mHealth applications; UGmHA

1. Introduction

In the recent era, the exponential evolution of technology is accelerating world development by making our life easier, more flexible and more effective. This development has been involved in the healthcare field in presenting electronic health (eHealth) technology. eHealth can enhance healthcare services using the internet and wireless technologies, usually through traditional desktops [1]. All over the world, countries are adopting eHealth services to improve the safety, standards and quality of their healthcare systems.

In relation to eHealth, a specific and essential part must be considered, called mobile health (mHealth) technology [2]. The concept of mHealth was defined and coined by Robert Istepanian in 2004 [3]. mHealth moves healthcare services from traditional desktops into mobile technologies, such as smartphones and wearable devices [4,5]. The key features of mHealth applications are the ability to monitor chronic disease by patients and facilitate communication between patients and doctors [6].

Despite the valuable benefits of mHealth applications, they can harm a patient’s life and health if they do not work as expected, such as erroneous results or wrong actions, whether by patients or clinical staff [7]. To the best of our knowledge, most mHealth applications are developed without using standard and specialized mHealth usability guidelines [8]. A set of usability guidelines is an evaluation method used to determine the extent to which a mHealth application’s interface is safe, efficient and usable by different kinds of people [9,10]. Accordingly, mHealth applications need to be regulated by applicable usability guidelines in order to assess their efficiency and safety for patients.

Many kinds of studies used various existing guidelines to design and evaluate mHealth applications; however, there are three main problems with these guidelines, which are:

Some of the guidelines are general, which means they are not specified for mHealth applications, such as Nielsen’s principles [11]. Since mHealth applications are intended to affect the health and the body of people, they need to have more rigorous and restricted guidelines.
Some of the guidelines are specified for mHealth applications; however, they cover only a few features of mHealth applications, such as:
(a)
Xcertia [12], which covered access to personal health information and notification features, while other features, such as self-monitoring, were not covered.
(b)
HE4EH [6], which covered self-monitoring, health goals and tips and biometric measurement features, while other features, such as consultations, were not covered.
(c)
Telemedicine recommendations [13], which covered consultations and booking appointment features, while other features, such as self-monitoring, were not covered.
Some of the guidelines did not consider accessibility features for the elderly and people with special needs, such as [11,13]. Accessibility is important to ensure that the application is accessible to people of different ages and needs.

In this study, our goal is to develop a comprehensive set of usability guidelines for mHealth applications, titled (UGmHA), that satisfy the following criteria:

Simple to follow and understandable by designers, developers and evaluators.
Accessible to all people, including elderly people and people with special needs.
Specialized in designing and evaluating mHealth applications that can cover different features. The following are examples of features gathered from different studies:
- Self-monitoring for chronic diseases [14,15].
- Online consultations [14,16].
- Sharing data with health care providers (HCP) [14,15].
- Booking appointments [16].
- Biometric measurements [14].
- Health goals and tips [14,15].
- Notifications and reminders [14].
- Access to personal health information [15].

This study has two main objectives. The first objective is to develop UGmHA using the systematic and formal methodology of Quinones et al. [17]. The second objective is to validate UGmHA by comparing UGmHA against the Nielsen and Xcertia guidelines, which are well-known and global guidelines. The experiment was conducted in Saudi Arabia by two authors of the paper using two local applications, which are named “Sehhaty” and “Sokry”.

The rest of the paper is organized as follows. Section 2 presents a background and literature review of mHealth-related guidelines as well as common development approaches and validation methods for new usability guidelines. Section 3 illustrates how UGmHA was developed and validated. Section 4 shows the results of the development and validation of UGmHA. Section 5 discusses the results obtained from the development and validation of UGmHA. Finally, our conclusions and future work are provided in Section 6.

2. Literature Review

This section presents the background information related to eHealth, mHealth and usability. We also review the existing guidelines related to mHealth applications, common development approaches and validation methods for new usability guidelines.

2.1. Electronic Health (eHealth) and Mobile Health (mHealth)

The term eHealth was established by Mitchell in 1999 [18] and is a broad term that integrates healthcare and technology to improve healthcare efficiency and reduce costs. The main objectives of eHealth are to improve patient safety and provide accurate diagnostics and appropriate treatment [19]. eHealth offers many different services and functionalities, such as electronic health records (EHR), archiving, electronic prescribing, order-entry systems and computerized decision support systems (CDS) [20]. Therefore, countries worldwide have adopted eHealth services to improve the standards and quality of their healthcare systems.

mHealth is a specific branch of electronic health (eHealth) technology. The “mHealth” terminology was coined and established by Robert Istepanian in 2004 [3]. mHealth can be defined as collecting and processing health data using smartphones, tablets and wearable devices [5]. mHealth applications are crucial since they enable a patient to monitor their own chronic disease and activities, support behavior changes for patients, improve their lifestyle and facilitate communication between patients and doctors [6,21]. Moreover, they provide doctors with greater mobility to help them care for their patients from different areas [5].

2.2. Usability and Guidelines

Usability is a key feature of a successful system because it is essential to create a well-designed and highly usable system [22,23]. According to ISO 9241-11 standard, usability is defined as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [24].

Usability can be implemented and evaluated effectively using usability guidelines. The usability guidelines have the advantage that they can be applied early in the design and used to evaluate and discover usability issues before application release [25]. Nielsen’s principles are the first and best-known usability guidelines, which were introduced by Nielsen and Molich in 1990 [26]. These guidelines are considered traditional and general for most user interfaces [27,28].

However, there is evidence in the literature confirming that general guidelines are not appropriate to design and evaluate domain-specific applications and may overlook essential elements that need to be considered in specific applications [27,28,29,30]. Moreover, Hermawati et al. [28] indicated that any set of usability guidelines should depend on the specific requirements of the user, task and environment of use. This means that any change in these requirements may introduce new features and issues of usability that are not considered by general guidelines. Therefore, it is crucial to develop mHelath-specific guidelines to ensure that usability features related to mHealth applications are identified.

Usability guidelines may include checklist items that work as a guide to evaluate domain-specific applications [17]. The checklist items can add more details and specifications for usability guidelines to make them easily tailored to the specific features of an application [31]. Without using checklist items, there is the potential for the wrong association of usability issues to the corresponding guideline, missed domain-specific usability issues and unreliable findings due to the lack of evaluator consensus [32].

Several guidelines have been used by studies for the design and evaluation of mHealth applications.

Nielsen’s principles [11] comprises ten usability guidelines, which are:

Visibility of system status.
Match between system and the real world.
User control and freedom.
Consistency and standards.
Error prevention.
Recognition rather than recall.
Flexibility and efficiency of use.
Aesthetic and minimalist design.
Help users recognize, diagnose and recover from errors.
Help and documentation.

Despite Nielsen’s principles being globally well-known for designing and evaluating most user interfaces, they are general and not specialized for distinct features of mHealth applications [27,28]. Furthermore, accessibility features for the elderly and people with special needs are not considered. However, several studies used Nielsen’s principles as a basis for generating new usability guidelines by modifying or adding new guidelines or checklist items of domain-specific applications [27].

Roy et al. [33] discussed the framework of the Xcertia guidelines, which were founded by leaders of the healthcare industry, such as the Association of American Medical Colleges, U.S. Food and Drug Administration (FDA) and Healthcare Information and Management Systems Society (HIMSS). The Xcertia guidelines aim to fulfill two primary purposes, which are assessing how an mHealth application is designed to be safe and optimizing mHealth applications to be used by specified users within a specified environment [33].

As shown in Figure 1, Xcertia includes five key workgroups: privacy, security, content, usability and operability. Each workgroup includes a set of guidelines, and each guideline can be measured by a set of performance requirements or checklists. Nonetheless, this research will focus only on the “usability” workgroup because it is within our scope. The “usability” workgroup includes ten guidelines with checklist items for each guideline. Below are the guidelines of the “usability” workgroup of Xcertia:

Visual design.
Readability.
App navigation.
Onboarding.
App feedback.
Notifications, alerts and alarms.
Help resources and troubleshooting.
Historical data.
Accessibility.
Ongoing app evaluation.

Xcertia guidelines are developed mainly for mHealth applications and the “usability” in Xcertia considers some of the accessibility features for people with special needs. However, despite Xcertia guidelines being developed mainly for mHealth applications, most of the usability guidelines are still general and do not evaluate specific features of mHealth applications, such as self-monitoring, consultations, health advice and booking appointments.

Khowaja et al. [6] presented a modified set of specialized guidelines for mHealth applications called (HE4EH). HE4EH includes 25 guidelines with checklist items. The guidelines are:

Visibility of system status.
User control and freedom.
Match between system and real world.
Consistency and standards.
Error prevention.
Help users recognize, diagnose and recover from errors.
Recognition rather than recall.
Flexibility and efficiency of use.
Aesthetic and minimalist design.
Help and documentation.
Privacy.
Skills.
Pleasurable interaction.
Accessibility.
Compatibility between different platforms.
Minimized human–device interaction.
Physical interaction and ergonomics.
Readability and layout.
Non-interrupting app information visualization.
Content.
Display.
Navigation.
Interactivity.
Behavior change.
Self-monitoring.

However, the study only focused on self-monitoring of diabetes and blood glucose levels without considering hypertension, obesity, etc. [6]. Moreover, since the scope of the study was about self-monitoring and behavioral-change features, the other features related to mHealth applications, such as consultations and booking appointments, were not covered. Furthermore, the study included more than 16 guidelines, while it is recommended to keep the number of usability guidelines between 10 and 16 [17].

Aldekhyyel et al. [13] evaluated the usability of mHealth applications with telemedicine features deployed during the COVID-19 pandemic in Saudi Arabia. The mHealth applications were “Sehha”, “Cura” and “Dr. Sulaiman Alhabib”, which were evaluated using Nielsen’s ten principles with a five-point severity rating scale (SRS). Then, the study provided usability design recommendations for each application based on the discovered usability issues. However, since the scope of the study was about telemedicine applications, it focused only on consultations, appointments, etc., without considering other mHealth-specific features, such as self-monitoring and health advice. Moreover, accessibility features for the elderly and people with special needs were not covered.

Al-Razgan et al. [34] converted and adapted some of the mobile phone usability guidelines and recommendations to develop modified usability guidelines for mobile launchers designed for elderly people. The study included 13 guidelines, which are classified into three sections as shown below:

Look and Feel
- Make elements on the page easy to read.
- Easy recognition and accessibility.
- Make clickable items easy to target and hit.
- Use language and culture appropriate for the elderly; minimize technical terms.
Interaction
- Provide clear feedback on actions.
- Provide preferable gestures for the elderly.
- Provide the elderly with information on launcher/elderly status.
- Use conventional interaction items.
- Ergonomic design.
Functionality
- Provide functions that reduce elderly memory load.
- Elderly does not feel lost or stuck (elderly control and freedom).
- Prevent errors from occurring.
- Provide necessary information and settings.

However, despite the study being mainly about mobile launchers’ usability for elderly people, we can make use of the guidelines to add the accessibility features for elderly people.

Arachchi et al. [35] integrated different learning theories and usability guidelines to develop an eLearning module for people with intellectual disability needs. The guidelines considered the specific abilities and context of learners, such as attention, mobility, cognitive capacity, learning ability and ability to read and write. The study incorporated nine guidelines with checklist items. The guidelines are:

Minimised distractions.
Easy to operate.
Readability and visualization.
Consistency.
Feedback.
Navigability.
Personalizing.
Easy to understand and relevant.
Learner-friendly.

However, despite the study being mainly about eLearning usability for intellectual disabilities, which is a part of special needs, we can make use of its guidelines to add accessibility features to support people with special needs.

2.3. Common Approaches for Developing New Usability Guidelines

Some common approaches are used to develop domain-specific usability guidelines, such as methodologies, literature review, usability problems, existing guidelines, interviews, theories and mixing processes [27]. However, the methodologies approach attracts the interest of most studies since it is the only formal approach that follows a systematic and clearly defined process. On the other hand, all other approaches are considered informal since their activities and processes are performed in a non-systematic way [27].

Different methodologies can be followed to generate a new set of usability guidelines, such as the Rusu methodology [10], which consists of six stages. This methodology is the most popular one [27,28] and is used by several studies, such as [36,37,38,39]. These studies emphasized the importance of the methodology to facilitate the generation of domain-specific guidelines. Nevertheless, Rusu et al.’s methodology [10] has been modified and adapted by the same authors with the support of usability experts and researchers to generate a new and examined methodology, here called Quinones et al.’s methodology [17]. Quinones et al.’s methodology consists of seven main stages, which are [17]:

Exploratory stage: to collect existing studies related to the main topics of the research, such as general or related usability guidelines, principles, application features and attributes.
Descriptive stage: to highlight the most important information of the previously collected studies to formalize the main concept of the research.
Correlational stage: to determine the characteristics that the usability guidelines of the specific domain should have based on traditional guidelines.
Selection stage: to keep, adapt or discard the existing sets of usability guidelines that were selected in the descriptive stage.
Specification stage: to formally specify the set of proposed guidelines using a standard template.
Validation (experimental) stage: to test the proposed guidelines against traditional ones through experiments.
Refinement stage: to refine the proposed guidelines based on the validation-stage results.

The advantage of employing Quinones et al.’s methodology is that it provides a standard template, which is used to explain more details about an individual guideline, including the ID, priority, name, definition, explanation, application features, examples, benefits, checklists and usability attributes. Moreover, it is possible to perform as many iterations as needed for all or some stages to improve the usability guidelines or the performance of experiments used for the validation stage [17].

2.4. Common Methods for Validating New Usability Guidelines

Three types of validation methods are proposed in Quinones et al.’s methodology. The first type is recommended by the methodology, and the other two types are for obtaining additional feedback as shown below: [17]:

Guideline evaluation: to check the proposed guidelines against traditional or specialized guidelines in terms of the number of discovered usability issues (The recommended method).
Expert judgment: using a questionnaire that assesses expert and evaluator perceptions of the proposed usability guidelines (To receive additional feedback).
User testing: to obtain users’ opinions about the proposed set of usability guidelines (To receive additional feedback).

However, the first type offers more opportunities to provide in-depth information about the effectiveness of the usability guidelines [17,28]. The comparison can be achieved by using a between-subject or within-subject study [40]:

Between-subject: each participant is exposed to only one condition of testing.
Within-subject: each participant is exposed to more than one of the conditions of testing.

The obtained results can be analyzed using some of the following criteria [17]:

Number of the discovered usability issues.
Number of discovered specific usability issues.
Number of severe usability issues.
Number of critical usability issues.

The majority of studies have applied the comparison method to validate their guidelines. However, the number of evaluators, compared guidelines and applications vary between the studies as shown in Table 1.

3. Materials and Methods

This section presents the methodology used to develop and generate UGmHA guidelines as well as the methods used to validate the resulting UGmHA guidelines.

3.1. Developing UGmHA

In the previous section, the literature review showed different guidelines used to evaluate mHealth applications. Some of these guidelines are general but used by various studies to evaluate mHealth applications, such as Nielsen’s principles [11]. On the other hand, some of the guidelines are used mainly to evaluate mHealth applications, such as Xcertia [12], HE4EH [6] and telemedicine mobile applications [13].

Each related work provides a diverse wealth of information and ideas around designing and evaluating mHealth applications. However, the guidelines and principles have limitations as described in Section 2. Inspired by previous work, we developed a set of comprehensive usability guidelines for mHealth applications (UGmHA), consisting of guidelines and checklist items generated from various studies and existing guidelines.

In order to develop our proposed guidelines effectively and efficiently, we employed the formal methodology of Quinones et al. [17], which was explained in Section 2. This methodology involves seven main stages, which were applied to fit within the context of our situation and research goals. The first five stages (exploratory, descriptive, correlational, selection and specification) were implemented by the first author of this paper and supervised by the other authors, while the sixth stage (validation) was performed by the first and third authors of the paper.

3.1.1. Stage 1: Exploratory Stage

In this stage, a systematic literature review was performed to collect information related to mHealth applications, consisting of application features, usability attributes, guidelines and recommendations. Since our study aims to make the applications accessible to different kinds of people, the reviewed literature is also related to elderly people and people with special needs.

In order to perform the systematic review, the method presented by Kitchenham and Charters [45] was employed using the following keywords:

For features: (“mHealth applications” OR “mobile health applications”) AND “features”.
For attributes: (“mHealth applications” OR “mobile health applications”) AND “attributes”.
For guidelines and recommendations: (“mHealth applications” OR “mobile health applications”) AND (“usability guidelines” OR “usability principles” OR “usability heuristics” OR “design recommendations”).
For accessibility guidelines: (“mHealth applications” OR “mobile health applications”) AND “accessibility guidelines”.

All keywords were used to search for relevant information using multiple databases, including MDPI, IEEE, Springer Link, JAMIA and JMIR.

The inclusion criteria for this study were: (1) studies related to mHealth applications; (2) studies related to the accessibility guidelines; (3) studies with a clear description of features, attributes, guidelines or recommendations; (4) studies that are publicly available; and (4) studies written in English. The exclusion criteria were (1) studies that were unavailable as a full-text; (2) studies not specified for mHealth applications or accessibility guidelines; (3) studies that did not provide enough information about targeted topics; and (4) survey papers.

As shown in Figure 2, a total of 54 studies were reviewed from databases, including features, attributes and guidelines related to mHealth applications and accessibility. We excluded eight studies due to the duplication of information. Then, we applied the inclusion and exclusion criteria based on the title, abstract and full text; a total of 34 studies additional were excluded. Finally, 12 studies were included for the goal of our study, consisting of 3 studies for features, 3 for attributes and 6 for guidelines.

The obtained information from this stage is shown below:

Features: these features are related to mHealth applications: [14,15,16], which will be discussed in the next stage.
Usability attributes, which include:
–
ISO attributes (effective, efficiency and satisfaction) [46].
–
Nielsen’s attributes (learnability, efficiency, memorability, errors and satisfaction) [47].
–
PACMAD attributes, which combine both attributes of ISO and Nielsen in addition to the cognitive load attribute (effective, efficiency, learnability, memorability, errors, satisfaction and cognitive load) [48].
Usability guidelines and design recommendations, which include:
–
Usability guidelines related to mHealth applications, which are Nielsen [11], Xcertia [12], HE4EH [6] and telemedicine mobile application recommendations [13].
–
Usability guidelines related to elderly people [34].
–
Usability guidelines related to people with special needs [35].

Section 2 presents more details regarding guidelines and design recommendations relevant to our study.

3.1.2. Stage 2: Descriptive Stage

In this stage, we highlighted and selected the essential information from the collected literature review in the previous stage.

For features: we gathered all features from [14,15,16] and summarized them to obtain a list of integrated features. Additionally, we added two more features related to our research, which were accessibility to elderly people and accessibility to people with special needs. Below are the selected features:
- Booking appointments [16].
- Consultation with HCP via text, voice messages and video calls [14,16].
- Sharing data with HCP [14,15].
- Access to personal health information [15].
- Self-monitoring of chronic disease [14,15].
- Allowing uploading and viewing of biometric measurements [14].
- Graphic display of patient’s information [14].
- Set health goals and treatment plan [14,15].
- Track health progress [15].
- Reminders and notifications [14].
- Health tips and motivation [14,15].
- Sharing health data with friends [15].
- Interactive prompts [14].
- Earn rewards [15].
- Bluetooth technology connection [14].
- Accessibility to elderly people.
- Accessibility to people with special needs.
For usability attributes: we selected only the PACMAD [48] attributes since they combine attributes of both ISO [46] and Nielsen [47]. Below are the selected attributes and their definition based on PACMAD [48].
–
Effectiveness: the ability to produce the desired outputs.
–
Efficiency: the reduction of wasted materials, such as time, money and effort.
–
Satisfaction: the fulfillment of the user’s needs and desires.
–
Learnability: a user can learn how to use the application easily.
–
Memorability: a user can remember how to use the application after a while.
–
Errors: minimizing the user’s error rate while using the application.
–
Cognitive Load: the ability to use a mobile application while doing daily activities.
For usability guidelines: we classified the studies into the main guidelines, which will be used as basic guidelines for the next stages, and checklist items, which help to make the main guidelines more specific to the mHealth applications:
–
Main guidelines: we selected all of Nielsen’s principles [11] as well as the “usability” workgroup of Xcertia guidelines [12]. The reason for selecting these two guidelines is that Nielsen’s principles are well-known guidelines that are used to evaluate most user interfaces. Moreover, Xcertia is specified to evaluate mHealth applications in general and developed by authorized organizations in the US.
–
Checklist items: we selected [6,12,13], which are related to mHealth applications. Additionally, we selected the guidelines related to elderly people [34] and people with special needs [35].

On the other hand, we discarded all other workgroups of Xcertia [12], which were “privacy”, “security”, “content” and “operability” since they are out of the scope of this study. Moreover, we discarded some checklist items in [6] that are very specific to diabetes and cannot be generalized to all chronic diseases, and we also discarded checklist items in [34,35] that cannot be used for mHealth applications. The results of this stage are shown in Table 2.

3.1.3. Stage 3: Correlational Stage

In this stage, the selected features, attributes and the main usability guidelines were matched as shown in Table 3. The purpose of this stage is to identify the specific features and attributes of mHealth applications that need to be evaluated by the main guidelines [17]. The process of matching should be controlled by features, which means that we chose to match the attributes that can measure a specific feature. Then, we chose the main guidelines (from Xcertia and Nielsen) that evaluate attributes with a feature as shown in Figure 3.

Let us take the booking appointments feature as an example: the most suitable attributes and guidelines can be matched with:

Attributes:
–
Effectiveness: to measure if the appointment is booked at the right date and time selected by the user.
–
Efficiency: to measure if this feature saves users’ time and effort without the need to go to the clinic.
Guidelines:
–
Visibility of system status in Nielsen and App feedback in Xcertia: to evaluate booking appointments with the effectiveness attribute if the application provides feedback to the user regarding the right date and time of the appointment.
–
Notifications and alarms in Xcertia: to evaluate booking appointments with the efficiency attribute if the application notifies a user after an appointment is booked. Thus, the users’ time and effort to remember the appointment are minimized.
–
Historical data in Xcertia: to evaluate booking appointments with the efficiency attribute if the application saves previous and upcoming appointments. Thus, the users’ time and effort to call the clinic to ask for their appointments are minimized.

Another example, is the earn rewards feature, which can be measured by the satisfaction attribute because, when the user earns rewards for a good action or habit, they will be more satisfied with the application. However, there is no matched guideline from Nielsen and Xcertia to evaluate this feature (which will be discussed in the next stage).

3.1.4. Stage 4: Selection Stage

In this stage, an evaluation and determination were made for each main guideline identified in the previous stage [17]:

Keep: without any change if the guideline is clear and correctly matched to the specific feature of mHealth applications.
Adapt: if the guideline needs a change to be more related to the mHealth applications.
Eliminate: if the guideline is redundant or not relevant to the features of mHealth applications.
Create: if a new guideline is required to evaluate the specific feature of the mHealth applications that the main guidelines cannot evaluate.

In this sense, we adapted all of Nielsen’s guidelines as well as some of the Xcertia guidelines, including onboarding, notifications, historical data and accessibility by adding more checklist items from different studies to be more specific to mHealth applications. Furthermore, we kept the ongoing app evaluation guideline in Xcertia. On the other hand, we eliminated the other Xcertia guidelines, which were visual design, readability, app navigation, app feedback and help resource and troubleshooting because we considered them and their checklist items as parts of Nielsen’s usability guidelines.

Therefore, we merged the checklists of each eliminated guideline in Xcertia with similar guidelines in Nielsen. For example, we found that the checklist items of visual design guideline in Xcertia satisfy the definition of Nielsen’s consistency and standards, recognition rather than recall and error prevention guidelines. Hence, the visual design guideline was eliminated, and its checklist items were moved to the corresponding guideline in Nielsen. This process was applied to all eliminated guidelines in Xcertia. However, F12 and F14 features were not covered by any of the selected main guidelines.

That is why we needed to create a new guideline (Interactive and motivations) that evaluates these features. After that, we determined the applicability of each guideline to identify its importance, whether (1) useful, (2) important or (3) critical based on the guideline definition and checklists. The results of this stage are described in detail in Table 4, Table 5 and Table 6.

3.1.5. Stage 5: Specification Stage

In this stage, the new set of usability guidelines was formally defined. Quinones et al. [17] recommended keeping the number of usability guidelines between 10 and 16 because it is difficult to employ many guidelines in practice and recommended using checklists to add more details to the guidelines. Therefore, we integrated the guidelines of Nielsen [11] and Xcertia [12] to form the main guidelines of UGmHA and keep them within the recommended range.

However, since Nielsen’s guidelines are general and most of Xcertia guidelines and checklist items are not specific to the features of mHealth applications, we adapted and extended them by adding more checklist items from different studies, which were [6,13,34,35]. The reasons for adding these checklist items are to make the main guidelines more specific to the features of mHealth applications and support the involvement of accessibility features for the elderly and people with special needs. The obtained results of this stage are 16 usability guidelines as shown in Table 7 in addition to 154 checklist items classified by the main guidelines.

The resulting template of this stage will be discussed in Section 4.

3.2. Validating UGmHA

3.2.1. Stage 6: Validation Stage

The validation stage was divided into three phases as suggested by Quinones et al.’s methodology [17]:

Guideline evaluation phase: check the new set of UGmHA against Nielsen and Xcertia guidelines regarding the number of identified usability issues and severity.
Expert judgment phase: evaluate the usefulness, efficiency and effectiveness of UGmHA guidelines by questionnaire or interview. Then, the results will be used in the seventh stage (refining stage) of Quinones et al.’s methodology [17].
User testing phase: enhance the usability of an existing application using refined guidelines of UGmHA. Then, perform user testing for the enhanced application.

However, the first phase of validation (guideline evaluation) will be presented in this paper, while the second and third phases (expert judgment and user testing) are ongoing work that will be published as soon as they are finished. Below are the details of the first phase of the validation.

Experiment Design:
–
Participants: The evaluators who participated in this experiment are two of the authors of this research. The first evaluator was Mrs. Eman Nasr, a master’s student in the Faculty of Computing and Information Technology (FCIT) at King Abdulaziz University with a medium knowledge of usability evaluation. The second evaluator was Dr. Duaa Sinnari, an assistant professor in FCIT at King Abdulaziz University with high experience in usability evaluation and human–computer interaction (HCI).
–
Guidelines: UGmHA was tested against Nielsen’s principles and the Xceria guidelines. The reason for selecting these two guidelines is that UGmHA is based mostly on Nielsen’s and Xcertia guidelines and because Nielsen’s principles are well-known guidelines and Xcertia is specified to evaluate mHealth applications.
–
Applications: We selected the “Sehhaty” and “Sokry” applications. Figure 4 shows the home screen of (a) the first application (Sehhaty) and (b) the second application (Sokry) on the Android operating system in the Arabic language. The reasons for selecting these two applications are because they are simple to use and to achieve diversity between the selected applications based on governmental/ private, functionality and popularity as shown in Table 8.
Experiment Procedure: To assess the performance, each of the evaluators evaluated both applications individually in a within-subject study that compared the UGmHA, Xcertia and Nielsen guidelines. First, the evaluators evaluated the “Sehhaty” and “Sokry” applications individually using Nielsen’s guidelines and wrote down the discovered usability issues for each application. Second, the assessments of each evaluator using Nielsen’s guidelines were grouped together to generate a single list of usability issues for each application.
Third, the evaluators worked together to rate each usability issue in the generated list based on the Severity Rating Scale (SRS). The SRS has five points (0–4) that can be used to prioritize and estimate which usability issues are important to be fixed as shown in Table 9 [51]. Then, the same aforementioned processes were repeated using Xcertia followed by the UGmHA guidelines. From this experiment, we produced six lists of usability issues, which are:
- Using Nielsen’s guidelines: list of usability issues of the “Sehhaty” application.
- Using Nielsen’s guidelines: list of usability issues of the “Sokry” application.
- Using Xcertia guidelines: list of usability issues of the “Sehhaty” application.
- Using Xcertia guidelines: list of usability issues of the “Sokry” application.
- Using UGmHA guidelines: list of usability issues of the “Sehhaty“ application.
- Using UGmHA guidelines: list of usability issues of the “Sokry” application.

3.2.2. Stage 7: Refining Stage

In this stage, the expert and user feedback obtained from stage 6 (validation stage) are used for the refining [17]. The process of refining is described as follows:

Document the changes that should be made to the guidelines.
Define the guidelines to be created, refined or deleted.
Iterate and repeat some stages again, if necessary.

However, since the experts’ judgment and user testing in the validation stage are future work. Accordingly, the refining stage is future work for this study.

4. Results

This section presents the results of developing and validating the UGmHA guidelines.

4.1. Results of UGmHA Development

The standard template of Quinones et al. [17] can contain different elements. Still, it depends on the researcher to determine whether to use the complete template or choose the needed elements [52]. Therefore, we decided to select the ID, priority, guideline name, guideline definition, application features and checklist items as described in Table 10. Appendix A shows a detailed description of each guideline in UGmHA.

4.2. Results of the UGmHA Validation

After receiving the usability results from the sixth stage (validation stage) described in Section 3, we defined the following findings:

4.2.1. Number of Usability Issues among the Three Guidelines

As depicted in Table 11, the numbers of usability issues discovered in the “Sehhaty” application were 73 using UGmHA, 22 using the Xcertia guidelines and 17 using Nielsen’s guidelines. On the other hand, the numbers of usability issues discovered in the “Sokry” application were 95 using UGmHA, 28 using the Xcertia guidelines and 25 using Nielsen’s guidelines.

4.2.2. Severity of Usability Issues among the Three Guidelines

As shown in Table 12 and Figure 5, the number of issues with a “Catastrophic” rating was the highest using all guidelines for both applications. Except for Nielsen’s guidelines, on “Sehhaty”, the highest number of issues were rated with a “Major” rating. However, there were no issues with the “no problem” rating for all guidelines on both applications.

For the UGmHA guidelines, as shown in Figure 6, the most unapplied guidelines in “Sehhaty” were accessibility with 11 issues and then user control and freedom and aesthetic and minimalist design with 9 issues, while the most unapplied guidelines in “Sokry” were the visibility of system status and accessibility with 13 issues and then user control and freedom with 11 issues.

However, the least unapplied guidelines in “Sehhaty” were error diagnosis and recovery with one issue, while the least unapplied guideline in “Sokry” was match between system and real world with one issue.

5. Discussion

This section presents the discussion of results obtained from developing and validating the UGmHA guidelines.

5.1. UGmHA Development Discussion

If we compare the resulting guidelines of UGmHA with the Nielsen and Xcertia guidelines, we find that the number of guidelines in UGmHA (16 guidelines) is greater than the number of guidelines in Nielsen (10 guidelines) and Xcertia (10 guidelines). The reason is that UGmHA integrated Nielsen and Xcertia by adapting and eliminating their guidelines through Quinones et al.’s methodology [17] as well as created a new guideline that covers features that Nielsen and Xcertia did not cover. Table 13 shows the UGmHA guidelines with the corresponding guidelines in Nielsen and Xcertia.

Furthermore, the number of checklist items in UGmHA (154 items) is greater than the number of checklist items in Xcertia (60 items), while there are no checklist items in Nielsen. As mentioned before, Nielsen’s guidelines are general, and a small number of checklist items in Xcertia guidelines are specific to the features of mHealth applications.

Therefore, we adapted and extended the guidelines by adding more checklist items that cover specific features of mHealth applications and features of accessibility for the elderly and people with special needs. The checklist items were extracted from the following studies:

HE4EH [6], which covers self-monitoring, biometric measurements, etc.
Telemedicine of mHealth applications [13], which covers consultation, booking appointments, etc.
Elderly people [34], which covers the feature of accessibility for elderly people.
People with special needs [35], which covers the feature of accessibility for people with special needs.

This leads to the main contribution of UGmHA by including more guidelines, checklist items and mHealth-specific checklist items than Nielsen and Xcertia as shown in Table 14.

5.2. UGmHA Validation Discussion

From the first experiment of UGmHA validation, we conclude that UGmHA discovered more usability issues than Xcertia and Nielsen for both applications. This is because UGmHA includes more guidelines and checklist items than the others. Moreover, the usability issues discovered in the “Sokry” application are more than those in the “Sehhaty” application for the three guidelines, which means that UGmHA can perform like Nielsen and Xcertia to determine which application is more usable.

For the severity of usability issues, we found that UGmHA found more severe usability issues than Xcertia and Nielsen. Again, this is because UGmHA includes more checklist items and more mHealth-specific checklist items.

For the UGmHA guidelines, we found that the highest number of issues were with the accessibility guideline in both applications. The reason is that the specifications of special needs and elderly people were not considered in the design, such as color preferences, font sizes and the number of words and sentences. This highlights the importance of the accessibility guideline and its checklist items that determine to what extent the application is accessible. For the newly created guideline Interactivity and motivations, we found several usability issues in both applications. This also determines to what extent the applications motivate users to take action and achieve a specific goal.

However, we could not use the Ongoing app evaluation guideline in the experiment since it is up to the application team. Therefore, we will leave it up to the expert judgment phase to acquire their opinion and suggestion about it. In the near future, we plan to provide design recommendations for “Sehhaty” or “Sokry” based on the discovered usability issues and, then, to provide these recommendations to the application team to take their opinion.

One of the challenges we faced was that we could not perform phases 2 and 3 of the validation stage. The reason is that we contacted the user experience (UX) team of the “Sehhaty” application to provide their opinion regarding UGmHA and discovered usability issues, they were willing to cooperate with us; however, right now, they are busy with government projects. Therefore, we postponed the implementation of phases 2 and 3 for a few months in future work.

6. Conclusions and Future Work

With the evolution of mobile applications that have become a supportive technology for different fields, mobile health (mHealth) applications are important to improve the efficiency of healthcare delivery using mobile devices, such as smartphones and wearable devices.

mHealth allows patients to monitor and track their health data and allows communication between patients and their physicians without meeting face to face. However, mHealth applications can harm users if they are not working as intended. Although some studies have discussed general and specific mHealth guidelines, there are certain limitations to them.

In this research, we proposed comprehensive usability guidelines for mHealth applications (UGmHA) that address some of the limitations of previous studies and cover various features related to mHealth applications. The proposed guidelines consist of sixteen (16) guidelines with 154 checklist items built from global and well-known guidelines, such as Nielsen and Xcertia.

In order to validate the effectiveness of the UGmHA guidelines, an experiment was conducted by measuring the performance of the proposed guidelines through comparing the outcomes of the UGmHAto worldwide guidelines. This comparison was applied to two mHealth applications in Saudi Arabia, “Sehhaty” and “Sokry”, to determine which guidelines could identify more usability issues. The results showed that the proposed guidelines discovered more usability issues compared with the others. Thus, UGmHA can assist expert evaluators, designers and developers in designing or evaluating mHealth applications by measuring the usability issues that can influence the user experience of mHealth applications.

For future work, further steps will be conducted to validate and refine UGmHA, which include:

Generating design recommendations based on the discovered usability issues for “Sehhaty” or “Sokry” and obtaining the application team’s opinion.
Expert judgment by reviewing guidelines by user experience (UX) experts and acquiring their advice and comments to perform the Refining stage.
User testing by enhancing the usability of an existing application using refined guidelines of UGmHA and, then, performing user testing for the enhanced application.

Author Contributions

Conceptualization, E.N., W.A. and D.S.; methodology, E.N.; validation, E.N. and D.S.; formal analysis, E.N. and D.S.; resources, E.N.; writing—original draft preparation, E.N.; writing—review and editing, E.N., W.A. and D.S.; supervision, W.A. and D.S.; project administration, W.A. and D.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

eHealth	Electronic health
mHealth	Mobile health
UGmHA	Usability guidelines for mHealth applications
FDA	Food and Drug Administration
HIMSS	Healthcare Information and Management Systems Society
SRS	Severity rating scale
HCP	Healthcare provider

Appendix A. Final Usability Guidelines for mHealth Applications (UGmHA)

This appendix includes sixteen (16) proposed usability guidelines for mHealth applications (UGmHA). The description and details of each guideline are provided based on Quinones et al.’s template [17]. The template consists of the guideline ID, guideline name, guideline definition, covered application features based on the provided checklist and guidance checklist to evaluate each guideline as shown in Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, Table A12, Table A13, Table A14, Table A15 and Table A16.

Table A1. Descriptive details of the first guideline.

ID	MHP1
Priority	(3) Critical
Name	Visibility of system status (Application Feedback)
Definition	Ensure that the application keeps users aware of what is going on in the application by including appropriate feedback on user actions.
Features	Booking appointments, consultation with HCP via text, voice messages and video calls; sharing data with HCP; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; set health goals and treatment plans; track health progress; Bluetooth technology; and accessibility by elderly people.
Checklists	Does the application clearly identify the patient’s name or ID [12]? Are feedback messages displayed in a predictable and consistent area within the interface in order to be noticed by the patient [12]? Does successful feedback appear when an action is completed, such as a successful profile update or appointment booking [12]? Is there a confirmation message for critical actions, such as appointment cancellation or data deletion [34]? Is the time between actions and results appropriate for the user’s cognitive processing (not too slow or too fast) [6,12]? Are several redundant identifiers—such as color, iconography, labeling and additional text—used to denote vital or safety-related information [12]? Do feedback messages that are not urgent or not related to medical safety risks avoid interfering with the application’s operations [12]? Do ongoing processes (greater than 20 s) utilize ongoing feedback, such as a progress bar or show the time remaining to complete the downloading process [6,12]? Is the application feedback appropriate and understandable by elderly people [34]? Does the application add real-time updates that signify the available clinics (currently online) vs. unavailable clinics (offline) [13]? For self-monitoring, does the application acknowledge the performance of the user’s recorded behavior and the goals achieved [6]? For self-monitoring, does the application inform users if the readings will vary depending on the tool used, method or time for taking measurements [6]? If the application uses sensors, are users informed for how many days this sensor works [6]? For a consultation session, does the application show a confirmation message to the user before starting the session [13]? For a consultation session, does the application show a confirmation message before closing the session [13]? For a consultation session, does the application show a timer and wait time to see a doctor or consultation duration [13]?

Table A2. Descriptive details of the second guideline.

ID	MHP2
Priority	(2) Important.
Name	Match between the system and the real world (Metaphor).
Definition	Ensure that the application speaks the users’ language, employs terminology and concepts that the user is familiar with and arranges information in a natural and logical sequence.
Features	Accessibility by elderly people and accessibility by people with special needs.
Checklists	Is the application speaking the native language of the country that it is intended for [12]? Is written content free of jargon, acronyms and text that may not be understandable to users with no clinical knowledge [12]? Does the application use terminology that is readable and understandable by users and avoid medical terminology [35]? Does the application use terms that older adults are familiar with [34]? Are the terms used in the application consistent through all screens [13]? If there are services or options used in the application that seem to be the same, but they are actually different, does the application provide a description for these options or services [13]?

Table A3. Descriptive details of the third guideline.

ID	MHP3
Priority	(2) Important.
Name	User control and freedom (Navigation).
Definition	Ensure that the application has proper navigation and provides an “emergency exit” to smoothly leave an unwanted state.
Features	Booking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; and accessibility by elderly people.
Checklists	Are users able to easily recognize where they are in the application and how to navigate to various destinations [12]? Are the number of taps, swipes or screens required for the navigation minimized [12]? Does the application provide reversible actions, such as allowing the user to return to previous pages [11,12]? Does using the back button always take the patient back to the page they were on before [34]? Does the application include clear instructions and emergency exits to leave an unwanted state [34]? Is the main menu provided in order to navigate easily within the application [12,34]? Is the main menu located on the top and left-hand side [12]? Are menu options labeled intuitively [12]? Does the navigation bar exist on every page in the application [34]? Does the navigation text clearly identify the destination [34]? Is the search service for information in the application provided, such as consultations, the name of the doctor and historical information [13,34]? Are the user’s search results presented in a proficient way (for instance, in alphabetical order, relevancy order or last updated order) [13,34]? Is the search box easily recognizable and accessible [13,34]? Does the size of the search box fit on the screen [13,34]? For appointment scheduling, is the patient able to choose among physician’s specialties [13]? For appointment scheduling, is the patient able to see a physician’s detailed information (such as name, education experiences and achievements [13]? Is a patient able to view past consultations and upcoming appointments [13]? Is a patient able to view the prescription and investigations ordered by their consulted physicians [13]? Is a patient able to view files, measurements and data shared with their HCP [13]? Is there any indicator for users where they can find their history of measurements [13]?

Table A4. Descriptive details of the fourth guideline.

ID	MHP4
Priority	(2) Important.
Name	Consistency and standards.
Definition	Ensure that the application’s elements are consistent and follow the standard design of the application.
Features	Accessibility by elderly people and accessibility by people with special needs.
Checklists	Is the text aligned right or left based on the direction of the local language [12]? Is the application’s logo positioned on the top center, top right or top left [12]? Is the main menu indicated by a three-bar “hamburger” icon, which is a familiar icon to the users [12]? Is the text hierarchy maintained, so that larger font sizes represent headings and smaller font sizes represent paragraph text [12]? Are all screens designed and elements maintain the same format [34,35]? Does the application use consistent font sizes and colors across all screens [35]? Are the titles, menus and forward, backward and save buttons positioned consistently [35]? Do error messages follow traditional visuals, such as bold and red text? [12]? Do the elements in the application screen fit the normal posture of the hand and finger [34]?

Table A5. Descriptive details of the fifth guideline.

ID	MHP5
Priority	(3) Critical.
Name	Error prevention.
Definition	Ensure that the application prevents problems from occurring by careful design.
Features	Booking appointments, self-monitoring of chronic diseases, allowing uploading and viewing of biometric measurements, accessibility by elderly people and accessibility by people with special needs.
Checklists	Does the application minimize the possibility of data entry error by providing users with selectable options rather than demanding text entries, such as selecting a doctor’s specialty to book an appointment [12]? Does the application inform users of data entry requirements, such as password needs letters, numbers and symbols [12]? Is an early error message displayed by the application when unacceptable letters or symbols are entered [13]? For self-monitoring, is a user informed of the requirements, best conditions, usual time and frequency of testing the biometric measurements [6]? For self-monitoring, is a user informed through the application of the desired and normal test results [6]?

Table A6. Descriptive details of the sixth guideline.

ID	MHP6
Priority	(2) Important.
Name	Recognition rather than recall (Memory).
Definition	Ensure that the application minimizes the user’s memory load.
Features	Booking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; accessibility by elderly people; and accessibility by people with special needs.
Checklists	Does the application support recognition over recall by keeping important information on screen rather than requiring users to remember it [12]? Is the application providing support to remember functions easily [34]? Are the icons or images on a button easy to guess what it does? [34]? Are items placed in recognizable places? [34]? Does the application group similar functions together, such as group consultations for specific specialties together [34]?

Table A7. Descriptive details of the seventh guideline.

ID	MHP7
Priority	(1) Useful.
Name	Flexibility and efficiency of use (Efficiency).
Definition	Ensure that the application provides an accelerator that shortcuts some actions and allows users to customize the application based on their needs and preferences.
Features	Consultation with HCP via text, voice messages and video calls; sharing data with HCP; access to personal health information; self-monitoring of chronic diseases; accessibility by elderly people; and accessibility by people with special needs.
Checklists	Does the application provide a flexible and usable interface to operate in both landscape and portrait orientations [12]? Does the application provide clear and readable content across different sizes of mobile screens and operating systems [12]? Are users allowed to customize the application based on their preferences, such as colors and font size [12,34]? Are the application’s default settings easy to use for the elderly [34]? For a consultation session, is a patient able to disable the camera and mute voice through consultation [13]? For a consultation session, does the application support text messaging through consultation [13]? Is a patient able to attach and send files to the physician through consultations [13]? Does self-monitoring of chronic diseases address each individual’s specific educational and behavioral requirements [6]? For self-monitoring, does the application provide a facility to set easy tasks and increase the difficulty level until the target goal is achieved [6]?

Table A8. Descriptive details of the eighth guideline.

ID	MHP8
Priority	(1) Useful.
Name	Aesthetic and minimalist design (Design).
Definition	Ensure that the application does not contain useless or irrelevant information. Ensure that the visual design adheres to the contrast, repetition, alignment and proximity rules.
Features	Access to personal health information, accessibility by elderly people and accessibility by people with special needs.
Checklists	Does the application follow contrasting colors, such as a white background and dark text for easy readability [12,34,35]? Does the application minimize the use of extraneous text, graphics and animation to avoid distracting the patients or cluttering the screen [12,34,35]? Are the critical elements to the application functionality and content understandability placed in a recognizable position, such as above the scroll line to minimize the opportunity for missed information? [12,34]? Does the application include multimedia, graphics, images and icons to make the content clear and descriptive [35]? Does the application indicate actionable elements and differentiate interactive elements from non-selectable content [12,34]? Is the content organized around big ideas with a holistic approach [35]? Do the options/information follow logical sequences [34]? Is the amount of text reduced (only essential information presented) [34]? Does the application use small information chunking or lists and tables to promote learnability and memorability when huge amounts of data must be displayed [12,34,35]? Does the application incorporate spacing allowances between lines to allow for breathability and readability [12,35]? Are the labels described clearly [34]? Are the buttons large enough to see the image or text on them [34]? Is the button size suitable for finger touch [34]? Is there enough space between the buttons to avoid accidentally pressing numerous or incorrect buttons [34]?

Table A9. Descriptive details of the ninth guideline.

ID	MHP9
Priority	(3) Critical.
Name	Error diagnosis and recovery (Recovery).
Definition	Ensure that the application expresses the error messages in plain language (with no codes), accurately describes the issue and positively suggests a solution.
Features	Accessibility by elderly people and accessibility by people with special needs.
Checklists	Are error messages clear, concise and explain the problem [12,34]? Are error messages informing users of required corrective actions in a clear and understandable way [12,34]? Are error messages understandable by the elderly and people with special needs [34,35]?

Table A10. Descriptive details of the tenth guideline.

ID	MHP10
Priority	(1) Useful.
Name	Help and documentation (Help).
Definition	Ensure that the application incorporates clear help and troubleshooting tools to assist users when necessary.
Features	Self-monitoring of chronic diseases, interactive prompts, accessibility by elderly people and accessibility by people with special needs.
Checklists	Does the application include a help section that compiles all the patient-assistance information [12]? Are help pop-ups for simple information and links for complex information embedded in the application when patients may be likely to need them [12]? Does the application use graphics or videos to enhance or replace text for complicated assistance instructions [12]? Does the application provide customer services, such as e-mail and online chat [12]? Are experienced users allowed to bypass step-by-step walkthroughs or detailed instructions [12]? Does the application give easy-to-follow steps and avoid text-heavy paragraphs for instructional information [12]? For self-monitoring, does the application provide instructions on how to perform a specific behavior [6]? For self-monitoring, is the behavior of an expert provided in an application that shows a person in a video for how to correctly perform a test or do a behavior [6]?

Table A11. Descriptive details of the eleventh guideline.

ID	MHP11
Priority	(3) Critical.
Name	Notifications, alerts and alarms (Notifications).
Definition	Ensures that the application’s notifications, alerts and alarms take both safety and usability into account to notify users when their attention is required. Notifications (general reminders for users). Alerts (non-urgent signs meant to catch user attention). Alarms (urgent signs for safety-critical messages).
Features	Booking appointments, consultation with HCP via text, voice messages and video calls; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; reminders and notifications; and interactive prompts.
Checklists	Do safety-critical alarms send out redundant user signals [12]? Are users required to accept alarms before proceeding with other tasks [12]? Is it possible for users to opt out of non-critical notifications and alerts [12]? Are users notified if an app deletes data after a certain period of time [12]? Are the volume and vibration of audible notifications customizable [12]? Are notifications, alerts and alarms stored as historical data for future reference [12]? For consultations, is a user informed through sound alerts that the physician is present in the session [13]? For consultations, is the user notified via alerts that the consultation’s session will finish if no response is received from the user [13]? For self-monitoring, is a user notified through a sound alarm of high and low measurements and how far their measurements are from the target range [6]?

Table A12. Descriptive details of the twelfth guideline.

ID	MHP12
Priority	(2) Important.
Name	Onboarding.
Definition	Ensure that the application facilitates launching, registering and preparing for first time use.
Features	Access to personal health information.
Checklists	Does the application provide a launch screen that clearly states its name and objectives? [12]? Does the application give the user intuitive choices for initial use, such as preferred language [12]? Does the application provide opportunities for users either to access detailed application instructions or immediately begin working with the application [12]? Does the application allow users to bypass personal data entry if it is not critical to the application’s functionalities [12]? For the data needed for the application functionality, does the application mark it as mandatory [12]? For applications with complex onboarding processes, is the data entered stored so that users can recover and avoid reentry if they are disconnected during the onboarding [12]? When setup is complete, does the application provide options for a walkthrough or tutorial on app use [12]? Does the application bypass onboarding for returning users [12]? Does the application allow users to update the data entered through onboarding on the “Profile” or “Setting” page [12]?

Table A13. Descriptive details of the thirteenth guideline.

ID	MHP13
Priority	(3) Critical.
Name	Historical data (History).
Definition	Ensure that the application stores historical data that allow users to access, read and understand these data easily.
Features	Booking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; graphic display of patient’s information; and track health progress.
Checklists	For large data history, are historical data sortable and filterable based on certain categories [12]? Are historical data displayed in a suitable way to the patients and customized depending on the data type (For example, the daily collected historical data can be represented in alphabetical order so that a patient has easy access to the most recent data collected) [12]? Are patients informed if the application has a limited amount of data storage [12]? For self-monitoring, does the application show results and reading history as a graph in order to be readable and understandable for patients [6]? Does the application show a clear and readable graph and charts for historical biometric measurements [6]? For self-monitoring, does the application inform patients of how many hours and days the results are stored and displayed [6]? For self-monitoring, is a review of previous goals provided in the application [6]?

Table A14. Descriptive details of the fourteenth guideline.

ID	MHP14
Priority	(3) Critical.
Name	Accessibility.
Definition	Ensure that the application is usable by all users, including elderly people and people with special needs.
Features	accessibility by elderly people and accessibility by people with special needs.
Checklists	Does the application provide accessibility for people with special needs [12]? Does the application use text alternatives or captions for multimedia, such as images or videos [12]? Is the data entry process easy for the elderly and people with special needs [34,35]? Does the application allow a variety of input modalities, such as gestures, handwriting and speech recognition [12]? Does the application provide a validation of the entered input modalities [12]? Does the application facilitate one-handed use [12,34]? Does the application consider customization of colors to be suitable for visual impairments, such as avoiding red and green colors [12]? Is it possible to increase the font size [34]? Does the application consider screen reader accessibility [12]? Does the application include less than four colors to not distract people with intellectual disabilities [35]? Are content written in large font sizes, such as larger than 14 points [35]? Are short sentences employed in the application <7 words [35]? Are the minimum number of sentences employed in the application <4 for each screen [35]? When changes are made to facilitate accessibility, does the content preserve functioning and readability [12]? Are the menus and buttons large enough for the elderly and those with special needs to navigate [34,35]? When the text size is increased, do buttons and icons enlarge [34]? Does the application allow to enable tap gestures for elderly people [34]? Do gestures work correctly and smoothly [34]?

Table A15. Descriptive details of the fifteenth guideline.

ID	MHP15
Priority	(2) Important.
Name	Ongoing app evaluation (Evaluation).
Definition	Ensure that the application undergoes iterative evaluation and follows a user-centered design.
Features	Important to design and evaluate all features.
Checklists	Are the characteristics of the target users studied before application envelopment [12]? If the application is intended for use in a clinical environment, does it consider how the app may realistically fit into the clinical workflow [12]? Does the application meets known usability guidelines [12]? Does the application use tools for failure modes or fault-tree analysis to determine which user tasks can cause risks for patients [12]? IS user testing conducted iteratively with the target end users of the application [12]? Is a final summative (validation) test conducted to ensure that the application can be used safely and successfully [12]?

Table A16. Descriptive details of the sixteenth guideline.

ID	MHP16
Priority	(1) Useful.
Name	Interactivity and motivations (Interactivity).
Definition	Ensure that the application motivates users and allows for communication between patients to share their experiences.
Features	Consultation with HCP via text, voice messages and video calls; sharing data with HCP; self-monitoring of chronic diseases; health tips and motivation; sharing health data with friends; interactive prompts; and earning rewards.
Checklists	For consultations, is the patient provided a satisfaction survey at the end of the consultation [13]? For self-monitoring, does the application ask patients to assess the relationship of self-monitoring for specific diseases with exercise, food, medications and stress [6]? For self-monitoring, is social support provided by allowing users to communicate and share their experience of how they change their behavior to offer help to patients [6]? For self-monitoring, are contingent rewards in terms of praise or encouragement given that are explicitly linked to the achievement of a specified goal [6]? For self-monitoring, are opportunities for social comparison provided by allowing users to view the performance of non-expert users [6]? For self-monitoring, is motivational interviewing supported in-app by asking the user to provide self-motivating statements to minimize the change resistance [6]? Are users invited to share content and provide feedback about their experiences [6]?

References

Risling, T.; Martinez, J.; Young, J.; Thorp-Froslie, N. Evaluating patient empowerment in association with eHealth technology: Scoping review. J. Med. Internet Res. 2017, 19, e329. [Google Scholar] [CrossRef] [PubMed]
Dicianno, B.E.; Parmanto, B.; Fairman, A.D.; Crytzer, T.M.; Yu, D.X.; Pramana, G.; Coughenour, D.; Petrazzi, A.A. Perspectives on the evolution of mobile (mHealth) technologies and application to rehabilitation. Phys. Ther. 2015, 95, 397–405. [Google Scholar] [CrossRef] [Green Version]
Istepanian, R.S.; Jovanov, E.; Zhang, Y. Guest editorial introduction to the special section on m-health: Beyond seamless mobility and global wireless health-care connectivity. IEEE Trans. Inf. Technol. Biomed. 2004, 8, 405–414. [Google Scholar] [CrossRef] [PubMed] [Green Version]
World Health Organization. mHealth: New Horizons for Health Through Mobile Technologies; World Health Organization: Geneva, Switzerland, 2011. [Google Scholar]
Barutçu, S. mHealth apps design using quality function deployment. Int. J. Health Care Qual. Assur. 2019, 32, 698–708. [Google Scholar] [CrossRef] [PubMed]
Khowaja, K.; Al-Thani, D. New checklist for the heuristic evaluation of mHealth apps (HE4EH): Development and usability study. JMIR mHealth uHealth 2020, 8, e20353. [Google Scholar] [CrossRef]
Cook, V.E.; Ellis, A.K.; Hildebrand, K.J. Mobile health applications in clinical practice: Pearls, pitfalls, and key considerations. Ann. Allergy Asthma Immunol. 2016, 117, 143–149. [Google Scholar] [CrossRef]
Larson, R.S. A path to better-quality mHealth apps. JMIR mHealth uHealth 2018, 6, e10414. [Google Scholar] [CrossRef]
Andrade, F.; Nascimento, L.; Wood, G.; Calil, S. Applying heuristic evaluation on medical devices user manuals. In Proceedings of the World Congress on Medical Physics and Biomedical Engineering, Toronto, ON, Canada, 7–12 June 2015; Springer: Cham, Switzerland, 2015; pp. 1515–1518. [Google Scholar]
Rusu, C.; Roncagliolo, S.; Rusu, V.; Collazos, C. A Methodology to establish usability heuristics. In Proceedings of the ACHI 2011: The Fourth International Conference on Advances in Computer-Human Interactions, Guadeloupe, France, 23–28 February 2011; IARIA: Wilmington, NC, USA, 2011; pp. 59–62. [Google Scholar]
Nielsen, J. Ten Usability Heuristics. 2005. Available online: http://www.nngroup.com/articles/ten-usability-heuristics (accessed on 5 November 2021).
Xcertia mHealth App Guidelines. 2019. Available online: https://www.himss.org/sites/hde/files/media/file/2020/04/17/xcertia-guidelines-2019-final.pdf (accessed on 7 November 2020).
Aldekhyyel, R.N.; Almulhem, J.A.; Binkheder, S. Usability of Telemedicine Mobile Applications during COVID-19 in Saudi Arabia: A Heuristic Evaluation of Patient User Interfaces. Healthcare 2021, 9, 1574. [Google Scholar] [CrossRef]
Donevant, S.B.; Estrada, R.D.; Culley, J.M.; Habing, B.; Adams, S.A. Exploring app features with outcomes in mHealth studies involving chronic respiratory diseases, diabetes, and hypertension: A targeted exploration of the literature. J. Am. Med. Inform. Assoc. 2018, 25, 1407–1418. [Google Scholar] [CrossRef] [Green Version]
Lazard, A.J.; Brennen, J.S.B.; Belina, S.P. App Designs and Interactive Features to Increase mHealth Adoption: User Expectation Survey and Experiment. JMIR mHealth uHealth 2021, 9, e29815. [Google Scholar] [CrossRef]
Shati, A. Mhealth applications developed by the Ministry of Health for public users in KSA: A persuasive systems design evaluation. Health Inform. Int. J. 2020, 9, 1–13. [Google Scholar] [CrossRef]
Quiñones, D.; Rusu, C.; Rusu, V. A methodology to develop usability/user experience heuristics. Comput. Stand. Interfaces 2018, 59, 109–129. [Google Scholar] [CrossRef]
Mitchell, J. From Telehealth to e-Health: The Unstoppable Rise of e-Health; Department of Communications, Information Technology and the Arts: Canberra, Australia, 1999. [Google Scholar]
Rooij, T.v.; Marsh, S. eHealth: Past and future perspectives. Pers. Med. 2016, 13, 57–70. [Google Scholar] [CrossRef] [PubMed]
Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K.; Bokun, T.; McKinstry, B.; Procter, R.; Majeed, A.; Sheikh, A. The impact of eHealth on the quality and safety of health care: A systematic overview. PLoS Med. 2011, 8, e1000387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Prado, L.; Carpentier, C.; Préau, M.; Schott, A.M.; Dima, A. mHealth Apps for Self-Management of Chronic Conditions in France: What Is Out There? In MEDINFO 2019: Health and Wellbeing e-Networks for All; IOS Press: Amsterdam, The Netherlands, 2019; pp. 1970–1971. [Google Scholar]
Markus, M.L.; Keil, M. If we build it, they will come: Designing information systems that people want to use. MIT Sloan Manag. Rev. 1994, 35, 11. [Google Scholar]
Tan, W.s.; Liu, D.; Bishu, R. Web evaluation: Heuristic evaluation vs. user testing. Int. J. Ind. Ergon. 2009, 39, 621–627. [Google Scholar] [CrossRef]
International Organization for Standardization. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs); ISO: Geneva, Switzerland, 1992. [Google Scholar]
Bevana, N.; Kirakowskib, J.; Maissela, J. What is usability. In Proceedings of the fourth International Conference on HCI, Stuttgart, Germany, 1–6 September 1991; pp. 1–6. [Google Scholar]
Nielsen, J.; Molich, R. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA, 1–5 April 1990; pp. 249–256. [Google Scholar]
Quiñones, D.; Rusu, C. How to develop usability heuristics: A systematic literature review. Comput. Stand. Interfaces 2017, 53, 89–122. [Google Scholar] [CrossRef]
Hermawati, S.; Lawson, G. Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Appl. Ergon. 2016, 56, 34–51. [Google Scholar] [CrossRef]
Vieira, E.A.O.; Silveira, A.C.D.; Martins, R.X. Heuristic evaluation on usability of educational games: A systematic review. Inform. Educ. 2019, 18, 427–442. [Google Scholar] [CrossRef]
Paz, F.; Paz, F.A.; Pow-Sang, J.A.; Collantes, L. Usability heuristics for transactional web sites. In Proceedings of the 2014 11th International Conference on Information Technology: New Generations, IEEE, Las Vegas, NV, USA, 7–9 April 2014; pp. 627–628. [Google Scholar]
Leverenz, T. The Development and Validation of a Heuristic Checklist for Clinical Decision Support Mobile Applications. Ph.D. Thesis, Wichita State University, Wichita, KS, USA, 2019. [Google Scholar]
Chattratichart, J.; Lindgaard, G. A comparative evaluation of heuristic-based usability inspection methods. In CHI’08 Extended Abstracts on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, USA, 2008; pp. 2213–2220. [Google Scholar]
Roy, B.; Call, M.; Abts, N. Development of usability guidelines for mobile health applications. In Proceedings of the International Conference on Human–Computer Interaction, Orlando, FL, USA, 26–31 July 2019; Springer: Cham, Switzerland, 2019; pp. 500–506. [Google Scholar]
Al-Razgan, M.S.; Al-Khalifa, H.S.; Al-Shahrani, M.D. Heuristics for evaluating the usability of mobile launchers for elderly people. In Proceedings of the International Conference of Design, User Experience, and Usability, Heraklion, Crete, Greece, 22–27 June 2014; Springer: Cham, Switzerland, 2014; pp. 415–424. [Google Scholar]
Arachchi, T.K.; Sitbon, L.; Zhang, J. Enhancing access to eLearning for people with intellectual disability: Integrating usability with learning. In Proceedings of the IFIP Conference on Human–Computer Interaction, Mumbai, India, 25–29 September 2017; Springer: Cham, Switzerland, 2017; pp. 13–32. [Google Scholar]
Sanz, F.; Galvez, R.; Rusu, C.; Roncagliolo, S.; Rusu, V.; Collazos, C.A.; Cofré, J.P.; Campos, A.; Quiñones, D. A set of usability heuristics and design recommendations for u-learning applications. In Proceedings of the Information Technology: New Generations: 13th International Conference on Information Technology, Las Vegas, NV, USA, 11–13 April 2016; Springer: Cham, Switzerland, 2016; pp. 983–993. [Google Scholar]
Campos, A.; Rusu, C.; Roncagliolo, S.; Sanz, F.; Gálvez, R.; Quiñones, D. Usability heuristics and design recommendations for driving simulators. In Proceedings of the Information Technology: New Generations: 13th International Conference on Information Technology, Las Vegas, NV, USA, 11–13 April 2016; Springer: Cham, Switzerland, 2016; pp. 1287–1290. [Google Scholar]
Gale, N.; Mirza-Babaei, P.; Pedersen, I. Heuristic guidelines for playful wearable augmented reality applications. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, London, UK, 5–7 October 2015; pp. 529–534. [Google Scholar]
da Hora Rodrigues, K.R.; Teixeira, C.A.C.; de Almeida Neris, V.P. Heuristics for assessing emotional response of viewers during the interaction with TV programs. In Proceedings of the Human–Computer Interaction. Theories, Methods, and Tools: 16th International Conference, HCI International 2014, Heraklion, Crete, Greece, 22–27 June 2014; Proceedings, Part I 16. Springer: Cham, Switzerland, 2014; pp. 577–588. [Google Scholar]
Charness, G.; Gneezy, U.; Kuhn, M.A. Experimental methods: Between-subject and within-subject design. J. Econ. Behav. Organ. 2012, 81, 1–8. [Google Scholar] [CrossRef]
Kuparinen, L.; Silvennoinen, J.; Isomäki, H. Introducing usability heuristics for mobile map applications. In Proceedings of the International Cartographic Conference, Dresden, Germany, 25–30 August 2013; International Cartographic Association: Bern, Switzerland, 2013. [Google Scholar]
Munoz, R.; Chalegre, V. Defining virtual worlds usability heuristics. In Proceedings of the 2012 Ninth International Conference on Information Technology-New Generations, IEEE, Las Vegas, NV, USA, 16–18 April 2012; pp. 690–695. [Google Scholar]
Moraes, M.C.; Silveira, M.S. How am I? Guidelines for animated interface agents evaluation. In Proceedings of the 2006 IEEE/WIC/ACM International Conference on Intelligent Agent Technology, IEEE, Hong Kong, China, 18–22 December 2006; pp. 200–203. [Google Scholar]
Muhanna, M.; Masoud, A.; Qusef, A. Usability heuristics for evaluating Arabic mobile games. Int. J. Comput. Games Technol. 2022, 2022, 5641486. [Google Scholar] [CrossRef]
Keele, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering; Technical Report, Ver. 2.3; EBSE: Durham, UK, 2007. [Google Scholar]
Iso, W. 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs). Int. Organ. Stand. 1998, 45, 22. [Google Scholar]
Usability 101: Introduction to Usability. 2012. Available online: https://www.nngroup.com/articles/usability-101-introduction-to-usability/ (accessed on 8 March 2021).
Harrison, R.; Flood, D.; Duce, D. Usability of mobile applications: Literature review and rationale for a new usability model. J. Interact. Sci. 2013, 1, 1. [Google Scholar] [CrossRef] [Green Version]
Sehhaty Application. 2022. Available online: https://play.google.com/store/apps/details?id=com.lean.sehhaty (accessed on 6 December 2022).
Sokry Application. 2022. Available online: https://play.google.com/store/apps/details?id=com.sokry (accessed on 6 December 2022).
Severity Ratings for Usability Problems. 1994. Available online: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ (accessed on 5 September 2021).
Quiñones, D.; Rusu, C. Applying a methodology to develop user eXperience heuristics. Comput. Stand. Interfaces 2019, 66, 103345. [Google Scholar] [CrossRef]

Figure 1. Xcertia workgroups.

Figure 2. Flowchart for the systematic literature review of UGmHA.

Figure 3. Matching process for features, attributes and guidelines.

Figure 4. (a) The home screen of the first application. (b) The home screen of the second application.

Figure 5. Severity of issues in the “Sehhaty” and “Sokry” applications among the three guidelines.

Figure 6. Number of usability problems found by UGmHA.

Table 1. Review of the validation process in different studies.

Ref	Validation Process	Validation Result
[41]	Four evaluators evaluated an application by comparing Nielsen guidelines against the authors’ guidelines.	The authors’ guidelines were more effective than Nielsen’s since they discovered more usability issues (19 issues, 6 of them were severe) than Nielsen’s guidelines (15 issues, 5 of them were severe).
[42]	Six evaluators evaluated two applications by comparing Nielsen guidelines against the authors’ guidelines.	The author’s guidelines worked better than Nielsen’s since they discovered more usability issues (App1: 43, App2: 45) than Nielsen’s guidelines (App1: 28, App2: 24)
[43]	Two evaluators evaluated four animated agents through new guidelines.	The results showed which agent’s design was better.
[44]	Eighteen evaluators evaluated two Arabic applications by comparing the authors’ guidelines against Nielsen’s guidelines and game usability principles (two applications and three guidelines).	The author’s guidelines worked better than the others since they discovered more usability issues (Game1: 13, Game2: 12) than Nielsen (Game1: 6, Game2: 5) and the game usability principles (Game1: 10, Game2: 8).

Table 2. The information highlighted in the descriptive stage.

Topic	Collected Information	Selected Information
Features	Features provided by [14]	Features of these studies are combined and
	Features provided by [15]	summarized in Table 3.
	Features provided by [16]
	Accessibility to elderly people	Selected.
	Accessibility by people with special needs	Selected.
Attributes	Usability attributes proposed by ISO standard [46] (three attributes): effective, efficiency and satisfaction	Unselected.
	Usability attributes proposed by Nielsen [47] (five attributes): learnability, efficiency, memorability, errors and satisfaction	Unselected.
	Usability attributes proposed by Harrison [48] (seven attributes): effective, efficiency, learnability, memorability, errors, satisfaction and cognitive load	Selected because it combines both attributes of ISO and Nielsen in addition to the cognitive load.
Existing guidelines	Nielsen’s guidelines (10 guidelines) [11]	The set of Nielsen’s principles are selected.
	Xcertia guidelines (five workgroups: privacy, security, content, usability and operability) [12]	The only selected workgroup is “usability” and discards all others since our focus is only on usability.
	HE4EH (25 guidelines) [6]	Select some checklists that cover the feature of self-monitoring in general and discard checklists that cannot be generalized to all chronic diseases.
	Telemedicine mobile application design recommendations [13]	All recommendations are selected.
	Elderly people guidelines (13 guidelines) [34]	Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the mobile launchers.
	Intellectual disabilities guidelines (31 guidelines) [35]	Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the eLearning systems.

Table 3. Matching among features, attributes and guidelines.

No.	Features	Attributes	Nielsen	Xcertia
F1	Booking appointment [16]	Efficiency and effectiveness	NH1	XU5, XU6, UX8
F2	Consultation with HCP via text, voice messages and video calls [14,16]	Efficiency and effectiveness	NH1	XU5
F3	Sharing data with HCP [14,15]	Memorability and efficiency	NH1, NH6	XU5
F4	Access to personal health information [15]	Memorability, error, efficiency and effectiveness	NH1, NH5, NH6, NH9	XU4, XU5, XU8
F5	Self-monitoring of chronic disease [14,15]	Learnability, cognitive load, efficiency and effectiveness	NH1, NH6	XU6, XU8
F6	Allowing uploading and viewing of biometric measurements [14]	Memorability, error, efficiency and effectiveness	NH1, NH3, NH6	XU3, XU6, XU8
F7	Graphic display of patient’s information [14]	Memorability, satisfaction, effectiveness and efficiency	NH2, NH4, NH6, NH8	XU1, XU2, XU8
F8	Set health goals and treatment plan [14,15]	Satisfaction, learnability, cognitive load, effectiveness	NH1, NH2	XU2
F9	Track health progress [15]	Memorability, efficiency and cognitive load	NH1	XU5, XU6, XU8
F10	Reminders and notifications [14]	Memory and cognitive load	NH6	XU6
F11	Health tips and motivation [14,15]	Satisfaction and effectiveness	NH2	XU2
F12	Sharing health data with friends [15]	Satisfaction and learnability	-	-
F13	Interactive prompts [14]	Learnability	NH10	XU6, XU7
F14	Earn rewards [15]	Satisfaction	-	-
F15	Bluetooth technology connection [14]	Efficiency and effectiveness	NH1	XU5
F16	Accessibility to elderly people	Satisfaction, efficiency, effectiveness and memorability	-	XU9
F17	Accessibility by people with pecial needs	Satisfaction, efficiency, effectiveness and memorability	-	XU9

Table 4. Guideline selection process (Nielsen) [11].

ID	Guideline Name	Action	Covered Features	Applicability
NH1	Visibility of system status	Adapt *	F1–F6, F8, F9, F15	(3) Critical
NH2	Match between system and the real world	Adapt *	F2, F7, F8, F11	(2) Important
NH3	User control and freedom	Adapt *	F1, F4, F5, F6	(2) Important
NH4	Consistency and standards	Adapt *	F7, F11	(2) Important
NH5	Error prevention	Adapt *	F5, F6	(3) Critical
NH6	Recognition rather than recall	Adapt *	F4, F5	(2) Important
NH7	Flexibility and efficiency of use	Adapt *	F3, F4, F5, F8	(1) Useful
NH8	Aesthetic and minimalist design	Adapt *	F4, F7	(1) Useful
NH9	Help users recognize, diagnose and recover from errors	Adapt *	F1, F4	(3) Critical
NH10	Help and documentation	Adapt *	F13	(2) Important

* Add more checklist items related to mHealth applications from different studies.

Table 5. Guideline selection process (Xcertia) [12].

ID	Guideline Name	Action	Covered Features	Applicability
XU1	Visual design	Eliminate (the checklists were moved to Nielsen’s NH4, NH5 and NH6 guidelines)	F7	-
XU2	Readability	Eliminate (the checklists were moved to Nielsen’s NH8 guideline)	F7, F8, F11	-
XU3	App navigation	Eliminate (the checklists were moved to Nielsen’s NH3 guideline)	F6	-
XU4	Onboarding	Adapt *	F4	(2) Important
XU5	App feedback	Eliminate (the checklists were moved to Nielsen’s NH1 guideline)	F1–F4, F9, F15	-
XU6	Notifications, alerts and alarms	Adapt *	F1, F5, F6, F9, F10, F13	(3) Critical
XU7	Help resource and troubleshooting	Eliminate (the checklists were moved to Nielsen’s NH10 guideline)	F13	-
XU8	Historical data	Adapt *	F1, F2, F4–F7, F9	(2) Important
XU9	Accessibility	Adapt *	F16, F17	(3) Critical
XU10	Ongoing app evaluation	Keep (no change for this guideline)	F1–F17	(2) Important

* Add more checklist items related to mHealth applications from different studies.

Table 6. Guideline selection process (new created guidelines).

ID	Guideline Name	Action	Covered Features	Applicability
N1	Interactivity and Motivations	Create (to cover F12 and F14 features since they are covered neither in Nielsen’s nor in the Xcertia guidelines.)	F12, F14	(1) Useful

Table 7. The set of proposed usability guidelines for mHealth applications (UGmHA).

ID	Guideline Name
MHP1	Visibility of system status
MHP2	Match between system and the real world
MHP3	User control and freedom
MHP4	Consistency and standards
MHP5	Error prevention
MHP6	Recognition rather than recall
MHP7	Flexibility and efficiency of use
MHP8	Aesthetic and minimalist design
MHP9	Error diagnosis and recovery
MHP10	Help and documentation
MHP11	Notifications, alerts and alarms
MHP12	Onboarding
MHP13	Historical data
MHP14	Accessibility
MHP15	Ongoing app evaluation
MHP16	Interactivity and motivations

Table 8. Details of the “Sehhaty” and “Sokry” applications.

	Sehhaty	Sokry
Government/ Private	A government mHealth application that belongs to the Ministry of Health (MOH) in Saudi Arabia.	A private mHealth application.
Functionality	Provides different services, such as booking appointments, children’s vaccines, immediate consultations to make video and audio calls with doctors, COVID-19 services and reviewing health data, including insurance and registered information in MOH.	Helps users manage their diabetes by recording blood sugar readings, meals, exercises and medications. In addition, provides health advice related to diabetes.
Popularity	Popular with the people since it is connected with health records of MOH and considered an essential application for people in Saudi Arabia (+10 million downloads) [49].	Less popular to the people (+50,000 downloads) [50].

Table 9. Severity Ranking Scale (SRS) [51].

Rating	Description
0	I do not agree that this is a problem at all.
1	Cosmetic problem only. Need not be fixed unless extra time is available in the project.
2	Minor usability problem. Fixing this should be given low priority.
3	Major usability problem. Important to fix so it should be given high priority.
4	Usability catastrophes. Imperative to fix this before the product can be released.

Table 10. Descriptive elements for the UGmHA guidelines.

ID	Guideline ID
Priority	(3) Critical, (2) Important or (1) Useful.
Name	Name of the guideline that resulted from the integration of Nielsen [11] and Xcertia [12].
Definition	Identify the guideline and its purpose.
Features	The selected features covered by the guideline.
Checklists	The checklist items selected from [6,12,13,34,35] to add more details to the guideline and to make it more related to the features of mHealth applications and accessible to the elderly and people with special needs.

Table 11. Numbers of usability issues of both applications based on the three guidelines.

	Sehhaty	Sokry	Total
UGmHA	73	95	168
Xcertia	22	28	50
Nielsen	17	25	42

Table 12. Severity of usability issues of the “Sehhaty” and “Sokry” applications.

	Sehhaty			Sokry
Rating	UGmHA	Xcertia	Nielsen	UGmHA	Xcertia	Nielsen
No Problem	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Cosmetic	7 (10%)	2 (9%)	0 (0%)	6 (6%)	2 (7%)	0 (0%)
Minor	10 (14%)	1 (5%)	0 (0%)	17 (18%)	4 (14%)	2 (8%)
Major	17 (23%)	6 (27%)	12 (71%)	26 (27%)	10 (36%)	9 (36%)
Catastrophic	39 (53%)	13 (59%)	5 (29%)	46 (48%)	12 (43%)	14 (56%)
Total	73	22	17	95	28	25

Table 13. Relation of the UGmHA guidelines with Nielsen and Xcertia.

ID	UGmHA	Corresponding Guideline (Nielsen)	Corresponding Guideline (Xcertia)
MHP1	Visibility of system status	Visibility of system status	App feedback
MHP2	Match between system and the real world	Match between system and the real world	-
MHP3	User control and freedom	User control and freedom	App navigation
MHP4	Consistency and standards	Consistency and standards	Visual design
MHP5	Error prevention	Error prevention	Visual design
MHP6	Recognition rather than recall	Recognition rather than recall	Visual design
MHP7	Flexibility and efficiency of use	Flexibility and efficiency of use	-
MHP8	Aesthetic and minimalist design	Aesthetic and minimalist design	Readability
MHP9	Error diagnosis and recovery	Error diagnosis and recovery	-
MHP10	Help and documentation	Help and documentation	Help resource and troubleshooting
MHP11	Notifications, alerts and alarms	-	Notifications, alerts and alarms
MHP12	Onboarding	-	Onboarding
MHP13	Historical data	-	Historical data
MHP14	Accessibility	-	Accessibility
MHP15	Ongoing app evaluation	-	Ongoing app evaluation
MHP16	Interactivity and Motivations	-	-

Table 14. Guidelines and checklist items of UGmHA, Xcertia and Nielsen.

	UGmHA	Xcertia	Nielsen
Number of guidelines	16	10	10
Number of checklist items	154	60	0
Number of mHealth-specific checklist items	44	6	0
Number of accessibility checklist items	18	7	0

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Nasr, E.; Alsaggaf, W.; Sinnari, D. Developing Usability Guidelines for mHealth Applications (UGmHA). Multimodal Technol. Interact. 2023, 7, 26. https://doi.org/10.3390/mti7030026

AMA Style

Nasr E, Alsaggaf W, Sinnari D. Developing Usability Guidelines for mHealth Applications (UGmHA). Multimodal Technologies and Interaction. 2023; 7(3):26. https://doi.org/10.3390/mti7030026

Chicago/Turabian Style

Nasr, Eman, Wafaa Alsaggaf, and Doaa Sinnari. 2023. "Developing Usability Guidelines for mHealth Applications (UGmHA)" Multimodal Technologies and Interaction 7, no. 3: 26. https://doi.org/10.3390/mti7030026

APA Style

Nasr, E., Alsaggaf, W., & Sinnari, D. (2023). Developing Usability Guidelines for mHealth Applications (UGmHA). Multimodal Technologies and Interaction, 7(3), 26. https://doi.org/10.3390/mti7030026

Article Menu

Developing Usability Guidelines for mHealth Applications (UGmHA)

Abstract

1. Introduction

2. Literature Review

2.1. Electronic Health (eHealth) and Mobile Health (mHealth)

2.2. Usability and Guidelines

2.3. Common Approaches for Developing New Usability Guidelines

2.4. Common Methods for Validating New Usability Guidelines

3. Materials and Methods

3.1. Developing UGmHA

3.1.1. Stage 1: Exploratory Stage

3.1.2. Stage 2: Descriptive Stage

3.1.3. Stage 3: Correlational Stage

3.1.4. Stage 4: Selection Stage

3.1.5. Stage 5: Specification Stage

3.2. Validating UGmHA

3.2.1. Stage 6: Validation Stage

3.2.2. Stage 7: Refining Stage

4. Results

4.1. Results of UGmHA Development

4.2. Results of the UGmHA Validation

4.2.1. Number of Usability Issues among the Three Guidelines

4.2.2. Severity of Usability Issues among the Three Guidelines

5. Discussion

5.1. UGmHA Development Discussion

5.2. UGmHA Validation Discussion

6. Conclusions and Future Work

Author Contributions

Funding

Institutional Review Board Statement

Informed Consent Statement

Data Availability Statement

Conflicts of Interest

Abbreviations

Appendix A. Final Usability Guidelines for mHealth Applications (UGmHA)

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI