TBM PRACTICE TOOL

Evaluating and selecting mobile health apps: strategies

for healthcare providers and healthcare organizations
Edwin D Boudreaux, PhD,1 Molly E Waring, PhD,2 Rashelle B Hayes, PhD,3 Rajani S Sadasivam, PhD,4
Sean Mullen, PhD,5 Sherry Pagoto, PhD

1
Departments of Emergency Abstract
Medicine, Psychiatry, and Mobile applications (apps) to improve health are Implications
Quantitative Health Sciences, proliferating, but before healthcare providers or Researchers: Prior to designing and testing new
University of Massachusetts Medical mobile health (mHealth) applications (apps), re-
School, LA-189, 55 Lake Avenue organizations can recommend an app to the patients they
North, Worcester, MA 01655, USA serve, they need to be confident the app will be user-
searchers should systematically evaluate the apps
2
Division of Epidemiology of Chronic
that already exist in their target domain.
friendly and helpful for the target disease or behavior.
Diseases and Vulnerable
This paper summarizes seven strategies for evaluating
Populations, Department of
Quantitative Health Sciences, and selecting health-related apps: (1) Review the
University of Massachusetts Medical
Practitioners: mHealth holds tremendous poten-
scientific literature, (2) Search app clearinghouse
School, Worcester, MA, USA tial for improving patient health; however,
3
websites, (3) Search app stores, (4) Review app healthcare providers and healthcare organizations
Division of Preventive and
Behavioral Medicine, Department of
descriptions, user ratings, and reviews, (5) Conduct a will need to standardize their identification, evalu-
Medicine, social media query within professional and, if available, ation, and selection of health related apps to max-
University of Massachusetts Medical patient networks, (6) Pilot the apps, and (7) Elicit imize their utility, safety, and impact.
School, Worcester, MA, USA
4
feedback from patients. The paper concludes with an
Division of Health Informatics and
Implementation Science, Department illustrative case example. Because of the enormous range
of Quantitative Health Sciences, of quality among apps, strategies for evaluating them will Policymakers: Policymakers and influential
University of Massachusetts Medical be necessary for adoption to occur in a way that aligns healthcare organizations should consider evaluat-
School, Worcester, MA, USA
5 with core values in healthcare, such as the Hippocratic ing and vetting health related apps using standard-
Department of Kinesiology &
principles of nonmaleficence and beneficence. ized evaluation criteria, such as those used by the
Community Health,
University of Illinois at Urbana- UK’s National Health Service (http://apps.nhs.uk).
Champaign, Urbana, IL, USA
Correspondence to: E Boudreaux Keywords
edwin.boudreaux@umassmed.edu technology and features, makes it difficult for con-
Mobile health, e-health, Application, Health sumers to keep abreast of which apps are available
Cite this as: TBM 2014;4:363–371 promotion, Health behavior
doi: 10.1007/s13142-014-0293-9 for use by patients. While the first priority in the field
should be creating a primary evidence base by study-
INTRODUCTION ing apps in well-controlled studies, the pace of science
The use of mobile technologies to track and improve is incongruent with that of the business sector and
health outcomes, referred to as mHealth, is a rapidly consumer demand. Healthcare providers may want
expanding practice [1]. The advent of smartphone to stay informed due to interest in recommending
technologies that enable quick and easy access, trans- apps as part of their treatment planning [5], and
fer, and tracking of information, as well as interactive healthcare systems, health promotion organizations,
displays and interventions that can be highly engag- and insurance providers (hereafter referred to as
ing, have promoted adoption. In 2012, 85 % of adults healthcare organizations collectively) may want to
in the USA owned a mobile phone, of which 53 % provide recommendations regarding health apps as
were smartphones [2]. Additionally, nearly one-fifth part of their health promotion services. However,
(19 %) of US adults who owned a smartphone had at guidance on how to judge the validity or worth of
least one application (app) designed to promote health commercially available apps is lacking [6]. The deci-
behavior or health maintenance (health app) on their sion to recommend an app to a patient can have
phone, with the majority being related to diet and serious consequences if its content is inaccurate or if
physical activity. By 2017, 50 % of mobile phone users the app is ineffective or even harmful. For example, a
are projected to have downloaded at least one health recent study found that a mobile app claiming to
app [3]. provide diagnostic recommendations for suspected
As of May 2012, more than 40,000 health apps melanoma had very low sensitivity and was therefore
existed in the USA, and the rate of proliferation is likely to miss many melanomas [7]. Use of this app
nearly exponential [4]. This incredible volume of had the potential to delay diagnosis and treatment for
apps, combined with the rapid evolution of a condition in which early detection has a significant
TBM page 363 of 371
 PRACTICE TOOL

impact on survival rates. Another recent study showed Elicit feedback from patients. These proposed strate-
that only 37 % of physicians recruited from an online gies are summarized in Table 1.
social network prescribed health apps to their patients,
42 % said they would not prescribe apps without
regulatory oversight, and 37 % said they have no idea Review the scientific literature
what apps are available [8]. Thus, healthcare providers
Systematic reviews of health apps are increasingly
and healthcare organizations are in a quandary: in-
appearing in the literature. Most reviews evaluate
creasingly, patients are using the countless existing
whether apps for a particular condition or health be-
health apps, but providers and organizations are hesi-
havior are rooted in evidence-based strategies or the-
tant to act because these apps are shrouded in quality
oretical models of behavior change. Examples include
and validity concerns, and they do not know which
reviews of apps for weight loss [10, 11], cancer preven-
ones to recommend.
tion and management [13], mental health [14], medi-
This paper outlines a practical guide for healthcare
cation self-management [15], chronic medical condi-
providers and healthcare organizations on the evalua-
tions [16, 17], HIV prevention [18], sports injury pre-
tion and selection of publically available apps targeting
vention [19], and smoking cessation [20]. Reviews are
health behavior or health maintenance. It will focus
likely to reveal the capabilities and limitations of apps
only on apps designed for use by patients to promote
and this information can be incorporated into clinical
patient health and not on those that provide information
decisions and patient recommendations. The same
directly to healthcare providers, such as apps designed
criteria used to judge other systematic reviews should
to avoid medication errors or for medical education. In
be used to evaluate the worth of app review papers,
addition, it will not provide information on evaluating
including the comprehensiveness of the search criteria,
medical apps regulated by the Food and Drug Admin-
the rigor of the evaluation methods, and the journal’s
istration (FDA) [9]. Apps regulated by the FDA include
peer-review process. In the absence of reviews, pro-
only those that are intended as an accessory to a regu-
viders can search for randomized trials, which are also
lated medical device or that transform a mobile plat-
increasingly common. Search terms “randomized tri-
form into a regulated medical device, such as an app
al,” “mobile app*,” and the name of the condition or
intended to diagnose cardiac arrhythmias. Most health-
health behavior (e.g., tobacco use) should produce the
related apps commonly available in apps stores do not
available literature for that condition.
fit this definition and are therefore not under the pur-
The main weaknesses with reviewing the scientific
view of the FDA. The paper is divided into two sections.
literature are: (1) despite the increasing attention, there
The first section proposes practical search and selection
remains only a very small published literature on health
strategies, while the second provides an illustrative ex-
apps, (2) it can be time consuming to sift through and
ample using a case study of a patient needing dietary
digest the literature, (3) evaluations are often focused on
assistance to manage celiac disease.
one operating system (iPhone or Android), and (4) the
literature becomes outdated quickly because of the
constantly evolving market. In addition, some reviews
do not list the names of the specific apps, and when
PRACTICAL SEARCH AND SELECTION STRATEGIES
moving from development to market, apps often
Currently, little or no quality control or regulations change names, making it difficult to know exactly what
exist to ensure health apps are user-friendly, accurate app to recommend. Contacting the corresponding au-
in content, evidence-based, or efficacious. For exam- thors of review papers may be one way to find out the
ple, a search using the term “weight loss” on iTunes apps included in the review and the ones they recom-
(January 15, 2014) revealed more than 3800 iPhone mend based on their review, but this is time consuming
apps alone. Reviews of commercially available weight and requests are not always acknowledged.
loss or weight management apps have found that most
reflect only a narrow range of evidence-based behav-
ioral strategies (e.g., tracking and goal setting) and Search app clearinghouse websites
some include strategies not rooted in any evidence Because of the difficulties inherent in searching the
base [10–12]. The following seven strategies balance evidence, notable efforts by public and private organi-
the ease and efficiency of the search process against the zations have begun to help organize, review, and “cer-
need to understand the accuracy, evidence base, and tify” health apps. The premise of “certifying” apps is
efficacy of the app. While they could be viewed as similar to existing web certification, such as the Health
sequential steps, they are not designed as such to on the Net Foundation Code of Conduct (http://
promote flexibility and to acknowledge that search www.healthonnet.org) and the Information Standard
strategies might depend upon the purpose and base- (http://www.theinformationstadard.org), where the
line knowledge of the consumer. These strategies are: credibility and reliability of health information is eval-
(1) Review the scientific literature, (2) Search app uated against predetermined standards. Ultimately, the
clearinghouse websites, (3) Search app stores, (4) Re- goal of app clearinghouses is to help consumers,
view app descriptions, user ratings, and reviews, (5) healthcare providers, and healthcare organizations feel
Conduct a social media query in professional and, if confident about their app selections by evaluating the
available, patient networks, (6) Pilot the apps, and (7) usability, functionality, accuracy of the content, or
page 364 of 371 TBM
 PRACTICE TOOL

Table 1 | Summary of strategies

1. Review the scientific literature: Search the scientific literature for papers reviewing apps in a content domain or strong
clinical trials
2. Search app clearinghouse websites: Clearinghouses that review apps can help with identifying strengths and
weaknesses
3. Search app stores: App stores are challenging to navigate, so it is important to fine-tune and filter app searches with the
most relevant and targeted key words, including words keyed to the pathological state or target behavior
4. Review app descriptions, user ratings, and reviews: Publicized ratings and user reviews can offer evidence of app
usability, functionality, and efficacy, which can help to narrow the pool of candidate apps
5. Conduct a social media query within professional and, if available, patient networks: Social networks may reveal new
app trends, likability by certain user groups, and other substantive data
6. Pilot test the app: Apps may be piloted by the healthcare provider or a designee, including examinations of functionality,
accuracy of content, and usability
7. Elicit feedback from patients: Patients may be able to provide valuable insights after they have used the app a provider
recommends

evidence base supporting the app. Because of the lack Apps Library (http://apps.nhs.uk), launched in March
of FDA regulation of most apps, and the burden asso- 2013, aims to provide a library of apps endorsed by the
ciated with some of the other strategies we outline, NHS. The library started with 70 apps and 1 year later
such as reviewing the medical literature, clearing- only had about 100 apps, which is few compared to the
houses hold much promise in helping providers and thousands of apps that are commercially available. The
organizations select apps to recommend to their evaluation process can also be subject to error.
patients. Happtique (www.happtique.com), a mobile health ap-
Table 2 describes several existing health app plication store, recently suspended their app certifica-
clearinghouses, including the website, a general tion program when independent groups found security
description, and, importantly, their review stan- flaws in apps that Happtique had “certified” as secure
dards and who conducts the evaluation. The infor- [21]. Finally, other app clearinghouses primarily focus
mation in Table 2 was abstracted from each clear- on a particular health topic, like EatRight (Table 2).
inghouse’s website as of July 2014. We have also This clearinghouse may be quite useful for identifying
included points to consider when using clearing- an app to help patients lose weight but does not provide
houses to select health apps to recommend to their suggestions for apps to manage stress or help patients
patients. The clearinghouses and websites listed quit smoking. Still others are targeted primarily to the
were those known to us authors and/or discovered healthcare provider or organization only (e.g., IMS
by searching Google and PUBMED (e.g. “mhealth www.imshealth.com; Table 2) and are not readily avail-
app”; “mhealth applications”; “mhealth”; “mhealth able to patients. Recently, academic institutions or other
c l e a r i n g h o u s e s ” ) . We a l s o r e v i e w e d nonprofit organizations have begun to offer listings of
mobihealthnews (http://mobihealthnews.com/). mhealth resources (e.g., http://www.zurinstitute.com/
Given that these searches were conducted in mentalhealthapps_resources.html) or apps for particu-
2014, our list of clearinghouses may not be exhaus- lar behaviors or health conditions (e.g., http://
tive at the time of this publication as mhealth and diabetes.ufl.edu/my-diabetes/diabetes-resources/dia-
clearinghouse development is constantly growing. betes-apps/). However, many of these listings may lack
Nevertheless, we encourage providers and organi- systematic independent reviews; a description of their
zations to consider the elements listed in Table 2 search and vetting processes are unclear, as well as their
when searching any app clearinghouse not de- process for updating their lists. Nonetheless, these list-
scribed here. ings may be helpful for providers when generating an
Advantages of using clearinghouses include efficien- initial pool of candidate apps to consider for further
cy, with numerous apps summarized in one place, ac- evaluation. In summary, providers and organizations
cess to systematic evaluations, and, in some cases, con- should become familiar with clearinghouses, including
tinual updating as new apps become available. Howev- their strengths and limitations. Given the differing pur-
er, the use of clearinghouses also has limitations. The poses, review standards, reviewers, and strengths and
quality of the app reviews is dependent on the evalua- weaknesses of each clearinghouse, providers and orga-
tion methods used by each clearinghouse. Reviewing nizations are encouraged to search multiple clearing-
apps can be resource intensive and time consuming, so houses for candidate apps.
the app listing may not reflect the range of available
apps on the market at a given time. There is an inherent
tradeoff between the thoroughness of review, timeliness Search app stores
of updates, and number of apps reviewed, and different If the scientific literature or clearinghouses do not list
clearinghouses may yield more useful recommenda- good-fit apps for a specific need, app stores can be
tions when searching for different types of apps. For searched directly. App stores are specific to the mobile
example, the UK’s National Health Service (NHS) device platform, with Apple apps available from
TBM page 365 of 371
page 366 of 371
PRACTICE TOOL

Table 2 | Available clearinghouses for selection of health mobile apps: description and review standards

Clearinghouse (website) Target audience Description Review standards Who reviews Considerations
National Health Service (NHS) UK (citizens, providers, Offers a library of health apps for All submitted apps are evaluated Reviewed by a NHS The Health Apps library was launched
Health Apps Library (http:// organizations) patients and aims to provide to make sure they are: (1) Clinical team that consists of doctors, in March 2013 and is still in
apps.nhs.uk) quality assurance to ensure that relevant to those living in nurses, and safety specialists. development, therefore, there are
apps are clinically safe. England. (2) comply with data Consumers are able to rate and review relatively few apps that have been
Apps are organized into listings by protection laws, and (3) comply apps on website. reviewed. Apps are submitted by
Conditions (e.g., diabetes, with trusted information sources. app developers and may not
insomnia), Healthy Living (e.g., If these minimum requirements are include all relevant apps.
alcohol, fitness, weight loss), met, NHS evaluates if a person’s The review process focuses on the
Health Information (e.g., symptom health or condition could be harmed. clinical safety of app; evidence base
checker), and Social Care (e.g. A clinical team from NHS works with for promoting or managing health is
service finder). the app developer to make sure the not applied.
Each app’s information is app adheres to safety standards.
presented by screenshots, a brief Apps are also reviewed on an ongoing
description from the developer, basis and/or if users flag concerns.
and consumer reviews. Consumers
can also search for apps by phone
platform and costs.
Happtique (www.happtique.com) USA—patients, physicians, and A mobile health application store Includes apps that have been The content of apps are evaluated by Provides an Engagement Score
medical organizations and app management solution developed by a source Happtique internal partners and in indicating an app’s potential to
that allows organizations to considers reputable, including partnership with medical compel consistent use. However, it
create individually branded, professional associations, professional societies. is unclear how the Engagement
secure, multi-platform universities/colleges, nonprofit Technical aspects of evaluation are Score is calculated, as the process
application stores for staff and associations, insurance managed by a 3rd party developer. and algorithm is proprietary.
patients. companies, federal government, state The company does not evaluate
Allows physicians to send their government, or pharmaceutical effectiveness or safety of the app for
patients a “digital health industry. Provides an Engagement its intended use.
prescription” with the apps, Score reflective of content validity,
documents, and videos of their persuasion techniques, and
choice. Initially offered an app psychosocial factors.
certification program based on a
19-page document specifying
specific certification standards, but
this was discontinued.
iMedicalApps USA—physician, patients, and This is an independent online Reviews are based on the reviewers’ Reviewers are 4 editors and 5 writers Reviews appear to be based on the
(www.imedicalapps.com) mHealth analysts medical publication. Its goal is own hospital and clinical all of whom are physicians, opinions of the reviewer only. App
to provide reviews, research, experiences. It is unknown physicians-in-training, allied reviews are categorized by medical
and commentary of mobile whether each reviewer has similar health professionals, or mHealth specialty, and are primarily
medical technology. specific standards when reviewing, analysts. focused on medical education.
Consumers can read app reviews though many reviews will include Still, it is difficult to find apps for
categorized by medical specialty an overall summary that may health behavior change (e.g.,
and phone platform. consist of the following: app price, weight loss, smoking cessation,
likes and dislikes, user interface, fitness/physical activity).
multimedia usage, and overall May be beneficial to know the name
score. of the app before using their search
tool.
Website is hard to navigate and has
undergone several changes since its

TBM
launch.
TBM
Eat Right (www.eatright.org/ Worldwide Sponsored by the world’s largest Reviewers write a brief evaluation The website specifically names 3 Apps are limited to those regarding
appreviews) organization of food and based on (1) overall synopsis of reviewers all registered dieticians. weight loss, diabetes
nutrition professionals, the app, (2) pros and cons, and the management, or gluten-free diets
Academy of Nutrition and (3) bottom line. Reviewers also rate only.
Dietetics apps from 1 to 5 stars. Apps are limited to iPhone or iPhone
Reviewed apps are described as compatible platforms; not useful for
top-rated, free iPhone apps for patients with Android phones or
those looking to lose weight, tablets.
manage their diabetes, or eat
gluten-free.
IMS Health’s AppScript Customers include pharmaceutical, IMS Health is an information, services, and More than 40,000 IMS With regard to wellness, prevention, and
(http://www.imshealth.com/portal/ medical device, and consumer health technology company. healthcare apps from iOS and treatment regiments, physicians can
site/imshealth/menuitem. manufacturers and distributors, Offers IMS Health AppScript, Android platforms, are assessed organize apps into formularies based
3e17c48750a3d98f53 providers, payers, government an electronic prescribing using their proprietary IMS Health on their patient’s needs.
c753c71ad8c22a/?vgnextoid= agencies, policymakers, researchers, service for apps. App Score. IMS Health App Scores are not readily
123ee61fec4b2410Vgn and the financial community. Physicians are able to securely prescribe, The institute published a 65 page available to the consumer and do not
VCM10000076192ca2R reconcile, and track patient app use report that provides the results consider physician evaluation or
CRD&vgnextfmt=default) from a mobile device. of an extensive content analysis efficacy evidence.
of all apps as of June 2013.
The overall IMS Health App
Score is based on a combination
of (1) functionality, (2) peer and
patient reviews, (3) certifications,
and (4) their potential to improve
outcomes and lower the cost of care.
Scoring criteria include, but are not
limited to, the type and quantity of
information provided by the app,
how the app tracks or captures user
data, the communication processes
utilized by the app, and the quantity
of device capabilities included in
the app. In addition, weightings are
applied specific to the category of
the app. For example, in a condition
management app there will be a higher
value placed on results that focus on
doctor communications than there
would be for a reference/education app.
HealthTap’s AppRx USA—primarily for patients/consumers This is a specific function of the larger Doctors in the network review the apps Physicians from a network of >60,000 Have to download free HealthTap app
(https://www.healthtap.com/) HealthTap app. The AppRx function based on 3 questions: (1) Is the app provide reviews, but it is unclear who and then create an account and log in
allows consumers to read health and medically sound?, (2) Is the app are the physicians and if they all in order to access app reviews.
medical app evaluation/ useful?, and (3) Is the app easy to use provide a similar review structure.
recommendations from a network of and understand?
60,000+ physicians. Doctors have the option to write a review,
but reviews do not enter the system
until 30 reviews have been made and
the HealthTap medical review board
has approved them.
Patients are also able to recommend the
app using an “applaud an app” function.
PRACTICE TOOL

page 367 of 371

 PRACTICE TOOL

iTunes and Android apps available from Google Play. star and five-star ratings). In the latter case, informa-
Unfortunately, no method exists for searching all app tion contained in reviews may indicate whether these
stores together. The primary advantage of this strategy low ratings were due to a glitch-prone update, which
is its ease and relevance. It is the only way to identify can be fixed, or truly polarized opinions of the app,
the apps that are currently available to patients. As which cannot be fixed. Changes in the distribution of
with other strategies, there are limitations. An impor- ratings over time may indicate changes in functionality
tant limitation is the lack of systematic evaluation of or interface over time, such as solving glitches or, in
the apps, so the accuracy and evidence base are diffi- contrast, an unpopular update. One can start with the
cult to evaluate. Complicating the evaluation process, most popular apps in the health behavior domain, as
app stores rely on ranking algorithms such that apps these apps are likely to be already used by some of
that appear first are often those that are more popular. patients, and represent apps with longevity.
Thus, apps designed for smaller audiences, such as for Healthcare providers and healthcare organizations
specific diseases or behaviors that may be less com- may need to consider price; patients may be more
mon in the population, or those that are new to the willing to try a free app than an app that costs money.
market will be harder to find. Lower ranks in the app A recent review of weight loss mobile apps showed
lists should not be viewed as synonymous with poor that paid apps did not include more evidence-based
quality. strategies than free apps [10]. While reviewing app
Search terms can be important when searching app descriptions, user ratings, and reviews has the benefit
stores. Entering a term that is too broad likely will of convenience and ease, because the information is
result in much irrelevant information. For example, readily obtained and easily understood, the primary
entering “alcohol” as a search term will lead to drink- disadvantage associated with this strategy is that it does
ing games along with apps designed to promote absti- not consider the evidence base, validity, or accuracy of
nence [22]. Search terms aligned with the pathological the app. In addition, even though the information is
state or disorder, or with the goal state, such as “alco- easily obtained, reviewing this information can be time
holism” or “alcohol abstinence,” are likely to reduce consuming.
this outcome. However, even highly specific search
terms can yield apps that are not a good fit because
Conduct a social media query within professional
of the way apps are indexed, which is often determined
and, if available, patient, networks
by the app developer, and will include a broad list of
terms to catch a broader consumer base. The term Although, as mentioned above, app clearinghouses
“alcohol abstinence,” for example, yields apps provide structured evaluations and recommendations
targeting smoking cessation as a result of the vagaries from providers and other regarded experts, healthcare
of indexing. Consequently, the initial resulting pool providers and healthcare organizations may be able to
from these searches may need to be refined through leverage their professional social media networks to
additional search terms. learn what their peers are recommending. Social me-
dia is increasingly used by healthcare providers to
connect with colleagues. About 25 % of physicians
use social media daily as part of their clinical practice
Review app descriptions, user ratings, and reviews
[23]. A large medical community is represented on
Any search strategy is likely to reveal more than one Twitter and hashtags used to identify these communi-
app that targets the behavior of interest. To narrow ties are archived at Symplur.com [24]. Physician-
apps to one or two “best of breed,” the content of the specific social networks like Sermo (www.sermo.com)
resulting apps can be reviewed by reading its brief might also be used to query colleagues about which
description in the app store, and, in some cases, one apps they recommend to patients or elicit feedback
can test drive the app, such as many of those found about a specific app. Also, providers can search online
through Amazon.com. In addition, while user ratings social networks that patients use (e.g., Twitter) with the
are generally not focused on the accuracy of the infor- app hashtag as a keyword (e.g., #myfitnesspal) to ob-
mation or the evidence base supporting the app, they serve conversations about the app. This will reveal if
can be helpful in determining usability and function- users are reporting generally positive or negative ex-
ality. Rating information generally includes the total periences and give a sense of whether the app is used
number and average rating, typically as a number of widely. They can tweet any users they discover and ask
stars ranging from 0 to 5. The number of ratings for experiences using the app. This approach, may,
provides a relative measure of popularity and/or lon- however, oversample patients with positive experi-
gevity of the app. This may be more important when a ences with a particular app, and may fail to elicit
patient is considering paying for an app, as users may experiences that resulted in the patient no longer using
feel more comfortable purchasing a highly rated app the app for health promotion.
that has been rated numerous times. In addition to the
average rating, the distribution of ratings, which is
typically presented, may also be useful. For example, Pilot the apps
an average rating of three stars might indicate a medi- Pilot testing, or using an app to assess its features,
ocre app (mostly three-star ratings), or it might repre- functionality, usability, and content accuracy [25], can
sent an app with highly polarized ratings (many one- be completed by a healthcare provider or other
page 368 of 371 TBM
 PRACTICE TOOL

designee, such as health informatics personnel. While identified the highest rated app on this website. Be-
piloting an app may seem laborious, many apps can be cause EatRight.org has individual dietitians rate apps
evaluated with only a few minutes of navigation, and it based on a scoring system with no stated criteria, she
rarely takes more than 1 day of using an app before the dec ided t o cr os s check w ith iMedi calapp s
major pros and cons are observed [26]. Providers (www.imedicalapps.com), but this app is not in the
should evaluate the candidate health app for the accu- iMedicalapps database. Her next strategy was to ex-
racy of the information provided, as well as the usabil- amine the user ratings in iTunes. Of 212 reviews, the
ity and inclusion of desired functionality. Desired func- average user rating of the app identified at
tionality depends not only on the health problem or EatRight.org is 1.5 out of 5 stars. She is concerned
behavior (e.g., weight loss, smoking cessation) but also about such a low score and is now reticent to recom-
how the provider intends the patient to use the app. mend this to her patients. The next highly rated app on
For example, an app for electronically tracking dietary EatRight.org is not in iTunes and the third highly rated
intake as a replacement for paper records requires app has no user ratings on iTunes. She has two choices,
different functionality than an app that provides more to select the fourth highly rated app on Eatright.org or
comprehensive assistance with weight loss, which also proceed to investigate the most highly rated app fur-
generally entails increasing caloric expenditure. ther. She chose the latter.
Dr. Benson decided to tap her online social network.
With hundreds of followers on Twitter, many of whom
Elicit feedback from patients are physicians with specialty treating celiac disease,
Once a provider has recommended an app to a patient patients with celiac disease, and professional organiza-
or discovered that a patient is using an app to manage tions dealing with celiac disease, she sent a tweet asking
their health, follow-up is an important step. The pro- for recommendations and for experiences using this
vider should review the app with the patient and elicit app. She performed a search on Twitter of the app
feedback on its usefulness. Because individuals vary in name to see the conversations about this app. Her social
their ability to evaluate an app, and different patients media search revealed positive experiences and recom-
may find a particular app more or less useful depending mendations of the app, which was reassuring.
on the exact nature of their health condition, behavior, Dr. Benson decided to use the app for a day herself
and personal factors, we recommend that providers to evaluate its usability. She discovered no significant
elicit feedback evaluations from multiple patients to usability issues and the content was factually accurate
increase confidence of an app’s potential benefit. Be- so she recommended her patient pilot test it herself.
cause the time spent with an individual patient is limit- She showed the patient how to find the app and sug-
ed, questions about the app’s utility will need to be gested that it might be helpful in managing her gluten-
maximally informative, such as asking whether the free diet. Dr. Benson agreed to follow-up with the
patient found the app to be useful, whether he or she patient in a week to hear the pros and cons of using
is continuing to use it, and whether the patient would the app. The patient reported a positive experience
recommend the app to others. Another more direct with the app and her symptoms related to celiac de-
indicator of success is whether the patient has success- creased over time. Dr. Benson then felt comfortable
fully changed his or her behavior or exhibits clinical recommending the app to other patients.
improvement while using the app. Providers or perhaps
larger organizations may even want to develop their
own standard clearinghouses of apps, and then accu-
mulate feedback from providers and patients as they LIMITATIONS
use the apps to guide updates of their clearinghouses.
The strategies outlined are not without limitations. The
most important limitation is that some of the strategies
may be impractical for individual providers, because
CASE STUDY they can be time consuming, especially if one ap-
Dr. Benson treats patients with celiac disease and proaches the seven strategies as a step-by-step plan.
many of them express difficulty adhering to a gluten- We encourage readers to view the strategies as a menu,
free diet. Dr. Benson heard about diet mobile apps for rather than a stepwise approach, with providers or
self-management of a gluten-free diet but had no idea if organizations selecting one or more strategies that are
these were safe and evidence-based. Table 1 summa- feasible in the context of their needs and constraints.
rizes our proposed framework for recommending Moreover, these strategies can be used by a group of
apps for Dr. Benson’s patient. Her first step was to individuals to spread the burden. For example, a task
contact a medical librarian to help her search for force or performance improvement team can be
systematic reviews and, if none, randomized clinical commissioned by a healthcare system to complete
trials for dietary apps targeting a gluten-free diet. Un- reviews of various clearinghouses and pilot the most
fortunately, no studies had evaluated such apps. She highly ranked apps for a particular health behavior or
moved to another strategy of searching clearinghouses disease. Another notable limitation includes costs.
(Table 2). She began with EatRight.org, a website from While many apps are free, some are not, which may
the Academy of Nutrition and Dietetics, since this is a hinder piloting the apps, or use by patients if such apps
reputable nutrition professional organization. She were recommended.
TBM page 369 of 371
 PRACTICE TOOL

CONCLUSION Mullen: Declares that he has no conflict of interest.

Pagoto: Dr. Pagoto is on the advisory board for Empower Fitness®, has
With thousands of health apps in the marketplace, consulted for Apple®, and receives funds to produce social media content for
healthcare providers and healthcare organizations Sears® Fitstudio.
need guidance on identifying apps that are effective,
provide accurate information, and are user-friendly
[6]. In addition, more primary research is needed to
establish evidence for health apps efficacy. This re-
1. Nilsen W, Kumar S, Shar A, et al. Advancing the science of mHealth.
search can begin with the very strategies outlined J Health Commun. 2012;17(Suppl 1):5-10.
herein to help the researcher become better 2. Fox S, Duggan M. Mobile Health 2012. Washington, DC: Pew Re-
search Center's Internet and American Life Project; 2012.
acquainted with the strengths and weaknesses of 3. Leroux E, Rivas H. Mobile health without borders, evidence-based
existing apps and to help guide the design of the next mHealth. Stanford University; 2014.
4. West DM. How mobile devices are transforming healthcare issues in
generation of health apps. The guidance in the current technology innovation. Issues Technol Innov. 2012;18:1-14.
paper is similar to steps a provider or organization 5. Wang A, An N, Lu X, Chen H, Li C, Levkoff S. A classification scheme
would use to identify evidence-based interventions of for analyzing mobile apps used to prevent and manage disease in
late life. JMIR MHealth UHealth. 2014;2(1):e6.
any kind, in that the research evidence is the first place 6. Powell AC, Landman AB, Bates DW. In search of a few good apps.
to look, and then in the absence of evidence, defaulting JAMA. Published online March 24, 2014.
7. Wolf JA, Moreau JF, Akilov O, et al. Diagnostic inaccuracy of
to other reliable sources, case studies, and recommen- smartphone applications for melanoma detection. JAMA Dermatol.
dations coming from professional networks. As the 2013;149:422-426.
evidence base for health mobile apps continues to 8. Gold A. Physicians split on use of mHealth apps
FierceMobileHealthcare. 2014.
grow, guidelines and app clearinghouses will likely 9. U.S. Food and Drug Administration. FDA issues final guidance on
be put forth by professional organizations and identi- mobile medical apps, 2013. http://www.fda.gov/downloads/
MedicalDevices/…/UCM263366.pdf. Accessed 19 August 2014.
fying high-quality health apps will become much eas- 10. Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-
ier for providers. Building better app clearinghouses based strategies in weight-loss mobile apps. Am J Prev Med.
2013;45:576-582. mHealth Eval 2014.enl Page 2.
may represent the most important direction of the 11. Breton ER, Fuemmeler BF, Abroms LC. Weight loss—there is an app
field, because it relieves some of the burden from the for that! But does it adhere to evidence-informed practices? Trans
provider. Some of the strategies we describe, such as Behav Med: Pract Policy Res. 2011;1(4):523-529.
12. Azar KMJ, Lesser LI, Laing BY, et al. Mobile applications for weight
reviewing the empirical literature to identify evalua- management: theory-based content analysis. Am J Prev Med.
tions of apps, may apply to building better clearing- 2013;45:583-589.
13. Bender JL, Yue RY, To MJ, Deacken L, Jadad AR. A lot of action, but not
houses that review apps not only for usability, privacy, in the right direction: systematic review and content analysis of
security, and functionality but also on how strongly smartphone applications for the prevention, detection, and man-
aligned the app is with the available evidence base. agement of cancer. J Med Internet Res. 2013;15:e287.
14. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR,
This will be particularly important for providers who Christensen H. Smartphones for smarter delivery of mental
are leery of prescribing apps to their patients because health programs: a systematic review. J Med Internet Res.
2013;15:e247.
of concerns about validity and lack of sufficient regu- 15. Bailey SC, Belter LT, Pandit AU, Carpenter DM, Carlos E, Wolf MS. The
lation. The FDA does not regulate most of these types availability, functionality, and quality of mobile applications
of apps, and the FDA’s website suggests that it does not supporting medication self-management. J Am Med Inform Assoc.
2014;21(3):542-546.
intend to expand regulation to nonmedical device 16. Martinez-Perez B, de la Torre-Diez I, Lopez-Coronado M. Mobile
apps [27]. Consequently, barring a significant change health applications for the most prevalent conditions by the World
Health Organization: review and analysis. J Med Internet Res.
in FDA policy, providers and healthcare organizations 2013;15:e120.
will have to use alternative methods of finding apps for 17. El-Gayar O, Timsina P, Nawar N, Eid W. Mobile applications for
diabetes self-management: status and potential. J Diabetes Sci
use with their patients, such as clearinghouses. Finally, Technol. 2013;7:247-262.
these strategies may also be appropriate for individual 18. Muessig KE, Pike EC, Legrand S, Hightow-Weidman LB. Mobile
patients and caregivers. Patients and caregivers are phone applications for the care and prevention of HIV and other
sexually transmitted diseases: a review. J Med Internet Res.
likely to have unique insights into the behavior or 2013;15:e1.
disease area, and slight modifications to using these 19. van Mechelen DM, van Mechelen W, Verhagen EA. Sports injury
prevention in your pocket?! Prevention apps assessed against the
strategies may be necessary. For example, a patient/ available scientific evidence: a review. Br J Sports Med.
caregiver is likely to rely less on reviewing the pub- 2014;48(11):878-882.
lished literature and rely more on searching for a 20. Abroms LC, Padmanabhan N, Thaweethai L, Phillips T. iPhone apps
for smoking cessation: content analysis. Am J Prev Med.
credible source for the information and/or piloting 2011;40(3):279-285.
the app to determine its utility and effectiveness. 21. Dolan B. Happtique suspends mobile health app certification pro-
gram. Mobihealthnews. 2013.
22. Cohn AM, Hunter-Reel D, Hagman BT, Mitchell J. Promoting behavior
change from alcohol use through mobile technology: the future of
Acknowledgments: The authors would like to thank Brianna Haskins, MS, ecological momentary assessment. Alcohol Clin Exp Res.
for her editorial assistance. 2011;35:2209-2215.
Financial disclosure: Boudreaux: Dr. Boudreaux owns stock in and receives 23. McGowan BS, Wasko M, Vartabedian BS, Miller RS, Freiherr
consulting fees from Polaris Health Directions, a small business that produces DD, Abdolrasulnia M. Understanding the factors that influ-
software for clinical outcomes management. They do not produce mobile ence the adoption and meaningful use of social media by
health apps. physicians to share medical information. J Med Internet Res.
Waring: Dr. Waring is supportedby National Institutes of Health grants 2012;14:e117.
24. O'Connor ME. 100 healthcare and digital health influencers to follow
KL2TR000160 and 1U01HL105268. She has no other financial disclosures.
in 2014. http://www.slideshare.net/ennoconn/health-care-social-
Hayes: Declares that she has no conflict of interest. media-influencers. Accessed 26 August 2014.
Sadasivam: Dr. Sadasivam is supported by National Institutes of Health, 25. Bastien JMC. Usability testing: a review of some methodological and
National Cancer Institute Career Development Award (K07CA172677). He technical aspects of the method. Int J Med Inform. 2010;79(4):e18-
has no other financial disclosures. e23. doi:10.1016/j.ijmedinf.2008.12.004. Epub 2009 Apr 2.
page 370 of 371 TBM
 PRACTICE TOOL

26. Boudreaux ED, Haskins B. Evaluation of websites and web-based 27. United States Food and Drug Administration. http://www.fda.gov/
applications targeting alcohol and drug use. Poster to be presented MedicalDevices/ProductsandMedicalProcedures/
at the 35th Annual Meeting and Scientific Session of the Society of ConnectedHealth/MobileMedicalApplications/ucm255978.htm.
Behavioral Medicine. Philadelphia, PA; 2014. Accessed 26 August 2014.

TBM page 371 of 371