Tuot et al.

BMC Health Services Research (2015) 15:568

DOI 10.1186/s12913-015-1233-1

RESEARCH ARTICLE Open Access

Facilitators and barriers to implementing

electronic referral and/or consultation
systems: a qualitative study of 16 health
organizations
Delphine S. Tuot1,2*, Kiren Leeds2, Elizabeth J. Murphy2,3, Urmimala Sarkar4, Courtney R. Lyles4,
Tekeshe Mekonnen2 and Alice Hm Chen2,4

Abstract
Background: Access to specialty care remains a challenge for primary care providers and patients. Implementation
of electronic referral and/or consultation (eCR) systems provides an opportunity for innovations in the delivery of
specialty care. We conducted key informant interviews to identify drivers, facilitators, barriers and evaluation metrics
of diverse eCR systems to inform widespread implementation of this model of specialty care delivery.
Methods: Interviews were conducted with leaders of 16 diverse health care delivery organizations between January
2013 and April 2014. A limited snowball sampling approach was used for recruitment. Content analysis was used to
examine key informant interview transcripts.
Results: Electronic referral systems, which provide referral management and triage by specialists, were developed to
enhance tracking and operational efficiency. Electronic consultation systems, which encourage bi-directional
communication between primary care and specialist providers facilitating longitudinal virtual co-management, were
developed to improve access to specialty expertise. Integrated eCR systems leverage both functionalities to enhance
the delivery of coordinated, specialty care at the population level. Elements of successful eCR system implementation
included executive and clinician leadership, established funding models for specialist clinician reimbursement, and a
commitment to optimizing clinician workflows.
Conclusions: eCR systems have great potential to streamline access to and enhance the coordination of specialty care
delivery. While different eCR models help solve different organizational challenges, all require institutional investments
for successful implementation, such as funding for program management, leadership and clinician incentives.
Keywords: Electronic referral, Electronic consultation, Access to care, Health technology, Specialty care,
Implementation, Health system redesign

Background communities [1–3], and highly prevalent gaps in co-

Suboptimal delivery of specialty care is one of the ordination and inter-provider communication. The
most pressing issues facing our health care system current primary-specialty care interface results in
today. Studies of diverse health delivery organizations avoidable specialist visits, duplicate testing, and de-
have documented poor access to specialty care, with layed diagnoses [4]. In turn, this leads to inefficient
wait times as high as 6–12 months in some use of scarce specialty resources [5], preventable harm
to patients, and unnecessary costs. In the United
* Correspondence: Delphine.tuot@ucsf.edu States, passage of the Affordable Care Act is antici-
1
Division of Nephrology at San Francisco General Hospital, University of
California, San Francisco, San Francisco, CA 94110, USA pated to increase demand for specialty care. New
2
Center for Innovation in Access and Quality at San Francisco General models for providing access to timely, coordinated,
Hospital, University of California, San Francisco, San Francisco, CA 94110, USA cost-effective specialty care are needed. Widespread
Full list of author information is available at the end of the article

© 2015 Tuot et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Tuot et al. BMC Health Services Research (2015) 15:568 Page 2 of 10

adoption of electronic medical records (EMR) has fos- given their very different financial structures. We did
tered a growing interest in the development of elec- not continue to sample for additional interviewees after
tronic referral and/or consultation (eCR) systems to April 2014 when we determined that no new informa-
enhance communication among providers as well as tion was emerging from the interviews and thematic sat-
streamline access to and improve coordination of spe- uration had been reached.
cialty care delivery.
In 2007, San Francisco General Hospital (SFGH) im- Definitions
plemented eReferral, a home-grown integrated eCR sys- In the United States, there are no standard definitions
tem that has enhanced access to and the delivery of for electronic referrals or electronic consultations. Per
coordinated specialty care while achieving high levels of the Affordable Care Act, a referral is a written order
satisfaction among primary care providers (PCPs) and from a primary care doctor for a patient to see a special-
specialists [6, 7]. With this eCR, specialist clinicians re- ist or get certain medical services [11]. Consultations are
view each referral and use the system to schedule a rou- not clearly defined by the healthcare.gov website;
tine or expedited clinic visit, recommend additional however, they are generally considered to reflect com-
diagnostic evaluation before scheduling a clinic visit munication between clinicians about general or patient-
(pre-consultation exchange), or provide education and specific questions [12]. We build upon these ideas and
management without a visit (virtual co-management). define an electronic referral as a technology-enabled
Since its implementation, we have received inquiries structured request by a referring provider, most of often
from a wide range of organizations with differing pay- a PCP, to a specialist with an expectation that the patient
ment structures interested in adopting the eReferral will be seen in person by the specialist. Electronic refer-
model. Because eCR systems result in new responsibil- ral systems provide an efficient mechanism for referral
ities for both PCPs and specialists, creating such a sys- management, tracking, and, if reviewed by specialists,
tem is disruptive to the traditional specialty care model triage. An electronic consultation is defined as a request
and thus challenging to implement. These challenges by a provider for a patient’s condition and treatment to
have been documented by individual organizations in be evaluated by a specialist with management advice if
Australia [8] and Canada [9] but a more comprehensive appropriate; it does not carry the expectation that a spe-
examination of implementation challenges is lacking cialist will see the patient. While both electronic referral
[10]. To better understand the drivers, facilitators and and electronic consultation systems allow specialists to
barriers to adopting eCR platforms across diverse health participate in pre-consultative exchange to ensure ad-
care delivery systems in the United States, as well as to equate diagnostic workup prior to an in-person visit,
garner best practices, we conducted key informant inter- only electronic consultation systems encourage ongoing,
views with leaders of health delivery organizations who bi-directional communication between primary care and
had expressed interest in implementing an eCR system. specialist providers. This functionality promotes virtual
co-management, whereby specialists provide longitu-
Methods dinal guidance to PCPs for patient management without
Design and sampling the need for an in-person patient evaluation. Electronic
The study was approved by the Committee on Human referral systems may be used in parallel to electronic
Research at the University of California, San Francisco. consultation systems, with different portals for referring
Using a limited snowball sampling approach, we con- providers. Integrated eCR systems, such as SFGH’s eRe-
tacted leaders of 22 health delivery organizations who ferral, have a single portal of entry for the referring pro-
had previously contacted SFGH to learn about its eCR vider and do not require providers to distinguish
system. Represented organizations were diverse, includ- referrals from consultations. These systems rely upon
ing academic medical centers, health plans, public health specialist review of every request for specialty expertise,
care delivery systems and community health networks, thereby leveraging the functionalities of both electronic
and were at different stages of eCR implementation at referral and electronic consultation systems for the de-
the time of contact, ranging from pilot projects to full livery of specialty care.
expansion. Email invitations to participate in key inform-
ant interviews about each organization’s progress Data sources
towards adopting an electronic referral and/or consult- Phone interviews were conducted by two investigators
ation system were sent between January 2013 and April (KL and DST). Audio-recorded interviews consisted of
2014. Those interviewees referred us to leaders of two nine questions that covered the following topics: drivers
other organizations who we subsequently contacted. Of for implementing an electronic referral, consult, or inte-
the 24 health care delivery organizations contacted, two grated eCR system, facilitators and barriers to imple-
were international and were excluded from the analysis mentation, evaluation metrics, and key lessons learned
Tuot et al. BMC Health Services Research (2015) 15:568 Page 3 of 10

(Additional file 1: Table S1). Each conversation lasted Description and functionalities of electronic referral,
30–60 min and was professionally transcribed. Informed consultation, and integrated eCR systems
consent was obtained by all participants. Electronic referral
Of participating organizations, five had either piloted
(n = 1) or implemented (n = 4) an electronic referral
Analysis system (Additional file 1: Table S3a). These included
We analyzed interview transcripts using directed content relatively small programs, including one health plan
analysis [13]. We developed a list of previously published and one academic medical center, and three larger
factors that influence eCR adoption and use [8] prior to programs implemented by safety-net institutions, en-
the first interview (Additional file 1: Table S2). This de- tities that provide a patchwork of services to the un-
ductive coding scheme allowed us to apply four major insured, under-insured and indigent populations who
categories for the analysis: drivers (which included the would otherwise have little access to health care ser-
primary reason during the planning phase for pursuing vices. Organizations relied on different IT platforms:
eCR systems), barriers and facilitators to the actual im- three were integrated into the EMR and two were
plementation process, and evaluation metrics for under- standalone systems. All systems had the same core
standing how and when eCR systems were effective. functionality of enabling PCPs and clinic coordinators
Then, during the analysis, we used open coding within to track referrals. PCPs, not clinic coordinators, were
each of these four categories to describe the emerging expected to submit the initial referral, which consisted
ideas from respondents, and to compare how the themes of either a structured template (n = 2) or a free text
differed by type of eCR being implemented and form (n = 3). Those with referral templates also had
organizational type. Two investigators (KL and DST) referral guidelines embedded in their electronic work-
coded the first few transcripts separately to determine flow to help PCPs complete medical workups prior to
themes, ensuring that these transcripts included all types referring to specialty care. In all systems, administra-
of healthcare delivery systems in the sample (community tive staff reviewed referral requests and distributed
health network, county public system, academic medical them to the appropriate specialist. Specialists engaged
center, and health plan). The entire coding framework in pre-consultative exchange in only one system.
was shared with all co-authors and agreed upon after
analyzing the first six transcripts.
Electronic consultation
Six systems developed electronic consultation plat-
Results forms, including three small systems used by aca-
Participating organizations demic medical centers and three larger systems
Of the 22 organizations contacted in the United implemented by safety-net organizations: one small
States, 16 (73 %) agreed to participate. Participating advocacy organization and two county public health-
organizations included academic medical centers (n = care delivery systems (Additional file 1: Table S3b).
4), health plans (n = 2), public health care delivery While the platform used by the advocacy organization
systems (n = 7), community health networks (n = 2), is no longer active, the remaining five systems have
and other nonprofit organizations (n = 1). been implemented for a varying number of specialties,
Organizational size varied considerably, ranging from ranging from 12 to 39. Several different IT platforms
400 to 1,300,000 patients served annually, as did the were used: three were integrated into the EMR and
number of patients actually served by each eCR sys- the others were standalone web-based systems. All
tem (range 47 to 100,000 (Table 1)). electronic consultation systems enabled bi-directional
Six organizations did not participate. Leaders of two of communication between referring and specialty
the organizations replied to initial emails inquiring about providers. Referrals were submitted by either the PCP
their eCR platforms, but interviews were not scheduled (n = 5) or a referral coordinator (n = 1). In five of the
due to logistical difficulties. Of those two, one has imple- six systems, reports/studies or lab results could be
mented an electronic referral system and the other has appended to the consultation request. By reviewing
an electronic scheduling system. The status of the other the consult question and appended data when applic-
four organizations’ eCR systems are unknown, as able, specialists in each system could identify a sub-
organizational representatives did not respond to mul- group of patients that did not need a face-to-face
tiple email inquiries and invitations to schedule a key in- visit and would benefit from electronic consultation
formant interview. We could find no evidence from alone. As with the electronic referral systems, organi-
literature and web searches that they have implemented zations that provided referral guidelines (n = 2) also
an eCR. required their PCPs to use specific referral templates.
Tuot et al. BMC Health Services Research (2015) 15:568 Page 4 of 10

Table 1 Characteristics of participating health delivery organizations, ordered by type of eCR system. SFGH included as reference
Geographic Area Type of Organization Safety Patients Type of electronic referral and Number of patients with electronic
net Served consultation system (eCR) referrals or consultations submitted
Annually
Colorado Public system – Yes 133,000 None none; pre-pilot
county
Washington Public system – Yes 240,000 None none; pre-pilot
county
Hawaii, all islands Health plan No 314,500 Referral 100 patients in referral pilot as of October
2013
Central Academic medical No 1,000,000 Referral Appointment requests for 300 patients as
Massachusetts center of March 2014
Southern Public system – Yes 500,000 Referral 18,000 consults in 2013
California county
Southern Public system – Yes 240,000 Referral 59,400 in 2013
California county
Northern Network of Yes 31,000 Referral 4000 patients with consults
California community health
centers
Boston, Academic medical No data not Consultation 47 consults in 3-month pilot as of October
Massachusetts center available 2013
Southern Academic medical No 971,000 Consultation 330 consults
California center
Northern Academic medical No 232,774 Consultation 550 consults between April 2012 and May
California center 2013
Southern Advocacy organization Yes 400 Consultation 250 patients with consults in 2012
California
Northern Public system – Yes 100,000 Consultation data not available; significant volume
California county
Northern Public system – Yes 80,000 Consultation data not available
California county
Connecticut Network of Yes 130,000 Integrated eCR 125 consults for 120 patients
community health
centers
Southern Health plan Yes 1,300,000 Integrated eCR 100,000 consults
California
Southern Public system – Yes 850,000 Integrated eCR 60,000 consults
California county
San Francisco Public system – Yes 123,500 Integrated eCR 58,000 yearly
General Hospital County

Integrated eCR systems Drivers for implementing electronic referral, consultation,

Three organizations implemented integrated eCR sys- or integrated eCR systems
tems similar to SFGH’s eReferral, all of which served While many leaders expressed similar challenges regard-
safety-net populations (Additional file 1: Table S3c). ing the delivery of specialty care, they cited different rea-
The eCR platform was embedded into the EMR for sons for implementing electronic referral versus
only one system. All integrated eCRs had the same electronic consultation or integrated eCR systems. Elec-
core functionalities of referral management/tracking tronic referral systems were primarily implemented to
and the possibility of bi-directional communication enhance operational and clinical efficiency (Table 2).
for all referrals. PCPs submitted electronic referrals to Prior referral methods, which relied on paper and fax,
designated specialist reviewers who determined contributed to inefficient in-person specialty visits be-
whether a patient needed to be seen for a face-to-face cause of inadequate workup and illegible communication
visit or could be managed via electronic consultation between primary care and specialty care providers. In
alone. The smallest organization did not have referral turn, this led to poor specialist satisfaction.
guidelines or templates, while the other two organiza- By contrast, the main driver of implementing elec-
tions did. tronic consultation or integrated eCR systems was poor
Tuot et al. BMC Health Services Research (2015) 15:568 Page 5 of 10

access to specialty care, particularly evidenced by long electronic consultation systems were expected to enhance
patient wait times for in-person specialty appointments. primary care capacity to longitudinally manage conditions
Not only were these wait times vexing for health system with specialist support as needed.
leaders concerned about the delivery of timely, coordi-
nated specialty care, but they were also thought to con- Facilitators for implementing electronic referral,
tribute substantially to poor patient satisfaction scores consultation, or integrated eCR systems
and to negatively impact revenue because of leakage of Organizations that achieved high levels of sustained pro-
patients to neighboring systems with quicker access to vider participation in their eCR systems cited engaged
specialty care appointments. Less prevalent reasons driv- leadership as the most important facilitator (Table 2). In
ing adoption of electronic consultation or integrated most circumstances, leaders included a combination of
eCR systems included the desire to leverage existing spe- high-level executives, such as Chief Medical Officers, as
cialty capacity to care for a larger number of patients and well as physician leaders who were early adopters. Ex-
to enhance communication among referring and consult- ecutive leaders prioritized implementation of the pro-
ing providers. By facilitating virtual co-management, gram in response to specific organizational challenges

Table 2 Main drivers, facilitators and barriers to implementation of referral, consultation and integrated eCRs and representative
quotes
Drivers Representative quotes
Referral systems Enhance operational and clinical “Specialists wanted to have all of the relevant clinical history for patients prior to a visit;
efficiency productive visits are key and prior communication [wasn’t] sufficient.”
Consultation systems Poor access to specialty care “The service was developed to improve access to high-need specialties with long wait
and Integrated eCRs times.”
Leakage of patients to other “Drivers included coordinating and improving integration of care with the goal of
systems retaining patients; payor data showed that approximately 30 % of patients were going
outside of [organization’s name] for specialty care.”
Enhance care coordination and “We’re trying to build good relationships with our community clinics to create an
communication integrated safety net care system.” “I think efficiency helps the supply–demand
mismatch… we have driven down the mismatch with eConsult, not by inventing new
specialists, but by using our existing specialists in a better way.”

Facilitators Representative quotes

Referral systems Engaged executive leadership “For our program, it was important to make eConsults/eReferrals mandatory [by the
leadership]. We found that [others] that did not do this had low uptake.”
Early clinician adopters “Having a physician leader who was able to have dedicated time to have lots of the
conversations with people, to message it, to overcome concerns and resistance, to
really be the driving force behind it, I think was critical in our implementation.”
Consultation systems Provider incentives “[Our system] is now funding reimbursement of specialists’ time for using the service.”
and Integrated eCRs
User-friendly technology “The template was easy to build and make friendly for staff and the doctors.”
Platform integrated into “We have a platform available within our electronic health record program that we
electronic medical record were able to adapt to our need/s.”

Barriers Representative quotes

Referral systems Provider resistance to change “As you well know, unless you can mandate, it is very difficult to get PCPs to adapt if
workflows they view this as taking any more time.”
Lack of eCR integration into “With no shared [technology], it has been difficult to get providers to [move past the
electronic medical record workflow issues] and see the benefit of … improved integration of care.”
Consultation systems or Primary care provider resistance “Many physicians didn’t want to submit [the referral] themselves.”
integrated eCRs to change in workflows
Lack of reimbursement “The biggest barrier to adoption we faced was reimbursement. … It is this funding
mechanisms issue that is preventing expansion of the program to include additional specialties.” “In
order to support adoption of the electronic consult system, we obtained grant funding.
We are currently using the results … to build a case for the state reimbursing electronic
referrals.”
Specialist provider liability “When we implemented … we got quite a lot – not surprising, but consistent –
concerns feedback or questions or skepticism from specialists with the liability, specialist
skepticism about whether the PCP [would] be able to provide reliable information.”
Tuot et al. BMC Health Services Research (2015) 15:568 Page 6 of 10

and used organizational priorities, such as operational individual health care delivery systems, all organizational
efficiency, to drive adoption. The physician leaders, by leaders expressed a commitment to program evaluation.
contrast, served as clinical champions, increasing uptake Interviewees highlighted the importance of evaluation
among colleagues through modeling. Existing collegial metrics for ongoing quality improvement activities to
relationships with specialists were also deemed import- ensure that their system met organizational needs and
ant by all interviewees. enhanced the primary-specialty care interface. All sys-
Leaders from organizations that implemented elec- tems, regardless of their functionality, had operational
tronic consultation or integrated eCR systems also em- metrics in place from inception that included: referral
phasized the importance of provider incentives and volume, number and type of services available, number
reimbursement mechanisms. In many cases, PCPs were of referring primary care sites and providers, and time
given productivity “credit” or access to referral managers from referral to treatment or first visit. With respect to
to expedite the referral process and specialists were paid common scheduling and clinical metrics, organizations
to perform pre-consultative exchange and virtual co- with all eCR types examined the number of days needed
management. Of interest, leaders of organizations whose to confirm patient appointments, the clinical reason for
electronic consultation systems had been in place for consultations and PCP satisfaction.
many years had started thinking about ways to Unique evaluation metrics pertinent to electronic re-
incentivize quality of electronic communication. Other ferral, consultation, and integrated eCR systems are pre-
facilitators included having user-friendly technology, as sented in Table 3. Many referral systems examined
well as technology that could be integrated into the or- percentage of referrals completed electronically vs. fax/
ganization’s EMR and embedded into existing workflows paper and patient disposition as determined by a referral
with dedicated program support staff to perform management department, as well as unique scheduling
outreach. metrics including the percentage of clinic template slots
used for patients referred electronically rather than via
Barriers to implementing electronic referral, consultation paper or fax. By definition electronic systems did not in-
or integrated eCR systems clude communication metrics. Health care systems that
All organizations encountered challenges. Resistance to implemented electronic consultation or integrated eCRs
change, particularly to changes in PCP workflow, often examined additional operational metrics related to
emerged prominently during our interviews (Table 2). specialty care access, such as time to first specialist re-
Without exception, with every eCR, PCP workload in- sponse and patient disposition as determined by the spe-
creased, as they were expected to navigate new technol- cialist reviewer. Most systems did not yet have
ogy to enter a referral question and pertinent patient benchmark goals for these metrics, with the exception of
data. Additionally, PCPs acquired extra work in man- time to specialist response, where the goal was typically
aging conditions that they used to refer. Specialists also 2–3 business days. Many organizations with consultation
experienced greater workload in the form of pre- and integrated eCRs also examined communication quality.
consultative exchange and virtual management, which In particular, several organizations looked at the number of
also served as a barrier to implementation. Working exchanges between a PCP and specialist per consult, PCP
with standalone eCR systems that were not integrated expectation with consultation requests, and quality of
with existing EMRs was also a challenge shared by many content in PCP referral and specialist response.
organizations, which resulted in duplication of work for As with traditional primary care-specialty care interac-
providers and/or staff. tions, measuring the impact of eCR systems on patient
The most oft-cited barrier to widespread implementa- outcomes and PCP/specialist capacity were frequently
tion of electronic consultation or integrated eCR systems cited as the most clinically relevant but challenging met-
was a lack of resources. Specifically, lack of reimburse- rics to capture. Many interviewees expressed the desire
ment mechanisms for specialists, inadequate funding for to measure whether their system had met patient needs,
on-going costs to support the technology platform, and but determining whether the right patient received spe-
a dearth of administrative support to maintain the sys- cialty expertise “by the right specialist at the right time,
tem were mentioned by many study participants in the right way” remained elusive. To date, only one
(Table 2). Another challenge unique to electronic con- organization with an integrated eCR has measured true
sultation and integrated eCR systems but not referral clinical outcomes, comparing PCP prescription of
systems was specialist concern about liability. guideline-concordant cardiovascular medications among
those who sent electronic consultations to cardiologists
Evaluation metrics versus those who referred patients via usual methods
While combinations of drivers, facilitators and barriers (Table 3). Adverse cardiac outcomes were also compared
to adoption of eCR systems may have differed among among patients who were referred electronically versus
Tuot et al. BMC Health Services Research (2015) 15:568 Page 7 of 10

Table 3 Unique existing evaluation metrics pertinent to referral, consultation and Integrated eCR systems, by domain
Operational Scheduling Clinical Communication
Referral • Percent of referrals completed • Percent of • Patient follow-up by electronic vs. paper/fax
electronically vs. fax/paper eReferral referral
slots used
• Disposition (scheduled vs not scheduled) • Slot
as determined by referral management availability
department for
eReferred-
patients
• Wait time for specialty service by
insurance status
Consultation • Time to first specialist response • Self-reported PCP ability to manage a patient with • Specialist
and specialist guidance satisfaction
integrated
eCR • Disposition (e.g., scheduled immediately, • Emergency department utilization • PCP
scheduled after review, consultation only) expectation for
as determined by specialist reviewer referral
• Time spent by specialist • Cardiac outcomes: appropriate diagnoses, percent of • Quality of PCP
patients with blood pressure control, PCP referral
prescription of guideline-conoirdance cardiovascular
medicationsa
• Economic impact of provider • Specialty clinic complexitya • Quality of
reimbursement strategy specialist
response
• Patient leakage to other health systems • Number of
for specialty care exchanges per
consult
• Primary care clinic adoption of systema • Number of
consults with
document
uploadsa
a
Denotes metrics that were only examined among integrated eCR systems

traditional methods. Three organizations examined other actively changing the primary care-specialty care inter-
types of clinical metrics that were not directly patient- face across diverse health care delivery settings.
facing, including specialty clinic complexity and impact on We found that integrated eCRs have been imple-
PCP capacity to manage “specialty problems”. Notably, mented mainly in public health care systems that are re-
none of the systems have been able to measure cost savings sponsible for a defined population of individuals and
or calculate a “return on investment” for their eCR. traditionally have had difficulties with specialty access.
On the one hand, requiring specialty review of each re-
Discussion ferral request entails additional work for both PCPs and
To date, the term “electronic referral system” has been the specialist reviewers. This requires additional re-
used in the United States to describe information tech- sources to compensate specialist reviewer time, and may
nology systems with a wide range of functionality and prompt PCPs who are unable or unwilling to absorb the
specialist involvement. Some eCR systems are standa- additional work of managing conditions they would usu-
lone web-based programs, while others are fully embed- ally refer, to seek alternate consultants. On the other
ded into EMRs. Some are purely referral tracking tools, hand, having a single entry point for all referrals to a
without any specialist clinician involvement prior to an given specialty provides a population health perspective
appointment, while others offer electronic consultation for the system and better ensures that scheduled spe-
as a means of providing timely specialty expertise for pa- cialty visits are appropriate with complete pre-visit eval-
tients with low complexity issues. A small number of or- uations. In this manner, integrated eCRs enable health
ganizations have implemented integrated eCR systems care organizations to act as stewards of scarce specialty
where a specialist reviews each referral, thereby maxi- resources and more effectively match supply of and de-
mizing opportunities for pre-consultative guidance and mand for specialist expertise at a large scale, thereby
virtual co-management; these are most similar to maximizing population health [14]. That said, some Vet-
SFGH’s eReferral system. While offering solutions to dif- erans Administration and Kaiser Permanente sites, two
ferent organizational challenges, all eCR systems are integrated health systems touted for their population-
Tuot et al. BMC Health Services Research (2015) 15:568 Page 8 of 10

health management, have implemented eCRs that main- and sustainability. Leaders of organizations that success-
tain separate electronic referral and consultation portals fully implemented eCR systems also cultivated a collab-
rather than an integrated eCR platform [15, 16]. It ap- orative approach, involving primary care and specialty
pears that these health systems encourage but do not care providers in the development of systems policies
mandate specialist review of all referral requests and do and referral guidelines to fully support stakeholder buy-
not compensate providers for this additional effort. in and end-user adoption. Many of the essential imple-
Per our interviews across diverse health care models, mentation elements in Fig. 1 are very similar to what
eCR systems that have developed into sustainable, suc- has been previously cited for successful implementation
cessful systems are those that provide both electronic re- of other health system changes in the United States,
ferral and consultative activities, including those that such as adoption of electronic health records [17],
maintain the two as separate entities, as well as those transformation of primary care practices into patient-
systems that have integrated them into a single portal of centered medical homes [18], development of Account-
entry. The technical platform and workflow inherent to able Care Organizations [19] and creation of care
an electronic referral system are often prerequisites for coordination agreements among primary care and spe-
electronic consultative activities, but they are not suffi- cialty care providers [20].
cient. The important elements for successful implemen- Importantly, elements of successful eCR implementa-
tation of electronic consultation systems that emerged tion were consistent across diverse health care delivery
from our data include funding for clinician reimburse- models, including academic medical centers, private hos-
ment as well as program management, a marketing plan pitals, and public delivery systems across the United
with positive messaging to stakeholders, a commitment States. Our study is timely, given the widespread need
to create efficient clinician workflows via system re- and interest for systems that enhance the coordination
design and dedication to on-going quality improvement and value of specialty care. The American College of
in response to key evaluation metrics (Fig. 1). Integration Physicians’ Patient Centered Medical Home-
of the consultative system with existing EMR platforms Neighborhood framework for care delivery and the Na-
is also a facilitator for implementation, though it may tional Council on Quality Assurance’s Patient-Centered
not be absolutely necessary, given the success of some Specialty Practice Recognition program both promote
programs using standalone technologies. But, having a specialty care coordination through tracking and coord-
user-friendly, affordable eCR system is key for adoption ination of referrals with the goals of enhanced

Fig. 1 Elements of successful implementation of electronic consultation systems

Tuot et al. BMC Health Services Research (2015) 15:568 Page 9 of 10

communication and population management for spe- incentives. As organizations gain greater experience with
cialty practices [21, 22]. eCR systems and other tech- existing systems, more data will emerge with regards to
nologies that encourage knowledge sharing in new and key eCR platform functionality, costs, and clinical im-
efficient ways have great potential to advance these goals pact. Meanwhile, policy makers and payers should en-
[23]. Highlighting this point is a $7 million Centers for courage ongoing development and evaluation of eCR
Medicare and Medicaid Innovation grant recently systems. Clarification of the medico-legal implications of
awarded to the American Association of Medical Col- electronic consultation, for example, could mitigate one
leges to implement and evaluate eCR platforms in five barrier to eCR implementation. Reimbursement of spe-
U.S. academic medical centers [24]. cialist reviewer effort and/or care coordination and
Many questions regarding eCRs remain unanswered. financial support for the development of eCR software
Financial evaluation of eCR systems is nascent at this that can integrate with existing EMRs could also spur
time. Data from our organization and others have dem- further innovation in this area of health system redesign.
onstrated improved access to and efficiency of specialty
care delivery with electronic consultation systems, with a Additional file
shift from in-person visits to virtual co-management,
which presumably decreases overall costs to the system Additional file 1: Table S1. Key informant interview guide. Table S2. A
priori analytic codes. Table S3a–c. Description, functionality, workflow,
[25, 26]. But, eCR systems increase PCP costs in the and implementation status of electronic referral, consultation and
form of a greater burden to provide care that would have integrated eCR systems. (DOCX 128 kb)
otherwise been provided by specialists and specialist costs
in the form of time spent reviewing and responding to Abbreviations
electronic consultations [15]. Robust measures of patient eCR: electronic referral and consultation; EMR: electronic medical record;
PCP: primary care provider; SFGH: San Francisco General Hospital.
experience as well as of clinical impact on both individual
patient outcomes and population health are needed. Competing interests
Like all qualitative studies, there are limitations to our The authors do not have any relevant financial or non-financial competing
interest to report.
results. Interviews were conducted with only 1–2 leaders
of each organization while eCR systems rely on support Authors’ contributions
from many leaders in a health care system. Thus re- DST and AHC had full access to all of the data and take full responsibility for
the integrity of the data and accuracy of the data analysis. AC conceived of
sponses reflect the views and knowledge of those indi-
the study and participated in its design. DST and KL performed key
viduals and may not be representative of others in informant interviews. DST, KL, US and CL participated in the qualitative
leadership roles. However, we believe that we identified analysis. All authors read, contributed to the drafting of the manuscript and
approve of the final product.
the individuals most knowledgeable and responsible for
eCR system implementation within each organization. Acknowledgements
After data analysis, we became aware of additional orga- This work was supported by the California Healthcare Foundation. The
nizations that had implemented eCR systems, including sponsor had no role in study design or conduct; data collection,
management, analysis or interpretation; manuscript preparation, review or
some Veteran’s Administration and Kaiser Permanente approval. DST is supported by K23DK094850 as well as the National Center
sites [12, 15, 27]. Given the similar drivers, barriers and for Advancing Translational Sciences at the National Institutes of Health,
facilitators noted by leaders from diverse systems in- through UCSF-CTSI Grant Number UL1 TR000004. The contents of this
manuscript are solely the responsibility of the authors and do not represent
volved in various stages of eCR maturity (i.e., from pilot the official views of the NIH.
studies to full expansion), the themes presented in this
analysis are likely applicable to those organizations as Author details
1
Division of Nephrology at San Francisco General Hospital, University of
well. California, San Francisco, San Francisco, CA 94110, USA. 2Center for
Innovation in Access and Quality at San Francisco General Hospital,
Conclusion University of California, San Francisco, San Francisco, CA 94110, USA.
3
Division of Endocrinology at San Francisco General Hospital, University of
In summary, eCR systems have great potential to California, San Francisco, San Francisco, CA 94110, USA. 4Division of General
streamline access to and enhance the coordination and Internal Medicine at San Francisco General Hospital, University of California,
appropriateness of specialty care delivery. With several San Francisco, San Francisco, CA 94110, USA.

different eCR models from which to choose, health care Received: 24 February 2015 Accepted: 11 December 2015
leaders interested in implementing an electronic referral
or consultation systems would be wise to begin with a
References
clear understanding of their organization’s challenges 1. Cook NL, Hicks LS, O’Malley AJ, Keegan T, Guadagnoli E, Landon BE. Access
and what problems they are seeking to address with an to specialty care and medical services in community health centers. Health
eCR. In addition, there are key institutional investments Aff (Millwood). 2007;26(5):1459–68.
2. Forrest CB, Nutting PA, von Schrader S, Rohde C, Starfield B. Primary care
that are required for successful implementation, such physician specialty referral decision making: patient, physician, and health
as funding for program management and clinician care system determinants. Med Decis Making. 2006;26(1):76–85.
Tuot et al. BMC Health Services Research (2015) 15:568 Page 10 of 10

3. Felt-Lisk S, McHugh M, Howell E. Monitoring local safety-net providers: do

they have adequate capacity? Health Aff (Millwood). 2002;21(5):277–83.
4. Singer SJ, Reyes Nieva H, Brede N, Ling J, Leydon N, Weissman JS, et al.
Evaluating ambulatory practice safety: the PROMISES project administrators
and practice staff surveys. Med Care. 2015;53(2):141–52.
5. Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW.
Communication breakdown in the outpatient referral process. J Gen Intern
Med. 2000;15(9):626–31.
6. Kim Y, Chen AH, Keith E, Yee Jr HF, Kushel MB. Not perfect, but better:
primary care providers’ experiences with electronic referrals in a safety net
health system. J Gen Intern Med. 2009;24(5):614–9.
7. Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee Jr HF, Kushel MB.
Evaluating electronic referrals for specialty care at a public hospital. J Gen
Intern Med. 2010;25(10):1123–8.
8. Warren J, White S, Day KJ, Gu Y, Pollock M. Introduction of electronic referral
from community associated with more timely review by secondary services.
Appl Clin Inform. 2011;2(4):546–64.
9. Liddy C, Maranger J, Afkham A, Keely E. Ten steps to establishing an e-
consultation service to improve access to specialist care. Telemed J E
Health. 2013;19(12):982–90.
10. Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al.
Electronic consultations (e-consults) to improve access to specialty care: A
systematic review and narrative synthesis. J Telemed Telecare. 2015;21:323–30.
11. 111 United States Congress: Patient Protection and Affordable Care Act.; 2010.
12. Horner K, Wagner EH, Tufano J: Electronic Consultations between primary
and specialty care clinicians: early insights. The Commonwealth Fund Issue
Brief; October, 2011.
13. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis.
Qual Health Res. 2005;15(9):1277–88.
14. Tuot DS, Murphy EJ, McCulloch CE, Leeds K, Chan E, Chen AH: Leveraging
an electronic referral and consultation system to provide high quality
corrindated specialty care. Health Care: The Journal of Delivery Science and
Innovation In press.
15. McAdams M, Cannavo L, Orlander JD. A Medical Specialty e-Consult
Program in a VA Health Care System. Fed Pract. 2014;31:26–31.
16. Palen TE, Price D, Shetterly S, Wallace KB. Comparing virtual consults to
traditional consults using an electronic health record: an observational
case–control study. BMC Med Inform Decis Mak. 2012;12:65.
17. McGinn CA, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, et al.
Comparison of user groups’ perspectives of barriers and facilitators to
implementing electronic health records: a systematic review. BMC Med.
2011;9:46.
18. Scholle SH, Asche SE, Morton S, Solberg LI, Tirodkar MA, Jaen CR. Support
and strategies for change among small patient-centered medical home
practices. Ann Fam Med. 2013;11 Suppl 1:S6–13.
19. Wan TT, Demachkie Masri M, Ortiz J, Lin BY. Willingness to participate in
accountable care organizations: health care managers’ perspective. Health
Care Manag (Frederick). 2014;33(1):64–74.
20. Carrier E, Dowling MK, Pham HH. Care coordination agreements: barriers,
facilitators, and lessons learned. Am J Manag Care. 2012;18(11):e398–404.
21. Laine C. Welcome to the patient-centered medical neighborhood. Ann
Intern Med. 2011;154(1):60.
22. National Committee for Quality Assurance. [http://www.ncqa.org/Programs/
Recognition/Practices/PatientCenteredSpecialtyPracticePCSP.aspx]
23. Kirsh SR, Ho PM, Aron DC. Providing specialty consultant expertise to
primary care: an expanding spectrum of modalities. Mayo Clin Proc. 2014;
89(10):1416–26.
24. Davis A, Gilchrist V, Grumbach K, James P, Kallenberg R, Shipman SA. Submit your next manuscript to BioMed Central
Advancing the Primary/Specialty Care Interface through Econsults and
Enhanced Referrals. Ann Fam Med. 2015;13(4):387–8. and we will help you at every step:
25. Chen AH, Murphy EJ, Yee Jr HF. eReferral–a new model for integrated care.
• We accept pre-submission inquiries
N Engl J Med. 2013;368(26):2450–3.
26. Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building Access to • Our selector tool helps you to find the most relevant journal
specialist care through e-consultation. Open Med. 2013;7(1):31–8. • We provide round the clock customer support
27. North F, Uthke LD, Tulledge-Scheitel SM. Integration of e-consultations into
• Convenient online submission
the outpatient care process at a tertiary medical centre. J Telemed Telecare.
2014;20:221–9. • Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research

Submit your manuscript at

www.biomedcentral.com/submit