Abstract

Background

‘One-stop shops’ for research evidence can allow health system policymakers, stakeholders, and researchers to find and use the best available research evidence efficiently in the limited time they have available to make, inform, or advocate for a decision. Such resources have been highlighted as a central pillar of broader efforts to support evidence-informed decision-making about health systems [1]. One-stop shops are critical to ensuring that policymakers have timely access to research evidence when pressing issues emerge; this is one of the key factors found to increase the prospects for research use by policymakers [2]. However, while one-stop shops have been developed to address questions regarding clinical programs and services and prescription drugs [3–7], as well as to address questions regarding public health programs and services [8, 9], no similar resource existed for questions about health systems (i.e., how we strengthen healthcare and public health systems or how we get cost-effective healthcare and public health programs and services, as well as drugs and other technologies, to those who need them).

Developing a one-stop shop for health system policymakers, stakeholders, and researchers requires addressing three challenges. First, these groups need to be able to find research evidence about health systems using an easily understood taxonomy of topics. Such a taxonomy would ideally be organized in a way that reflects the ways in which these groups think about health systems (i.e., by focusing on ‘policy levers,’ which can include both health system arrangements and implementation strategies) and using terminology they are familiar with. While a number of taxonomies exist [10], they tend either to lack the specificity needed to capture the many different types of policy levers that exist (e.g., World Health Organization (WHO) ‘building blocks of health systems’ taxonomy [11]) or the breadth of levers available in health systems (e.g., taxonomies focusing on particular health system domains such as human resources or pharmaceutical policy [12, 13]).

Second, health system policymakers, stakeholders, and researchers need to know when they have conducted a comprehensive search of the many types of research evidence relevant to them. Providing such reassurance with a single one-stop shop means ensuring that it incorporates the many types of research evidence needed to answer their questions and contains comprehensive inventories of each of these types of research evidence. Many existing one-stop shops address questions about the effectiveness and cost-effectiveness of policy options (through systematic reviews, single studies such as impact evaluations and economic evaluations, or both), but not questions about the policy problem (e.g., stakeholders’ views and experiences with the problem) or about many features of policy options (e.g., how and why options work) and implementation considerations (e.g., barriers to implementing a particular options) [14, 15]. Moreover, many existing one-stop shops do not contain comprehensive inventories and, as a result, users are left uncertain about how many other databases they should search to address the same question.

Third, these groups need to be able to rapidly identify decision-relevant information in the results of their search of a given one-stop shop. For systematic reviews, for example, a busy policymaker would ideally like to be able to quickly scan the search results to know whether the retrieved reviews are of high quality, whether the searches for research evidence were conducted recently, and whether the studies included in the reviews were conducted in their own health system or in health systems that share characteristics that are likely to influence the applicability of the findings to their own system [2]. The same policymaker would ideally also like to read a structured decision-relevant summary of a particular review of interest; however, eight different groups in the world now prepare such summaries and they are not available on a single site [14].

Our objective was to develop and refine the methods for an easily searched, comprehensive, free, one-stop shop for research evidence that could provide decision-relevant information about the many types of questions asked by policymakers, stakeholders, and researchers regarding health systems. To address this objective, we proceeded in four stages: i) developing a taxonomy of governance, financial, and delivery arrangements within health systems and of implementation strategies within health systems; ii) building content by identifying, selecting, and categorizing content and by adding value to that content; iii) expanding the types of content; and iv) continuously updating the resulting one-stop shop (Table 1).

Table 1

Stages in the creation of the one-stop shop

Stage	Activities	Activities by source	Outcomes
1 – Developing a taxonomy	• Developed and iteratively refined a taxonomy of health system topics during stage 1 of content building	Taxonomy development	• Prototype for a taxonomy
		• Drew on existing categorization schemes	• Included systematic reviews (n=616)
		o System-wide schemes such as WHO’s “building blocks” [11]	o Reviews of effects* (n=513)
		o Domain-specific schemes such as those related to human resources policy [13], pharmaceutical policy [12], and implementation strategies [7]	■ Cochrane reviews of effects (n=184)
	• Began building a repository of systematic reviews and systematic review protocols	• Used it to code progressively larger bundles of systematic reviews and systematic review protocols and made adjustments as needed	o Reviews addressing questions other than effects (cumulative n=103)
		Preliminary content building (phase 1)	• Included Cochrane protocols of reviews of effects (n=64)
		• Searched Medline (OVID) from 1966 to September 2006 (n=848 reviews requiring eligibility assessment and, if eligible, coding)
		• Hand searched the Cochrane Database of Systematic Reviews (CDSR) up to Issue 3, 2007 (n=3,308 reviews and review protocols requiring eligibility assessment and, if eligible, coding)
2 – Building content and adding value to that content	• Expanded breadth of searches	Content building (phase 2)	• Current taxonomy and cross-cutting taxonomy (Table 2)
	• Expanded scope of documents to include review-derived products	• Hand searched key databases and journals	• Comprehensive set of:*
		o CDSR for overviews of systematic reviews, systematic reviews and systematic reviews in progress (i.e., protocols) up to issue 7, 2012 and monitored each issue thereafter	o Systematic reviews (n=4,240):
	• Iteratively refined the taxonomy during phase 2 of content building and complemented it with a cross-cutting taxonomy	o Database of Abstracts of Reviews of Effects (DARE) for systematic reviews up to April 2012 and monitored CRD News or a direct feed thereafter	■ Reviews of effects (n=3,378 of which 585 are Cochrane reviews)
		o Rx for Change up to 2010 and monitored annual or semi-annual updates thereafter	■ Reviews addressing questions other than effects (n=862)
		o Cochrane Qualitative and Implementation Methods Group’s reference database for qualitative reviews up to July 2012 and monitored it regularly thereafter	o Systematic reviews in progress (n=423)
	• Began providing added value for existing content (e.g., for systematic reviews: assessing their methodological quality, coding them according to how recently searches were conducted and the countries in which included studies were conducted, and linking them to structured decision-relevant summaries)	o 15 journals for qualitative reviews from the first issue of 2004 to the last issue of 2008** (but hand searching not continued thereafter because of overlap with other sources)	o Review-derived products (n=183)
		• Continuous scanning of listservs	■ Overviews of systematic reviews (n=58)
		o Evidence Updates and KT+ for systematic reviews identified by McMaster PLUS	■ Evidence briefs (n=125)
		o Listservs administered by PAHO EQUIDAD, PHCRIS, Sax Institute among	• ‘Added value’ coding available for all existing content
		• Review of websites for systematic reviews and reviewed-derived products (unless otherwise noted) up to June 2012 then continuous scanning
		o 3Ie (International Initiative for Impact and Evaluation)
		o Alliance for Health Policy and Systems Research
		o Campbell Collaboration
		o Canadian Institutes of Health Research ‘Evidence on Tap’ and ‘KT Synthesis’ programs
		o Department for International Development (UK)
		o Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre)
		o Evidence-Based Practice Centers (EPC), Agency for Healthcare Research and Quality (AHRQ)
		o Evidence-Informed Policy Networks for evidence briefs [18]
		o McMaster Health Forum for evidence briefs
		o WHO Regional Office for Europe/Health Evidence Network/European Observatory on Health Systems and Policies for evidence briefs
3 – Expanding the types of content	• Expanded scope of documents to include:	Content building – phase 3	• Comprehensive set of:*
		• Hand searched CDSR up to issue 10, 2012 of the CDSR for systematic reviews being planned (i.e., registered titles) and monitored each issue thereafter	o Systematic reviews being planned (n=220)
	o Systematic reviews being planned		o Health reform descriptions (n=1,107)
	o Economic evaluations and costing studies	• Hand searched two databases maintained by the Centre for Reviews and Dissemination	o Health system descriptions (n=232)
	o Health reform descriptions	o PROSPERO up to October 2012 for systematic reviews being planned, and monitor it regularly thereafter	o Economic evaluations and costing studies published since 2003 (n=2,236)
	o Health system descriptions	o Economic Evaluation Database up to October 2012 for economic evaluations and costing studies and monitored it regularly thereafter	• Partial (and soon to be comprehensive) set of links to studies included in systematic reviews, with the links assigned names based on the countries where the studies were conducted to facilitate immediate access to more locally applicable studies (91% of reviews now have links)
	• Providing more added value for existing content (e.g., links to studies included in systematic reviews)	• Hand searched Health Policy Monitor up to October 2012 for descriptions of health system reforms and monitored it regularly thereafter
		• Hand searched the websites of the European Observatory on Health Systems and Policies, World Health Organization headquarters and all regional offices and World Bank up to October 2012 for health system descriptions and monitored them regularly thereafter
4 – Continuously updating the one-stop shop	• Refined and executed procedures for continuously updating the one-stop shop in all seven supported languages	• Continue to monitor sources from the above stages, assess eligibility, categorize them, and add value to the content	• Procedures document for maintenance (with excerpts available upon request)
		• Continue to identify new potential sources and more efficient ways of accessing these sources, as well as new potential types of documents	• Agreements with partners to ensure the one-stop shop is functional in Arabic, Chinese, English, French, Portuguese, Russian and Spanish

Open in a separate window

*All numbers in this list are cumulative and include the documents identified in the previous stage.

**We hand searched three journals that had published two or more qualitative reviews of which we were already aware, namely the International Journal of Nursing Studies, Journal of Advanced Nursing, and Patient Education and Counseling. We also hand searched 12 journals that had published two or more policy-relevant reviews of any type, namely the American Journal of Managed Care, Evidence-based Healthcare & Public Health, Health Expectations, Health Policy, Health Policy & Planning, International Journal for Quality in Health Care, International Journal of Technology Assessment in Health Care, Journal of Health Services Research & Policy, Medical Care, Medical Care Research and Review, Psychiatric Services, and Social Science & Medicine.

Methods

Continuously updating the one-stop shop

Finally, we developed an approach to continuously updating the online ‘one-stop shop’ in all six WHO official languages (Arabic, Chinese, English, French, Russian, and Spanish) as well as Portuguese (Table 1, Figure 1) and, through user testing, training workshops and other interactions with our target audiences, continuously adapting it based on their input. We continue to monitor all of the sources used in previous stages, assess eligibility, categorize them, and add value to the content. We also continue to identify new potential sources of documents and more efficient ways to access these sources, such as direct feeds. Finally, we remain open to suggestions for new types of documents that could address the questions being asked by health system policymakers, stakeholders, and researchers both within and across countries. For example, in Canada, these groups wanted to be able to access policy-relevant documents that provided key contextual information about the problems, policy options, and implementation issues being considered. We were able to rapidly develop an approach to identifying, categorizing, and coding these documents, integrating them into the one-stop shop, and making them visible to users registered as being based in Canada or who are accessing Health Systems Evidence from an IP address in Canada [24]. We also identified a similarly complementary set of policy-relevant documents about health system strengthening prepared by international agencies. The Intergovernmental Organizations’ Health Systems Documents Portal allows users to look for research evidence in the domains covered by 22 types of international agency documents, including World Health Assembly resolutions and WHO guidance.

Open in a separate window

Figure 1

Process for maintaining Health Systems Evidence.

Results

The methodological developments described in this paper have led to the creation of Health Systems Evidence (http://www.healthsystemsevidence.org) – a one-stop shop for research evidence to support policymakers, stakeholders, and researchers interested in how to strengthen or reform health systems or in how to get cost-effective programs, services, and drugs to those who need them. The one-stop shop contains many functionalities designed to address the three challenges described earlier in this paper: health system policymakers, stakeholders, and researchers can now find research evidence about health systems using an easily understood taxonomy of topics, know when they have conducted a comprehensive search of the many types of research evidence relevant to them, and rapidly identify decision-relevant information in the results of their search (Table 3).

The taxonomy at the center of Health Systems Evidence is organized by governance, financial, and delivery arrangements and by implementation strategies (Table 2), and it is complemented by a taxonomy of the cross-cutting domains of diseases, technologies, sectors, and providers (Table 2). The one-stop shop now contains a comprehensive inventory of evidence briefs, overviews of systematic reviews, systematic reviews, systematic review protocols, registered systematic review titles, economic evaluations and costing studies, health reform descriptions, and health system descriptions, as well as many types of added-value coding (Table 1, last column). It is continuously updated using a variety of regularly scheduled hand searches, direct feeds, and other approaches and regular translation of new content into Arabic, Chinese, English, French, Portuguese, Russian, and Spanish (Figure 1). We provide a complete description of the contents of Health Systems Evidence in a separate paper [25].

Conclusions

Policymakers and stakeholders can now easily access and use a wide variety of types of research evidence about health systems to inform decision-making and advocacy. Moreover, they can also access and use complementary content, such as WHO policy-relevant documents. Researchers and research funding agencies can use Health Systems Evidence to identify gaps in the current stock of research evidence and domains that could benefit from primary research, systematic reviews and overviews of reviews.

The main strengths of our approach include: i) developing and iteratively refining a taxonomy based on existing categorization schemes and the practical challenges that arise when using it to code a heterogeneous body of research evidence; ii) building content through systematic searches of a broad array of sources, adding value to that content in ways that have been identified as important to policymakers and stakeholders, and having two reviewers independently participate in each step of the process; iii) expanding the types of content included in the one-stop shop as we achieve comprehensiveness for existing content and recognize the next most pressing gap faced in the type of research evidence needed; iv) designing the one-stop shop in a way that uses links to drive traffic to the websites of those groups involved in producing and/or disseminating these documents (e.g., Cochrane Library, Centre for Reviews and Dissemination, Rx for Change) or in preparing decision-relevant summaries (e.g., Australasian Cochrane Centre and SUPPORT collaboration) so that these groups get credit for their work in the form of website hits; and v) continuously updating the one-stop shop in all seven supported languages.

The limitations of our study are: i) having some sub-categories in the taxonomy containing very large numbers of documents, which could frustrate busy policymakers, stakeholders, and researchers; ii) potentially missing review-derived products, particularly evidence briefs [17], which are a new type of research product and not easily found through traditional search mechanisms; and iii) having a user interface that can appear overly complicated to new users. We are now working to address these limitations, particularly the user interface by engaging in ongoing user testing with health system policymakers and stakeholders.

Electronic supplementary material

Acknowledgments

Abbreviations

Footnotes

Contributor Information

John N Lavis, Email: ac.retsamcm@jsival.

Michael G Wilson, Email: ac.retsamcm@2mosliw.

Kaelan A Moat, Email: ac.retsamcm@aktaom.

Amanda C Hammill, Email: ac.retsamcm@alimmah.

Jennifer A Boyko, Email: moc.liamg@okyobnnej.

Jeremy M Grimshaw, Email: ac.irho@wahsmirgj.

Signe Flottorp, Email: on.teretnesspaksnnuk@prottolf.engis.

References