Using Pragmatic Clinical Trials To Test The Effectiveness of Patient-Centered Medical Home Models in Real-World Settings PCMH Research Methods Series
Using Pragmatic Clinical Trials To Test The Effectiveness of Patient-Centered Medical Home Models in Real-World Settings PCMH Research Methods Series
Using Pragmatic Clinical Trials To Test The Effectiveness of Patient-Centered Medical Home Models in Real-World Settings PCMH Research Methods Series
Agency for Healthcare Research and Quality Advancing Excellence in Health Care www.ahrq.gov
Using Pragmatic Clinical Trials to Test the Effectiveness of Patient-Centered Medical Home Models in Real-World Settings
This brief focuses on using pragmatic clinical trials (PCTs) to evaluate patient-centered medical home (PCMH) models. It is part of a series commissioned by the Agency for Healthcare Research and Quality (AHRQ) and developed by Mathematica Policy Research under contract, with input from other nationally recognized thought leaders in research methods and PCMH models. The series is designed to expand the toolbox of methods used to evaluate and refine PCMH models. The PCMH is a primary care approach that aims to improve quality, cost, and patient and provider experience. PCMH models emphasize patient-centered, comprehensive, coordinated, accessible care, and a systematic focus on quality and safety.
Though not discussed in this brief, observational and other nonexperimental evaluation approaches may provide biased estimates of program effects because (1) it is difficult to develop a fair comparison group for a highly motivated intervention group and (2) other external factors may contribute to the outcomes of interest but may not be accounted for in the analysis (Tunis, Stryer, and Clancy, 2003).
Measure effects on quality, cost, and patient and provider experience. Study end points in traditional RCTs typically include measures such as mortality and major morbidity but exclude outcomes that are sometimes of greatest relevance to decisionmakers deciding whether to adopt and spread PCMH models (Tunis, Stryer, and Clancy, 2003). Outcomes of interest to decisionmakers include quality, cost, and patient and provider experience. Adapt to the local context and evolve over time. Traditional RCTs deliver the same intervention across a range of settings and do not vary the intervention over time. However, there is consensus among researchers and decisionmakers that PCMH approaches need to be operationalized differently for different practices, to reflect their settings, resources, and patient panels. For example, practices with a higher proportion of low income and chronically ill patients may need to provide more coordination with social services. Similarly, rural practices may manage more of their patients care needs internally due to shortages of specialist care. In addition, implementers want to be able to refine a model over time based on operational realities and learning as the intervention unfolds, rather than keeping it static. In contrast to traditional RCTs, PCTs (sometimes called practical RCTs in the literature) are designed to directly inform health policy and decisionmaking. Below, we describe how to use PCTs to study PCMH models, as well as their advantages and limitations. Although this brief focuses on PCTs, many of the recommendations for using this approach to improve upon the limitations of traditional RCTs also apply to nonexperimental evaluation designs.
See Peikes, Dale, Lundquist, et al. (2011) for a description of how evaluators can measure different outcomes for different samples to enhance the statistical power to detect true effects.
Recruit from a variety of practice settings. PCTs recruit practices from the community that do not traditionally participate in research. Although it is challenging to recruit from a wide range of practices, doing so improves external validity and ensures the representativeness and real-world applicability of study results. In the PCMH context, a PCT approach would include smaller, independent practicesthe practices that deliver much of the countrys primary care. Most early tests of PCMH models were completed in integrated delivery systems because these systems could reallocate the potential downstream savings from reduced use of acute and specialty care to pay for the PCMH intervention. However, we also need to know what happens in smaller, independent practices. In an effort to include multiple practice settings, evaluators could consider: inviting all practices in a community to participate; assessing the take-up rate; and examining how the outcomes vary for practices that had different degrees of readiness and motivation to become a PCMH, as well as different degrees of primary care functionality before the intervention. Measure a broad range of outcomes. PCTs choose outcomes that measure the anticipated effects of the intervention and that can inform decisionmaking by stakeholders. PCTs may also include a longer followup period than traditional RCTs. In the PCMH context, a PCT approach would ideally collect evidence on a full range of outcomes important to various stakeholders. For example, insurers might be primarily interested in the total medical costs they paid and in patient satisfaction; employers might be concerned with absenteeism and presenteeism among members of their workforce; primary care providers might focus on practice revenue and profit, quality of care, and staff satisfaction; and labor unions might focus on patient functioning, satisfaction, and out-of-pocket costs. At the outset of an evaluation, evaluators should elicit input from stakeholders about the key information needed to guide decisions. Additionally, evaluators should dedicate sufficient resources to collect a comprehensive set of data. AHRQ recommends that evaluations measure the effects of PCMH models on quality, health services utilization and cost, and patient and provider experience (Peikes, Zutshi, Genevro, et al., 2012b). AHRQ released the PCMH-Consumer Assessment of Healthcare Providers and Systems in 2011 to measure patient experience and has efforts under way to develop survey instruments to measure provider experience. Additionally, the Commonwealth Fund developed a set of core measures of utilization, cost, and clinical quality (Rosenthal, Abrams, Bitton, et al., 2012). Adapt the intervention being tested to the local context. In addition to the four considerations discussed above, it is also important to generate evidence on PCMH models that have been adapted to the local context. PCTs can be used to help test the effectiveness of a PCMH model that is tailored to individual practices and their markets. Using a large sample of practices allows evaluators to analyze how different practices adapt the PCMH model, and how the practices organizational structure, initial characteristics, and adaptations affect outcomes. To do so, the evaluation needs a strong implementation component that includes collecting data about differences among practices prior to the intervention in terms of size, medical neighborhood, patient composition, and initial primary care capabilities and functionalities. In addition, the evaluation should include an implementation analysis to capture information about how different practices operationalize each component of the medical
home.3 The impact analysis can then use this information to identify the effects of different variants of the PCMH in different settings.
III. Advantages
Below we describe two main advantages to using PCTs when evaluating PCMH models. Increases relevance of study results for key stakeholders. The main advantage of PCTs is that the findings are more relevant to clinicians, patients, and decisionmakers than those of traditional RCTs. Since PCTs test the intervention on typical patients and practices, and assess a PCMH model that is refined as it is implemented, they can more closely mirror the real world and provide a better sense of what is actually happening, what is feasible, and whether an intervention will be successful and useful in various practice settings. Emphasizes important contextual factors to enhance replicability. Because PCTs emphasize contextual factors and real-world applicability, they can measure and document factors such as the amounts of resources, staff, and time needed to implement a PCMH model. This information can help other clinicians, evaluators, and policymakers determine whether an intervention is replicable in their settings, and if so, indicate how to do so.
IV. Limitations
There are three main limitations, discussed below, to using PCTs when evaluating PCMH models. Entails a tradeoff between internal and external validity. The main limitation to the PCT approach is that it may reduce internal validity to gain the external validity that allows for more real-world applicability of the results. PCTs provide results that are more generalizable to other settings and contexts, but it may be harder to detect effects if the intervention is not always delivered as planned or if it is not uniform across practices. It is important to point out that an intervention tested in a PCT may be morerather than lesspotent than one tested in a strictly defined RCT if practices develop better interventions that meet their own specific resources and patient needs. If the effect size is larger, it could be easier to detect effects. Requires adequate resources. There are significant costs associated with conducting PCTs, including (1)implementing random assignment in real-world settings with representative populations of practices and patients, (2) collecting comprehensive outcomes data, and (3) carefully measuring implementation. Documenting implementation is critical to determine what was actually tested and to explore the sources of variation in outcomes across practices.
Other briefs in this series provide guidance about approaches for measuring implementation, as well as incorporating implementation with impact findings.
Increases difficulty of specifying the exact intervention delivered. A third limitation is that a PCT may make it difficult to tease out the effects of different components of the intervention in different settings. A PCT approach allows for more heterogeneous intervention components, which practices are encouraged to adapt to their specific contexts. It is possible for some components (or combinations) to be responsible for most of the observed intervention effects, while others have no effect, and for the patterns to vary across practice settings. Two methods can help researchers discern the best approaches for operationalizing the PCT model in different settings: (1) orthogonal design, which can be used at the outset of a study, and (2) fuzzy set qualitative comparative analysis, which can be used during the analysis phase after the intervention has taken place. This series includes briefs on both methods (see http://pcmh.ahrq.gov).
V. Conclusion
In contrast to a traditional RCT, a PCT can provide evidence that is more generalizable by measuring a broad range of policy-relevant outcomes across a diverse study population of practices and patients. This approach carefully considers the local practice context while still capitalizing on the methodological rigor of randomization, thus providing information that is most useful to clinicians, patients, payers, and other decisionmakers. For these reasons, future evaluations of PCMH models should consider investing the resources to conduct comprehensive PCTs, which will ultimately strengthen the evidence base on PCMH approaches.
VI. References
Agency for Healthcare Research and Quality (AHRQ). Consumer Assessment of Healthcare Providers and Systems (CAHPS) Web site. https://www.cahps.ahrq.gov/Surveys-Guidance/CG/PCMH.aspx. Glasgow RE. Pragmatic trials. Presented at the Training Institute for Dissemination and Implementation Research in Health; 2012 July 9-13; San Jose, CA. http://conferences.thehillgroup. com/OBSSRinstitutes/TIDIRH2012/presentations/Day4_Glasgow_CER%20&%20Pragmatic%20 Trials.pdf. Godwin M, Ruhland L, Casson I, et al. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003;3:28. Peikes D, Dale S, Lundquist E, et al. Building the evidence base for the medical home: what sample and sample size do studies need? Rockville, MD: Agency for Healthcare Research and Quality; 2011. AHRQ Publication No. 11-0100-EF. Peikes D, Zutshi A, Genevro J, et al. Early evaluations of the medical home: building on a promising start. Am J Manag Care. 2012a;18(2):105-16. Peikes D, Zutshi A, Genevro J, et al. Early Evidence on the Patient-Centered Medical Home. Final Report (Prepared by Mathematica Policy Research, under Contract Nos. HHSA290200900019I/ HHSA29032002T and HHSA290200900019I/HHSA29032005T). Rockville, MD: Agency for Healthcare Research and Quality. February 2012b. AHRQ Publication No. 12-0020-EF.
Roland M, Torgerson DJ. What are pragmatic trials? BMJ 1998;316(7127):285. Rosenthal M, Abrams MK, Bitton A, Patient-Centered Medical Home Evaluators Collaborative. Recommended core measures for evaluating the patient-centered medical home: cost, utilization, and clinical quality. The Commonwealth Fund, May 2012. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutic trials. J Chronic Dis 1967; 20(8):637-48. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003;290(12):1624-32.
VII. Resources
General Overview Information on PCTs
Glasgow, RE. Pragmatic trials. Presented at the Training Institute for Dissemination and Implementation Research in Health; 2012 July 9-13; San Jose, CA. http://conferences.thehillgroup. com/OBSSRinstitutes/TIDIRH2012/presentations/Day4_Glasgow_CER%20&%20Pragmatic%20 Trials.pdf. Roland M, Torgerson DJ. What are pragmatic trials? BMJ 1998;316(7127):285. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutic trials. J Chronic Dis 1967; 20(8):637-48. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003;290(12):1624-32.
This brief was prepared by Deborah Peikes, Ph.D. (dpeikes@mathematica-mpr.com); Kristin Geonnotti, Ph.D.; and Winnie Wang, B.A. (Mathematica Policy Research). Suggested Citation: Peikes D, Geonnotti K, Wang W. Using Pragmatic Clinical Trials to Test the Effectiveness of Patient-Centered Medical Home Models in Real-World Settings. Rockville, MD: Agency for Healthcare Research and Quality. February 2013. AHRQ Publication No. 13-0030-EF. This brief and companion briefs in this series are available for download from pcmh.ahrq.gov.