See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/40897605

Are Key Principles for Improved Health

Technology Assessment Supported and Used by
Health Technology...

Article in International Journal of Technology Assessment in Health Care January 2010

DOI: 10.1017/S0266462309990833 Source: PubMed

CITATIONS READS

35 93

7 authors, including:

Peter Neumann Bryan R Luce

Tufts Medical Center United BioSource Corporation
392 PUBLICATIONS 10,745 CITATIONS 21 PUBLICATIONS 1,392 CITATIONS

SEE PROFILE SEE PROFILE

Uwe Siebert
Private Universitt fr Gesundheitswissensc
463 PUBLICATIONS 9,406 CITATIONS

SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate, Available from: Uwe Siebert
letting you access and read them immediately. Retrieved on: 10 November 2016
International Journal of Technology Assessment in Health Care, 26:1 (2010), 7178.
Copyright

c Cambridge University Press, 2010
doi:10.1017/S0266462309990833

POLICIES

Are Key Principles for improved

health technology assessment
supported and used by health
technology assessment
organizations?
The International Working Group for HTA Advancement

Peter J. Neumann
Tufts Medical Center
Michael F. Drummond
University of York
Bengt Jonsson
Stockholm School of Economics
Bryan R. Luce
United BioSource Corporation
J. Sanford Schwartz
University of Pennsylvania
Uwe Siebert
University for Health Sciences, Medical Informatics and Technology
Sean D. Sullivan
University of Washington

Previously, our groupthe International Working Group for HTA Advancementproposed

a set of fifteen Key Principles that could be applied to health technology assessment (HTA)
programs in different jurisdictions and across a range of organizations and perspectives.
In this commentary, we investigate the extent to which these principles are supported and

The International Working Group for HTA Advancement was established in July 2007 with unrestricted funding from the Schering Plough Corporation.
The mission of the Working Group is to provide scientifically based leadership to facilitate significant continuous improvement in the development and
implementation of practical, rigorous methods into formal health technology assessment (HTA) systems and processes, by facilitating development and
adoption of high quality, scientifically driven, objective, and trusted HTA to improve patient outcomes, the health of the public and overall healthcare quality
and efficiency. We are grateful to Andrew Mitchell, Jill Sanders, Tony Tarn, and Bong-Min Yang for providing feedback on selected HTA organizational
practices. The evaluations and views expressed in this study are those of the authors and do not necessarily reflect the opinions of any of these individuals or
their organizations. We are also grateful to Hannah Auerbach for excellent research assistance and to the anonymous referees for constructive comments on
an earlier draft.

71
Neumann et al.

used by fourteen selected HTA organizations worldwide. We find that some principles are
broadly supported: examples include being explicit about HTA goals and scope;
considering a wide range of evidence and outcomes; and being unbiased and transparent.
Other principles receive less widespread support: examples are addressing issues of
generalizability and transferability; being transparent on the link between HTA findings and
decision-making processes; considering a full societal perspective; and monitoring the
implementation of HTA findings. The analysis also suggests a lack of consensus in the
field about some principlesfor example, considering a societal perspective. Our study
highlights differences in the uptake of key principles for HTA and indicates considerable
room for improvement for HTA organizations to adopt principles identified to reflect good
HTA practices. Most HTA organizations espouse certain general concepts of good
practicefor example, assessments should be unbiased and transparent. However,
principles that require more intensive follow-upfor example, monitoring the
implementation of HTA findingshave received little support and execution.

Keywords: Cost-effectiveness analysis, Health technology assessment, Health policy

Health technology assessment (HTA) has been defined as a Selecting HTA Organizations
multi-disciplinary field of policy analysis, studying the med- for the Exercise
ical, economic, social and ethical implications of develop- We selected a variety of HTA organizations against which to
ment, diffusion and use of health technology (11). A variety evaluate support and use of the Key Principles. We attempted
of public and private sector organizations, advisory commit- to capture a sample of agencies worldwide, and to include
tees and regulatory bodies now conduct HTA worldwide. examples of both established and emerging entities with dif-
Previously, our groupthe International Working Group for ferent roles and objectives. We included both traditional HTA
HTA Advancementproposed a set of fifteen Key Princi- agencies, as well as reimbursement agencies focusing mainly
ples to guide HTA assessments (5). In this commentary, we on drugs. We included public and private organizations: in
investigate the extent to which these principles have been most nations, the organizations that perform HTAs are public
supported and implemented at selected HTA organizations sector groups, reflecting the host countrys public financing
around the world. and/or provision of health care. However, private sector or-
ganizations also undertake HTAs, particularly in the United
DATA AND METHODS States, where private health insurance is common (13;16;18).
Finally, we also focused on HTA organizations, based on spe-
The Key Principles cific knowledge of members of our group.
In developing the Key Principles, our group observed that Ultimately, we included fourteen HTA organizations for
HTA was a dynamic and rapidly evolving process, embracing the exercise: the Centers for Medicare and Medicaid Ser-
different types of assessments. We also noted that HTA orga- vices (CMS) (U.S.); the Washington State Medicaid pro-
nizations, which themselves vary in substantial ways, were gram/Drug Effectiveness Review Project (DERP) (U.S.);
increasingly undertaking or commissioning HTAs to inform Wellpoint (U.S.); Blue Cross Blue Shield Associations,
a variety of health policy decisions. We further noted that Technology Evaluation Center (U.S.); National Institute for
the landscape for HTA was changing rapidly in the United Health and Clinical Excellence (NICE) (England and Wales);
States, Europe, and parts of Asia and Latin America (5). Institute for Quality and Efficiency in Health Care (IQWiG)
The Key Principles were designed to build upon other (Germany); German Agency for Health Technology Assess-
efforts, which sought to describe HTA systems or to iden- ment at the Institute for Medical Documentation and Infor-
tify appropriate and inappropriate practice for the conduct of mation (DAHTA@DIMDI) (Germany); Council on Technol-
HTAs (3;6;7;10). In developing the principles, our main focus ogy Assessment in Health Care (SBU) (Sweden); Dental and
was on those HTA activities that were linked to, or included a Pharmaceutical Benefits Agency (TLV) (Sweden); Canadian
particular resource allocation decision. We also emphasized Agency for Drugs and Technologies in Health (CADTH)
that it was important to consider the link between the HTA (Canada); Department of Health Technology Assessment,
and the decisions that followed them. A detailed description Health Insurance Review Agency (HIRA) (South Korea);
of the principles and the rationale for each is provided else- National Health Surveillance Agency (Anvisa) (Brazil); Cen-
where (5). Table 1 enumerates the principles, divided into ter for Drug Evaluation (CDE) (Taiwan); and Pharmaceutical
four sections: structure of HTA programs, methods of HTA, Benefits Advisory Committee (Australia).
processes for conducting HTA, and the use of HTA for deci- We recognize that these organizations are not necessar-
sion making. ily representative of the entire universe of HTA agencies.

72 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010

 Table 1. Support and Use of Key HTA Principles Across Selected Organizations

Washington DAHTA@
CMS Medicaid/ WellPoint BCBS TEC NICE IQWiG DIMDI TLV SBU CADTH HIRA PBAC Anvisa DHTA
(US)a DERP (US)b (US) (US) (UK) (Germany) (Germany) (Sweden) (Sweden) (Canada) (Korea) (Australia) (Brazil) (Taiwan)c

Year of inception 1999 2003 2009 1985 1999 2004 2000 2002 1987 1990 2008 1992 1999 1998

Key Principle

Structure of HTA
program
1 The goal and scope of ++ ++ ++ ++ ++ ++d + + + ++ ++ ++ + ++
the HTA should be
explicit and relevant to
its use
2 HTA should be an ++ ++ ++ ++ ++ ++ ++ ++ ++ + +
unbiased and
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010

transparent exercise
3 HTA should include all ++ ++ ++ ++ ++ ++ + +
relevant technologies
4 A clear system for + + ++ + ++ + ++ ++ ++
setting priorities for
HTA should exist
Methods of HTA
5 HTA should incorporate + ++ +d ++ ++ + ++ + ++ +
appropriate methods
for assessing costs and
benefits
6 HTAs should consider a ++ + + ++ ++ ++d ++ ++ ++ ++ + + ++

Are HTA principles supported by HTA organizations?

wide range of evidence
and outcomes
7 A full societal + ++ ++ +
perspective should be
considered when
undertaking HTAs
8 HTAs should explicitly ++ ++ + + ++ + ++ +
characterize
uncertainty
surrounding estimates
9 HTAs should consider + ++ + + ++ + +
and address issues of
generalizability and
transferability
Processes for
conducting HTA
10 Those conducting HTAs ++ + ++ ++ ++ + ++ +
should actively engage
all key stakeholder
groups
73
74

Neumann et al.
Table 1. Continued

Washington DAHTA@
CMS Medicaid/ WellPoint BCBS TEC NICE IQWiG DIMDI TLV SBU CADTH HIRA PBAC Anvisa DHTA
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010

(US)a DERP (US)b (US) (US) (UK) (Germany) (Germany) (Sweden) (Sweden) (Canada) (Korea) (Australia) (Brazil) (Taiwan)c

Year of inception 1999 2003 2009 1985 1999 2004 2000 2002 1987 1990 2008 1992 1999 1998

11 Those undertaking ++ + ++ ++ ++ + ++ ++ + ++ + ++
HTAs should actively
seek all available data
12 The implementation of + + ++
HTA findings needs to
be monitored
Use of HTA in decision
making
13 HTA should be timely ++ ++ + + ++ + ++ + + + + + ++
14 HTA findings need to be ++ ++ ++ + ++ ++ + +
communicated
appropriately to
different decision
makers
15 Link between HTA + + + ++ + ++ ++
findings and decision-
making processes
needs to be transparent
and clearly defined
Note. + signifies that the organization supported the principle in question in written guidelines or other form, regardless of whether they actually follow it.
++ means that the organization implemented the principle in published reports and decisions based on these reports demonstrate adoption of the specific principle.
CMS = Centers for Medicare and Medicaid Services.
DERP = Drug Effectiveness Review Project.
BCBS TEC = Blue Cross Blue Shield Associations, Technology Evaluation Center.
NICE = National Institute for Health and Clinical Excellence.
IQWiG = Institute for Quality and Efficiency in Health Care.
DIMDI = German Agency for Health Technology Assessment at the Institute for Medical Documentation and Information.
TLV = The Dental and Pharmaceutical Benefits Agency.
SBU = The Council on Technology Assessment in Health Care.
CADTH = Canadian Agency for Drugs and Technologies in Health.
HIRA = Department of Health Technology Assessment, Health Insurance Review Agency.
Anvisa = National Health Surveillance Agency (Office of Economic Evaluation of Health Technologies).
PBAC = Pharmaceutical Benefits Advisory Committee.
DHTA = Division of Health Technology Assessment.
a This table refers to CMSs HTA process for national coverage decision making.
b Washington Medicaid is one of 14 participants in the DERP. DERP researchers conduct health technology assessments for drug classes. Participants in the DERP, such as the Washington Medicaid
program, retain local authority for interpreting DERP reports and for decision making regarding which drugs to pay for. Though the Medicaid program makes decisions on all technologies, the DERP
focuses only on drugs.
c DHTA is within the Center for Drug Evaluation in Taiwan.
d There was disagreement in the group about whether IQWiG warranted a plus for principles 1 and 6, and whether methods for assessing costs and benefits (principle 5) were appropriate.
Moreover, at the time of the evaluation, IQWiG had not yet performed a cost-effectiveness assessment so implementation of this principle could not be judged.
 Are HTA principles supported by HTA organizations?

Other investigators might have selected other representative blush effort to analyze support and use of the principles to
organizations among the many dozens of existing HTA en- advance the discussion about good HTA practices and to
tities worldwide. Our goal was to produce a list that was facilitate their acceptance and adoption. In the discussion
diverse with respect to geography, scope, and stage of de- section, we expand upon the effort that might be required to
velopment; included leading organizations familiar to group undertake a formal benchmarking exercise.
participants; and comprised a useful set for assessing the
adequacy of the principles.
RESULTS

Evaluating Support and Implementation Table 1 indicates that there is considerable variation around
of the Principles support of the principles. Some principles are broadly sup-
ported. Examples include being explicit about HTA goals
We investigated the extent to which each of the fifteen prin- and scope (supported by all of the fourteen organizations an-
ciples have been supported and implemented by the fourteen alyzed), being timely in assessments (13 of the 14), consider-
HTA organizations. By supported, we meant that the orga- ing a wide range of evidence and outcomes (13/14), seeking
nization embraces the principle in written guidelines or other all available data (12/14), and being unbiased and transparent
form, regardless of whether they actually follow it. By im- (11/14). Other principles have received less widespread sup-
plemented, we meant that published reports and decisions port. Examples include addressing issues of generalizability
based on these reports demonstrate adoption of the specific and transferability (7/14), being transparent about the link be-
principle. tween HTA findings and decision-making processes (7/14),
We evaluated each principle for each organization in a considering a full societal perspective (4/14), and monitoring
two-stage process: we gave a plus sign to a principle ref- the implementation of HTA findings (3/14).
erenced by an organization in its published charter or guide- There also is variation in terms of the level of imple-
lines, or if we could infer from other information available mentation of the principles. Examples of principles being
that the principle was relevant; (i.e., supported); we con- implemented more widely include being explicit about
ferred a second plus sign if the principle was actually adhered the goals and scope of HTA (implemented by 9 of the 14
to by the organization in practice (i.e., implementation). organizations), being unbiased and transparent (9/14), and
The two-stage process enabled us to capture the extent considering a wide range of evidence and outcomes (9/14).
to which the principles have been supported and, the extent to Implementation of other principles is lagging, including,
which they have been realized in practice. Our intention was being transparent about the link between HTA findings and
to focus on uptake and use of the principles, rather than to decision-making processes (3/14), considering a full societal
issue a verdict or report card on the HTA entities evaluated. perspective (2/14), considering issues of generalizability and
One of the co-authors took the lead for the evaluation of transferability (2/4), and monitoring HTA implementation
the principles for each organization. Each evaluator reviewed (1/14).
the Web site, mission, and activities of the organization in There is variation in the degree to which the HTA or-
question. In many cases, the co-author conducting the eval- ganizations we examined are supporting and implementing
uation had participated in technology assessments for the the key principles. Some agencies, such as NICE, IQWiG,
organization and/or had written about the HTA process at the DAHTA@DIMDI, SBU, CADTH, and PBAC support twelve
organization and about particular decisions. Based on this or more of the principles, for example. Others, such as Wash-
knowledge, the evaluator issued an overall judgment about ington Medicaid/DERP; Blue Cross Blue Shield TEC; and
the extent to which published reports and decisions based DHTA (Taiwan) have supported six or fewer.
on these reports were in line with the specific principle. As
a rule, we chose not to ask the respective organizations ei-
ther to self-evaluate or to review and comment on our as- DISCUSSION
sessments. However, in a few cases, we sought input from
individuals affiliated with an HTA organization in question Support and Use of the Principles
to clarify certain practices. Those individuals are noted in the The Key Principles were intended to provide a set of univer-
acknowledgment section of the study. The judgments were sally applicable guideposts for HTA programs in different
then discussed in a group consensus meeting of the working jurisdictions and across a range of organizations with differ-
group. All final evaluations are from the group and do not ent roles, objectives, and perspectives. We previously argued
necessarily reflect the views of others. The evaluations are that application of the principles had the potential to improve
current as of August 15, 2009. clinical and policy decisions, to enhance access to clinically
We recognized at the outset that, despite our attempt to effective and cost-effective care, to improve the efficiency of
provide objective and thorough assessments, our judgments care, and to advance the health of the public (5). We also rea-
were by nature somewhat impressionistic. We present this soned that adoption of the principles could help enhance the
exercise in the spirit of a commentary, intended as a first- quality and credibility of HTAs organizations, while building

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010 75

Neumann et al.

greater trust in and support for HTA programs. While other goal in this study was to advance the practice of HTA and to
groups have examined differences across HTA entities in stimulate informed discussion through an extended and inter-
terms of organizational aspects and certain methodological active process (5). We acknowledge that there is subjectivity
practices (4;8;14;17;18), this commentary represents a first in our evaluations and that other researchers or the agen-
examination of support and use of the key principles. cies themselves may be more or less strict about whether
The analysis illustrates a mixed picture, with some prin- a particular principle has been supported or implemented.
ciples receiving broad support and others much less so. If In some cases, we judged our assessments to be more or
there is a pattern here, perhaps it is that most HTA organiza- less straightforward, while in others, the evaluation required
tions espouse certain general concepts of good practicefor greater discretion, and caused considerable debate within the
example, that goals and scope of HTA should be explicit group. In particular, there was disagreement around whether
and relevant to its use and that HTA should be unbiased IQWiG has supported various principles, including whether
and transparent. On the other hand, principles that require the goal and scope was explicit and relevant to its use (prin-
more intensive follow up are not regarded as the primary ciple 1); whether its methods for assessing costs and benefits
responsibility of the agencyfor example, monitoring the and Germanys decision not to consider health resource allo-
implementation of HTA findings or communicating findings cation decisions across diseases were appropriate (principle
to different decision makershave received little support and 5); and whether the agency considers a wide range of evi-
execution. The analysis also suggests a lack of consensus in dence and outcomes (12).
the field about certain principlesfor example, considering The two-stage process for judging support and imple-
a societal perspectiveor perhaps a need for more guidance mentation has its pros and cons. Generally, it was relatively
about how to implement certain principlesfor example, easy to agree on support, because these assessments are based
addressing issues of generalizability and transferability. mainly on written documentation. On the other hand, agree-
ment on implementation was more difficult, because it de-
Cross-National Differences in HTA pends partly on local knowledge and perceptions, as well as
actual evidence of performance. In addition, it is important to
Application of the principles also serves to illustrate varia-
recognize that our applications are point-in-time snapshots
tions in the experiences and performances of HTA agencies.
in a very dynamic field in which organizations and programs
Perhaps not surprisingly, no single organization supports
are continually evolving. In some instances, such as the case
and applies all of the principles. In general, there appears
of IQWiGs assessments of costs and benefits, the agency has
to be more support and implementation of the key princi-
not yet performed economic evaluations, so it remains to be
ples among the HTA organizations we evaluated in Europe,
seen whether it will follow this principle. Yet another issue is
Canada, and Australia, compared with those in the United
that some of the organizations we included (e.g., BCBS TEC,
States, Brazil, and Asia. Notably, three of the four large
DERP) are not the ones themselves making the resource allo-
U.S.-based organizations that we examined (CMS, Wash-
cation decisions. Thus, it is not straightforward and perhaps
ington Medicaid/DERP, and Blue Cross Blue Shield) do not
not even fair to judge them against all of the principles in-
generally support the assessment of costs and benefits.
cluded here. Still, we included them as examples of leading
We have pointed out previously that there is no sin-
HTA entitiesand because the exercise serves to highlight
gle way to conduct HTAs that will meet all of the needs of
challenges in moving HTA organizations to support and use
all decision makers, stakeholders, and societies (5). Thus,
universal principles.
one does not expect all HTA organizations to conduct as-
We also recognize that not everyone agrees with the fif-
sessments in the same way given their many differences,
teen principles themselves. Indeed, we have received many
including differences across jurisdictions in health systems,
constructive comments and suggestions since their publica-
institutional structures and governance, statutory authority,
tion. Some observers have argued that certain principles (e.g.,
data availability and resources, cultures, traditions, incomes,
stakeholder involvement) are too general, while other prin-
and local practice patterns, prices, and preferences. More-
ciples (e.g., dissemination of findings) lack clarity as to the
over, the HTA organizations we considered are in different
intended audience (2,9). Others have questioned whether the
stages of development and vary considerably in the resources
principles are realistic and whether it is fair to evaluate orga-
available to them. Some, such as PBAC and NICE (estab-
nizations relative to all of the principles (2). One reviewer,
lished in 1992 and 1999, respectively), are well established
for example, pointed out that many HTA agencies would
while others, such as HIRAs HTA unit (2008), are emerging
find the full process not feasible, opting instead for quick
and have only recently adopted a set of publicly available
and dirty assessments, which would be better than none (2).
guidelines.
Several critics emphasized that there is a general tradeoff
between rigor and inclusiveness of HTAs on the one hand,
Key Challenges and timeliness on the other (1;2;9).
Undoubtedly, some will take issue with some of our assess- A few observers pointed out that the key principles
ments. As was the case in developing our key principles, our omit potentially important items, such as the degree to

76 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010

 Are HTA principles supported by HTA organizations?

which HTAs incorporate new assessment methods (e.g., for some principles will lend themselves more readily to such
evaluating patient preferences and patient reported out- benchmarking than others. Still, we believe that it will prove
comes), or the extent to which HTAs consider social, psy- a useful endeavor.
chological, or ethical dimensions (2;15). Some argued that We continue to believe that defining, gaining consensus,
the principles focus undue attention on resource allocation and adopting good HTA principles are essential for con-
as the basis for HTA (2). Finally, we have heard informally sistent, informed, and rational decision making. This study
from some who take issue with the premise of our initiative demonstrates considerable variation in uptake of the princi-
itself. Those individuals have argued that a statement of key ples we have proposed, and, to the extent that the principles
principles carries an implicit endorsement of a centralized have merit, there is much room for improvement. We also
HTA process including assessment of cost-effectiveness as believe that there is a need for flexibility as well as a need to
the gold standard. These critics charge that such efforts serve revisit continually these and other Key Principles and to apply
governmental and payer interests and short-term budgetary them to other HTA organizations not evaluated here. Above
mandates rather than patient needs, and that such principles all, there is a need to engage in thoughtful and constructive
should not be championed, even if they incorporate so-called dialogue and debate with colleagues across disciplines and
good practices. those with different viewpoints.
To these and other criticisms, we counter that scientif-
ically rigorous and comprehensive HTA is a critical ingre- CONTACT INFORMATION
dient for improving the effectiveness and efficiency of the
healthcare system. Regarding criticisms that the principles Peter J. Neumann, ScD (pneumann@tuftsmedicalcenter.
are unrealistic, idealistic, or unachievable, we would empha- org), Professor, Tufts University School of Medicine; Di-
size that principles are just that. . .principles. They are not rector, Center for the Evaluation of Value and Risk in Health,
standards or requirements. Furthermore, the principles we Tufts Medical Center, 800 Washington Street #63, Boston,
put forth are intended to be forward looking and designed to Massachusetts 02111
guide the conduct and evaluation of HTA, specifically relative Michael Drummond, BSc (md18@york.ac.uk), MCom,
to resource allocation decisions. We believe that the intended DPhil, Professor of Health Economics, Centre for Health
audiences of the principles include those working in HTA as Economics, University of York, Heslington, York, North
well as all decision makers, including patients. We also be- Yorkshire YO10 5DD, UK
lieve that a focus on resource allocation is justified, given that Bengt Jonsson, PhD (bengt.Jonsson@hhs.se) Professor, De-
HTAs are not simply reports disseminated to the public but partment: Department of Economics, Stockholm School of
increasingly influencing coverage and reimbursement deci- Economics, 65, Sveavagen, Stockholm, SE 11383 Sweden
sions. Moreover, because all clinical decisions affect the use Bryan R. Luce, PhD, MBA (bryan.luce@unitedbiosource.
of healthcare resources, an explicit consideration of resource com), Senior Vice President, Science Policy, United
allocation is justified (5). BioSource Corporation, 7101 Wisconsin Avenue, Bethesda,
Maryland 20817; Adjunct Senior Fellow, Leonard Davis In-
stitute of Health Economics, University of Pennsylvania,
Next Steps 3641 Locust Walk, Philadelphia, Pennsylvania 19104
Our intention is to stimulate debate and to lead policy mak- J. Sanford Schwartz, MD (schwartz@wharton.upenn.edu),
ers to reflect on ways to improve the concepts and practice Professor of Medicine, Health Care Management, and Eco-
of HTA. Ideally, other researchers will conduct their own nomics, School of Medicine and Wharton School, University
studies of HTA principles and HTA organizations. It will be of Pennsylvania, Blockley Hall Suite #1120, Philadelphia,
useful to have a broader discussion on ways to move HTA or- Pennsylvania 19104
ganizations to adopt or adapt principles that currently do not Uwe Siebert, MD, MPH, MSc, ScD (public-health@umit.
receive widespread support, such as considering a societal at), Professor of Public Health (UMIT), Chair, Department of
perspective and monitoring results of HTAs. Public Health, Information Systems and Health Technology
In the future, researchers might also consider how to Assessment, UMITUniversity for Health Sciences, Medical
undertake more formal benchmarking exercises. It will be Informatics and Technology, Eduard Wallnoefer Center 1,
useful to prespecify and quantify more precisely the criteria Hall i.T., Austria, A-6060; Adjunct Professor of Health Pol-
for achieving a positive verdict on support and use of the prin- icy and Management, Center for Health Decision Science,
ciples. For example, an evaluation of whether an HTA orga- Department of Health Policy and Management, Harvard
nization has successfully implemented principle 13 (timely School of Public Health, 718 Huntington Avenue, Boston,
HTA) might stipulate a period (e.g., 6 months) for producing Massachusetts 02115
HTA reports, and a criterion (e.g., that 75 percent of reports Sean D. Sullivan, PhD (sdsull@u.washington.edu) Profes-
must have been done within the 6-month window) for an sor, Pharmaceutical Outcomes Research and Policy Program,
HTA to achieve a favorable evaluation. Such an evaluation University of Washington, 1959 NE Pacific Avenue, Box
was beyond the scope of our exercise. We recognize that 357630, Seattle, Washington 98195

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010 77

Neumann et al.

REFERENCES 10. Hutton J, McGrath C, Frybourg JM, et al. Framework for de-
1. Aronson N. Establishing key principles for health technology scribing and classifying decision-making systems using tech-
assessment. ISPOR 13th Annual International Meeting. 2008. nology assessment to determine the reimbursement of health
2. Banta HD. Commentary on the article, Key principles for technologies (fourth hurdle systems). Int J Technol Assess
the improved conduct of health technology assessments for re- Health Care. 2006;22:10-18.
source allocation decisions. Int J Technol Assess Health Care. 11. International Network of Agencies for Health Technology
2008;24:362-368. Assessment (INAHTA). http://www.inahta.org/HTA (accessed
3. Busse R, Orvain J, Velasco M, et al. Best practice in undertak- March 31, 2008).
ing and reporting health technology assessments. Int J Technol 12. Jonsson B. IQWiG: An opportunity lost? Eur J Health Econ.
Assess Health Care. 2002;18:361-422. 2008;9:205-207.
4. Drummond M, Grubert N. International trends in the use of 13. Luce B, Cohen RS. Health technology assessment in the United
health economic data. Waltham MA: Spectrum Report, Deci- States. Int J Technol Assess Health Care. 2009;25(Suppl 1):33-
sion Resources; 2007. 41.
5. Drummond MF, Schwartz JS, Jonsson B, et al. Key principles 14. Martelli F, La Torre G, Di Ghionno E, et al. Health technology
for the improved conduct of health technology assessments for assessment agencies: An international overview of organiza-
resource allocation decisions. Int J Technol Assess Health Care. tional aspects. Int J Technol Assess Health Care. 2007;23:414-
2008;24:244-258. 424.
6. Emanuel EJ, Fuchs VR, Garber AM. Essential elements 15. Neuhauser D. Commentary on the article, Key principles for
of a technology and outcomes assessment initiative. JAMA. the improved conduct of health technology assessments for re-
2007;298:1323-1325. source allocation decisions. Int J Technol Assess Health Care.
7. European Federation of Pharmaceutical Industries and Asso- 2008;24:362-368.
ciations. The use of health technology assessments (HTA) to 16. Neumann PJ, Sullivan SD. Economic evaluation in the US:
evaluate medicines: EFPIA key principles. Brussels: EFPIA; What is the missing link? Pharmacoeconomics. 2006;24:1163-
2005. 1168.
8. Garcia-Altes A, Ondategui-Parra S, Neumann PJ. Cross- 17. Schwarzer R, Siebert U. Methods, procedures, and contextual
national comparison of technology assessment pro- characteristics of health technology assessment and health pol-
cesses. Int J Technol Assess Health Care. 2004;20:300- icy decision making: Comparison of health technology assess-
310. ment agencies in Germany, United Kingdom, France, and Swe-
9. Hailey D. Commentary on the article, Key principles for the den. Int J Technol Assess Health Care. 2009;25:305-314.
improved conduct of health technology assessments for re- 18. Sullivan SD, Watkins J, Sweet B, Ramsey SD. Health technol-
source allocation decisions. Int J Technol Assess Health Care. ogy assessment in health care decisions in the United States.
2008;24:362-368. Value Health. 2009;12:s39-s44.

78 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 26:1, 2010