Step by Step – Evaluating Violence and Injury Prevention Policies

Brief 2: Planning For Policy Evaluation

This brief provides an overview of the first three steps in the program
evaluation Framework as they apply to policy evaluation: engaging
stakeholders, describing the policy, and focusing the evaluation design.

Step 1: Engage Stakeholders

Because multiple sectors participate in injury prevention policy efforts,
it can be challenging to come up with a complete list of potential
stakeholders. Consider the following types of stakeholders:
ƒƒ Policy experts ƒƒ Subject matter experts
ƒƒ Evaluation experts ƒƒ Implementers

For examples, see Appendix E. It can also be valuable to involve some of the stakeholders who were or are
opposed to the policy.1 They can provide valuable insight into initial or on-going resistance to the policy and
implementation, and their involvement can lend credibility to evaluation findings.

Roles and Responsibilities

Team members need a clear understanding of their degree of involvement and specific responsibilities.
Establish clear goals and expectations for each of the team members to keep the process on track and hold
members accountable.1 Consider which participating stakeholders are appropriate and available for the core
team.1,2 The core team should include stakeholders who are involved in the evaluation from beginning to
end and will assist with design and implementation of the evaluation as well as analysis and dissemination of
results. Other stakeholders may be more appropriate for specific steps of the evaluation process. Select one
or more evaluators to lead on the core team to coordinate the evaluation efforts. The lead evaluator is often
responsible for activities including planning, budgeting, developing the evaluation plan, guiding the team in
selecting evaluation questions and design, addressing data collection issues, compiling results, facilitating
discussion about interpretation of results, and preparing final evaluation results.1 Key considerations for
selecting an evaluator can be found in Appendix F.

Step 2: Describe the Policy Being Evaluated

When planning the policy evaluation, it’s important to have clarity and consensus on the components of the
policy being evaluated, what it is supposed to accomplish, and its underlying logic (i.e., why should this policy
achieve the intended impact?). Describing the policy can also assist with selecting appropriate indicators
and points of measurement. A logic model can be useful in describing the policy because it articulates the
underlying logic, the assumed causal pathways between a policy or policies and behaviors, and the links
between those behaviors and long-term impacts such as injury rates.3 A logic model helps to identify:

1 MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowksi, K., & Turner, K. (2001). Introduction to program evaluation for comprehensive
tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/tobacco/tobacco_
control_programs/surveillance_evaluation/evaluation_manual/pdfs/evaluation.pdf
2 W. K. Kellogg Foundation (1998, rev. 2004). Evaluation handbook. Battle Creek, MI: Author. Retrieved from http://www.wkkf.org/knowledge-
center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx
3 Milstein, B., & Chapel, T. (2012). Developing a logic model or theory of change. In The Community Toolbox (Part A, Chapter 2, Section 1; V.
Renault & S. Fawcett, Eds.). Retrieved from http://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspx

National Center for Injury Prevention and Control

ƒƒ Inputs - the information or resources required for developing and implementing a policy.
ƒƒ Activities - the actions that are carried out in order to implement the policy.
ƒƒ Outputs - the direct results of these action steps.
ƒƒ Outcomes are short-term and intermediate changes in target audience behaviors, awareness of risk factors,
attitudes, and knowledge.
ƒƒ Impacts are long-term changes in indicators.
ƒƒ Indicators are specific, observable, measurable characteristics of changes that demonstrate progress toward
outcome or impact.
The logic model process is an easy way to ensure that all
stakeholders have the same understanding of the policy
and its intended outcomes.1 For an overview of logic model Questions To Guide Selection Of
components as well as a template and examples, see Evaluation Focus
Appendices G, H, I, J, and K. Appendix I provides an example
of a logic model developed in a more traditional format, ƒƒ What type of policy is being
while Appendix J presents a logic model developed using an evaluated (legislative, regulatory, or
alternative format. Understanding the policy components organizational)?
and implementation requirements will ensure that you are ƒƒ What level of policy is being
planning a thorough evaluation. evaluated (local, state, national)?
ƒƒ What type of evidence base exists
Step 3: Focus the Evaluation Design for this policy?
Once the core team is assembled and stakeholders have a ƒƒ How complex is the theory of
clear understanding of the policy and roles, it is important change and the implementation of
to clarify the goals and expectations of the evaluation and the policy?
identify the focus for the evaluation (content, implementation, ƒƒ What phase is the policy in? Has it
or impact). This clarity guides the selection of evaluation been fully implemented?
questions and appropriate design, and it helps ensure efficient ƒƒ How will the evaluation be used and
use of resources.4 Suggestions for implementing Step 3 for who is the potential audience?
each of the three main phases (content, implementation, and
impact) are discussed in Briefs 3, 4, and 5.

To ensure a thorough understanding of a policy, you will often need to conduct evaluations for all three of
the phases discussed in Brief 1 (content, implementation, and impact).5 It is important to follow the steps
separately for each phase to ensure a clear match between the evaluation focus, questions, and design. Results
from each of the evaluations can then inform interpretation of the other evaluations.5 Brief 5 and Appendix O
provide information about different types of evaluation designs.

4 Newcomer, K. E. (2009, May). Enhancing the usefulness of evidence to inform practice (National Collaborative on Childhood Obesity Research
[NCCOR] Obesity-Related Policy Evaluation Webinar Series, Session 3). Retrieved from: http://www.nccor.org/downloads/Webinar_3.pdf
5 Her Majesty’s Treasury (2011). The magenta book: Guidance for evaluation. London, UK: Author. Retrieved from http://www.hm-treasury.gov.
uk/data_magentabook_index.htm

2 Step by Step – Evaluating Violence and Injury Prevention Policies

Selecting Evaluation Questions
Once the team has pinpointed the focus of the evaluation, the next step is to identify the specific evaluation
questions. Having clear evaluation questions helps to establish the boundaries of an evaluation.6 The logic
model of the policy may be a helpful tool as you identify and select specific policy sections or components to
evaluate.2 In addition, the focus of the evaluation will influence the evaluation questions chosen. Examples
of evaluation questions within each of the three types of policy evaluation are provided in Briefs 3, 4, and
5. When selecting evaluation questions (and designs), it can be useful to consider the evaluation standards
of feasibility and utility. Feasibility considerations include available resources, data, and complexity.7 Utility
focuses on the extent to which an evaluation meets stakeholder needs.

Selecting an Evaluation Design

The evaluation design, which can have a huge impact on evaluation results, should balance utility and
feasibility.4 The design influences the types and amounts of data required, the analysis techniques used, and
the conclusions that can be drawn. Keep in mind that there is no one “right” design. It is important to find the
most appropriate design for answering the evaluation questions and meeting the needs of the stakeholders.2
If resources allow, choose a mixed-methods approach to balance the pros and cons of the different methods.
Suggested evaluation designs for each type of evaluation are found in Briefs 3, 4, and 5.

Selecting Meaningful Indicators

Once you have selected the focus, questions, and design, the next step is to define outcomes and measurable
indicators. Your answers will depend on the type of policy, the phase of the policy, and what data is available.
Monitoring short-term and intermediate outcomes of a policy is just as important as knowing its long-
term impacts. If the policy is based on strong scientific evidence, measuring short-term and intermediate
outcomes can provide further evidence that a policy will have an influence on injury-related behaviors (i.e,.
long-term outcomes). Because seeing a change in impacts may take a long time, evaluation of short-term and
intermediate outcomes can be useful in providing support for a particular policy approach in the meantime.5

The team should also identify the indicators that will be used to measure progress toward selected outcomes.
These are specific, observable, measurable variables that show the progress a policy is making toward
achieving a specified outcome.1 Identify meaningful indicators along each step of the logic model that will
allow an assessment of the planned work and the intended results.3 Doing so will ensure the collection of
relevant data and selection of the most appropriate design.5 Select indicators that will realistically be affected
by the policy within the evaluation time frame.2,4,5 Research the field of interest to identify any well-established
outcomes and indicators that are part of federal or large-scale initiatives. Appendix L provides examples of
outcomes and impacts, indicators, and data sources.

6 CDC. (1999, September). Framework for program evaluation in public health (MMWR Recommendations & Reports vol. 48, No. RR-11).
Retrieved from ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf
7 Chapel, T. J. (2008). Evaluation 101: An overview for new evaluation practitioners. Presented at the American Evaluation Association Summer
Institute, Atlanta, GA. Retrieved from http://www.eval.org/summerinstitute/06SIHandouts/SI06.Chapel.TR1.Online.pdf

3 Step by Step – Evaluating Violence and Injury Prevention Policies

Tips for Developing an Evaluation Plan
Ideally, evaluation planning should begin during the development of the policy, prompting the inclusion of
data collection, implementation, and resources in the policy content.5 One way to document the important
decisions is by creating an evaluation plan. The main components include the following:
ƒƒ Evaluation team members. ƒƒ Data collection and analysis plan.
ƒƒ Evaluation goals and questions. ƒƒ Dissemination plan.
ƒƒ Evaluation methodology, design, and timetable. ƒƒ Resources.

Remember that evaluating impact prematurely—before implementation has begun—will result in erroneous
or nonexistent findings.

Timeframes for Policy

Enactment to Implementation
The following timeline on the
Massachusetts “Return to Play”
law illustrates the extended
time that may elapse between
enactment and implementation.
It is critical to consider this
time frame when planning the
evaluation.

Consider the following when developing the evaluation timeline:

ƒƒ Time required for evaluation planning
ƒƒ Time for realistic change in indicators5
ƒƒ Time required to fully implement the policy4,5
ƒƒ Availability of data
It is also important to consider the resources available for conducting the evaluation:
ƒƒ What funding is available for the evaluation?
ƒƒ Who is responsible for conducting the evaluation?
ƒƒ What resources are required for data collection?
ƒƒ Do you need to consult internal or external experts?
Examples for maximizing and supplementing available resources are provided in Appendix M.

4 Step by Step – Evaluating Violence and Injury Prevention Policies

Action Steps
ƒƒ Identify stakeholders involved in policy implementation and define key roles and responsibilities.
ƒƒ Describe the policy by developing a preliminary logic model and identifying meaningful indicators.
ƒƒ Identify the initial evaluation focus and evaluation questions.
ƒƒ Identify resources available within and outside your agency to conduct policy evaluation.

ADDITIONAL RESOURCES
CDC Program Evaluation Page: Overview of Framework Steps 1, 2 and 3
http://www.cdc.gov/eval/steps/engagingstakeholders.PDF
http://www.cdc.gov/eval/steps/Describingtheprogram.PDF
http://www.cdc.gov/eval/steps/focusingtheevaluationdesign.pdf
Developing an Effective Evaluation Plan. From the National Center for Chronic Disease Prevention and Health
Promotion, Office on Smoking and Health, and the Division of Nutrition, Physical Activity, and Obesity (Both
CDC, 2011). Available at http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluation/
evaluation_plan/index.htm

5 Step by Step – Evaluating Violence and Injury Prevention Policies