Review of Literature On Evaluation Methods Relevant To Extension
Review of Literature On Evaluation Methods Relevant To Extension
Review of Literature On Evaluation Methods Relevant To Extension
www.g-fras.org
GFRAS is the Global Forum for Rural Advisory Published by:
Services. The GFRAS forum is made up of various GFRAS – Global Forum for Rural Advisory Services
stakeholders worldwide who have an interest and c/o Agridea
role in rural advisory services (RAS). The mission Eschikon 28
of this forum is to provide advocacy and leadership 8315 Lindau
on pluralistic, demand-driven rural and agricultural Switzerland
advisory services within the global development Tel: +41 (0)52 354 97 64
agenda. GFRAS advocates for better recognition of Info@g-fras.org
the essential role of RAS and to see this recogni- www.g-fras.org
tion reflected in the policies of international devel-
opment organisations, better investment in RAS, Text contributors
and more influence by RAS providers in interna- Adrienne Martin*
tional fora. The GFRAS vision is to see rural adviso- Sabine Gündel
ry services effectively contributing to the sustain- Essie Apenteng
able reduction of hunger and poverty worldwide. Barry Pound
3
opmental efficiency, effectiveness, impact addition, this trend was mirrored by re-
and sustainability. An evaluation should ductions in developing country national
provide information that is credible and budget allocations for agriculture (Global
useful, enabling the incorporation of les- Donor Platform for Rural Development
sons learned into the decision-making pro- 2008). Agriculture has suffered dras-
cess of both recipients and donors (OECD tic declines in development investment
1991, 5) and assistance (Global Donor Platform
for Rural Development 2006), with far-
The materials reviewed include academic reaching consequences for agricultural
papers, existing evaluation toolboxes and infrastructure, including extension ser-
major evaluation guidelines and frame- vices. Over the last decade, projects and
works. Firstly, the context and scope of programmes in agricultural extension
the review are discussed, followed by have been to a large extent superseded
sections addressing the purposes, users by other development approaches such
and uses of evaluation, evaluation stand- as environmental protection, good gov-
ards and criteria, approaches, rigour and ernance, poverty reduction, etc. (World
attribution. The final three sections dis- Bank 2008). These initiatives were often
cuss the principles for evaluation of rural established as separate programmes
advisory services in highly complex sit- rather than building on and integrating
uations, the strengths, weaknesses and with existing rural services.
gaps in existing approaches, and the
ways forward. These trends have resulted in severe un-
der-funding or closure of national exten-
sion services and also in a lack of research
and publications in this field. There is now
The changing orientation a renewed interest in, and recognition of,
of extension the role of agriculture in food security and
Extension services in the past were close- wider environmental services, which has
ly linked to the agricultural sector and to put this sector back on the global agenda
agricultural development programmes. (Nagel 2003; Anderson 2007; World Bank
Despite signing up for the United Nations 2008). Whereas in the past the focus was
‘halving hunger’ goals in 1996 and again on production increases through modern
in 2000, donor countries reduced official agricultural technologies, the recent focus
aid to agriculture from 16.8% of all of- takes into account the wider set of func-
ficial development spending in 1979, to tions agriculture has in provision of eco-
just 3.4% in 2004 (ActionAid 2009). In system services and in rural livelihoods.
4
Birner et al. (2006) describe the shift in for the complexity of agricultural systems
focus as follows: “This shift emphasizes within diverse social and ecological con-
the continued need for agricultural advi- texts” ( IAASTD 2009, 4). This diversity
sory services as a means of promoting of contexts requires regional or even local
agricultural productivity, increasing food extension approaches, as no single ap-
security, improving rural livelihoods, and proach can simultaneously increase mar-
promoting agriculture as an engine of pro- ket orientation, food security and mitigate
poor economic growth. Agricultural advi- climate change (Christoplos 2010).
sory services are also needed to meet the
new challenges agriculture is confronted The definition below by Leeuwis and van
with: changes in the global food and ag- den Ban (2004) emphasises the complex-
ricultural system, including the rise of su- ity in terms of the actors and innovation
permarkets and the growing importance of requirements involved.
standards and labels; growth in non-farm
rural employment and agribusiness; con- Extension [is] a series of embedded com-
straints imposed by HIV/AIDS, and other municative interventions that are meant,
health challenges that affect rural liveli- among others, to develop and/or induce
hoods; and the deterioration of the natural innovations which supposedly help to re-
resource base and climate change.” solve (usually multi-actor) problematic situ-
ations (Leeuwis and van den Ban 2004).
This shift in focus is also reflected in
the changing perspective on extension In addition to the above, The Common
systems. Whereas production-orient- Framework on Agricultural Extension
ed agricultural focus was served by lin- (Neuchâtel Group 1999 and 2000) empha-
ear, commodity-focused, top-down exten- sises the following drivers for change in
sion approaches (e.g. 1970s and 1980s, extension services:
the Training and Visit system (T&V)) and • Decentralization, liberalization, privati-
later, by more participatory bottom-up sation, and democratization
approaches such as Farmer-Field Schools • New actors are becoming involved in
(FFSs), the recent focus on the multi-func- “extension” activities
tionality of agriculture requires a more • Public spending on extension is shrinking
pluralistic and demand-driven extension • The aims of official development assis-
or rural advisory approach, “accounting tance are becoming more focused
5
GFRAS interprets Rural Advisory Services The changing orientation
(RAS) as follows: of evaluation
RAS are about strengthening capacities,
empowering rural people, and promoting The importance of evaluation has long
innovations. RAS support people to ob- been recognized, but there have been
tain skills and information, and to address some important shifts in the understand-
challenges so as to improve their liveli- ing of its function and significance in the
hoods and well-being. Traditionally, RAS context of globalization and changing pol-
disseminate information about technolo- icy objectives, international aid modali-
gies, markets, inputs and financial ser- ties, and the Paris Declaration and Accra
vices, and assist farmers to develop their Agenda for Action, with their focus on de-
farming and management skills. But RAS livering and accounting for results (OECD
also broker interactions between farmers, 2005/2008). Evaluation practice has had
the private sector, research, education, to adjust to accommodate this level of
and government. RAS coach different ac- complexity. It has to play its traditional
tors to improve market access, deal with role of generating information on the im-
changing patterns of risk, and protect the plementation and results of a programme
environment.’ or project, but in addition assess policy
Source: GFRAS web site, http://www.g-fras.org/ impacts and also provide the basis for im-
Furthermore, the Neuchâtel Group (2006) proved management and decision-making
identified the following principles for as well as for transparent accountability
Demand-Driven Service Delivery Systems: to citizens (Commission of the European
• Services shall be driven by user demand Communities 2007).
• Service providers shall be accountable
to the users There have been shifts in international
• Users shall have a free choice of service donor support to agriculture, from pro-
providers ject-based approaches to sector sup-
port programmes (e.g. Uganda, Rwanda,
The change in terminology tries to cap- Tanzania and Mozambique), trends to-
ture the more holistic perspective and wards decentralisation of agricultural ser-
complexities of modern extension service, vice provision through local government,
although so far it seems difficult to leave and a new emphasis on the inter-connect-
the well-established terminology behind. edness of local agricultural development
6
with wider national and international pro- also concerned with the effects of poli-
cesses. A further change has been in cy changes relating to agricultural service
the conceptualization of extension as in- provision and the extent to which these
volving complex processes of innovation foster flexible and equitable relationships
– multiple and new sources of informa- among different service providers, in-
tion, multiple and differentiated actors, cluding the private sector (Swanson and
market-driven trends in production and Rajalahti (2010). A growing interest in par-
supply, linked to national and internation- ticipatory evaluation (Rajalahti et al. 2005,
al trade policy and regulations. Similarly, 35) has developed, partly in response to
the recognition of the range of actors the empowerment agenda.
participating has widened to include pri-
vate sector companies, non-governmen- It is also clearly recognized that there is a
tal bodies, national and international difference between evaluations which are
NGOs and charitable foundations, and a intended to account to investors and the
range of government service providers public (including donors, governments, tax
and political actors. Accompanying this payers, farmers’ organizations, etc.) for
has been the mainstreaming of concerns money that has been appropriately spent
for capacity strengthening among actors in accordance with agreed deliverables
at different levels, and for the ‘empow- providing a satisfactory return on invest-
erment’ of farmers, particularly through ment, and those which are more learning-
farmers’ groups and associations. oriented and concerned with what works
and what doesn’t work and what adjust-
The changes have required a different ap- ments should be made.
proach to evaluation of agricultural devel-
opment projects, including those involved There have been different phases of
in agricultural extension and advisory ser- donor interest in evaluation theory and
vice provision. These have included shifts approaches. The current resurgence of
from single technology focused evalua- interest in evaluation and impact assess-
tions, e.g. introduction of a new crop va- ment in general, is linked in part to the
riety or new methods of crop or livestock international financial crisis, the need for
management, to more complex assess- expenditure cuts and a renewed emphasis
ments of changes in agricultural institu- on value for money. It is consistent with
tions, farming systems, human resource the emphasis on measuring progress and
capacity and trade relationships, together development effectiveness articulated in
with the impact on the incomes and live- the Paris declaration (OECD/DAC Working
lihoods of participants. Evaluations are Party on Aid Effectiveness 2008).
7
The scope of evaluation practice has ex- (White 2009a). In particular, methods for
tended to more sophisticated ex ante assessing projects emphasizing processes
evaluation of proposed projects, more of change, complex interrelationships and
emphasis on outcomes and impacts rather learning are less well developed and are
than the immediate outputs of projects – perceived as lacking the rigour of methods
particularly in view of concerns on aid ef- used for assessing the outcomes and im-
fectiveness and contribution to the MDGs. pacts of technical interventions. However,
There have been considerable advances in it is increasingly argued that multiple and
the theory and practice of impact evalu- complementary evaluation approaches are
ation and a particular interest in evalua- needed to address the complexity of RAS.
tions which can attribute changes in se-
lected outcomes to a specific intervention. A recent statement by Anderson (2007)
This reflects an interest in understanding indicates that ‘hard’ data on newly emerg-
whether development interventions actu- ing RAS are still in short supply and are
ally have a significant effect on outcomes urgently needed to support the function-
and whether they are cost effective. ing of RAS and inform extension policy.
He cites the framework proposed by the
Thus there has been a co-existence of Extension Research Group at IFPRI for ad-
methods and approaches in evaluation – dressing this knowledge gap by assisting
some providing detailed quantitative as- the design and analysis of pluralistic ex-
sessments of economic benefits, others tension systems. The aim of this frame-
using qualitative methods (Patton 2002a) work is to identify and learn from “best-fit”
to examine transformation in agricultural solutions.
extension processes and access to ser-
vices and to learn lessons about effective The framework “disentangles” the major
approaches to building capacity and em- characteristics of agricultural adviso-
powerment, etc. Underlying these are dif- ry services: (1) governance structures,
ferent values and objectives in evaluation (2) capacity, management and organiza-
and in development (Alkin and Christie tion, and (3) advisory methods. Four sets
2004). While it could be argued that the of frame conditions have been identified
range of methods and approaches pro- that need to be considered when decid-
vides a more effective choice depending ing on these characteristics: (1) the policy
on the context and purpose of an evalu- environment, (2) the capacity of potential
ation, in some cases the result has been service providers, (3) the type of farming
polarization and contestation concerning systems and the market access of farm
evaluation methods, or merely confusion households; and (4) the nature of the local
8
communities, including their ability to co- agenda that promote sustainable rural
operate. The framework suggests an im- growth and help the poor’, this review
pact-chain approach to analyze the per- focuses specifically on recent and cur-
formance and impact of advisory services. rent literature on evaluation which is
The framework can be applied in a dy- relevant to rural advisory services char-
namic perspective to analyze processes of acterized by a high level of social and
change over time. Focusing on the ques- technical complexity, as noted in Section
tion “What works, where, and why?”, the 2 above.
framework aims to support a shift from a
“best practice” or “one-size-fits-all” to a The diagram below taken from Patton
“best fit” approach in the reform of public (2007) depicts a range of extension and/
advisory services (Anderson 2007; Birner or evaluation situations, which can range
et al. 2006). from “simple”, in terms of their level of
social and technical complexity, to “com-
plicated”. The social dimension is shown
Scope of review on the vertical axis and technical on the
horizontal. As social complexity increases,
Taking as a guiding principle the purpose agreements between the range of differ-
of GFRAS (2010b), which is ‘to provide ent stakeholders are more of a challenge,
a space for advocacy and leadership on and with greater technical complexity,
pluralistic, demand-driven rural advisory technical interventions may require fur-
services within the global development ther experimentation and adaptation.
Figure 1
This spectrum could be further increased
Far from
Socially
by taking into account uncertainties such
Complex / uncertain
as potential climate tipping-points or glob-
al economic developments, which leads
Agreement
9
Table 1 Key elements of simple, complicated and complex situations
10
A range of extension approaches and The purposes, users
methods have been developed, imple- and uses of evaluation
mented and evaluated for “simple” situ-
ations, where a specific technology has Purposes of evaluation
been promoted by a single agent to a Reflection on objectives, users and uses
specific target group. Although these sit- of evaluation should be the first impor-
uations and approaches may occupy a tant step in designing an evaluation ap-
certain niche in emerging rural advisory proach for extension (Deshler 1997).
services, the main challenges will be en- Conventionally, evaluations were conduct-
countered in socially complicated situa- ed at the end of an intervention (ex post)
tions, with a range of RAS actors and com- in order to generate evidence to support
munication technologies and in a context claims about its overall achievements (re-
of uncertainty in terms of technological sults) which were mainly directed towards
appropriateness and market availability, donors and policy makers. These evalua-
which results in highly complex situations. tions were usually conducted by external
For this reason the review concentrates evaluators. More recently the purpose of
on literature covering the latter situations. evaluation in extension systems has shifted
towards a stronger client/ user focus with
In order to learn from a diverse range of a broader set of evaluation objectives, in-
situations we consider in our literature re- cluding learning, transparency and capacity
view lessons learned from developed and development. This shift from a focus purely
developing countries as well as sources re- on results to include an evaluation of pro-
lating to different sectors. The main sourc- cess (in the sense of understanding what is
es used in the meta-review are academic happening in the course and context of a
papers, existing evaluation toolboxes and programme or intervention), has also tend-
major evaluation guidelines and frame- ed to increase the involvement of internal
works (e.g. DAC). evaluators (OECD 2010a).
11
Table 2 Types of evaluation and their purposes
12
Developmental evaluation (not to be evaluation develops measures and track-
confused with the evaluation of devel- ing mechanisms as outcomes emerge.
opment) is where an evaluator working
as part of a team facilitates a process These different types of evaluations are also
of discussion around evaluative ques- linked to potential uses. One important dis-
tions and data-based decision-mak- tinction between evaluation objectives can
ing in the developmental process. This be drawn between the intended uses of
type of evaluation is seen as relevant evaluation findings and the intended pro-
to the complex, dynamic and unpredict- cess use (Hoffmann et al. 2009), which are
able scenarios of RAS (Donaldson et al. ultimately linked to the question whether
2010). In contrast to summative evalua- the main aim of the evaluation is to prove or
tion which measures performance and to improve (Rennekamp and Arnold 2009).
success against predetermined goals The table below summarizes the uses of
and defined outcomes, developmental evaluation based on this distinction.
13
It is important to note that these two pur- • The importance of evaluation as a
poses are not mutually exclusive, as those means to demonstrate and rein-
commissioning evaluations (whether do- force accountability in the achieve-
nors, governments, NGOs etc.) will con- ment of its objectives by assessing the
tinue to request cost effectiveness and impact of funded activities. It is seen as
impact evaluations whilst recognizing that generating impartial and unbiased infor-
evaluations for process uses are increas- mation and documentation to publically
ingly important to respond to the grow- account for aid expenditure, processes
ing complexity of extension situations. and outcomes. It can support mutual
However, there are likely to be trade- offs accountability between stakeholders in
and possible tensions between these two development interventions.
aspects of evaluations. • Systematic identification and dis-
semination of the lessons learned
Important factors underlying any evalua- from experience, both successes and
tion include the policies, requirements and shortcomings, “to tell it the way it is”
associated guidelines of the commission- and show what is working and what is
ing organization. These perspectives on not. Some donors see this as strength-
the purpose of evaluation play an impor- ening their institutional learning culture
tant role in determining how the evalua- and that of their development partners,
tion is designed and conducted. For ex- which can lead to improved design and
ample, a review of the stated purposes of delivery of future initiatives and activi-
evaluation of some of the major interna- ties. It can generate general knowledge
tional development donors3 reveals sev- on development which can contribute
eral common factors: to maintaining and improving the qual-
• An emphasis on objectivity and in- ity of aid.
dependent assessment of the per- • A related purpose is to improve man-
formance of programmes and projects, agement and decision making by
the extent to which activities have been providing information for managing
performed as intended and expected re- projects and programmes and evidence
sults have been achieved together with on which to base decisions by devel-
assessment of their efficiency and ef- opment partners as well as lessons to
fectiveness. Qualities such as reliability guide future decisions.
and rigour are valued. The assessment
is seen as contributing to enhanced rel-
evance, effectiveness and improved 3
EC, World Bank, IFAD, UNDP, FAO and bilateral
performance. donors – CIDA, GTZ, NORAD, DANIDA, SIDA, IDRC.
14
Other objectives less frequently cited by opment community to households and in-
donor agencies were to enhance capac- dividuals. Barker (2007) emphasized that
ity in projects and programmes and to stakeholders are essential to extension in-
assess the sustainability of results, in- itiatives as they provide the link between
dicating that these dimensions have re- extension priorities or purposes and the
ceived less emphasis. The relative em- target community. Patton (2008) suggests
phasis on accountability versus learning that stakeholder involvement is also cru-
within evaluation commissioning organi- cial for evaluating extension initiatives and
zations will influence the formulation of emphasizes the role of utilization-focused
the objective for a specific evaluation and evaluation in this context (see box below).
hence the evaluation questions, criteria
and approach. A particular strength of utilization-focused
evaluation is a high degree of situational
responsiveness, which guides the interac-
Users, uses and participation tive process between evaluator and pri-
A further important aspect for evaluation mary intended users (Patton 2002b). This
planning is the consideration and involve- is a key characteristic which is valuable in
ment of users. Users in the context of ex- highly complex situations, where any eval-
tension evaluation can range from donors, uation approach needs to be based on the
government, project staff, and the devel- local conditions and requirements.
15
Findings from the analysis of extension The importance of a strong user focus
case studies (Pound et al. 2011) suggest leads to the question of user involve-
that none of the extension projects evalu- ment. In the past the main evaluation
ated has implemented a stakeholder anal- mode was that of expert evaluations,
ysis as part of the evaluation process. which could be either externally or inter-
Furthermore, it was found that often only nally implemented. The table below sum-
the objectives were clearly stated, with lit- marizes the advantages and disadvan-
tle attempt to define use and users of the tages of expert modes of external and
evaluation (Pound et al. 2011), which was internal evaluations.
also reflected in a lack of specified dis-
semination strategies for the evaluation
findings and recommendations.
External Internal
Can take a fresh look at the programme Knows the programme only too well
Not personally involved, so it Finds it hardest to be objective
is easier to be objective
Is not part of the normal power structure Is a part of the power and
authority structure
Gains nothing from the programme but May be motivated by hopes
may gain prestige from the evaluation of personal gains
and the organisational or donor
connections which may affect response
Trained in evaluation methods. May May not be specifically trained in
have experience in other evaluations. evaluation methods. Has no more
Regarded as ‘expert’ by the programme. training than others in the programme.
An ‘outsider’ who may not understand Is familiar with and understands
the programme or the people involved the programme, and can interpret
personal behaviour and attitudes
May cause anxiety as programme staff Known to the programme, so
and participants are not sure of motives poses no threat of anxiety or
disruption. Final recommendations
may appear less threatening.
Source: Adapted from Hoffmann et al. (2009)
16
However, more recently, evaluations con- A third category could be seen in participa-
ducted by mixed teams of external and in- tory evaluations, which do not solely rely
ternal evaluators have become increasing- on ‘experts’ but rather on a range of differ-
ly common. A further variant is the form ent stakeholders present in an evaluation
of an internal, but independent evaluation context (Ljungman et al. 2005). Table 5
run by an independent evaluation depart- compares expert evaluation approaches
ment within an organization. with participatory evaluation approaches.
17
The degree of user involvement and the and clearly understood (whether based on
specific evaluation phases in which differ- random assignment, matched intervention
ent users are involved in the evaluation and ‘control’ groups or comparison over
process are important aspects which influ- time) and that baseline data are collected.
ence the evaluation process. Looking back This requires stakeholder participation.
at the complexity of many extension situ- Stakeholder participation in the design
ations, it becomes obvious that there is a and planning stages of evaluations can
range of different users and other stake- help to ensure the evaluation is both real-
holders who could contribute to and ben- istic in scope in relation to resources and
efit from involvement in different phases is of sufficient rigour and will provide the
of the evaluation process. Whereas in sim- expected information. It can identify addi-
ple extension situations it might be quite tional partners for an evaluation if particu-
obvious who should be involved, complex lar specialist areas are needed.
extension situations require a careful pro-
cess of stakeholder identification. A num-
ber of challenges need to be confronted
in these complex situations, ranging from Evaluation standards
the management of conflicting interests and criteria
and perspectives, to managing power in-
equalities and different motivations for This section initially examines the exist-
involvement. ing basic principles of evaluation and the
evaluation standards operating among
Ideally, planning for impact assessment the major international and bilateral
should begin at the early stages of a pro- donor agencies, since these are influen-
ject or programme and include those with tial in evaluation practice with respect
evaluation experience. This will help to en- to their funded programmes. It then dis-
sure that the basis of comparison for im- cusses the relevance of these criteria
pact assessment is appropriately designed for evaluation in the context of regional-
18
ly and nationally commissioned evalua- The aim is to determine the relevance
tions of RAS programmes with different and fulfilment of objectives, develop-
channels of accountability. For example, mental efficiency, effectiveness, impact
for complex extension programmes the and sustainability. An evaluation should
criteria would need to span a range of provide information that is credible and
actors (knowledge brokers, intermediar- useful, enabling the incorporation of les-
ies, entrepreneurs, producers, traders sons learned into the decision-making
etc) as well as the context of demand process of both recipients and donors.”
and the policy and legal context. The
challenge is how to capture and assess It defines the purposes of evaluation as:
complex social, behavioural and institu- to improve future aid policy, programmes
tional change. and projects through feedback of les-
sons learned, and to provide a basis for
accountability, including the provision of
Evaluation Principles information to the public. Other purposes
and Standards among major concern the promotion of process-orient-
international donors ed and participatory approaches; evalu-
An important document setting out ation as a management tool for decision
the DAC Principles for Evaluation of making and as a way of increasing knowl-
Development Assistance was produced in edge about development assistance and
1991 (OECD 1991). This was extensively its possibilities and limitations as an instru-
reviewed in 1998 (OECD 1998). The 1991 ment of economic and social change.
document defines evaluation as:
The document elucidates a number of
“an assessment, as systematic and ob- basic principles of evaluation.
jective as possible, of an on-going or • Impartiality and independence of
completed project, programme or policy, the evaluation from the process con-
its design, implementation and results. cerned with policy-making, and the de-
19
livery and management of development Other issues concerned the need for
assistance. This was seen as contribut- practical, action-oriented findings, rig-
ing to the credibility of evaluation and our in data collection and understand-
the avoidance of bias, providing legiti- ing of the institutional, historical, or local
macy, and reducing the potential for context.
conflict of interest. This principle im- • Participation of funders and stake-
plies an organizational structure which holders – whenever possible, both
clearly separates evaluation and opera- funders and programme stakeholders
tional responsibilities. The 1998 review should be involved in the evaluation
acknowledged that if institutional lesson process, with the issues of concern to
learning is an important objective, the each partner represented in the terms
principle of independence has to be bal- of reference. This provides an opportu-
anced with promotion of participation, nity for learning by doing and strength-
relevance and ownership of the evalu- ening skills and capacities in the recipi-
ation products and recommendations. ent countries. The 1998 review found
• Credibility and transparency are that evaluation was still predominantly
closely linked to impartiality and inde- a donor-driven process mainly due to
pendence. Transparency is helped by time constraints and high costs, com-
an open process including participation munication problems, and, occasional-
of recipient countries and supported by ly, political obstacles.
reporting both successes and failures, • Donor cooperation – The standards
making results widely available, dis- encourage donor collaboration to avoid
tinguishing between findings and rec- duplication of effort, to enhance joint
ommendations and including informa- learning, to develop evaluation methods
tion in a way that does not compromise and improve mutual understanding of
sources. procedures and approaches, share re-
• Usefulness – for evaluations to be use- ports and information, and improve ac-
ful, the findings must be perceived as cess to evaluation findings. Joint eval-
relevant. Their presentation should be uations also reduce the administrative
clear, concise, accessible and timely if burden on the recipient. However, there
they are to have an impact on decision- have been limited examples of success-
making and should reflect the different ful collaboration and commitment to
interests and needs of the many parties replace individual agency evaluations,
involved in the programme. A key test while joint evaluation has been slow to
of usefulness is whether action is under- develop. This issue has become more
taken according to recommendations. urgent with the development of multi-
20
donor support programmes (e.g. sector and the positive or negative impact of
support programmes, co-financing ar- external factors. The evaluator should
rangements etc.) assess whether the impacts justified the
• Evaluation programming – This prin- costs and whether there are better ways
ciple advises on the development of an of achieving the results. Sustainability
overall plan for the evaluation including of the achievements in the longer run
the activities to be evaluated, priorities is a further issue. While recognizing the
and timetable, taking into account the difficulty of attributing responsibility for
requirements of management and policy outcomes, the principles suggest that an
makers. These decisions should involve attempt to establish causal relationships
the users of the evaluation outputs. In must be made. Feedback from member
1998, users of evaluation reports sug- countries recommended a stronger em-
gested that more transparency in design phasis on results and performance rat-
and implementation of evaluation would ing and coverage of specific programme
be attained by using logical frame- interests, e.g. gender, environment and
works, indicators, and “success rating” poverty as well as adaptations for evalu-
systems. Guidelines and standards for ations of new forms of assistance, e.g.
evaluations, particularly for project eval- sector-wide assistance programmes, na-
uations, are common, although their ap- tional development programmes, etc.
plication is not always consistent. • Reporting, dissemination and feed-
• Design and implementation of eval- back – this principle noted that “dis-
uation – terms of reference for the semination and feedback must form
evaluation should define its purpose and a continuous and dynamic part of the
scope and the intended recipients or evaluation process. It emphasizes clear,
users. It should include the questions to jargon-free reporting, the inclusion of
be addressed in the evaluation, specifi- an executive summary a profile of the
cation of the methods, the performance activity evaluated, a description of the
assessment standards and the resourc- methods used, the main findings, les-
es and time allocated. It should examine sons learned, conclusions and recom-
the project/ programme rationale and mendations. Systematic dissemination
relevance of objectives and whether and feedback through different chan-
they are realizable; whether they have nels, formal and informal, would help
been achieved and the major factors ensure improved planning and imple-
influencing this. Other criteria are the mentation of policy and programmes.
impact and effects of the project/pro- This requires resources and senior man-
gramme, both intended and unintended agement support. The 1998 members’
21
survey indicated that little attention was roles, ethnicity, ability, age, sexual orienta-
being paid to the demand, use and us- tion, language and other differences when
ability of evaluations, and differentia- designing and carrying out the evaluation.
tion according to different audiences Furthermore, the standards recommend a
and lesson sharing. A more effective partnership approach, including different
approach to dissemination was needed; stakeholders, coordination with national
for example, through active promotion and local evaluation systems and support
by the members in their agencies in in- for capacity development among develop-
teraction with officials and professional ment partners. They advise quality control
groups in the partner countries. through internal and/or external review or
advisory panels.
A further key document by the OECD
Development Assistance Committee’s Individual donors’ guidelines closely relate
Network on Development Evaluation is to the above principles and standards.
the Quality Standards for Development
Evaluation (OECD 2010). This document At the European Commission (EC), eval-
provides guidance on good practice in uation is conducted in accordance with
evaluation, building on the 1991 prin- a set of evaluation standards (European
ciples. The standards were produced in Commission n.d.) which are expressed as
draft in 2006 and tested before finalizing a set of guiding principles that apply to
in 2010. They aim to improve the quali- both internal and external evaluations or
ty of development evaluation processes combinations. Different standards apply
and products, facilitate the comparison for ex-ante evaluations and impact as-
of evaluations across countries, support sessments. The evaluation principles
partnerships and collaboration on joint of European bilateral donors, for exam-
evaluations, and increase development ple, GTZ (GTZ 2006, 2007; Reuber and
partners’ use of each other’s evaluation Haas 2009), DANIDA (DANIDA 2006a and
findings. 2006b), Sida (Sida 2004) and NORAD
(NORAD 2006) closely follow the DAC
The standards deal with evaluation pro- principles and standards. NORAD (2006)
cesses and products, while reaffirming notes in addition the increasing impor-
principles of transparency, independence, tance of sector programs and budget sup-
credibility, and affirming the importance port and donor harmonization and the
of evaluation ethics and codes of conduct, need to adapt to these changes with more
respect for human rights and cultural and joint evaluations and closer cooperation
religious differences, mindful of gender within the OECD/DAC evaluation network.
22
Sida is committed to using country sys- types of evaluation are subject to the same
tems for monitoring, reviews and evalua- principles as the DAC principles above –
tions whenever feasible and appropriate. usefulness, credibility, transparency, and
independence (World Bank n.d.).
Similar principles inform the evaluation
practice of other donors. The IDRC Evaluation Unit follows stand-
ards of utility, feasibility, accuracy, and
United Nations agencies follow the United propriety (IDRC n.d. and 2007). They
Nations Evaluation Group standards emphasize the importance of participa-
(UNEG 2005) which identify 13 norms of tion by relevant users for the production
evaluation, including independence, trans- of relevant, action-oriented findings.
parency, consultation and follow up, eval- IDRC also emphasizes capacity strength-
uation ethics and contribution to knowl- ening, stating that evaluation process-
edge building. es should develop capacity in evaluative
thinking and evaluation use among IDRC
The evaluation methodology in use at the managers, programme staff, or project
IFAD Office of Evaluation (IFAD 2009) is partners – building an evaluation ‘cul-
based on the principles set out in IFAD’s ture’. Specific strategies can be built into
evaluation policy, approved in 2003 (IFAD evaluations to foster this.
2003). The manual builds on internation-
ally recognized evaluation practices and The principles which guide evaluation
criteria. Similarly, UNDP’s evaluation pol- among charities and foundations are
icy seeks to increase transparency, co- less readily accessible. The W. K. Kellogg
herence and efficiency in generating and Foundation (2004a) has detailed informa-
using evaluative knowledge for organiza- tion on evaluation and sees the guiding
tional learning and effective management principles as follows: strengthening pro-
for results, and to support accountability jects, using multiple approaches, address-
(UNDP 2009). These norms and standards ing real issues, and a participatory and
are largely in line with the standards of the flexible process.
OECD-DAC.
In summary, the development of evaluation
The World Bank distinguishes between principles and standards by the major fund-
self-evaluations by the units responsi- ing agencies contributes to a broad frame-
ble for particular programmes and activ- work for evaluation practice, applicable to
ities and independent evaluation by the programmes in different sectors. However,
Independent Evaluation Group (IEG). Both the particular emphasis on the different
23
standards will depend on who is driving the • Relevance – the extent to which the
evaluation and with what purpose. objectives of the development interven-
tion are consistent with beneficiaries’
needs and problems, country needs,
Evaluation criteria global priorities and partners’ and do-
Understanding of the evaluation princi- nors’ policies; whether the objectives
ples and standards – whether formalized continue to be relevant.
as discussed above, or implicit in organi- • Effectiveness – the extent to which
zational norms and practices – forms the the objectives of the development inter-
basis on which evaluation objectives are vention were achieved, or are expected
defined and specific evaluation questions to be achieved, taking into account their
developed for particular programmes or relative importance.
projects. Evaluation practice varies ac- • Efficiency – examines how resources
cording to whether standardized evalu- – inputs, funds, expertise, time – have
ation criteria are used and adapted or been converted to results and whether
whether the evaluation criteria and asso- the results were achieved at a reason-
ciated questions are developed as part of able cost.
the evaluation process, for example linked • Sustainability – the extent to which
to the analysis of the intervention logic. the benefits from a development inter-
vention continue after major develop-
The most widely used criteria are linked ment assistance has been completed
to the DAC principles and standards and the probability of continued long-
(EuropeAid 2006). The DAC evaluation term benefits.
criteria – relevance, efficiency, effective- • Impact – the positive and negative, pri-
ness, sustainability and impact4 – rep- mary and secondary long-term effects
resent different dimensions or perspec- produced by a development interven-
tives for assessing the performance of tion, directly or indirectly, intended or
a project. unintended.
4
To these, the EC add coherence/
complementarity and community/value added.
24
The additional EC criterion of coherence/ uation process is informed social participa-
complementarity is important for evalua- tion, making a contribution to better gov-
tions of complex programmes – examining ernance, and transparency (IDRC 2007).
the ‘fit’ of the programme with donor and
country policies and priorities. It can also Interestingly, the meta review of evalua-
refer to the logic of the relationship be- tion case studies conducted as part of the
tween activities, outputs and results, ex- same commission as this literature review
amining whether there are internal contra- (Pound et al. 2011) found that about half
dictions which affect the results. of the selected cases referred to some or
all of the five OECD DAC criteria, but in
The OECD DAC criteria are utilized, with most cases these were not linked to the
minor modifications, by many donors and evaluation questions. Relevance and ef-
development organizations, including the ficiency were less frequently mentioned
World Bank (World Bank n.d.), IFAD (2009), than the other criteria.
UNDP (UNDP 2009), FAO (2007 and 2009),
GTZ (2006 and 2007), NORAD (2006), and These are broad criteria for evaluation.
DANIDA (2006b). The main area in which There are several important methodologi-
additions have been made concerns insti- cal steps in moving towards developing
tutional development and partnerships. the detailed questions for an evaluation.
The World Bank adds ‘institutional devel-
opment impact’ and ‘bank and borrower
performance’. IFAD includes ‘innovations Impact evaluation
and performance of partners’, and FAO, An important debate relevant to evalu-
‘institutional strengths and weaknesses, ation of RAS concerns the meaning and
including institutional culture and inclu- expectations of ‘impact evaluation’. The
siveness of process’. CIDA includes part- term is often used broadly to refer to an
nership in their development results and analysis of outcomes and long-term ef-
success factors (CIDA 2004 and 2006). fects of an intervention (White 2009a).
For example, the DAC definition of impact
IDRC’s approach to evaluation is utiliza- is ‘positive and negative, primary and
tion-focused and does not advocate any secondary long-term effects produced by
particular evaluation content, model, a development intervention, directly or
method, or theory. Rather, the primary in- indirectly, intended or unintended’. Any
tended users of evaluations should select evaluation which refers to impact (or out-
the most appropriate focus, methodology come) indicators is thus, by definition, an
and approach. A key outcome of the eval- impact evaluation.
25
However, increasingly, ‘impact evalua- tribution requires an experimental or qua-
tion’ is being defined more specifically to si-experimental design involving the con-
refer to approaches which address the struction of a rigorous counterfactual and,
question of how outcomes such as par- where possible, the use of quantitative
ticipants’ well-being would have changed measures of impact.
if the intervention had not been under-
taken (White 2009a). This definition is Counter arguments cite the complexity
about attribution of changes in the out- of rural social structures and interaction
come to the programme or intervention. (particularly in the case of RAS) consider-
Reasons for interest in such approaches ing that there are far too many variables
are partly driven by policy concerns and and possibilities emerging and interacting
the need to demonstrate cost effective- dynamically to conceptualize simple coun-
ness and justify investment. terfactuals (Patton 2010). A further risk in
the context of RAS evaluation is that at-
To analyse attribution, there must be a tention to the broader goals of RAS – for
means of comparing the actual changes example, institutional change, empower-
brought about by the programme with ment, gender equity, strengthening so-
the situation if the programme had not cial capital and enhancing participation –
taken place (the counterfactual). This would be diverted in favour of those which
can be done in different ways; for ex- are more easily measurable; for example,
ample, comparing the situation before diffusion and adoption of technologies.
and after an intervention, or comparing
groups who were targeted by or par- The implication of the above is that the ap-
ticipated in the intervention with similar propriate approach and methods in evalua-
groups who were not. tion depend greatly on what kinds of ques-
tions are being asked and what type of
Because of the interest in establishing analysis is required. There is a growing suite
causality and the need to exclude influ- of methods, both qualitative and quantita-
ences which could affect results, but tive, and the challenge is to select the best
which are independent of the programme, methods and combinations of methods for
some evaluators assert that analysis of at- the purpose and resources available.
26
Analysing the intervention • It is difficult to encapsulate all the ele-
logic or programme theory ments of a complex programme in a log-
There is a broad consensus that the rea- frame, unless it has different interrelat-
soning behind a programme, project or ed levels. The logical hierarchy makes it
intervention needs to be clearly under- more difficult to represent multiple par-
stood. The evaluation literature and avail- allel processes – such as programmes at
able evaluation guidelines make a number different locations, interactions between
of suggestions which link the development different outputs and outcomes, and the
of evaluation criteria to a theory of change different roles of multiple stakeholders.
or intervention logic. This relates to how Complex statements can make the logi-
particular interventions or activities bring cal framework unreadable.
about certain results and what those re- • The assumptions column is often con-
sults will lead to. Other terminologies for a sidered ‘residual’ and important exter-
similar conceptualisation are ‘impact path- nal factors that condition or limit project
way’, ‘outcomes model’ and ‘chain of re- implementation and results are given in-
sults’. A theory-based approach can pro- adequate attention, although they may
vide a framework for an evaluation (White be issues that the project should seek
2009b). This can be conceptualised as a to influence.
series of cause and effect relationships • Responsibility and agency – who will do
linking inputs and activities to expected what among multiple partners – is not
results5 and impacts and to overall goals. always clear.
This is often captured in a project or pro- • Logical frameworks are not always
gramme logical framework with associ- ‘owned’ by the project team. While some
ated indicators, time-bound targets and may be a product of participatory plan-
assumptions, which are then used as the ning programme processes (e.g. ZOPP -
basis for evaluation. Some of the short- objective oriented project planning) they
comings of logical frameworks and their are often prepared as a separate activity
application are well known, but these are by a consultant to meet donor require-
particularly pertinent when it comes to ments, and are not used by the project
complex programmes or when they are team as a management and monitoring
used as the single tool. tool (Bakewell and Garbutt 2005).
5
The OECD DAC criteria utilise the concepts
outputs, outcomes and impacts
27
• The logframe identifies planned results Where a logical framework does not exist,
but tends to be inflexible with respect the intervention logic can be reconstruct-
to incorporating changes that were not ed using statements of objectives and an-
originally planned or unintended effects. ticipated results and impacts. This may in-
clude multiple stakeholders and be done
Use of the logical framework in evaluation as a participatory activity, or may sim-
is often emphasised where the main driver ply be conducted by the evaluation team
of the evaluation is upwards accountability based on project documentation and veri-
to the donor (Bakewell and Garbutt 2005). fied with project stakeholders. The pro-
Its advantage is to focus the enquiry on cess should clearly indicate what was stat-
whether the intervention is delivering the ed in documentation and what additional
outputs and impacts as proposed (or ‘con- results have been added. It can be pre-
tracted’), and to examine how outcomes sented as a diagram or tables. The evalu-
and goals have been delivered that can ation questions are then developed from
challenge the programme logic and ques- the intervention logic.
tion whether the project is in fact ‘doing
the right things.’ One consequence is that ‘Theory of change’ approaches or theory-
this focus on outcomes and impact may based evaluation (Chen 1990; Weiss 1995;
come at the expense of understanding White 2009b) are similar to logic models
and learning from the processes of pro- in defining how and why the programme
ject implementation – whether things works and what transforms actions into
‘were done right’ (Bakewell and Garbutt interim and long term outcomes. They ad-
2005). However, in practice, the upper dress processes in more detail, generally
levels of logical frameworks are often ne- involving an examination of the context,
glected and their indicators not taken very assumptions, and preconditions and pre-
seriously, especially where the logframe senting them in a more interlinked visu-
is seen as a donor-imposed requirement. al format. These approaches evolved in
Hence the focus tends to be more on pro- the charitable sector working on com-
ject implementation and whether ‘things munity development programmes and
were done right’. other complex initiatives (W. K. Kellogg
Foundation 2004b). While logic models
The use of logframes as the single basis have been used in extension evaluation,
for evaluation can also be problematic there is potential for the more flexible use
when the actual objectives and dynam- of programme theory to help in specify-
ics are determined by the playing out of ing causal connections, focusing attention
stakeholder interests (Gasper 2000). on certain possibly weaker parts of the
28
model, and identifying critical mediators This formulation is useful for monitoring
that are necessary to make change hap- and evaluating changes in relationships,
pen (Braverman and Engle 2009; Funnel and how such changes can be brought
and Rogers 2011). about. The methodology is helpful in ad-
dressing the challenge of monitoring and
A different approach, from an NGO, is evaluation within a complex multi-stake-
ActionAid’s Accountability, Learning and holder context where changes in relation-
Planning System (ALPS). This is a set of ships, processes, attitudes, capacities and
minimum requirements for planning and quality of communication are important
accountability which describes the reviews objectives.
and reports required within the organisa-
tion and a set of principles describing how Network models (www.mande.co.uk/
these requirements should be met. These networkmodels.htm) are useful where
are closely linked to ActionAid’s mission there are many actors (people and /or or-
and strategy, intended to promote greater ganisations) who are fairly autonomous
accountability to the people on whose be- and where there is no single authority
half money is raised, gender equity, trans- directing them, and where programmes
parency, and empowerment of the poor have multiple or competing objectives,
(Guijt 2004, 5). or where there complex connections be-
tween actors and outputs. This is particu-
More strongly actor-oriented frameworks larly the case in pluralistic RAS where dif-
are evolving. An example is Outcome ferent organizations are active in the same
Mapping (Earl et al. 2001) which focuses rural space.
on people and organizations rather than
assessing the products of a programme. An example is a study from Kenya (AED/
It starts by establishing a vision of the USAID n.d.) which used social network
human, social, and environmental bet- analysis as a participatory planning tool
terment to which the programme hopes for dairy feed system development.
to contribute and then focuses monitor- Information on relationships and infor-
ing and evaluation on changes in behav- mation exchange was mapped and dis-
iour, relationships, actions, and/or activi- cussed, identifying strengths weakness-
ties of the people and organizations with es and gaps in the network relationships.
whom a development programme works Collaborative actions were monitored and
directly and has opportunities for influ- changes in the network structure mapped.
ence. These latter are termed ‘boundary This creates a foundation for collaborative
partners’. evaluation of innovations in a single area.
29
The Innovation Systems Framework also ing groups, and building cultures (http://
offers a useful framework for analysis, appreciativeinquiry.case.edu/intro/defini-
monitoring and evaluation of networks tion.cfm). It searches for the positive in
of organizations in extension (Hall et al. people, their organizations, and the rel-
2003; World Bank 2006) and links local- evant world around them. It does this
ized networks to a broader enabling na- through a “4D” process to 1) Discover the
tional and international policy and trade “best of what is”; 2) Dream “what might
environment. Other methods of devel- be”; 3) Design “what should be”, and 4)
oping criteria for evaluation where these create a Destiny based on “what will be”.
have not been specified as part of a pro- AI can be used in evaluations to develop a
ject design and M&E system (for example, programme logic model, clarify the evalu-
in the evaluation of some humanitarian in- ation purpose, identify stakeholders, de-
terventions in conflict situations) include termine key questions, and develop meas-
the ‘Most significant change’(MSC) tech- ures and indicators and an evaluation plan
nique (Davies and Dart 2005). This is a (Preskill and Coghlan 2003).
form of participatory monitoring and eval-
uation in which many project stakehold-
ers are involved both in deciding the sorts Evaluation questions
of change to be recorded and in analys- Evaluation questions provide a transpar-
ing the data. It contributes to evaluation ent focus for the evaluation and should
because it provides data on impact and reflect the priority issues. The choice
outcomes that can be used to help assess of questions is particularly critical for
the performance of the programme as a evaluations of complex multidimen-
whole. The data are ‘significant change’ sional programmes – a narrow focus al-
stories collected from the field level, and lows more targeted data collection and
the systematic selection of the most sig- in-depth analysis, but may risk missing
nificant of these stories by panels of des- important factors, especially if the in-
ignated stakeholders or staff. These are tervention logic has been challenged.
shared and are the basis for discussions Hence the recommendation in the DAC
about the value of these reported chang- evaluation standards document that rel-
es and programme impact. MSC does not evant stakeholders, including the users
make use of pre-defined indicators, espe- of the evaluation report, have the op-
cially quantitative ones. portunity to contribute to evaluation de-
sign, including identifying issues to be
Appreciative inquiry is a technique for addressed and evaluation questions to
creating an organisational vision, align- be answered. Question areas for inclu-
30
sion might concern particularly innova- the project/programme not been in place
tive aspects of a programme which in- or had it worked with different partners.
terest stakeholders or topics that will This means considering whether the evi-
inform a decision or a policy debate. dence could be supported by alternative
explanations.
The evaluation questions relate to the cri-
teria for evaluation and the programme or For projects with logical frameworks con-
project logic. The EC evaluation guidelines taining indicators and targets, the as-
suggest questions, and associated sub- sessment criteria for the specified re-
questions should be linked to the DAC cri- sults should be explicit. Where these do
teria (see example in section 10), exploring not exist, assessment criteria, targets and
the extent to which the project activities indicators have to be developed and ac-
have led to the intended results (effec- cepted for consistency, transparency and
tiveness), whether the results have been feasibility of data collection.
achieved at reasonable cost (efficiency),
and whether these have actually been rel-
evant to local needs and with what levels
of impact and prospects of sustainability
Evaluation approaches
(EuropeAid 2006; European Commission This section discusses approaches to eval-
2004). The guidelines suggest each ques- uation in the sense of the overall orienta-
tion should focus on one evaluation criteri- tion toward the evaluation, including the
on as they will require different approach- underlying value orientation, purposes
es to collecting evidence. Translating the and methods. Approaches to evaluation
generic criteria into specific questions is are related to the purpose, uses and users
a challenge in itself, but other questions, of the evaluation as discussed in section
not directly linked to the intervention logic 4 above, which are in turn related to the
are also needed – concerning unexpected principles, standards and expectations
impacts and their benefits or lack of ben- of those commissioning evaluations and
efits and cross-cutting issues such as en- those involved in conducting them.
vironment, gender, good governance and
human rights. The different purposes of evaluation were
presented in Table 2 above. Different
As part of assessing the contribution purposes of evaluation will be relevant
and the rationale of the project or pro- for a programme according to its strate-
gramme, the questions may also explore gic needs. For example, while formative
what changes would have taken place had evaluation to ensure that the programme
31
is well-developed is clearly important in provide insight into why things happened
the early stages of programme devel- the way they did.
opment, it can be useful at points along
the programme implementation to im- Other types of evaluation include:
prove management performance and pro- • Empowerment evaluation – empha-
gramme targeting. Formative evaluation sizes that the evaluation process and
may include the development or revision methods should be empowering to those
of the programme logic model, examining who are being evaluated (Fetterman et
stakeholders’ views of the programme; re- al. 1996, 2004).
searching or reviewing information about • Stakeholder evaluation – looks at
needs; sharing experience and consolidat- the different perspectives of all of a pro-
ing learning within and across the pro- gramme’s stakeholders (where stake-
gramme; and developing data collection holders are those who have an interest
systems for M&E (Duignan 2009d). in the programme) (Greene 1988).
• Goal-free evaluation – in which the
In addition to types of evaluation de- evaluator’s task is to examine all of the
fined by purpose, there are several types outcomes of a programme, not just its for-
of evaluation characterised by a specif- mal outcomes as specified by programme
ic focus or approach. Utilization-focused planners in its objectives (Scriven 1972).
evaluation, Developmental evaluation and • Constructivist/naturalistic or
theory-based evaluation have been dis- fourth-generation evaluation – em-
cussed above. ‘Process evaluation’ is an phasizes the qualitative uniqueness of
evaluation approach which focuses on un- programmes and is a reaction against
derstanding programme or implementing the limitations of quantitative evaluation
organisations’ dynamics and decisions – approaches (Guba and Lincoln 1989).
why things happen/ed the way they do/ • Experimentalist evaluation – the
did. It can complement other types of traditional quantitative approach to
evaluations by helping to interpret the evaluation experiments or quasi-experi-
context and the interpersonal, institution- ments which puts a particular emphasis
al and political processes that influence on their use as ways of providing eval-
achievements. It is based on exploration uative information that can be used to
and analysis of stakeholder and participant attribute changes in outcomes to par-
perceptions about the programme, includ- ticular interventions (Cook and Campbell
ing documentation and communication. It 1979). Similar to some interpretations of
can be combined with other approaches to impact evaluation.
32
• Strategic evaluation – an approach Summative evaluations are about assess-
which emphasizes that evaluations ing the changes brought about by a pro-
should be driven by the strategic value gramme, intended or unintended, positive
of answering key sector information or negative. The difference between this
needs rather than by just focusing on approach and the specific requirements of
individual programmes (Duignan 2008). impact evaluation, which is concerned with
• Most significant change – collects measuring and attributing impact, have
‘significant change’ stories from the been discussed above. While both types of
field. The most significant of these are evaluation are concerned with identifying
selected systematically by panels of changes that can be plausibly attributed to
designated stakeholders or staff to con- the programme, impact assessment does
stitute data on impact and outcomes this by reference to a ‘counterfactual’ (‘what
that can be used to help assess the per- would have happened to beneficiaries in the
formance of the programme as a whole. absence of the programme’), (see http://
(Davies and Dart 2005). en.wikipedia.org/wiki/Impact_evaluation).
(source: adapted from Duignan 2009d)
33
Under the heading of summative evaluation ways; for example, comparing groups
approaches are included a range of meth- who were targeted by, or participated in
ods that rely on qualitative and sometimes the intervention with similar groups who
quantitative approaches. They often explore were not. There should be an assess-
participants’ and stakeholders’ experience ment of whether project participants
and assessments and examine institution- and non-participants are comparable
al change, incorporating ‘before’ and ‘after’ (usually involving some form of statis-
comparisons, or monitor and then evalu- tical matching) to exclude the possibil-
ate changes over time. These approaches ity that pre-existing differences bias the
can successfully identify and estimate the targeting of programme delivery. This
extent of change and assess performance, can involve complex experimental de-
but are not necessarily able to attribute the signs using control groups and sophisti-
change unambiguously to the programme cated statistical techniques. Both types
intervention. of evaluations are more straightforward
where relevant baseline data have been
In contrast, for impact evaluations which collected, but a good baseline study de-
are interested in analysing attribution, sign is the first step in the design of an
there has to be a means of comparing impact assessment.
the actual changes brought about by the
programme with the situation if the pro- These alternatives raise very different is-
gramme had not taken place (the coun- sues, including ethical and resource im-
terfactual). This can be done in different plications (Duignan 2009b). Decisions on
34
the appropriate approach for evaluation of single intervention programmes in fairly
complex programmes are particularly chal- controlled environments (W. K. Kellogg
lenging. Among the considerations are: Foundation 2004a).
• The acceptability of approaches to dif- • The scale of the evaluation, the feasibil-
ferent client groups – donors and coun- ity and the resources and time required
try partners; what is seen as constitut- for different approaches.
ing ‘evidence’ and the level of rigour
required (see below) An example of a decision tool to help in
• The ethical implications of ‘excluding’ decision-making on impact/outcome eval-
potential beneficiaries from a develop- uation design is given by Duignan 2009c.
ment programme in order to construct This looks at the relationship between dif-
a ‘control’ group. ferent evaluation types and the charac-
• The main function and purpose of teristics of an intervention – including the
the evaluation – including the relative level of control over access to an inter-
weight given to accountability or ‘prov- vention, ease of discerning causal mech-
ing’ impact, or to learning lessons for anisms, resource levels, the degree of
the future and ‘improving’ performance. stakeholders’ scepticism, and likelihood of
• The type of programme being evalu- completion. Duignan notes that particular
ated and the degree of complexity. For stakeholders may reject the last two de-
some evaluation approaches, the crite- signs or some of the others as being inad-
ria necessary to conduct these evalua- equate for robustly establishing causality
tions limit their usefulness to primarily in specific cases or in all cases.
35
Table 6: Impact/outcome evaluation design selection
36
Causal mechanisms Limited resources Powerful Concern
difficult for key for impact/ stakeholders about risk
informants/ outcome eval- sceptical about of impact/
experts to discern? uation? interven-tion? outcome
evaluation
not being
com-pleted?
May be appropriate May be more May be more Higher
expensive appropriate
easily amenable to randomised design or sion initiatives and not to retreat from the
control groups and where it is more dif- evaluation of complex programme areas.
ficult to be confident about attribution. For example, to understand changes in
Hence there is a need to consider combi- interrelationships and partnerships, out-
nations of methods in evaluation of exten- come mapping might be used.
37
Rigour describe how it is being delivered, in-
cluding potential differences between
Braverman and Arnold (2008, 82) de- programme delivery sites?
fine rigor as “a characteristic of evalu- • Programme participation and attrition:
ation studies that refers to the strength Are efforts made to reach those who did
of the design’s underlying logic and the not participate or whom the programme
confidence with which conclusions can did not reach: participants who didn’t at-
be drawn”. Funding agencies, grant re- tend regularly, who left the programme
viewers, legislators, and our academic midway, or who received different levels
departments desire methodologically of exposure to the programme? Or does
rigorous evaluations of extension pro- it just measure whoever happens to at-
grammes, that is, evaluations that are tend on the day of data collection?
technically sound and provide an oppor-
tunity to show solid, convincing evidence An inevitable dilemma in impact assess-
of a programme’s impact (Duniform et ment is the need to be rigorous and the
al. 2004). need to be comprehensive. Here, ‘rigor-
ous’ is meant in the narrow sense – in
Rigour contributes to evaluation quality, terms of obtaining representative and
and it can be described in terms of spe- convincing statistical data – implying a
cific elements related to the evaluation’s restricted scope. Comprehensiveness is
planning and implementation. Those criti- meant in terms of a broad range of tech-
cal elements include the following (see nical, educational, social and political im-
Braverman and Arnold 2008 for a more pacts. Studies have been designed to
detailed list): be either rigorous or comprehensive but
• Evaluation design: For programme im- never both (van de Berg 2004).
pact evaluations, how well does the de-
sign allow us to determine if the pro- In complex situations such as communi-
gramme itself was the cause of positive ty initiatives, or those equally applicable
change in the outcomes? to complex RAS, the conventional hypo-
• Measurement strategies: Will the pro- thetico-deductive, natural science model,
gramme outcomes be measured in a which results in high levels of statisti-
valid, reliable way that provides strong cal rigour, is ill-equipped to capture the
evidence for drawing conclusions? multi-dimensional impacts (W. K. Kellogg
• Programme monitoring: During the Foundation 2004b), leading to an incom-
evaluation, are we observing the pro- plete understanding of the intervention
gramme closely enough so that we can dynamics and system changes.
38
The consequences are: agenda to meet their own needs will be
• Exclusion or narrowing of the choice of judged as a failure of the project rather
alternative paradigms and associated than evidence of readiness to absorb feed-
questions which, if chosen, are seen as back. One lesson learned from project ex-
being of lesser value. perience in the Mekong Delta was how
• Since it is difficult or impossible to apply traditional, quantitative targets can rule
the dominant evaluation paradigm, out the investment required to develop
such initiatives may not be evaluated a relationship with the poor (Christoplos
at all and evidence of their effective- 1996). Emphasis on rigorous quantitative
ness will be lacking. Alternatively, using and cost-benefit evaluation of the impacts
the standard impact evaluation meth- of extension investments may cause the
ods, the range of impacts would not be extension service to adopt an elite bias,
captured. working with wealthier farmers who are
• This might prompt the revision of pro- more likely to show greater production in-
gramme activities and design away from creases for lower administrative and logis-
the original objectives. tical costs (Christoplos 1996). In contrast,
poorer farmers operating diversified sys-
Quantitative measures have a built-in tems, where the production is difficult to
bias toward high-external-input methods aggregate and quantify, are more likely to
which maximise production through po- be ignored.
tentially unsustainable techniques. If eval-
uation parameters based on production It is therefore necessary to decide on the
goals have been pre-set, the empower- ‘right’ level of rigour for a given situation,
ment of farmers to change the extension based on purpose, actors, etc.
39
Table 7
40
Other points to consider in evaluation plan- Attribution
ning are how the programme or interven-
tion was targeted to individuals, house- Attribution analysis aims to assess the ex-
holds, communities or other stakeholder tent to which the observed change can
groups – whether randomised assignment, really be attributed to the evaluated in-
or individual participant self selection, or tervention. Some writers consider this to
selection on the basis of programme or be synonymous with impact evaluation.
community determined criteria. One challenging aspect of evaluating new
models in agricultural service provision is
There are principles and approaches that they often represent different com-
that can ensure greater rigour in the use binations of change in governance struc-
of non-experimental and participatory tures, capacity, management and adviso-
methods. Triangulation of data collection ry methods. Moreover, which combination
is an important principle in using qualita- is most suitable for a given situation de-
tive methods. It is important to ensure pends on a number of frame conditions,
that a thorough stakeholder analysis is such as the type of farming system, soci-
carried out and the interactions based oeconomic conditions, and state capacity
on a selection of these groups (Guion (Birner et al. 2006). Hence it is inherently
2002). Triangulation of methods can add difficult to establish which factors account
further rigour by collecting similar data for observed changes in outcome if a new
using different methods, both quantita- model is introduced (Anderson 2007).
tive and qualitative, and by comparing
conclusions. Environmental triangulation In terms of the outcomes and impacts of
involves the use of different locations, agricultural service provision, there is a
settings and other key factors related to high level of risk and unpredictability as-
the environment in which the interven- sociated with factors outside the control of
tion took place and which may influence advisory service providers. Many of these
the information received during the eval- factors, for example climate variability, po-
uation (Guion 2002). litical instability, and market price volatil-
41
ity, can have serious negative direct and (from the original programme design or
indirect effects on the operation of other- constructed by the evaluation team) can
wise well-designed programmes. assist in this; however, alternative expla-
nations should also be considered
The level of attribution is related to the
choice of evaluation design discussed For complex RAS, isolating the impact or
above and the use of evidence to con- contribution of one service or one element
struct an ‘argument’ about the effects of is always challenging and sometimes im-
the programme interventions. Comparing possible. It certainly helps when the issue
evidence to programme theory diagrams is considered at project design stage.
42
Required principles • Demand-driven: A diverse range of de-
for RAS evaluation mands can be expected from diverse
user groups. Demand will differ based on
approaches in high
situational context (value chain, markets
complexity situations and market trends, pro-poor focus, etc.)
So far we have reviewed the different cri- • Pluralistic: The service will need to take
teria applied in evaluation and have estab- into account the co-existence of dif-
lished the relevance of the DAC evaluation ferent delivery agents and sources.
criteria in development aid, which includes Therefore, networking and cooperation
support to rural advisory services. Guiding are key elements.
principles for the RAS evaluation process • Learning and capacity focused: A key
should be the use of the DAC criteria as element of the service is to enhance
a framework to structure the evaluation users’ knowledge and capacity to ac-
content as well as the key characteristics cess and utilise RAS and to develop ad-
of a pluralistic and demand-driven exten- ditional capacities.
sion service. • Adaptive/ change oriented: No single
development pathway is likely to be
The key characteristics of RAS based suitable in complex social and environ-
on GFRAS (2010a), Anderson (2007), mental conditions. Therefore, the ser-
Hoffmann et al. (2009), and Patton (2008) vice needs to be adaptive and flexible in
can be summarised as follows: order to react to emerging opportunities
• Inclusive: RAS should embrace diversity and needs. Flexibility and practical re-
of users and needs to be suitable for sponses to risks and changes in circum-
different genders, ethnic groups and dif- stances are required.
ferent socio-economic categories, with
a particular focus on pro-poor inclusive- The table below is an example of how the
ness. Participation of users is a key ele- five DAC criteria could be linked to the at-
ment of such service. tributes of emerging RAS services.
43
Table 8: Agricultural extension evaluation matrix:
attributes of RAS by evaluation criteria
This matrix could serve as a framework for proach. The purpose, focus and questions
the evaluation design; however it does not of the evaluation, the resources at hand
on its own lead to a specific evaluation ap- (time, money, human resources) and the
44
Pluralistic delivery Learning / capacity Adaptive change
agents Partnerships / strengthening orientation
networking
Number and diversity Resources used to enable Response time of
of delivery agents. reflection and analysis reactions to changing
Frequency and number of performance circumstances – inertia
of farmer contacts
To what extent are the How well are lessons Do responses change
delivery agents different assessed and absorbed performance and
in performance? into practice? for how long?
45
Strengths, weaknesses Participation and user
and gaps in extension focus, demand-driven
Growing evidence suggests that stake-
evaluation approaches
holder involvement is a key requirement
suitable for complex for successful evaluation practices. Patton
situations (2008) stresses that the evaluation process
must discover the key stakeholders, name-
The Neuchâtel Group has identified the ly those who really want to know some-
need for common approaches to the thing, as this not only increases the chance
monitoring and evaluation of exten- of usefulness but also of ownership and ac-
sion activities. Their recommendation is ceptance. Ownership can lead to improved
that to improve the analysis of exten- uptake, whereas the lack of ownership of
sion schemes, clear frameworks should evaluations delivered externally can ex-
be offered to evaluators of projects with plain (at least partly) their lack of influence
an extension component. Donor agen- (Jones and Mendizabal 2010). External
cies could also devise common report- evaluations are intended to provide an in-
ing procedures (Neuchâtel Group 1999). dependent perspective (van de Berg 2004),
Although in theory this seems a desir- but it should be recognised that they can
able strategy, it might prove difficult to also contradict the aim of transferring
identify ‘best practice’ approaches to power to users. Furthermore, they can be
implement within such frameworks. As costly, and the lack of close association of
we have shown above, there are a range the evaluators with a programme can ham-
of possible approaches which could be per sample selection, the choice of param-
adopted for the evaluation of complex eters, and the interpretation of results (van
extension activities. In practice, it is de Berg 2004). In contrast, internal eval-
more a matter of defining ‘best-fit’ ap- uations are potentially more relevant and
proaches according to the specific cir- comprehensive, especially when primary
cumstances. However, a set of com- stakeholders are involved in designing the
monly adopted principles, including the evaluation (van de Berg 2004). However,
improved and coherent reporting re- other voices point out that direct or indirect
quirements mentioned above, could be stakes in a programme can cause results
an effective way to improve future ex- (e.g. benefits, disadvantages, costs) to be
tension evaluation strategies and facili- overstated or understated (van de Berg
tate joint learning. In this section we will 2004). A possible way forward is mixed-
identify strengths, weaknesses and gaps team evaluations, which can combine the
in current evaluation practice. benefits of both.
46
An important aspect to clarify here is ity to address future problems and leads to
the quality of stakeholder participation. local action to deal with current problems.
Participation in evaluation spans a gradi- These participatory processes, however,
ent from complete user (client)-controlled have methodological implications; for ex-
evaluations to initiatives by researchers ample, Carruthers and Tinning (2003) and
and development agents or governments Reed et al. (2005) suggest that the meth-
to consult users about the results of inter- ods used to collect, analyse, and interpret
ventions, to the participation of field work- data must be easy enough for active par-
ers and researchers in evaluation – as op- ticipation by local communities.
posed to external evaluations by funding
agencies with little focus on user involve- On the other hand, community indica-
ment. Lawrenz and Huffman (2003) sug- tors can be very specific to communities
gest the use of the following four criteria – based on their experiences and on the
to determine the extent of user partici- local context. This makes it difficult for the
pation: (1) type of evaluation information wider extrapolation and comparison of in-
collected, such as defining questions and dicators across communities and across
instruments; (2) participation in the eval- landscapes. Because of some of these
uation process; (3) decisions concerning shortcomings, expert-led or external-led
the data to provide; and (4) use of evalu- approaches argue the need for generic in-
ation information. dicators. Most of the discussion on com-
munity indicators and their use has been
Lessons from participatory evaluations in in the context of environmental sustaina-
community development projects outlined bility (Reed et al. 2005, 2006; Bossel 2001;
by Mancini and Jiggins (2008) established Fraser et al. 2005). There is a need to ex-
that participatory approaches to develop- plore these further in other contexts.
ing indicators provide opportunities for
community empowerment not otherwise
provided by top–down approaches (Fraser Accountability and learning
et al. 2005): they ensure that the indicators Accountability as a main purpose of eval-
are relevant to local situations; they meas- uation may contradict other purposes,
ure what is important to the community; as it is generally achieved through in-
and they are adapted over time as com- dependence and rigour, which can lead
munity circumstances change (Pretty 1995; to tensions with approaches adopted to
Carruthers and Tinning 2003; Freebairn pursue additional purposes. Baker et al.
and King 2003). The involvement of com- (2007) highlight potential conflicts be-
munities helps to build community capac- tween accountability and learning, and
47
Patton (2008) stresses that evaluations There are trends in institutional prac-
required by donors often become an end tices aimed at separating accountability
in themselves, which can undercut utility functions from wider evaluation purpos-
for other stakeholders. es. Examples here are UK, DEC and Sida,
the later having an Aid Watch Dog for ac-
Van de Berg (2004) describes a similar di- countability purposes and an internal eval-
lemma between rigour and comprehen- uation department focusing on utilization-
siveness, which can be caused by the based evaluation with a strong learning
limited understanding of the logical links orientation.
between input-output-outcome in result-
based evaluation frameworks (Rennekamp
and Arnold 2009). Despite wide application Learning/ capacity strengthening
of logic modelling in development inter- Very few of the case studies reviewed by
ventions, application in extension did not Pound et al. (2011) emphasize ‘learning’
happen until the mid-1990s (Taylor-Powell and ‘capacity building’ as an important
and Boyd 2008). If we are to learn from purpose of the evaluation process. This is
evaluations, they need to be designed in also reflected in the fact that procedures
such a way that the information they gen- and guidelines for sharing and dissemi-
erate helps to confirm the presumed link- nating evaluation findings are frequently
ages between actions and outcomes. Only lacking, which is an important weakness
then can the evaluation theory for exten- across different evaluation approaches.
sion systems be strengthened. Many of The identification of users and the plan-
the extensions reviewed by Pound et al. ning for communicating findings to these
(2011) fail to develop these linkages. It is users should be a key step in evaluation
important to develop more flexible and design. Capacity strengthening applies to
non-linear models which can incorporate the range of stakeholders and their organ-
various directions of behaviour change isations involved in the evaluation process.
and identify the ‘mediators’ between in- It can refer to improved capacities in eval-
terventions and behaviour changes and uation management of external and inter-
eventual outcomes (Braverman and Engle nal evaluators, but it can also refer to op-
2009). The approach also needs to be portunities for community empowerment
open to acknowledging a combination of (Fraser et al. 2005). Learning and capacity
factors such as commodity prices, mar- strengthening are closely linked to partici-
keting systems, level of education, policy pation, user involvement and demand ori-
context, appropriateness of innovation, entation as well as to accountability and
etc. (Hoffmann et al. 2009). learning, as discussed above.
48
Adaptive - change orientation holder analysis and careful identification
Evaluation approaches which are utiliza- of potential participants in the evaluation
tion-based and participatory are more like- process. It is therefore partly linked to the
ly to be able to adapt over time as commu- aspects discussed under the ‘participation’
nity circumstances change (Pretty 1995; section above. There are challenges to de-
Carruthers and Tinning 2003; Freebairn veloping wider inclusion of service provid-
and King 2003). Furthermore, due to their ers in RAS evaluations, particularly those
change orientation, they are more likely who are not defined in terms of ‘project
to build users’ capacities to address future partners’. There is still an important shift to
problems and lead to local action to deal be made in terms of thinking about RAS as
with current problems. The limitations of a network which operates across govern-
more conventional methods to capture un- ment and NGO agencies and the private
predictable but relevant effects have been sector and outside of particular projects.
criticized, as this leads to less adaptive in-
terventions with limited contributions of
the evaluation findings for programme im- Gaps
provement (Murray 2000). As pointed out by various authors, there is
a gap in evaluation studies which combine
different approaches in order to capture
Pluralistic delivery agents – the complexity of the RAS situation and to
partnerships/ networking cater for the demands and requirements
This is an aspect that has received little at- of different user groups. Findings from re-
tention in evaluation literature and can per- cent reviews of evaluations of Farmer Field
haps best be described by the additional EC Schools (FFS), which can be described as
criterion of coherence/ complementarity. multi-dimensional and complicated/com-
Emerging RAS systems consist of a range plex in terms of social and technical interac-
of potential delivery agents and this re- tions, suggest the need for a combination
quires collaboration and effective allocation of evaluation methods in order to evaluate
of tasks among them. In terms of evalua- impact. The Box below systematizes some
tion approach, it requires a detailed stake- of the key findings from recent studies:
49
Box 4: Lessons from evaluations of Farmer Field Schools (FFS)
A study of different evaluation methods for FFS conducted by Mancini and
Jiggins (2008) suggested that opinions on the evaluation approach are divergent
and will most likely remain divergent across stakeholders, and that there is a
need for a range of evaluation approaches and methodological innovations to
address evolving challenges posed by assessing complexities in development.
They conclude that mono-disciplinary studies with pre-determined objectives
are no longer considered sufficient to evaluate development interventions
centred on people’s empowerment. This is supported by van der Berg (2004),
who reviewed 25 case studies of FFS evaluations and found out that all the
studies were designed to be either statistically rigorous (but with a restricted
scope) or comprehensive (but with limited coverage), but never both, which
had negative consequences for their overall conclusions on FFS performance.
Similarly, Njuki et al. (2008), based on a study conducted in Malawi, emphasize
the need for ‘hybrid’ methodologies using elements of participatory and
conventional approaches for evaluating research and development programmes.
A further area of intervention should focus share findings not only within projects/
on better targeted dissemination of evalu- programmes but also with the wider ex-
ation findings to enhance learning. There tension evaluation community so that pro-
is an urgent need to disseminate and fessional practice can be further informed.
50
Ways forward tailored to facilitate exploration of the eval-
uation questions that have been defined
The review of the literature suggests that to within the time and resources available.
facilitate the identification and implementa-
tion of ‘hybrid’ approaches, there is a need Findings based on the literature have indi-
to develop guidance for evaluation plan- cated a particular need for principles and
ning which helps in the selection of evalu- guidance concerning:
ation approaches appropriate for complex • Decision-making on the level and extent
situations. An initial informal inquiry into of stakeholder participation in extension
the usefulness of an extension evaluation evaluation
toolbox has revealed that there is more • Development of flexible and non-linear
demand for a ‘concept and principle’ box, programme theory models
aimed at bringing different stakeholders’ • Designing mixed method evaluations
perspectives and evaluation demands clos- which address both the impacts of
er together, rather than the development of what has been done and the strate-
an additional evaluation toolbox. gic and institutional positioning of RAS
interventions
There are a large number of available • Procedures and guidelines for sharing
guidelines and tools available for evalua- and disseminating evaluation findings to
tion – some generic, like the EC evaluation different users
guidelines, and some more specific. The • Using the criteria of coherence and
specific tools and techniques used should complementarity.
be consistent with the principles underpin- • Combining and sequencing different ap-
ning the evaluation and its objectives and proaches and tools in evaluation
51
References
• ActionAid. 2009. Who’s really fighting hunger? Hunger Free
ScoreCard 2009. Johannesburg: ActionAid.
52
• Barker, B. S. 2007. Stakeholders’ input on 4-H science and technology program
area exploratory study. Journal of Extension [On-line], 45(2) Article 4RIB6.
Available at: www.joe.org/joe/2007april/rb6.php (accessed 18 July 2011)
• Braverman, M. T., and M. Engle. 2009. Theory and rigor in Extension program
evaluation planning. Journal of Extension 47(3).
www.joe.org/joe/2009june/a1.php (accessed 6 June 2011).
• Carruthers, G., Tinning, G., 2003. Where, and how, do monitoring and
sustainability indicators fit into environmental management systems?
Australian Journal of Experimental Agriculture 43, 307–323.
53
• CIDA. 2004. CIDA Evaluation Guide. Quebec: Canadian International
Development Agency (CIDA).
www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Performancereview5/$file/
English_E_guide.pdf (accessed 9 September 2010).
• Davies, R. J., and J. Dart. 2005. The most significant change (MSC) technique: A
guide to its use. www.mande.co.uk/docs/MSCGuide.pdf
(accessed 6 June 2011).
54
• Duignan, P. 2008. Reframing program evaluation as part of strategic information collec-
tion for sector decision-making. Outcomes Theory Knowledge Base Article No. 221.
http://knol.google.com/k/reframing-program-evaluation-as-part-of-
collecting-strategic-information-for# (accessed 19 July 2011)
• Earl, S., F. Carden, and T. Smutylo. 2001. Outcome Mapping: Building Learning
and Reflection into Development Programs. Ottawa: IDRC.
www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html (accessed 6 June 2011).
55
• EuropeAid 2006. Evaluation Methods.
http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_pro_en.htm
(accessed 14 July 2011)
• Fraser, E., Mabee, W., Figge, F., 2005. A framework for assessing the
vulnerability of food systems to future shocks. Futures 37, 465–479.
56
• GFRAS. 2010a. Five Key Areas for Mobilising the Potential of Rural Advisory
Services. GFRAS Brief 1, October 2010. Global Forum for Rural Advisory
Services (GFRAS), c/o Agridea, Eschikon 28, 8315 Lindau, Switzerland. www.g-
fras.org/en/knowledge/gfras-publications# (accessed 19 July 2011)
• GFRAS. 2010b. Providing space for advocacy and leadership on rural advisory
services Strategy and Work Plan for Inception Phase (January 2010 – June 2011).
Global Forum for Rural Advisory Services (GFRAS), c/o Agridea, Eschikon 28, 8315
Lindau, Switzerland. www.g-fras.org/fileadmin/UserFiles/GFRAS-documents/
GFRAS-Strategy-for-Inception-Phase_Jan2010-June2011.pdf (accessed 19 July 2011)
• GTZ. 2006. Working for sustainable results. Evaluation at GTZ. Eschborn: GTZ.
www2.gtz.de/dokumente/bib/06-0796.pdf (accessed 9 September 2010).
• GTZ. 2007. Results Monitoring 2007: Evaluation Report on the Work of GTZ
and Its Partners. 10th Cross-section Analysis 2006–2007. Eschborn: GTZ.
www.gtz.de/en/dokumente/Evaluation-Report-0801.pdf (accessed 9 September 2010).
• Guijt, I. 2004. ALPS in Action: A Review of the Shift in ActionAid towards a new
Accountability, Learning and Planning System. Action Aid International.
www.actionaid.org.uk/_content/documents/ALPSReview.pdf (accessed 14 July 2011)
57
• IAASTD. 2009. Agriculture at a Crossroads: Synthesis Report.
Washington: Island Press.
• IFAD. 2003. IFAD Evaluation Policy. Rome: International Fund for Agricultural
Development (IFAD). www.ifad.org/pub/policy/oe.pdf (accessed 9 September 2010).
• Leeuwis, C., and A. van den Ban. 2004. Communication for Rural Innovation:
Rethinking Agricultural Extension. 3rd ed. Oxford: Blackwell Publishing.
• Ljungman, Cecilia M., Helge Rønning, Tejeshwar Singh, Henrik Steen Pedersen et
al. 2005. Sida’s Work with Culture and Media. Main Report. Sida Evaluation 04/38.
58
• Narayan, Deepa. 1993. Participatory Evaluation: Tools for Managing
Change in Water and Sanitation. World Bank Technical Paper 207
• Neuchâtel Group. 2000. Guide for Monitoring, Evaluation and Joint Analyses of
Pluralistic Extension Support. Lindau: Neuchâtel Group.
www.g-fras.org/fileadmin/UserFiles/Documents/Frames-and-guidelines/M_E/
Guide-for-Monitoring-Evaluation-and-Joint-Analysis.pdf (accessed 6 June 2011).
• OECD. 1998. Review of the DAC Principles for Evaluation of Development Assistance.
Paris: DAC Working Party on Aid Evaluation.
www.oecd.org/dataoecd/63/50/2065863.pdf (accessed 6 June 2011).
• OECD. 2005/2008. The Paris Declaration on Aid Effectiveness and the Accra Agenda
for Action. Paris: OECD.
www.oecd.org/dataoecd/30/63/43911948.pdf (accessed 6 June 2011).
59
• OECD/DAC Working Party on Aid Effectiveness. 2008. Aid Effectiveness:
A Progress Report on Implementing the Paris Declaration. 3rd High level Forum
on Aid Effectiveness, September 2–4, 2008, Accra, Ghana.
http://siteresources.worldbank.org/ACCRAEXT/Resources/Progress_Report-Full-EN.pdf
(accessed 6 June 2011).
• Patton, M.Q. 2007. Process use as a usefulism. In J. B. Cousins (Ed.), Process use
in theory, research, and practice. New Directions for Evaluation, 116, 99–112.
• Rajalahti, R., J. Woelcke, and E. Pehu. 2005. Monitoring and Evaluation for
World Bank Agricultural Research and Extension Projects: A Good Practice Note.
Agriculture and Rural Development Discussion Paper 20. Washington: The
International Bank for Reconstruction and Development / The World Bank.
http://siteresources.worldbank.org/INTARD/Resources/ARD_DP20.pdf
(accessed 14 July 2011)
60
• Reed, M.S., Fraser, E.D.G., Morse, S., Dougill, A.J., 2005. Integrating
methods for developing sustainability indicators that can facilitate
learning and action. Ecology and Society 10 (1): r3 (online).
• Reuber, M., and O. Haas. 2009. Evaluations at the Deutsche Gesellschaft für Technische
Zusammenarbeit (GTZ GmbH) German Technical Cooperation. Eschborn: GTZ.
www.gtz.de/en/dokumente/en-evaluations-at-gtz-reuber-haas-2009.pdf
(accessed 9 September 2010).
61
• UNDP. 2009. Handbook on Planning, Monitoring and Evaluating for Development
Results. New York: UNDP.
www.undp.org/evaluation/handbook/documents/english/pme-handbook.pdf
(accessed 9 September 2010).
• van den Berg, H. 2004. IPM FFS: A synthesis of 25 impact evaluations. Report
prepared for the Global IPM Facility. Wageningen: The Netherlands.
62
• World Bank. 2008. World Development Report 2008: Agriculture for Development.
Washington: The International Bank for Reconstruction and Development /
The World Bank.
• World Bank Group. n.d. Evaluation for better results. IEG Brochure.
Washington: Independent Evaluation Group (IEG).
http://siteresources.worldbank.org/EXTOED/Resources/ieg_brochure.pdf
63
The Global Forum for Rural Advisory Services (GFRAS) has
commissioned work to develop guidance for the evaluation of
extension (projects, programmes, tools and initiatives). The
purpose of the overall project is to identify methods for better
evaluation of extension through the development of a Guidance
Kit for extension evaluation.
webhint.ch