Synonyms:
QUIP
The QUIP sets out to generate differentiated evidence of impact based on narrative causal statements elicited directly from intended project beneficiaries without use of a control group. Evidence of attribution is sought through respondents’ own accounts of causal mechanisms linking X to Y alongside Z rather than by relying on statistical inference based on variable exposure to X. This narrative data is intended to complement quantitative evidence on changes in X, Y and Z obtained through routine project monitoring.
QuiP includes strategies to reduce the risk of response bias – in particular by collecting data without reference to the project being evaluated.
The Qualitative Impact Assessment Protocol uses a Contribution Analysis approach; combining quantitative monitoring of key indicators with qualitative, self-reported attribution of impact (key informant attribution) gathered through interviews to provide sufficient evidence using process tracing to test the theory of change behind the activity being evaluated.
Resources
Discussion paper
The Qualitative Impact Assessment Protocol (QUIP) – This easy to read CDS Briefing Paper by James Copestake and Fiona Remnant presents an overview of the QUIP in three steps: the background to the QUIP and its main aims; the data collection and analysis methodology; and QUIP in the context of other approaches to evaluation, particularly Contribution Analysis and Process Tracing.
Example
Assessing rural transformations: piloting a qualitative impact protocol in Malawi and Ethiopia – This working paper by James Copestake and Fiona Remnant reports on findings from four pilot studies of a protocol for qualitative impact evaluation of NGO sponsored rural development projects in Malawi and Ethiopia
Guide
QUIP: Understanding clients through in-depth interviews – This Practice Note by Imp-Act gives a step-by-step guide to developing and conducting in-depth interviews using the QUIP approach, and analysing the information and making conclusions based on what you have learned.

Source:
Qualitative Impact Assessment Protocol (QUIP). (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/QUIP

Synonyms:
Use-focused evaluation, Utilisation Focused Evaluation, Utilization Focused Evaluation

Utilization-Focused Evaluation (UFE), developed by Michael Quinn Patton, is an approach based on the principle that an evaluation should be judged on its usefulness to its intended users.  Therefore evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance.

UFE has two essential elements.  Firstly, the primary intended users of the evaluation must be clearly identified and personally engaged at the beginning of the evaluation process to ensure that their primary intended uses can be identified.  Secondly, evaluators must ensure that these intended uses of the evaluation by the primary intended users guide all other decisions that are made about the evaluation process.

Rather than a focus on general and abstract users and uses, UFE is focused on real and specific users and uses.  The evaluator’s job is not to make decisions independently of the intended users, but rather to facilitate decision making amongst the people who will use the findings of the evaluation.

Patton argues that research on evaluation demonstrates that: “Intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process and findings [and that] [t]hey are more likely to understand and feel ownership if they’ve been actively involved. By actively involving primary intended users, the evaluator is preparing the groundwork for use.” (Patton, 2008, Chapter 3)

UFE can be used for different types of evaluation (formative, summative, process, impact) and it can use different research designs and types of data.

The UFE framework can be used in a variety of ways depending on the context and the needs of the situation.  Patton’s original framework consisted of a 5 step process which can be seen in the example below.  However, there is also a 12 step framework (see Utilization-Focused Evaluation (U-FE)Checklist in the resources below) and the latest update which is 17 steps is outlined below.

The 17 Step UFE Framework

• Assess and build program and organizational readiness for utilization-focused evaluation
• Assess and enhance evaluator readiness and competence to undertake a utilization-focused evaluation
• Identify, organize, and engage primary intended users: the personal factor
• Situation analysis conducted jointly with primary intended users
• Identify and prioritize primary intended uses by determining priority purposes
• Consider and build in process uses if and as appropriate
• Focus priority evaluation questions
• Check that fundamental areas for evaluation inquiry are being adequately addressed: implementation, outcomes, and attribution questions
• Determine what intervention model or theory of change is being evaluated
• Negotiate appropriate methods to generate credible findings that support intended use by intended users
• Make sure intended users understand potential methods controversies and their implications
• Simulate use of findings: evaluation’s equivalent of a dress rehearsal
• Gather data with ongoing attention to use
• Organize and present the data for interpretation and use by primary intended users: analysis, interpretation, judgment, and recommendations
• Prepare an evaluation report to facilitate use and disseminate significant findings to expand influence
• Follow up with primary intended users to facilitate and enhance use
• Meta-evaluation of use: be accountable, learn, and improve

Example

This example utilizes the 5 step UFE framework.

International Network for Bamboo and Rattan (INBAR)
In late 2006, the International Network for Bamboo and Rattan engaged one of the authors (Horton) to evaluate its programmes. Headquartered in Beijing, INBAR’s mission is to improve the wellbeing of bamboo and rattan producers and users while ensuring the sustainability of the bamboo and rattan resource base. The Dutch Government had initially requested and funded the evaluation as an end-of-grant requirement.

Step 1. Identify primary intended users. The first task was to ascertain the ‘real’ purposes and potential users of the evaluation. This process began with a face-to-face meeting with INBAR’s Director and a call to a desk officer at the Dutch Ministry of Foreign Affairs, which revealed that the intent of both parties was for the evaluation to contribute to strengthening INBAR’s programmes and management. During an initial visit to INBAR’s headquarters, additional stakeholders were identified, including INBAR board members and local partners.

Step 2. Gain commitment to UFE and focus the evaluation. From the outset, it was clear that key stakeholders were committed to using the evaluation to improve INBAR’s work. So the main task was to identify key issues for INBAR’s organizational development. Three options were used: (1) a day-long participatory staff workshop to review INBAR’s recent work and identify main strengths, weaknesses and areas for improvement; (2) interviews with managers and staff members; and (3) proposing a framework for the evaluation that covered the broad areas of strategy, management systems, programmes and results.

Step 3. Decide on evaluation options. After early interactions with the Dutch Ministry of Foreign Affairs on the evaluation Terms of Reference (ToR), most interactions were with INBAR managers, staff members and partners at field sites. It was jointly decided that INBAR would prepare a consolidated report on its recent activities (following an outline proposed by the evaluator) and organize a self-evaluation workshop at headquarters. The evaluator would participate in this workshop and make field visits in China, Ghana, Ethiopia and India. INBAR regional coordinators proposed schedules for the field visits, which were then negotiated with the evaluator.

Step 4. Analyze and interpret findings and reach conclusions. At the end of each field visit, a debriefing session was held with local INBAR staff members. At the end of the field visits, a half-day debriefing session and discussion was held at INBAR headquarters; this was open to all staff. After this meeting, the evaluator met with individual staff members who expressed a desire to have a more personal input into the evaluation process. Later on, INBAR managers and staff members were invited to comment on and correct a draft evaluation report.

Step 5. Disseminate evaluation findings. The evaluator met personally with representatives of three of INBAR’s donors to discuss the evaluation’s findings, and the final report was made available to INBAR’s donors, staff members and the Board of Trustees. A summary of the report was posted on the INBAR website.

Utility of the evaluation. The evaluation process helped to bring a number of issues to the surface and explore options for strengthening INBAR’s programmes. For example, one conclusion of the evaluation was that INBAR should seek to intensify its work in Africa and decentralize responsibilities for project management to the region. There has been a gradual movement in this direction, as new projects have been developed. INBAR has recently opened a regional office for East Africa, in Addis Ababa and is putting more emphasis on collaboration with regional and national partners.

Source: Patton, M.Q. and Horton, D. 2009. Utilization-Focused Evaluation for Agricultural Innovation. International Labor Accreditation Cooperation (ILAC) Brief No. 22. ILAC, Bioversity, Rome.

Advice

Advice when USING this approach

• While in principle there is a straightforward, one-step-at-a-time logic to the unfolding of a utilization-focused evaluation, in reality the process is seldom either as simple or as linear. For example, the evaluator may find that new users become important as the evaluation proceeds or that new questions emerge in the midst of options decisions. Nor is there necessarily a clear and clean distinction between the processes of focusing evaluation questions and making options decisions; questions inform options, and methodological preferences can inform questions.
• U-FE requires active and skilled guidance from and facilitation by an evaluation facilitator.
• Time resources available for the evaluation must be clearly negotiated, built in from the beginning, and stated in the agreement. The essentially collaborative nature of U-F evaluation demands time and active participation, at every step of the entire process, from those who will use the evaluation results. Additional resources may be needed if new uses or users are added after the evaluation has begun.
• Financial resources available for the evaluation must be clearly stated. They must include financial resources beyond mere analysis and reporting. Resources that facilitate use must be available.
• In conducting a U-F evaluation the evaluator must give careful consideration to how everything that is done, from beginning to end, will affect use.

Resources

Guides

• Utilization Focused Evaluation: A primer for evaluators: This book, written by Ricardo Ramírez and Dal Brodhead, is designed to support evaluators and program managers implement Utilization-focused evaluation (UFE). It includes detailed discussion of the 12 steps for implementing UFE and also provides a number of case studies to guide the user.
• Utilization-Focused Evaluation: 4th edition: this book from Michael Quinn Patton provides detailed advice for using UFE to conduct evaluation programs.  It includes a range of activities and a specific UFE checklist to ensure a well-designed program evaluation takes place.
• Essentials of utilization-focused evaluation: a primer: Drawing on Michael Quinn Patton‘s best-selling Utilization-Focused Evaluation, this book provides an overall framework and essential checklist steps for designing and conducting evaluations that actually get used.
• Utilization-Focused Evaluation for Agricultural Innovation: This paper introduces UFE to evaluation, outlines key steps in the evaluation process and provides specific examples.

Tools

• Utilization-Focused Evaluation (U-FE)Checklist:   designed by Michael Quinn Patton in 2002 this is a comprehensive checklist for undertaking a utilization focused evaluation. It draws attention to the necessary tasks and associated challenges of conducting Utilization-Focused Evaluation (UFE).

Examples

• Reflections on the Utilization-Focused Evaluation (UFE) Process: This presentation from the Strengthening ICTD Research Capacity in Asia (SIRCA) provides an overview of how UFE was used in their SIRCA programme.
• Case Study: The Information Society Innovation Fund (ISIF): This case study describes how the Information Society Innovation Fund (ISIF) used UFE for a project with the Developing Evaluation Capacity in ICTD (DECI) programme.

Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings.

Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?.

The complete realist question is: “What works, for whom, in what respects, to what extent, in what contexts, and how?”. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.

The realist understanding of how programmes work

Realist philosophy (Pawson and Tilley use the term ‘scientific realism’) considers that an intervention works (or not) because actors make particular decisions in response to the intervention (or not). The ‘reasoning’ of the actors in response to the resources or opportunities provided by the intervention is what causes the outcomes.

Strictly speaking, the term ‘generative mechanism’ refers to the underlying social or psychological drivers that ‘cause’ the reasoning of actors. For example, a parenting skills programme may have achieved different outcomes for fathers and mothers. The mechanism generating different ‘reasoning’ by mothers and fathers may relate to dominant social norms about the roles and responsibilities of mothers and fathers. Additional mechanisms may be situated in psychological, social or other spheres.

Context matters: firstly, it influences ‘reasoning’ and, secondly, generative mechanisms can only work if the circumstances are right. Going back to our example, there may different social beliefs about the roles and responsibilities of mothers and fathers in different cultures, which may affect how parents respond to the parenting programme. Whether parents can put their new learning into practice will depend on a range of factors – perhaps the time they have available, their own beliefs about parenting, or their mental health. Finally, the context may provide alternative explanations of the observed outcomes, and these need to taken into account during the analysis.

Undertaking a realist evaluation

Developing the initial programme theory

Realist evaluation starts with theory and ends with theory. In other words, the purpose of a realist evaluation is as much to test and refine the programme theory as it is to determine whether and how the programme worked in a particular setting.

The programme theory describes how the intervention is expected to lead to its effects and in which conditions it should do so. The initial programme theory may be based on previous research, knowledge, experience, and the assumptions of the intervention designers about how the intervention will work. The difference between realist and other kinds of programme theory-based evaluation approaches is that a realist programme theory specifies what mechanisms will generate the outcomes and what features of the context will affect whether or not those mechanisms operate. Ideally, these elements (mechanisms, outcome, context) are made explicit at the evaluation design stage, as it enables to design the data collection to focus on testing the different elements of the programme theory.

Choosing the evaluation methods

Realist evaluation is method-neutral (i.e., it does not impose the use of particular methods).

As with any evaluation, the choice of data collection, and analysis methods and tools should be guided by the types of data that are needed to answer the evaluation questions, or more specifically, to test the initial programme theory in all its dimensions.

Usually, both quantitative and qualitative data are collected in a realist evaluation, often with quantitative data being focused on context and outcomes and qualitative data on generative mechanisms. Because the realist analysis uses mainly intra-programme comparisons (i.e., comparisons across different groups involved in the same programme) to test the initial theory, a realist evaluation design does not need to construct comparison groups. Rather, the refined programme theory will be tested subsequently in a different context in a next study. Often the case study design is used, whereby case selection is typically purposive, as the cases should enable ‘testing’ of the initial programme theory in all its dimensions.

Using a realist data analysis approach

Realist data analysis is driven by the principles of realism: realist evaluation explains change brought about by an intervention by referring to the actors who act and change (or not) a situation under specific conditions and under the influence of external events (including the intervention itself). The actors and the interventions are considered to be embedded in a social reality that influences how the intervention is implemented and how actors respond to it (or not). The context-mechanism-outcome (CMO) configuration is used as the main structure for realist analysis.

In the first phase of analysis, data are organised in relation to the initial programme theory – that is, whether the data relate to what was done (the intervention activities) or to context, mechanism, outcome and (groups of) actors. Qualitative data are coded and appropriate methods for analysing quantitative data applied. The data on outcomes are disaggregated by sub-groups (which were selected on the basis of the programme theory).

Once patterns of outcomes are identified, the mechanisms generating those outcomes can be analysed, provided the right kinds of data are available. The contexts in which particular mechanisms did or did not ‘fire’ can then be determined. Contexts may relate to the sub-groups for whom outcomes were generated and/or to other stakeholders, processes of implementation, organisational, socio-economic, cultural and political conditions.

The analytic process is not necessarily sequential, but should result in a set of ‘context-mechanism-outcome’ (CMO) statements: “In this context, that particular mechanism fired for these actors, generating those outcomes. In that context, this other mechanism fired, generating these different outcomes.”

The last phase of the analysis consists of determining which CMO configuration(s) offers the most robust and plausible explanation of the observed pattern of outcomes. This resulting CMO configuration is then compared with the initial programme theory, which is modified (or not) in light of the evaluation findings.

Using the findings from realist evaluation

Both generative mechanisms and programme theories can be considered at different levels of abstraction, from very specific (particular individuals within particular programmes) to quite abstract (across different kinds of programmes). Pawson and Tilley argued that ‘middle range theories’* (MRT) are most useful.  MRTs are specific enough to generate particular propositions to test and general enough to apply across different situations. Typically, MRTs develop over time based on the accumulation of insights acquired through a series of studies allowing gradual specification of the realist findings. All kinds of theory in realist evaluation – programme theory, theories about particular mechanisms, CMOs, and formal theory– are most useful if developed at a middle level of abstraction.

Because realist evaluation uses the idea of generative causality (i.e. mechanisms only fire when the context is conducive), realists are modest in their claims, stating that an evaluation cannot produce universally applicable findings. At best, evaluation can make sense of the complex processes underlying programmes by formulating plausible explanations ex-post. It can indicate the conditions in which the intervention works (or not) and how they do so. This realistic specification allows decision makers to assess whether interventions that proved successful in one setting may be so in another setting, and assists programme planners in adapting interventions to suit specific contexts.

*A middle range theory is understood as “theory that lies between the minor but necessary working hypotheses …and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organization and social change” [Merton, R.K .(1968). Social theory and social structure. New York: The Free Press, p39].  In essence, “middle range” refers to the degree of abstraction and can refer to programme theory, generative mechanisms, CMOs, formal theories etc.

Image source: Mobius Transform of a regular circle packing, by fdecomite on Flickr

Source:
Realist Evaluation. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/approach/realist_evaluation

Synonyms:
Clinical Trial, Randomised Controlled Trial, Randomized Impact Evaluations

Randomized controlled trials (RCTs), or randomized impact evaluations, are a type of impact evaluation which uses randomized access to social programs as a means of limiting bias and generating an internally valid impact estimate.

An RCT randomizes who receives a program (or service, or pill) – the treatment group – and who does not – the control. It then compares outcomes between those two groups; this comparison gives us the impact of the program. RCTs do not necessarily require a “no treatment” control – randomization can just as easily be used to compare different versions of the same program, or different programs trying to tackle the same problem.

In this way, the control mimics the counterfactual. The counterfactual is defined as what would have happened to the same individuals at the same time had the program not been implemented. It is, by definition, impossible to observe – it’s an alternative universe! RCTs work by creating a group that can mimic it.

Many times, evaluations compare groups that are quite different to the group receiving the program. For example: if we compare the outcomes for women who take up microcredit to those that do not, it could be that women who choose not to take up microcredit were different in important ways that would affect the outcomes. For example, women who do not take up microcredit might be less motivated, or less aware of financial products.

Using a randomization approach means that a target population is first identified by the program implementer, and then program access is randomized within that population.

Instead of randomizing individuals, randomization can be done at cluster levels, such as villages, or schools, or health clinics. These are known as cluster randomized control trials.

There are two main reasons to randomize at a level larger than the individual. First, it can address contamination: where treated individuals mix and chat and potentially “share” treatment with individuals in the control group. This would “contaminate” our impact, and our control group would no longer be a good comparison. Randomizing at the village level may minimize the risk of this happening. Second, we might want to randomize at the level that the intervention would actually be implemented: for example, an intervention which provides electrification to schools. It is logistically impractical – if not impossible – to randomize electricity access over schoolchildren.

When randomizing at the cluster level, the unit of randomization is the unit at which we will randomly roll out the program; i.e. the cluster (in our example above, a school). The unit of analysis, defined as the unit at which we will collect data and compare outcomes, is usually the individual – for example, individual students’ test scores. This distinction will become important when we calculate the sample size needed. Among other things, sample size is affected by the intra-cluster correlation (ICC), which refers to how similar or dissimilar individuals within a cluster are. The ICC will determine how many individuals per cluster, and how many clusters, you will need to sample.

Simple example

Consider this hypothetical example: an NGO, iPads 4 All (i4A), plans to distribute iPads to low-income children in a developing country. i4A want to evaluate the impact that an iPad has on children’s education, health and future income levels. It’s likely that they will never have enough iPads to cover all the children that “deserve” one. Instead of ad hoc distribution to the children who express interest, or are nearby, or who the government determines as “neediest”, an RCT would randomize their access.

If they randomize at the individual level, they would put all the eligible children’s names into a bowl, or a list on a computer, and run a lottery. Some children would get an iPad. Some would not. If they randomize at the school level, they would do this for the school names and some schools would receive iPads. In a phase-in/pipeline design, the individuals or schools who did not receive an iPad initially would be placed in a queue to receive it if the study found them to be effective and funds were available.

Beyond this simplified example, the RCT methodology can be adapted to a wide variety of contexts.

As with all human subjects research, RCTs are subject to rigorous ethical reviews to ensure that no human subjects are harmed during the research process.

Steps of an RCT

• An optional prelude is a needs assessment, which can provide information on the context and its constraints. For example: a needs assessment could tell us how many children have received their full immunization course in rural Rajasthan. It could lead us to specify a hypothesis, or key evaluation question.
• A program theory is developed (alternatively, a logic model). This program theory describes the program, unpacking the pathways of its impact, and articulates all the risks and assumptions which could hamper a successful program. It is also useful, at this stage, to think of the indicators which could be collected at each step of the way.
• A baseline survey is conducted of the entire target sample. Data are collected on the relevant indicators.
• The sample is randomized into different groups. Randomization can be done using software like Excel, or Stata. To ensure that randomization has “succeeded”, check they are equivalent in terms of baseline indicators and contextual variables that might be important: they should be statistically identical – that is, the same average income, the same average health level, and so on. ​
• The program or intervention is implemented in the treatment group.
• During the program, it is strongly advisable to monitor the program’s implementation. This data will have three advantages. First, it becomes a type of monitoring, which is beneficial for the implementing organization’s operations and efficiency. Second, it provides intermediate indicators which allow evaluators to unpack the “black box” of impact (and follow along the theory of change). In other words, these intermediate indicators allow us to answer why a program had the effect it did. Third, and most importantly, it is necessary to monitor that the intervention is being adequately implemented to the treatment group(s), and the control group is not being contaminated (receiving the intervention through some other means).
• Following the program’s implementation, and depending on the context of the evaluation (e.g. some indicators are quick to respond, others slow), there is an endline, or follow-up, survey. Ideally, this survey will share many questions and characteristic with the baseline survey.
• Outcomes are then compared between treatment and control groups to derive the impact estimate. Results are reported to the implementing partner.

Examples

The RCT approach is flexible enough to accommodate a variety of contexts and sectors. It can be used in education, health, environment, and so on. With a little imagination, randomization can be adapted to a number of different circumstances. Constraints and what RCTs cannot do will be discussed below. For now, here is a short gallery of examples of what RCTs can do.

Mapping the approach in terms of tasks and options

RCTs share, with other impact evaluation methodologies, a number of the same tasks and options. For example, by definition, they must specify key evaluation questions.  These questions could be things like: will deworming pills lead to increased school attendance? Will they lead to improved educational outcomes as well? Does access to microfinance lead to greater business investments? Is iron-fortified salt an effective way of decreasing anemia rates in the rural population?

Furthermore, data collection and data analysis are integral parts of the RCT approach. A deep understanding of the sample is essential: who is the target population? Is the selected sample representative of the larger population? Following randomization of program access, are the treatment and control groups comparable along important indicators? Thinking deeply about indicators is also important: for example, how will women’s empowerment be measured? Cognitive ability? Financial literacy? How will data on these indicators be collected?

Finally, given the importance of establishing causality, it is useful to highlight the role of the control group as counterfactual 

Advice on choosing this approach

It is important to remember that, while RCTs can be a rigorous way to measure impact in certain circumstances, they are only one part of a wider array of evaluation tools. That is, they can be a useful addition to any portfolio of methods, but they are unlikely to be able to answer every question. In this section, we will describe some of the binding constraints which would prevent an evaluator from choosing the RCT approach.

Binding constraints: Sample size

One of the major constraints to any quantitative impact evaluation – not just RCTs – is sample size. In the case of RCTs, we are concerned with sample size along two dimensions: the unit of analysis, and the unit of randomization. Both the unit of analysis and the unit of randomization are integral in determining statistical significance and statistical power.

Statistical significance refers to the probability that the results we observe are not purely based on chance. Conventions in the literature state that significance levels above 90% – preferably at 95% – are sufficient. This means that, either 5% or 10% of the time, the results we observe are by chance.

Statistical power, instead, refers to the probability of detecting an impact when there is one. The inverse, then, is how likely are we to miss impact when it occurs (thus generating a “false negative”)? A number of factors determine statistical power: the sample size, the minimum detectable effect size (i.e. how sensitive must the test be), the outcome variable’s underlying variance, the proportion that are in treatment and control, and – if it is a cluster RCT – the intra-cluster correlation. Convention allows 80% to be a sufficient level of power.

There is an argument for saying that, for low levels of power, it is preferable not to conduct an impact evaluation – else resources will be wasted, resources which could be better used elsewhere (in conducting a good process evaluation, for example).

Binding constraints: Retrospective vs. Prospective

By design, RCTs cannot determine impacts of currently existing projects, that is, of programs that have already launched and did not, by chance, randomly deliver their services. (Most programs are, indeed, not delivered randomly – notable exceptions being Mexico’s PROGRESA and reservations for women and caste minorities under India’s 73rd amendment.) Given that randomization occurs at the moment of implementation, and randomization is integral to the RCT approach, they can only be planned ex ante – not ex post. Thus, for existing programs, RCTs can only be applied to either: roll-outs of the program into new areas, additions to the program (e.g. new products).

Advice when using this approach

There are a number of issues that may arise during the implementation of even the best-designed RCT. It is important, then, to be prepared and include plans to mitigate or control various risks.

Take-up rates

Take-up rates can sometimes be lower than expected, and this can have consequences on your effect size (and, following that, on your statistical power). It is worth noting that the relationship between take-up and power is exponential: a 50% drop in effect size will require a four-fold increase in sample size to achieve the same power.

For this reason, it is advisable to adequately anticipate – and, if anything, underestimate – take-up rates of the program. Choosing a conservative, even pessimistic, estimate for this may reward you with higher power down the line.

Non-compliance

Another issue which can compromise an RCT’s estimates is non-compliance by program participants. That is, while individuals may be assigned to treatment or control, these assignments are rarely required or controlled. Consider a microfinance program, which opens branches in randomly-selected “treatment” neighborhoods and does not do so in “control” neighborhoods. Individuals living in the latter may simply journey to the “treatment” neighborhoods in order to open an account at the microfinance branch office. In this case, the control group no longer serves as a true counterfactual.

Non-compliance, then, can threaten the integrity of randomization if individuals are able to self-select into groups. While non-compliance can never totally be eliminated, it can be minimized. One method is to choose a unit of randomization large enough such that the two groups are unlikely to mix. For example, in the microfinance example, if “treatment” and “control” neighborhoods were also reasonably far apart, we might expect non-compliance to remain low.

Note, however, when control group individuals take up the program, and treatment individuals do not, this resembles the encouragement design.

Attrition

Attrition occurs when parts of your sample are no longer available for follow-up, for example, because they’ve moved away. If attrition is driven by statistical differences in your treatment and control groups, we call this “differential attrition”. This can be especially concerning, because it essentially un-randomizes your sample, as people are self-selecting out of one group or the other. It is important to note that, while the rates of attrition may look the same, differential attrition may still be occurring if the reasons people are leaving the treatment or control groups is related to the treatment.

In the microfinance example, differential attrition could occur if some households in treatment neighborhoods obtain loans, grow their businesses, and become wealthy enough to leave the neighborhood – out of our sample. If this was the case, we would not be able to include them in our analysis, and thus our remaining “treatment group” would look a little poorer than it should (since all the wealthy households have moved away!). It is therefore very important to follow up with households, especially in the case of differential attrition.

Non-differential attrition occurs when attrition from the treatment or control groups occurs for reasons unrelated to the treatment: people may move away, die, or otherwise drop out of our sample, and it has nothing to do with whether they are in treatment or control. In this case, we would only worry if non-differential attrition erodes our sample size such that issues of statistical significance or power crop up.

Conducting a baseline survey

In theory, if randomization has been successfully implemented, an endline survey is sufficient in determining an internally valid impact estimate. However, baseline surveys, beyond providing empirical assurance that randomization has generated balanced treatment and control groups, provide an additional benefit in the form of increased power. In general, more frequent data collection (such as a baseline, midline, and endline) can give us the same power for a smaller sample size. Also, baseline results allow us to measure heterogeneous effects (i.e. subgroup analysis) when the groups are defined by variables that could change over time. For example it allows us to measure the test-score impact of an education innovation on the subgroup of children who scored poorly on the exam at baseline. Without the baseline, we would not be able to identify which children these were.

Comparing multiple treatments

If we want to detect the difference between two variations of a program, then we will need more power – and, consequently, a larger sample size. If we simply want to compare having a program to not having a program, then less power (and thus a smaller sample size, relatively) is sufficient.

Resources

Guides

Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit. Handbook of development economics, 4, 3895-3962.

Courses

The Abdul Latif Jameel Poverty Action Lab (J-PAL) offers a weeklong Executive Education course throughout the world and throughout the year. This course explores impact evaluation, focusing on when and how to use randomized impact evaluations. A free, archived version of the course can be found online at MIT’s Open CourseWare.

FAQ (Frequently Asked Questions)

1. Are RCTs a new approach in evaluation?

2. What should you do if randomization does not produce equivalent groups?

Source:
Randomized Controlled Trial (RCT). (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/rct

Positive Deviance (PD) refers to a behavioral and social change approach which is premised on the observation that in any context, certain individuals confronting similar challenges, constraints, and resource deprivations to their peers, will nonetheless employ uncommon but successful behaviors or strategies which enable them to find better solutions. Through the study of these individuals– subjects referred to as “positive deviants” – the PD approach suggests that innovative solutions to such challenges may be identified and refined from their outlying behavior.
Since its inception in nutrition research in the 1970s, the PD approach has been used extensively by researchers and evaluators in development. It is a strength-based approach based around five core principles: first, that communities possess the solutions and expertise to best address their own problems; second, that these communities are self-organising entities with sufficient human resources and assets to derive solutions to communal problems, third, that communities possess a ‘collective intelligence’, equally distributed through the community, which the PD approach seeks to foster and draw out; fourth, that the foundation of any PD approach rests on sustainability and the act of enabling a community to discover solutions to their own problems through the study of local “positive deviants”, and five, that behavior change is best achieved through practice and the act of “doing”.
In applying the PD approach, an investigator must first obtain an invitation from the community in question requesting their aid in addressing a problem they have collectively identified as by important Once this invitation is obtain, it is the investigator’s task to work as a facilitator in guiding community members through the four “Ds” of PD:

• “Define” the problem: in cooperation with the community, identify the problems(s) and establish a quantitative baseline to measure progress towards addressing them, while identifying key stakeholders and decision-makers.
• “Determine” the presence of PD individuals or groups: through observation and data analysis, the community works towards identifying key “positive deviants” amongst them.
• “Discover” existing uncommon practices or behaviors: a stage of community-led inquiry into the successful behavior, attitudes, beliefs and practices which define the “positive deviants”, emphasising the empowerment of the community through realisation that their problem(s) can be overcome through local solutions.
• “Design” and develop initiatives to leverage these solutions: Once successful strategies are identified, the community decides which ones to follow, and designs an implementation strategy.
• “Discern” the initiative’s effectiveness: The M&E aspect of the PD approach is participatory. Evaluation encourages the community by highlighting “social proof” – aspects of progress and reinforcing optimimism in their power for behavioral and social change

(Source: Positive Deviance Initiative.)

Brief Example
A 2008 ODI study of livelihoods, nutrition, and anti-retroviral therapy (ART) employed a PD approach to studying the relationship between HIV/AIDs and food security among Kenyan and Zambian subjects. By identifying successful positive deviants and examining their coping mechanisms, the study was able to isolate a series of promising strategies for promotion of food security among ART users (such as urban farming).
Source: “HIV, Food and Drugs: Livelihoods, Nutrition, and Anti-Retroviral Therapy (ART) in Kenya and Zambia”, Overseas Development Institute (ODI), 2008. Online

Advice

Advice for CHOOSING this approach (tips and traps)

• Be aware that PD is particularly time and labor intensive, and requires a skilled facilitator.
• The PD process occurs in a non-traditional, iterative fashion, and demands a degree of comfort in uncertainty from donors, planners, and implimeters as the PD study takes its time to scale up.

Advice for USING this approach (tips and traps)

• The PD approach is often messy, chaotic, and iterative – as such, it emphasizes process over efficiency. Investigators should be patient.
• An ideal PD facilitator is an active listener, who asks specific open-ended questions, demonstrates empathy and humility, and withholds their own expertise unless asked directly by the community. This can represent a challenge for researchers unprepared for such lack of control and power-sharing.
• As with all participatory M&E practices, be sure to keep non-literate community members in mind when designing evaluation exercises.

Resources

Guide and Example

• The Positive Deviance Initiative – Affiliated with Tufts University Friendman School of Nutrition Science and Policy, the Positive Deviance Initiative’s website offers a comprehensive collection of resources including manuals, slides, videos, both published and anecdotal case studies of PD in practice

Examples

• HIV, Food and Drugs: Livelihoods, Nutrition, and Anti-Retroviral Therapy (ART) in Kenya and Zambia – A report by the  Overseas Development Institute (ODI) in 2008,  Online
• Protecting Children from Exploitation and Trafficking Using the Positive Deviance Approach in Uganda and Indonesia – This book documents two child protection projects implemented by Save the Children using the Positive Deviance approach.

Source
Richard P., Sternin, J and Sternin, M. (2010) The Power of Positive Deviance: How Unlikely Innovators Solve the World’s Toughest Problems. (Boston: Harvard Business Press).
Sternin, J. (2002) “Positive deviance: A new paradigm for addressing today’s problems today.” Journal of Corporate Citizenship, pp.5:57-62.
Pant L.P. and Hambly Odame, H. (2009) “The Promise of Positive Deviants: Bridging Divides between Scientific Research and Local Practices in Smallholder Agriculture.” The Knowledge Management for Development Journal 5:2, pp. 138-150.
Positive Deviance. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/positive_deviance

“Participatory Rural Appraisal (PRA) recently renamed Participatory Learning for Action (PLA), is a methodological approach that is used to enable farmers to analyse their own situation and to develop a common perspective on natural resource management and agriculture at village level.
PRA is an assessment and learning process that empowers farmers to create the information base they need for participatory planning and action. Outsiders contribute facilitation skills and external information and opinions. Many different tools have been developed for use in PRA. There are four main classes: tools used in group and team dynamics; tools for sampling; options for interviews and dialogue; and options for visualisation and preparing diagrams. Most countries have had some experience with PRA and local publications are available. IIED regularly reports on new developments in its PLA notes (Pretty et al 1995).” (Bie, 1998)
Sources
Bie, S. W. (1998). Participatory learning, planning and action toward LEISA. LEISA in perspective: 15 years ILEIA, 14(2-3), 27-31. Retrieved from http://www.agriculturesnetwork.org/magazines/global/15-years-ileia/chapt…
Participatory Rural Appraisal. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/approach/PRA

Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study. A participatory approach can be taken with any impact evaluation design, and with quantitative and qualitative data. However, the type and level of stakeholder involvement will necessarily vary between different types, for example between a local level impact evaluation and an evaluation of policy changes (Gujit 2014, p.1). It is important to consider the purpose of involving stakeholders, and which stakeholders should be involved how, in order to maximise the effectiveness of the approach.
Campilan (2000) indicates that participatory evaluation is distinguished from the conventional approach in five key ways:

• Why the evaluation is being done
• How evaluation is done
• Who is doing the evaluating
• What is being evaluated
• For whom evaluation is being done.

It is often practiced in various ways, such as: self-assessment, stakeholder evaluation, internal evaluation and joint evaluation. In addtion, it can include individual story-telling, participatory social mapping, causal-linkage and trend and change diagramming, scoring, and brainstorming on program strengths and weaknesses.

Advantages of doing participatory evaluation

• Identify locally relevant evaluation questions
• Improve accuracy and relevance of reports
• Establish and explain causality
• Improve program performance
• Empower participants
• Build capacity
• Develop leaders and build teams
• Sustain organizational learning and growth

Challenges in implementing and using participatory evaluation

• Time and commitment
• Resources
• Conflicts between approaches
• Unclear purpose of participation, or a purpose that is not aligned with evaluation design
• Lack of facilitation skills
• Only focusing on participation in one aspect of the evaluation process, e.g. data collection
• Lack of cultural and contextual understanding, and the implications of these for the evalution designAs

Irene Gujit notes:

“The benefits of participation in impact evaluation are neither automatic nor guaranteed. Commissioning such approaches means committing to the implications for timing, resources and focus. Facilitation skills are essential to ensuring a good quality process, which in turn may require additional resources for building capacity.” (Gujit 2014, 18)

Example
Supporting indigenous governance in Colombia
In Colombia, ACIN, an association of indigenous people covering 13 communities, is involved in monitoring and evaluating its own multi-sectoral regional development plan. They are looking at links between productivity and environmental and cultural factors, tracking changes over time and comparing plans with results in a systematic way. This has helped communities recognise their strengths and improve their management capabilities, which, in turn, is leading to changes in power relationships. Links are being made between communities, providing the concerted voice needed in negotiations with national and provincial government, and the private sector. (Guijt and Gaventa, 1998)
Advice
Advice for CHOOSING this approach

There are a number of reasons to use this approach:

• Involving stakeholders in the process of an evaluation can lead to “better data, better understanding of the data, more appropriate recommendations, [and] better uptake of findings” (Gujit 2014, p.2)
• It is ethical to include the people to be affected by a programme or policy in the process to inform relevant decisions.
• The first step in a participatory approach is to clarify what value this approach will add, both to the evaluation and to the stakeholders who would be involved. Irene Gujit suggests three questions to ask when using this approach (2014, p.3):
  • What purpose will stakeholder participation serve in this impact evaluation?​
  • Whose participation matters, when and why?
  • When is participation feasible?

Advice for USING this approach

• It is important to pilot any method of participatory evaluation to ensure safe and open engagement with participants, and that relevant indicators are included.

• There are a number of ways to use participatory methods:
  • To collect qualitative and quantitative impact data
  • To investigate causality, for example through focus group discussions or interviews.
  • To negotiate differences and to validate key findings.
  • To score people’s appreciation of an intervention’s impact, such as a matrix ranking or spider diagram.
  • To assess impacts in relation to wider developments in the intervention area.
• You can find more information on these in the UNICEF Methodological Brief on Participatory Approaches in impact evaluation.

Using the BetterEvaluation Framework to answer: Whose participation matters, when and why?

UNICEF’s Methodological Brief on Participatory Approaches uses the BetterEvaluation Rainbow Framework to cluster a number of questions for someone seeking to use a participatory approach (Gujit 2014, pp.7-8):

Resources

Guides

• Participatory Approaches: This guide, written by Irene Guijt for UNICEF, looks at the use of participatory approaches in impact evaluation.
• Participatory Monitoring and Evaluation: Learning from Change
• Increasing Participation in Evaluating – Bruner Foundation Guide – This guide discusses how Organization Staff, Evaluators and Funders are involved in Participatory Evaluation.
• Jacobs, A., C. Barnett, et al. (2010). “Three Approaches to Monitoring: Feedback Systems, Participatory Monitoring and Evaluation and Logical Frameworks.” IDS Bulletin 41(6): 36-44.

Website

• The Collaborative, Participatory and Empowerment Evaluation Topical Interest Group (TIG) – designed to foster collaboration, share knowledge, and promote scholarship, this TIG is a division of the American Evaluation Association (AEA).

Sources
Campilan, D. (2000). Participatory Evaluation of Participatory Research. Forum on Evaluation of International Cooperation Projects: Centering on Development of Human Resources in the Field of Agriculture. Nagoya, Japan, International Potato Center. http://ir.nul.nagoya-u.ac.jp/jspui/bitstream/2237/8890/1/39-56.pdf
Chambers, R. (2009) Making the Poor Count: Using Participatory Options for Impact Evaluation in Chambers, R., Karlan, D., Ravallion, M. and Rogers, P. (Eds) Designing impact evaluations: different perspectives. New Delhi, India, International Initiative for Impact Evaluation. http://www.3ieimpact.org/admin/pdfs_papers/50.pdf
Guijt, I. and J. Gaventa (1998). Participatory Monitoring and Evaluation: Learning from Change. IDS Policy Briefing. Brighton, UK, University of Sussex. http://www.ids.ac.uk/files/dmfile/PB12.pdf
Guijt, I. (2014). Participatory Approaches, Methodological Briefs: Impact Evaluation 5, UNICEF Office of Research, Florence. Retrieved from:http://devinfolive.info/impact_evaluation/img/downloads/Participatory_Approaches_ENG.pdf
Zukoski, A. and M. Luluquisen (2002). “Participatory Evaluation: What is it? Why do it? What are the challenges?” Policy & Practice(5). http://depts.washington.edu/ccph/pdf_files/Evaluation.pdf
Participatory Evaluator. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/participatory_evaluation

Synonyms:
Outcome mapping (OM) is a methodology for planning, monitoring and evaluating development initiatives in order to bring about sustainable social change. As the name suggests, its niche is understanding outcomes; the so-called ‘missing-middle’ or ‘black box’ of results that emerge downstream from the initiative’s activities but upstream from longer-term economic, environmental, political or demographic changes.
At the planning stage, the process of outcome mapping helps a project team or program be specific about the actors it intends to target, the changes it hopes to see and the strategies appropriate to achieve these. For ongoing monitoring, OM provides a set of tools to design and gather information on the results of the change process, measured in terms of the changes in behaviour, actions or relationships that can be influenced by the team or program.
As an evaluation approach, OM unpacks an initiative’s theory of change, provides a framework to collect data on immediate, basic changes that lead to longer, more transformative change, and allows for the plausible assessment of the initiative’s contribution to results.
OM provides a set of tools that can be used stand-alone or in combination with other planning, monitoring and evaluation systems to:
identify individuals, groups or organisations with whom you will work directly to influence behavioural change;
plan and monitor behavioural change and the strategies to support those changes;
monitor internal practices of the project or program to remain effective;
create an evaluation framework to examine more precisely a particular issue.
OM is a robust methodology that can be adapted to a wide range of contexts. It enhances team and program understanding of change processes, improves the efficiency of achieving results and promotes realistic and accountable reporting.
Potential users of OM should be aware that the methodology requires skilled facilitation as well as dedicated budget and time, which could mean support from higher levels within an organisation. OM also often requires a “mind shift” of personal and organisational paradigms or theories of social change.

Steps of OM
OM involves 12 steps in three stages: intentional design, Outcome and performance monitoring and evaluation planning.
The Intentional Design stage is based on seven steps which are normally developed in a sequential order:
The vision describes the large-scale development changes that VECO hopes to encourage;
The mission spells out how VECO will contribute to the vision and is that ‘bite’ of the vision on which VECO’s programme is going to focus.
The boundary partners are those individuals, groups, or organisations with whom the programme interacts directly and with whom it anticipates opportunities for influence.
An outcome challenge statement describes the desired changes in the behaviour, relationships, activities, actions (professional practices) of the boundary partner. It is the ideal behavioural change of each type of boundary partner for it to contribute to the ultimate goals (vision) of the programme;
Progress Markers are a set of statements describing a gradual progression of changed behaviour in the boundary partner leading to the ideal outcome challenge. They are a core element in OM and the strength rests in their utility as a set of desired changes which indicate progression towards the ideal outcome challenge and articulate the complexity of the change process. They represent the information which can be gathered in order to monitor partner achievements. Therefore, progress markers are central in the monitoring process. Progress markers can be seen as indicators in the sense that they are observable and measurable but differ from the conventional indicators used in Logical Framework Approach (LFA). Progress markers can be adjusted during the implementation process, can include unintended results, do not describe a change in state and do not contain percentages or deadlines;
Strategy maps are a mix of different types of strategies used by the implementing team (VECO) to contribute to and support the achievement of the desired changes at the level of the boundary partners. OM encourages the programme identify strategies which are aimed directly at the boundary partner and those aimed at the environment in which the boundary partner operates.
Organisational Practices explain how the implementing team (VECO) is going to operate and organise itself to fulfil its mission. It is based on the idea that supporting change in boundary partners requires that the programme team itself is able to change and adapt as well, i.e., not only by being efficient and effective (operational capacities) but also by being relevant (adaptive capacities).
The monitoring stage involves four steps:
Monitoring priorities provides a process for establishing the areas of the project to be monitored.
Outcome journals are a tool for collecting data about the progress markers over time.
Strategy journals are a tool for collecting data about the activities of a project.
Performance journals are for collecting data about organisational practices.
The evaluation stage involves one step:
Evaluation plan provides a process and a tool for designing an evaluation using OM.
Uses of Outcome Mapping
Manage
OM includes two generic tools to support these decisions: monitoring plan for setting monitoring priorities (step 8) and the evaluation plan (step 12). These tools are based on the principles of utilisation-focused evaluation and both can be used for any kind of evaluation, not just those applying OM.
Define
The vision and mission steps of OM provide a useful way of succinctly describing the initiative. The particular way that OM uses these common tools, and the process it suggests for developing them, make them very effective at getting to the core of what an initiative is really about and what its core contributions are.
The outcome challenge and progress markers tools, together with the outcome journals, allows users to capture unintended changes in behaviour of crucial actors external to the programme, as the programme is running. They can also be used in a retrospective evaluation, to re-construct a process of change to bring up intended and unintended (positive and negative) outcomes.
The 7 steps of the intentional design stage of Outcome Mapping provide users with a guided process for developing a logic model based on the articulation of changes desired in direct partners, and the strategies employed by the initiative to support these. In particular, the Progress Marker tool (step 5) helps users develop a theory of change for particular actors based on concrete, observable behaviour changes.
Frame
The OM concept of boundary partners can be useful in identifying intended users of an evaluation by focusing on those who are directly associated with the initiative itself.
OM suggests a participatory approach to developing outcome challenges and progress markers together with boundary partners. This process is very effective at illuminating different perspectives of the initiative and the underlying values of different stakeholders.
Describe
Although OM doesn’t depend on any particular data collection option, it does suggest the use of journals for collecting qualitative data. Outcome journals are used to collect data about behavioural changes observed among boundary partners while the strategy journal is used to collect data about the activities completed. A third journal is also provided for collecting data about the internal performance of the initiative, in particular its learning function.
In cases where there are many boundary partners grouped together (perhaps because they play a similar role or the outcomes hoped for are similar), being able to see the outcome journals for each of them can provide a quick overview to compare across the set.
Understand causes
With the use of progress markers the process towards a specific outcome can be analysed independently from the intervention itself. If the journals are used well then ongoing monitoring will result in a record of incremental change that may or may not have been influenced by the initiative. This can then be used to reconstruct pathways of change. Likewise, a retrospective assessment based on the OM approach will generate alternative and complimentary explanations.
OM is explicit about the fact that change occurs as a result of many actors and factors. It is designed for the purpose of understanding an initiatives contribution to change in the context of other factors outside of its control and each step in the OM process builds on this idea.
Report & support use
Support use: OM provides a process and guidelines for continuous reflection among key actors involved in the initiative. By building in participation from the start, OM maximises the chances that findings will result in actual changes on the ground.
Resources
Sarah Earl Outcome Mapping: These three videos from Sarah Earl provide an introduction to the concepts of Outcome Mapping. Part 1; Part 2; Part 3
Outcome Mapping FAQs: This website from the Outcome Mapping Learning Community provides answers to 14 frequently asked questions about outcome mapping.
Outcome mapping: A method for tracking behavioural changes in development programs. This ILAC brief provides an overview of the process of outcome mapping including relevant examples.
Outcome mapping: Building learning and reflection into development programs: This book outlines a detailed step by step process for using outcome maps including a variety of worksheets and examples.
10 Years of Outcome Mapping: This webinar from the Outcome Mapping Learning Community (OMLC) presents the key findings from research conducted into the extent of Outcome Mapping use and the support required for its implementation.
Source:
Outcome Mapping. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/outcome_mapping

Outcome Harvesting collects (“harvests”) evidence of what has changed (“outcomes”) and, then, working backwards, determines whether and how an intervention has contributed to these changes.
Outcome Harvesting has proven to be especially useful in complex situations when it is not possible to define concretely most of what an intervention aims to achieve, or even, what specific actions will be taken over a multi-year period.
Contents
1 What is Outcome Harvesting?
2 Where does Outcome Harvesting come from?
3 When can Outcome Harvesting be used?
4 Who should be involved in Outcome Harvesting?
5 How is Outcome Harvesting done (overview)?
6 How is Outcome Harvesting done (six key steps)?
7 Where and by whom has Outcome Harvesting been used?
8 Strengths and limitations of Outcome Harvesting
9 Resources
1 What is Outcome Harvesting?
Outcome Harvesting is an evaluation approach in which evaluators, grant makers, and/or programme managers and staff identify, formulate, verify, analyse and interpret ‘outcomes’ in programming contexts where relations of cause and effect are not fully understood.
Outcomes are defined as changes in the “behaviour writ large” (such as actions, relationships, policies, practices) of one or more social actors influenced by an intervention. For example, a religious leader making a proclamation that is unprecedented and considered to be important; a change in the behaviour between organisations or between communities; changes in regulations, formal laws or cultural norms.
Unlike some evaluation approaches, Outcome Harvesting does not measure progress towards predetermined objectives or outcomes, but rather, collects evidence of what has changed and, then, working backwards, determines whether and how an intervention contributed to these changes. The outcome(s) can be positive or negative, intended or unintended, direct or indirect, but the connection between the intervention and the outcomes should be plausible.
Information is collected or “harvested” using a range of methods to yield evidence-based answers to useful, actionable questions (“harvesting questions”).
Outcome Harvesting can be used for monitoring as well as for evaluation (including developmental, formative or summative evaluation) of interventions or organisations.
2 Where does Outcome Harvesting come from?
The approach is inspired by Outcome Mapping and informed by Utilization-Focused Evaluation. It was developed by Ricardo Wilson-Grau and colleagues* who, as co-evaluators, applied it in different circumstances and further refined it over many years of evaluation practice.
*Barbara Klugman, Claudia Fontes, David Wilson-Sánchez, Fe Briones Garcia,Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, NataliaOrtiz, and Wolfgang Richert.
3 When can Outcome Harvesting be used?
Outcome Harvesting is particularly useful when outcomes, and even, inputs, activities and outputs, are not sufficiently specific or measurable at the time of planning an intervention. Thus, Outcome Harvesting is well-suited for evaluation in dynamic, uncertain (i.e., complex) situations.
Outcome Harvesting is recommended when:
The focus is primarily on outcomes rather than activities. Outcome Harvesting is designed for situations where decision-makers (as “harvest users”) are most interested in learning about what was achieved and how. In other words, there is an emphasis on effectiveness rather than efficiency or performance. The approach is also a good fit when the aim is to understand the process of change and how each outcome contributes to this change.
The programming context is complex. Outcome Harvesting is suitable for programming contexts where relations of cause and effect are not fully understood. Conventional monitoring and evaluation aimed at determining results compares planned outcomes with what is actually achieved, and planned activities with what was actually done. In complex environments, however, objectives and the paths to achieve them are largely unpredictable and predefined objectives and theories of change must be modified over time to respond to changes in the context. Outcome Harvesting is particularly appropriate in these more dynamic and uncertain environments in which unintended outcomes dominate, including negative ones. Consequently, Outcome Harvesting is particularly suitable to assess social change interventions or innovation and development work.
The purpose is evaluation. Outcome Harvesting can serve to track the changes in behaviour of social actors influenced by an intervention. However, it is designed to go beyond this and support learning about those achievements. Thus, Outcome Harvesting is particularly useful for on-going developmental, mid-term formative, and end-of-term summative evaluations. It can be used by itself or in combination with other approaches.
The timing of “the harvest” depends on how essential the harvest findings are to ensure the intervention is heading in the right direction. If the certainty is relatively great that doing A will result in B, the harvest can be timed to coincide with when the results are expected. Conversely, if much uncertainty exists about the results that the intervention will achieve, the harvest should be scheduled as soon as possible to determine the outcomes that are actually being achieved.
4 Who should be involved in Outcome Harvesting?
A highly participatory process is a necessity for a successful Outcome Harvesting process and product.
Depending on the situation, either an external or internal person (“the harvester”) is designated to lead the Outcome Harvesting process. Harvesters facilitate and support appropriate participation and ensure that the data are credible, the criteria and standards to analyse the evidence are rigorous, and, the methods of synthesis and interpretation are solid.
“Harvest users” are individuals or organisations requiring the findings to make decisions or take action. They should be engaged throughout the process. These users must be involved in making decisions about the design and re-design of the approach as both the process and the outcomes unfold. Also, the principal uses for the harvest may shift as findings are generated which, in turn, may require re-design decisions.
The harvester also needs to engage informants who are knowledgeable about what the intervention has achieved and how, and who are willing to share, for the record, what they know. Field staff who are positioned “closest to the action” tend to be the best informants.
5 How is Outcome Harvesting done (overview)?
The harvester obtains information from the individual(s) or organisation(s) implementing the intervention whose actions aim to influence outcome(s) in order to identify actual changes in the social actors and how the intervention influenced them.
“Outcome descriptions” are drafted to a level of detail considered useful for answering the harvesting questions. The descriptions may be as brief as a single sentence or as detailed as a page or more of text. They may include explanations of context, collaboration with or contributions from others to the specific outcome; diverse perspectives on the outcome; and, the importance of the outcome. The harvested information undergoes several rounds of revisions to make it specific and comprehensive enough to be verifiable.
Subsequently, the information about select outcomes is validated by comparing it with information collected from other knowledgeable and authoritative, but independent, sources.
Finally, the validated outcome descriptions are analysed and interpreted –either as an individual outcome or as a group of outcomes– for their significance in achieving a mission, goal or strategy and are used to answer the harvesting questions.

 
The Outcome Harvesting approach should be customised to the specific needs of the primary intended users/uses. The six “steps”, discussed below, are to be taken more as guiding principles rather than rigid formulae to follow. Nonetheless, the rigorous application of all six principles is necessary for a sound and credible outcome harvest.
Design the Outcome Harvest: The first step is to identify the primary intended users of the harvest and their principal intended uses for the harvest process and findings. Based on those, the harvest users and harvesters agree what needs to be known and write useful, actionable questions to guide the harvest (harvesting questions). For example, a useful question may be: What has been the collective effect of grantees on making the national governance regime more democratic and what does it mean for the portfolio´s strategy? Then, they agree what information is to be collected and from whom in order to answer the questions. At a minimum, this involves obtaining information about the changes in social actors and how the intervention influenced them.
Review documentation and draft outcome descriptions: From reports, previous evaluations, press releases and other documentation, harvesters identify potential outcomes (i.e., changes in individuals, groups, communities, organisations or institutions) and what the intervention did to contribute to them. For example, the change can be a president’s public commitment to being transparent (behaviour); two government agencies collaborating rather than competing (relationships); a minister firing a corrupt civil servant (action); the legislature passing a new anti-corruption law (policy); or a third successive government publishing its procurement records (practice). The influence of the change agent can range from inspiring and encouraging, facilitating and supporting, to persuading or pressuring the social actor to change.
Engage with informants in formulating outcome descriptions: Harvesters engage directly with informants to review the outcome descriptions based on the document review, and to identify and formulate additional outcomes. Informants will often consult with others inside or outside their organisation knowledgeable about outcomes to which they have contributed.
Substantiate: Harvest users and harvesters review the final outcomes and select those to be verified in order to increase the accuracy and credibility of the findings. The harvesters obtain the views of one or more individuals who are independent of the intervention (third party) but knowledgeable about one or more of the outcomes and the change agent’s contribution.
Analyse and interpret: Harvesters, classify all outcomes, often in consultation with the informants. The classifications are usually derived from the useful questions; they may also be related to the objectives and strategies of either the implementer of the intervention or other stakeholders, such as donors. For large, multidimensional harvests, a database is required to store and analyse the outcome descriptions. Harvesters interpret the information and provide evidence-based answers to the harvesting questions.
Support use of findings: Harvesters propose issues for discussion to harvest users grounded in the evidence-based answers to the harvesting questions. They facilitate discussions with users, which may include how they can make use of the findings.
These steps are not necessarily distinct; they may overlap and can be iterative; feedback can spark decisions to re-design a next step or return to or modify an earlier step. Typically, feedback from step 4 (substantiation) and step 5 (analysis and interpretation) does not influence the earlier steps; feedback from step 6 (support of use) only affects step 5 (analysis and interpretation). Nonetheless, feedback from all the steps can, of course, influence decisions about future harvesting for either monitoring or evaluation purposes.
It should be noted that “Outcome Harvesting” is done as often as necessary to understand what the change agent is achieving. Depending on the time period covered and the number of outcomes involved, the approach can require a substantial time commitment from informants. To reduce the burden of time on informants, outcomes may be harvested monthly, quarterly, biannually, or annually. Findings may be substantiated, analysed or interpreted less frequently.
The above steps are explained in more detail in Outcome Harvesting by Wilson-Grau R and Britt H (2013).
7 Where and by whom has Outcome Harvesting been used?
Nongovernmental organisations, community-based organisations, networks, government agencies, funding agencies, universities and research institutes have used Outcome Harvesting in over 143 countries. In 2013, the Evaluation Office of the United Nations Development Programme (UNDP) selected Outcome Harvesting as one of eleven promising innovations in monitoring and evaluation practice. After ten teams of the World Bank Institute piloted a customised version of the Outcome Harvesting, the approach and as customised for the Institute were listed amongst its resources for internal monitoring and evaluation. The US Agency for International Development (USAID) chose Outcome Harvesting as one of five approaches particularly suited for monitoring and evaluation in dynamic, uncertain situations.
Thus, Outcome Harvesting has proven to be a useful approach in a variety of locales and for diverse interventions. Some examples are:
Summative evaluation of the Oxfam Novib’s Global Programme 2005-2008, a €22 million investment to support 38 grantees working on sustainable livelihoods and social and political participation. Documentation of more than 300 outcomes from 111 countries.
Retrospective ‘Outcome Harvesting’: Generating robust insights. Report on the evaluation experience of the BioNET global network to promote taxonomy for development. Identification and documentation of 200 emergent outcomes of BioNET.
World Bank Institute Cases in Outcome Harvesting. Ten pilot experiences identify new learning from multi-stakeholder projects to improve results. There is an average of 30 outcomes per pilot.
8 Strengths and limitations of Outcome Harvesting
The following are recognized strengths of Outcome Harvesting:
Overcomes the common failure to search for unintended outcomes of interventions;
Generates verifiable outcomes;
Uses a common-sense, accessible approach that engages informants quite easily;
Employs various data collection methods such as interviews and surveys (face-to-face, by telephone, by e-mail), workshops and document review;
Answers actionable questions with concrete evidence.
Limitations and challenges include:
Skill and time, as well as timeliness, are required to identify and formulate high-quality outcome descriptions;
Only those outcomes that informants are aware of, are captured;
Participation of those who influenced the outcomes is crucial;
Starting with the outcomes and working backwards represents a new way of thinking about change for some participants.
9 Resources
What is Outcome Harvesting? Ricardo Wilson-Grau (Sept 2014) explains the essence of the Outcome Harvesting approach in this video (< 3 minutes). Outcome Harvesting. This brief, written by Ricardo Wilson-Grau and Heather Britt, introduces the key concepts and approach used by Outcome Harvesting (published by the Ford Foundation in May 2012; revised in Nov 2013). Source: Outcome Harvesting. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/plan/approach/outcome_harvesting

Synonyms:
MSC
The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why.
The are three basic steps in using MSC:
Deciding the types of stories that should be collected (stories about what – for example, about practice change or health outcomes or empowerment)
Collecting the stories and determining which stories are the most significant
Sharing the stories and discussion of values with stakeholders and contributors so that learning happens about what is valued.
MSC is not just about collecting and reporting stories but about having processes to learn from these stories – in particular, to learn about the similarities and differences in what different groups and individuals value.
It provides some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders. By itself it is not sufficient for impact evaluation as it does not provide information about the usual experience but about the extremes.
If you imagine a normal distribution of outcomes for individuals then the stories often come from the extremity of positive change. It can be useful to explicitly add a process to generate and collect stories from the extremity of little or negative change.
MSC can be very helpful in explaining HOW change comes about (processes and causal mechanisms) and WHEN (in what situations and contexts). It can therefore be useful to support the development of programme theory (theory of change, logic models).
Example
“In 1994 Rick Davies was faced with the job of assessing the impact of an aid project on 16,500 people in the Rajshahi zone of western of Bangladesh (6). The idea of getting everyone to agree on a set of indicators was quickly dismissed as there was just too much diversity and conflicting views. Instead Rick devised an evaluation method which relied on people retelling their stories of significant change they had witnessed as a result of the project. Furthermore, the storytellers explained why they thought their story was significant.
If Rick had left it there the project would have had a nice collection of stories but the key stakeholders’ appreciation for the impact the project would have been minimal. Rick needed to engage the stakeholders, primarily the region’s decision-makers and the ultimate project funders, in a process that would help them see (and maybe even feel) the change. His solution was to get groups of people at different levels of the project’s hierarchy to select the stories which they thought was most significant and explain why they made that selection.
Each of the 4 project offices collected a number of stories and were asked to submit one story in each of the four areas of interest to the head office in Dhaka. The Dhaka head office staff then selected one story from the 16 submitted. The selected stories and reasons for selection were communicated back to the level below and the original storytellers. Over time the stakeholders began to understand the impact they were having and the project’s beneficiaries began to understand what the stakeholders believed was important. People were learning from each other. The approach, called Most Significant Change, systematically developed an intuitive understanding of the project’s impact that could be communicated in conjunction with the hard facts.
Rick’s method was highly successful: participation in the project increased; the assumptions and world views surfaced, helping in one case resolve an intra-family conflict over contraceptive use; the stories were extensively used in publications, educational material and videos; and, the positive changes where identified and reinforced.”
Example taken from Anecdote.com: Evaluating the soft stuff
Advice
Advice for CHOOSING this approach(tips and traps)
MSC is particularly useful when you need different stakeholders to understand the different values that other stakeholders have in terms of “what success looks like” – criteria and standards for outcomes, processes and the distribution of costs and benefits.
MSC works best in combination with other options for gathering, analysing and reporting data. It doesn’t provide comprehensive information about the impacts produced by an intervention.
Advice for USING this approach(tips and traps)
Ensure the stories are not highjacked for other purposes such as for promotional material. Data can only be used for the original stated purpose, which in this case is evaluation unless other uses have been negotiated and agreed to at the time.
MSC is not a quick option. It takes time and an appropriate project infrastructure to generate understanding and value clarification (identifying what people think is important). The full MSC process involves analysis of stories and sharing with both contributors and stakeholders, which requires a programme with several structures in it (for example, local, regional and national project structures) and it needs to be repeated through several cycles.
There is scope to be innovative in this option. Your project may not have a hierarchical structure so there may be other ways of forming groups around which the stories can be discussed and the values identified.
It can be challenging to get engagement of the different groups involved in the process and to maintain their interest. Don’t have too many cycles of review.
Other Skills Necessary: Good facilitation skills are important along with the ability to identify priorities.
Resources
Guides
The ‘Most Significant Change’ (MSC) Technique – A Guide to Its Use – Guidelines from the Rick Davies and Jessica Dart who developed the concept of Most Significant Change

Strategy Development: Most Significant Change (MSC) – This guide to the Most Significant Change approach was written by the Overseas Development Institute (ODI). It provides an overview, a detailed description of the process, and an example of the technique in action.
Participatory Video and the Most Significant Change. A guide for facilitators – The toolkit is designed to support evaluators in planning and carrying out evaluation using Participatory Video with the Most Significant Change approach.

Example

This overview from Capacity.org outlines the process and findings of an NGO self-evaluation process which used Most Significant Change to evaluate the organisational development services offered by the group.
Source:
Most Significant Change. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/approach/most_significant_change