Managing an evaluation involves agreeing on how decisions will be made for each cluster of the evaluation (from framing an evaluation to reporting and supporting use) and ensuring they are implemented well.

As you work through the process of planning and implementing the evaluation, you may need to revisit and revise the choices you have made.

Tasks

The following tasks are involved in managing an evaluation, an evaluation system or ongoing evaluative activity:

1. Understand and engage stakeholders

Begin by clarifying who has an interest in the evaluation in addition to the primary intended users. An analysis of the interests of different stakeholders helps in thinking about how you will keep stakeholders engaged during the evaluation.

2. Establish decision making processes

Specify how decisions will be made about the evaluation. Who will provide advice, who will make recommendations, and who will make the actual decisions?

3. Decide who will conduct the evaluation

Who will actually undertake the evaluation? Will it be a group internal to the organization that is implementing a program, an external evaluator, or some combination of the two? Will the community and/or intended beneficiaries be involved in conducting the evaluation?

4. Determine and secure resources

Identify what resources (time, money, expertise) will be needed and available for the evaluation. Consider both internal resources  (e.g. staff time) and external resources (e.g. participants’ time to attend meetings to provide feedback).

5. Define ethical and quality evaluation standards

What will be considered appropriate quality and ethical standards for the evaluation and what will need to be done to ensure these standards are achieved? These decisions will be influenced by the purpose and timing of the evaluation, the level of resourcing available and the characteristics of the intervention being evaluated.

6. Document management processes and agreements

It is usually necessary to develop formal documents about an evaluation, including a brief, a plan, and a Terms of Reference. Additional documents will be needed if an evaluation is externally contracted, such as an Expression Of Interest, a Request For Proposal, a Request For Tender, and a contract.

7. Develop an evaluation plan, framework or policy

An evaluation plan sets out how an individual evaluation will be undertaken.  An evaluation framework guides a series of evaluations.

8. Review evaluation (do meta-evaluation)

Review the evaluation process, findings and conclusions drawn, when appropriate it is valuable to engage stakeholders in this task.

9. Develop evaluation capacity

How can your organization or group build on, or develop, human capital (knowledge and skills), organizational capital (technical infrastructure and processes) and social capital (supportive networks) for effectively managing, undertaking and using evaluations?

Stakeholders are people with a stake in the evaluation, including primary intended users and others.

Understanding and taking into account the priorities and concerns of different stakeholders informs evaluation planning, communication strategies during and after the evaluation and supports the utilisation of evaluation findings.

The primary intended users – people who will be making decisions on the basis of the evaluation findings – are a key group of stakeholders.(There’s a separate task of identifying primary intended users in the FRAME cluster of tasks).

Other stakeholders include people who will be affected by decisions made during or after the evaluation (program staff, program participants and beneficiaries) and secondary users of the evaluation findings. Evaluation findings are often of interest to policy makers and advocates for or against a particular course of action.

Different stakeholders can be engaged for different purposes and at different phases of evaluation planning and implementation. It may not be feasible or appropriate to engage all potential stakeholders.

Involving stakeholders during evaluation planning and implementation can add value by:

• providing perspectives on what will be considered a credible, high quality and useful evaluation
• contributing to the program logic and framing of key evaluation questions
• facilitating quality data collection
• helping to make sense of the data that has been collected
• increasing the utilization of the evaluation’s findings by building knowledge about and support for the evaluation.

Engaging stakeholders is also important for managing risks especially when evaluating a contentious program or policy in which key stakeholders are known to have opposing views. It is important to understand different perspectives on what will be considered credible evidence of outcomes and impacts.

Options

Understand stakeholders

• Community Scoping: developing a more in-depth understanding of a community of interest by providing information about its social diversity, history, existing networks, and overall socio-economic characteristics.
• Stakeholder mapping and analysis: identifying different stakeholders’ level of interest and influence.

Engage stakeholders

• Community fairs: organising a community event with the aim of providing information about the project and raising the awareness of relevant issues.
• Fishbowl technique:  managing group discussion about relevant issues.
• Formal meeting processes: guidance on processes for running formal meetings.
• Informal meeting processes: a conversation between an evaluator and a key stakeholder that is not conducted in a formal way but is still seeking the same outcomes.

Evaluations can be conducted by a range of different actors including: external contractors; internal staff; those involved in delivering services; by peers; by the community; and by a combined group. Therefore it is important to make decisions about who is best to conduct the evaluation.

Consider the relative importance of different types of expertise. Relevant expertise may include skills and knowledge in evaluation, in the specific domain (eg education) or program (e.g. delivering health services), or the local culture and context.

Consider the balance of distance and involvement that will be most suitable and that will support use of the evaluation findings. An external, unaligned evaluator may be viewed as more (or less) credible by different stakeholders. Involving staff and communities may be important for supporting cultural change, knowledge building and supporting the utilization of the evaluation findings.

Different management tasks arise depending on who is involved in which evaluative activities. For example, when using an external evaluator you will need to develop a process for selecting and managing them. If internal staff and/or intended beneficiaries are involved there may be a need to ensure processes are well documented and that relevant training in specific evaluation options is conducted to ensure that quality and ethical standards are maintained.

Decisions about who will conduct an evaluation, or components of an evaluation, will also be informed by timelines, resources, and the purpose of the evaluation.

Options

• Community: conducting an evaluation by using the broader community or groups of intended beneficiaries.
• Expert Review: conducting an evaluation by using someone with specific content knowledge or expert judgment and professional expertise.
• External Consultant: contracting an external consultant to conduct the evaluation.
• Hybrid – Internal and External Evaluation: a combination of internal staff and an external (usually expert) opinion to jointly conduct an evaluation.
• Internal Staff:conducting an evaluation using staff from the implementing agency.
• Learning Alliances:  bringing together different groups to conduct the evaluation
• Peer Review:conducting an evaluation using Individuals/organizations who are working on similar projects.

Approaches

• Horizontal Evaluation: conducting an evaluation through a a structured approach to peer learning.
• Positive deviance:involves intended evaluation users identifying ‘outliers’ – those with exceptionally good outcomes – and understanding how they have achieved these.
• Participatory evaluation:involving key stakeholders in the evaluation process.

Resources

Guides

• NSW Government Evaluation Toolkit:Part 3 of this web-based toolkit, developed by BetterEvaluation for the NSW Government, focuses on choosing the right consultant for an evaluation.  It provides detailed advice about what to look for in a consultant and how to go about choosing the right one for the job.

Key Considerations for Managing Evaluations: Part 4 of this guide from Pact South Africa focuses on selecting an evaluation team. It includes an outline of the skills required by different team members, what to look for when forming a selection committee and how to develop key selection criteria that ensure transparency and consistency.

The purpose and scope of the evaluation needs to be considered when determining the budget.  The amount of resources available may influence the level of an evaluation’s rigor or the certainty of its findings. The importance of the program, existing knowledge about the program from previous evaluations and the decisions to which the evaluation will contribute are important factors to consider.

A program that has been thoroughly tested in a context similar to the current implementation setting may require fewer resources to satisfy information needs. A higher proportion of funds may be warranted for:

• Evaluations that will contribute to important decisions, such as whether to roll out a program on a large scale
• Evaluations that require highly defensible findings or will come under scientific scrutiny
• Programs that have not been evaluated before

Very often the available resources (time, money and expertise) will restrict the scope of the evaluation (the number of questions, size of the sample, data collection and analysis options) or influence the choice of evaluation designs. Some organizations have a policy of setting aside a certain percentage of the total program budget for evaluation. Organizations often use a “rule of thumb” to specify considerations in making a budget estimate. Common budget estimates range between 5 – 20% of program costs.

When commissioning an evaluation it is wise to start the budgeting process by consulting with the budget, procurement and/or human resource offices within the organization in order to verify and understand budget process, rules, and stipulations. Engage project staff, stakeholders, and M&E staff or professionals to ensure that the budget is comprehensive and accurate.

The process of developing an evaluation budget may be an excellent opportunity to encourage stakeholders to agree on the value of the evaluation and the amount and type of resources necessary to support it. Sometimes after intended users are engaged and the evaluation purpose and questions decided there is scope to add additional resources in order to undertake the type of evaluation that is required.

Budgets are just as critical for planning an internal evaluation as an external one.  Although an internal evaluation draws primarily from resources within the organization, getting agreement on available resources will ensure the evaluation runs much more smoothly.  For example, staff may be more flexible than consultants, but developing an accurate calculation of staff time costs early in the process helps to enlist their commitment.

Options

Determine resources needed

• Evaluation budget matrix: creating a budget using an evaluation budget matrix.
• Evaluation costing: calculating the costs of an evaluation including time, money and expertise.
• Resources stocktake: stocktaking resources for an evaluation which can include people’s time and expertise, equipment and funding.

Secure resources needed

• Designated staff time for evaluation:At the proposal or planning stage, ensuring that staff have time to conduct, participate in or reflect on the results of evaluations as part of organizational learning.
• Grant funding for evaluation:Writing evaluation in to grants and / or writing a separate grant to request funding for an evaluation.
• Work with universities to staff the evaluation:collaborating with a local university, research institute, or think tank to conduct your evaluation.
• Institutionalised budget allocation: uses the rule of thumb approach to estimate the percentage of project funds to spend on evaluation.
• Leveraging partnerships:Many projects are undertaken by a consortium of organizations working together.  Consider approaching your implementing partners to pool resources and carry out the evaluation jointly.
• Strategies to reduce costs: reducing the costs is an option to consider if evaluation costs outweigh the predicted benefits.

For any evaluation there needs to be clarity about what will be considered a quality and ethical evaluation. In some organisations there is agreement about using particular evaluation standards and/or ethical guidelines to guide the evaluation and to evaluate it.

Many organizations have guidelines which address issues of quality and ethics together.  For example, the UNDP (United Nations Development Programme) Norms for Evaluation state that evaluation in UNDP should be:

• Independent — Management must not impose restrictions on the scope, content, comments and recommendations of evaluation reports. Evaluators must be free of conflict of interest.
• Intentional — The rationale for an evaluation and the decisions to be based on it should be clear from the outset.
• Transparent — Meaningful consultation with stakeholders is essential for the credibility and utility of the evaluation.
• Ethical — Evaluation should not reflect personal or sectoral interests. Evaluators must have professional integrity, respect the rights of institutions and individuals to provide information in confidence, and be sensitive to the beliefs and customs of local social and cultural environments.
• Impartial — Removing bias and maximizing objectivity are critical for the credibility of the evaluation and its contribution to knowledge.
• Of high quality — All evaluations should meet minimum quality standards defined by the Evaluation Office
• Timely — Evaluations must be designed and completed in a timely fashion so as to ensure the usefulness of the findings and recommendations
• Used — Evaluation is a management discipline that seeks to provide information to be used for evidence-based decision making. To enhance the usefulness of the findings and recommendations, key stakeholders should be engaged in various ways in the conduct of the evaluation.

Different evaluation associations have articulated what is expected of members in terms of conducting ethical and quality evaluations, for example, the American Evaluation Association’s five guiding principles are:

• Systematic Inquiry: Evaluators conduct systematic, data-based inquiries about whatever is being evaluated.
• Competence: Evaluators provide competent performance to stakeholders.
• Integrity/Honesty: Evaluators ensure the honesty and integrity of the entire evaluation process.
• Respect for People: Evaluators respect the security, dignity and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact.
• Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare.

Options:

• Cultural Competency:  ensuring the influence of culture on human behaviour is taken into consideration during the evaluation.
• Ethical guidelines:Institutional or organizational rules or norms that guide evaluation practice, especially regarding vulnerable populations.
• Evaluation standards: Core national or internationally agreed best practice for conducting evaluation.
• Institutional review board: A committee set up by an organization or institution to monitor the ethical and technical research and evaluation conducted by its members.

Sources

UNDP 2011, The Evaluation Policy of UNDP, Executive Board Document viahttp://web.undp.org/evaluation/  .

American Evaluation Association Guiding Principles via http://www.eval.org

It is important to document decisions about the management of evaluative activities, including any processes for monitoring compliance with ethical and quality standards during the evaluation.

These documents will also ensure that different stakeholders, whether funders, partner organisations, communities or expert advisors are clear about what is being done, how and when, and their responsibilities and accountabilities for the evaluation.

Different organisations have different forms of documents and different labels for the document that describes what is to be done – the purpose, Key Evaluation Questions and timeline.

Sometimes this document is referred to as Terms of Reference (ToR), Scope of Work (SOW), Statement of Work (SOW), Request for Proposal (RFP), Request for Quotation (RFQ), Invitation To Tender (ITT) or the evaluation brief.

This document can be used for any type of evaluation (internal, external, self-evaluation) but they are particularly useful as part of the process of engaging an external evaluator.

Other types of documents might be developed to formalise the relationships between different organisations working together on the evaluation.  These could include a Memorandum of Understanding or a Contractual Agreement.

Options

Document what is needed in an evaluation:

• Expression of Interest (EoI):An EoI is a way for an organisation to publish its intention to appoint an evaluation team to conduct an evaluation of a specific project or program.
• Request for Proposal (RFP):  RFPs (also known as Request for Application or Request for Quotation) are formal request for evaluators to prepare a response to a planned evaluation and are generally used to select the final evaluator for the evaluation.
• Scope of Work (SoW):a plan for conducting an evaluation which outlines the work that is to be performed by the evaluation team.
• Terms of Reference (ToR): sets out the purpose(s) of the evaluation and its key evaluation questions, as well as a timeline (and possibly the available budget and existing data resources); for external evaluations this can include contractual arrangements.  In some cases this includes an evaluation plan setting out the methodology.

Document how different organisations will work together:

• Contractual Agreement: creating formal contracts to engage external evaluators.
• Memorandum of Understanding (MoU): high level agreement between two or more organisations committing to work together

Source:
Document Management Processes and Agreements. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/manage/management_processes

An evaluation plan (for a particular evaluation) usually specifies: what will be evaluated; the purpose and criteria for the evaluation; the key evaluation questions; and how data will be collected, analyzed, synthesized and reported. It may include a program theory/logic model.

An evaluation framework (sometimes called a Monitoring and Evaluation framework, or more recently a Monitoring, Evaluation and Learning framework) provides an overall framework for evaluations across different programs or different evaluations of a single program (e.g. process evaluation; impact evaluation). This can include guidance on data sources and management processes, as well as an overall program theory/logic model.

However sometimes the term ‘evaluation framework’ is used to refer to a plan for a single evaluation or to an organisational policy.

Options

• Aide memoire:A short document that summarizes key findings and recommendations.
• Evaluation framework: provides an overall framework  across different evaluations, such as individual projects within a program, or across an organisation
• Evaluation plan: set out the proposed details of an evaluation including what, how and when an evaluation will take place.
• Evaluation work plan:involves the development of clear timeframes, deliverables and milestones.​
• Inception report: set out the evaluation’s conceptual framework, key questions and methodology, and timeframe.

Source:
Develop planning documents for the evaluation. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/manage/develop_framework

It is important to document decisions about the management of evaluative activities, including any processes for monitoring compliance with ethical and quality standards during the evaluation.

These documents will also ensure that different stakeholders, whether funders, partner organisations, communities or expert advisors are clear about what is being done, how and when, and their responsibilities and accountabilities for the evaluation.

Different organisations have different forms of documents and different labels for the document that describes what is to be done – the purpose, Key Evaluation Questions and timeline.

Sometimes this document is referred to as Terms of Reference (ToR), Scope of Work (SOW), Statement of Work (SOW), Request for Proposal (RFP), Request for Quotation (RFQ), Invitation To Tender (ITT) or the evaluation brief.

This document can be used for any type of evaluation (internal, external, self-evaluation) but they are particularly useful as part of the process of engaging an external evaluator.

Other types of documents might be developed to formalise the relationships between different organisations working together on the evaluation.  These could include a Memorandum of Understanding or a Contractual Agreement.

Options

Document what is needed in an evaluation:

• Expression of Interest (EoI):An EoI is a way for an organisation to publish its intention to appoint an evaluation team to conduct an evaluation of a specific project or program.
• Request for Proposal (RFP):  RFPs (also known as Request for Application or Request for Quotation) are formal request for evaluators to prepare a response to a planned evaluation and are generally used to select the final evaluator for the evaluation.
• Scope of Work (SoW):a plan for conducting an evaluation which outlines the work that is to be performed by the evaluation team.
• Terms of Reference (ToR): sets out the purpose(s) of the evaluation and its key evaluation questions, as well as a timeline (and possibly the available budget and existing data resources); for external evaluations this can include contractual arrangements.  In some cases this includes an evaluation plan setting out the methodology.

Document how different organisations will work together:

• Contractual Agreement: creating formal contracts to engage external evaluators.
• Memorandum of Understanding (MoU): high level agreement between two or more organisations committing to work together

Source:
Document Management Processes and Agreements. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/manage/management_processes

Reviewing an evaluation (also known as meta-evaluation) can be done before an evaluation (reviewing the plan) or afterwards. Prior to finalising the evaluation it is useful to review the reporting of the evaluation process, findings and conclusions in order to establish the validity of the findings and to ensure that the key messages from the evaluation are clear and consistent with the findings.

Many evaluation systems require evaluation reports to be formally reviewed as part of quality control before they are publicly released.

Reviewing the evaluation will also help to identify how key messages may be interpreted, concerns about the methodology that need to be discussed, and possible ways that the findings will be used. Being mindful of how the evaluation findings could be received helps in presenting the findings in a way that is likely to support use.

In particular, evaluations which find that a program is ineffective, inefficient, or has harmful unintended impacts need to be accurately and carefully reported.

Involving the primary intended users and other key stakeholders in a review of the evaluation also supports the use of the evaluation findings by building the ‘personal factor’ – the involvement of people who care about the evaluation and how the findings will be used.

Options

• Beneficiary exchange: seeking feedback from the beneficiaries by discussing the findings with them.
• Expert review for meta-evaluation:reviewing the evaluation by using subject experts, either individually or as a panel.
• Group critical reflection:facilitating a group stakeholder feedback session.
• Individual critical reflection:asking particular individual stakeholders for their independent feedback.
• Peer review for meta-evaluation:reviewing the evaluation by using peers from within the organisation or outside of the organisation.

Source:
Review Evaluation (do meta-evaluation). (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/manage/review_evaluation

It is important to develop and build on human capital (knowledge and skills), organizational capital (technical infrastructure and processes) and social capital (supportive networks) for effectively managing, undertaking and using evaluations. Evaluation capacity includes developing an evaluation culture of valuing evidence, valuing questioning, and valuing evaluative thinking. Some people also refer to evaluation capability – the ability to actually use capacity.

Options

• Community of practice:A community of practice allows a group of people with a common interest or concern to share and learn through a series of interactions, thus reflecting the social nature of human learning.
• Conferences:Attendance at professional conferences to understand how other evaluators frame and discuss their findings is a key component of building evaluation capacity.
• Coaching:Coaching involves supporting an individual during training or development in order for them to reach a specific personal or professional goal.
• Evaluation policy:An evaluation policy outlines the definition, concept, role and use of evaluation within an organisation.
• Evaluation Library:
• Evaluation societies and associations:
• Evaluator competencies:the skills, abilities, knowledge, experience, and/or qualifications that an evaluator is expected to have depending on their role in the evaluation process.
• Learning circle:A Learning Circle allows a group of individuals to meet and explore an issue and learn from each other in the process.
• Mentoring:supporting a colleague by sharing professional and personal experiences in order to support their development and growth.
• Organisational policies and procedures:Policies and procedures which lay out an organisation’s decision making processes.
• Peer Coaching:Peer coaching involves two or more colleagues working closely together to improve knowledge and skills and create solutions to workplace problems.
• Peer review for meta-evaluation:reviewing the evaluation by using peers from within the organisation or outside of the organisation.
• Reflective practice:Reflective practice involves an individual reflecting on their work allowing them to learn from their own experiences and insights and engage in a practice of continual learning.
• Supervised practice in teams:
• Training and formal education: developing people’s knowledge and skills in conducting and/or managing an evaluation.

Source:
Develop Evaluation Capacity. (n.d.). Retrieved January 26, 2017, from http://betterevaluation.org/en/plan/manage_evaluation/evaluation_capacity