Analysing data to summarise it and look for patterns is an important part of every evaluation. The options for doing this have been grouped into two categories – quantitative data (number) and qualitative data (text, images).

Options

Numeric analysis

Analysing numeric data such as cost, frequency, physical characteristics.

• Correlation: a statistical measure ranging from +1.0 to -1.0 that indicates how strongly two or more variables are related. A positive correlation (+1.0 to 0) indicates that two variables will either increase or decrease together, while a negative correlation (0 to -1.0) indicates that as one variable increases, the other will decrease.
• Crosstabulations: using contingency tables of two or more dimensions to indicate the relationship between nominal (categorical) variables. In a simple crosstabulation, one variable occupies the horizontal axis and another the vertical. The frequencies of each are added in the intersecting squares and displayed as percentages of the whole, illustrating relationships in the data.
• Data mining: computer-driven automated techniques that run through large amounts of text or data to find new patterns and information.
• Exploratory Techniques:taking a ‘first look’ at a dataset by summarising its main characteristics, often by using visual methods.
• Frequency tables: a visual way of summarizing nominal and ordinal data by displaying the count of observations (times a value of a variable occurred) in a table.
• Measures of central tendency:a summary measure that attempts to describe a whole set of data with a single value that represents the middle or centre of its distribution. The mean (the average value), median (the middle value) and mode (the most frequent value) are all measures of central tendency. Each measure is useful for different conditions.
• Measures of dispersion:a summary measure that provides information about how much variation there is in the data, including the range, inter-quartile range and the standard deviation.
• Multivariate descriptive: providing simple summaries of (large amounts of) information (or data) with two or more related variables.

• Multiple regression
• Factor analysis
• Cluster analysis
• Structural equation modelling

• Non-Parametric inferential statistics: methods for inferring conclusions about a population from a sample’s data that are flexible and do not follow a normal distribution (ie, the distribution does not parallel a bell curve), including ranking: the chi-square test, binomial test and Spearman’s rank correlation coefficient.
• Parametric inferential statistics: methods for inferring conclusions about a population from a sample’s data that follows certain parameters: the data will be normal (ie, the distribution parallels the bell curve); numbers can be added, subtracted, multiplied and divided; variances are equal when comparing two or more groups; and the sample should be large and randomly selected.
• Summary statistics: providing a quick summary of data which is particularly useful for comparing one project to another, before and after.
• Time series analysis: observing well-defined data items obtained through repeated measurements over time.

Textual analysis

Analysing words, either spoken or written, including questionnaire responses, interviews, and documents.

• Content analysis: reducing large amounts of unstructured textual content into manageable data relevant to the (evaluation) research questions.
• Thematic coding: recording or identifying passages of text or images that are linked by a common theme or idea allowing the indexation of text into categories.
• Framework matrices:a method for summarising and analysing qualitative data in a two-by-two matrix table. It allows for sorting data across case and by theme.
• Timelines and time-ordered matrices:aids analysis by allowing for visualisation of key events, sequences and results.

Resources

Websites

WISE: Web Interface for Statistics Education: This website organises a large amount of statistics resources into one central place. It is also home to a series of interactive, sequenced tutorials on key statistical concepts. On WISE, you can find WISE tutorials, WISE applets, excel downloads, teaching papers, quick guides, and publications.

Tools

For an overview of specialist tools for qualitative data analysis, see the CAQDAS site at the University of Surrey which compares ten packages including Atlas.Ti, HyperResearch and NVivo.

This cluster of evaluation tasks involves collecting or retrieving data and analyzing it to answer evaluation questions about what has happened – activities, outcomes and impacts –  and also important contextual information.

Tasks

Tasks related to this cluster are:

1. Sample

Sampling is the process of selecting units (e.g., people, organizations, time periods) from a population of interest, so that inferences can be drawn about the population. There are three clusters of sampling options: probability, purposive and accidental.

2. Use measures, indicators or metrics

Are any existing measures or indicators appropriate, or should new measures or indicators be developed for use in describing implementation or results? Can these be developed in advance or will they need to emerge during the evaluation?

3. Collect/ retrieve data

There are five clusters of options for collecting and/or retrieving data: information from individuals; information from groups; observation; physical measurements; and existing records and data.

4. Manage data

Managing data during an evaluation involves options for storing and organizing data, cleaning datasets using standardized procedures, documenting changes to how data is organized and retrieving data.

5. Combine qualitative and quantitative data

Collecting both quantitative data (numbers) and qualitative data (text, images) is important for most evaluations although they differ in terms of whether the data are equally important in the evaluation. Plan ahead how these wil be combined. We have grouped options into three groups which relate to: the sequence of when the qualitative and quantitative data are collected, when qualitative and quantitative data are combined, and the purpose for combining.

6. Analyse Data

There are many different ways of analyzing data. Here they are grouped into three  clusters of options: numeric analysis, mapping and textual analysis.

7. Visualise Data

Data visualization iis a particular type of analysis involving graphical analysis. Data visualization serves two purposes: to bring clarity during analysis and to communicate.

Sampling is the process of selecting units (e.g., people, organizations, time periods) from a population of interest, studying these in greater detail and then drawing conclusions about the larger population to study them in greater detail.

Options

Consider why you want to study your population of interest and what you want to do with the information that you have gathered, before you choose your option.

There are three clusters of sampling options: Probability; Purposive (or Purposeful); and Convenience.

Probability

Probability sampling options use random or quasi-random options to select the sample, and then use statistical generalization to draw inferences about that population. To minimize bias, these options have specific rules on selection of the sampling frame, size of the sample, and managing variation within the sample. The options include:

• Multi-stage: cluster sampling in which larger clusters are further subdivided into smaller, more targeted groupings for the purposes of surveying.
• Sequential: selecting  every nth case from a list (e.g. every 10th client)
• Simple random: drawing a sample from the population completely at random.
• Stratified random: splitting the population into strata (sections or segments) in order to ensure distinct categories are adequately represented before selecting a random sample from each.

Purposive (or Purposeful)

Purposive sampling options study information-rich cases from a given population to make analytical inferences about the population. Units are selected based on one or more predetermined characteristics and the sample size can be as small as one (n=1). To minimize bias, this cluster of options encourages transparency in case selection, triangulation, and seeking out of disconfirming evidence. The options are:

• Confirming and disconfirming: cases that match existing patterns (to explore them) and those that don’t match (to test them).
• Criterion: cases that meet a particular condition
• Critical case: a case of particular importance, or that can make a strong point
• Homogenous: cases that are very similar to each other.
• Intensity: selecting cases which exhibit a particular phenomenon intensely.
• Maximum variation: contains cases that are as different from each other as possible.
• Outlier: analysing cases that are unusual or special in some way, such as outstanding successes or notable failures.
• Snowball: asking initial informants to identify additional informants,  creating a snowball effect as the sample gets bigger and bigger
• Theory-based: selecting cases according to the extent to which they represent a particular theoretical construct.
• Typical case: developing a profile of what is agreed as average, or normal.

Convenience

Convenience sampling is a cluster of options that use samples which are readily available and which may not allow credible inference about the population. Convenience options are:

• Convenience: based on the ease or “convenience” of gaining access to a sample. simply in which data is gathered from people who are readily available.
• Volunteer: sampling by simply asking for volunteers

Resources

Probability

• Probability Sample:This entry from the Encyclopedia of Survey Research Methods provides a detailed overview of probability sampling and the different kinds of designs that can be used for gathering data for this option.

Purposive

• Qualitative Research & Evaluation Methods:This book from Michael Quinn Patton analyses and provides clear guidance and advice for using a range of different qualitative methods including some approaches and issues that may be involved when using them during fieldwork.
• Purposive Sampling:This entry from the Encyclopedia of Survey Research Methods provides a detailed overview of purposive sampling and how it can be used in evaluation.

Sources

1. Q. Patton (2001 )Qualitative Research and Evaluation Methods(3rd edition). Thousand Oaks, CA: Sage Publications.

As part of an evaluation, it is often important to either develop or use existing indicators or measures of implementation and/or results.

Using an existing indicator or measure can have the advantage of producing robust data which can be compared to other studies, as long as it is appropriate.

Considerable  work has been done to develop measures and indicators that can be used for the outcomes of development projects.

The terms “measure”, “metric” and indicator” are often used interchangeably and their definitions vary across different documents and organisations. Hence, it is always useful to check what these terms mean in specific contexts.

Terms that are commonly associated with measurements include:

• Atarget is the value of an indicator expected to be achieved at a specified point in time. Often a benchmark is used to mean the same thing.
• Anindex is a set of related indicators which intend to provide a means for meaningful and systematic comparisons of performance across programmes that are similar in content and/or have the same goals and objectives.
• Astandard is a set of related indicators, benchmarks or indices which provide socially meaningful information regarding performance.

Education and Training

Examples

• Outcomes and Performance Indicators for Youth Tutoring Programs: This set of outcome indicators, developed by the UrbanInstitute, is aimed at supporting supporting the development, monitoring and evaluation of Youth Tutoring Programs.
• Outcomes and Performance Indicators for Youth Mentoring Programs: This set of outcome indicators, developed by the UrbanInstitute, is aimed at supporting supporting the development, monitoring and evaluation of Youth Mentoring Programs.
• Outcomes and Performance Indicators for Employment Training/Workforce Development Programs:This set of outcome indicators, developed by the Urban Institute, is aimed at supporting supporting the development, monitoring and evaluation of Employment Training/Workforce Development Programs.

Governance

Examples

• Worldwide Governance Indicators: aggregates indicators from six areas of governance from 213 economies over the 1996 – 2010 period. (World Bank)
• The Ibrahim Index:provides a framework and tools that were developed in order to assess the delivery of public goods and services in Africa. (Mo Ibrahim Foundation)
• DataGov: allows users to interactively access and compare data for governance issues from around the world. (DataGov)

Guide

• Global Programme on Democratic Governance Assessments: designed to support developing countries produce governance indicators in order to allow the monitoring of performance toward democratic governance reforms. (UNDP)

Health

Examples

• Outcomes and Performance Indicators for Health Risk Reduction Programs:This set of outcome indicators, developed by the UrbanInstitute, is aimed at supporting supporting the development, monitoring and evaluation of Health Risk Reduction Programs.

Inequality

Guide

• Policy Impacts on Inequality: Simple Inequality Measures: demonstrates a range of ways to measure inequality by using the statistical concepts of location, shape and variability. (Food and Agriculture Organization)

Poverty

Tool

• Handbook on Poverty and Inequality: provides a range of tools which allow the user to measure, describe, monitor, evaluate, and analyze poverty. (World Bank)

Example

• Multidimensional Poverty Index: aims to capture the multiple aspects that constitute poverty. (Oxford Poverty & Human Development Initiative)

Welfare

Examples

• Outcomes and Performance Indicators for Transitional Housing Programs:This set of outcome indicators, developed by the Urban Institute, is aimed at supporting supporting the development, monitoring and evaluation of Transitional Housing Programs.
• Outcomes and Performance Indicators for Prisoner Re-entry Programs: This set of outcome indicators, developed by the UrbanInstitute, is aimed at supporting supporting the development, monitoring and evaluation of Prisoner Re-entry Programs.
• Outcomes and Performance Indicators for Emergency Shelter Programs: This set of outcome indicators, developed by the UrbanInstitute, is aimed at supporting supporting the development, monitoring and evaluation of Emergency Shelter Programs.

Wellbeing

Example

• 2012 California County Scorecard of Children’s Well-Being: provides an interactive display of statistics reporting on the wellbeing of children in California. (Children Now)

World Peace

Example

• Global Peace Index:The Global Peace Index, an initiative of Institute for Economics and Peace (IEP), provides a ranking for each nation in regards to their peacefulness.

This task focuses on ways to collect and/or retrieve data about activities, results, context and other factors. It is important to consider the type of information you want to gather from your participants and the ways you will analyze that information, before you choose your option. You should also consider triangulating your options in order to ensure multiple data sources and perspectives.

Options

There are five clusters of options listed under this task:

1. Information from individuals
2. Information from groups
3. Observation
4. Physical measurements
5. Reviewing existing records and data

Information from individuals

• Deliberative Opinion Polls: providing information about the issue to respondents to ensure their opinions are better informed.
• Diaries: monitoring tools for recording data over a long period of time.
• Goal Attainment Scales: recording actual performance compared to expected performance using a 5 point scale from -2 (much less than expected) to +2 (much more than expected).
• Hierarchical card sort:Hierarchical Card Sorting (HCS) is a participatory card sorting option designed to provide insight into how people categorise and rank different phenomena.
• Interviews with individuals:
• Convergent Interviews: asking probing questions to interviewees and then using reflective prompts and active listening to ensure the conversation continues.
• In-depth Interviews:using probing and multiple interview sessions to collect detailed responses from participants beyond initial answers to questions.
• Key Informant Interviews: interviewing people who have particularly informed perspectives.
• Keypad technology: gauging audience response to presentations and ideas in order to gain provide valuable feedback from large group settings.
• Mobile Data Collection: Targeted gathering of structured information using devices such as smartphones, PDAs, or tablets.
• PhotoVoice: promoting participatory photography as an empowering option of digital storytelling for vulnerable populations.
• Photolanguage: eliciting rich verbal data where participants choose an existing photograph as a metaphor and then discuss it.
• Polling Booth:collect sensitive information from participants anonymously
• Postcards: collecting information quickly in order to provide short reports on evaluation findings (or an update on progress).
• Projective Techniques: participants selecting one or two pictures from a set and using them to illustrate their comments about something (also known as photo-elicitation).
• Questionnaires(or Surveys)
• Email Questionnaires:distributing questionnaires online via email.
• Face to Face Questionnaires:administering questionnaires in real time by a researcher reading the questions.
• Internet Questionnaires: collecting data via a form (with closed or open questions) on the web.
• Mobile Questionnaires:using mobile phones to distribute surveys, either by linking with an adapted internet-based survey or through a specific survey app.
• Mail questionnaires:posting hard copies to participants to be returned.
• Telephone Questionnaires: administering questionnaires by telephone.
• Seasonal Calendars: analysing time-related cyclical changes in data.
• Sketch Mapping: creating visual representations (‘map’) of a geographically based or defined issue.
• Stories (Anecdote): providing a glimpse into how people experience their lives and the impact of specific projects/programs.

Information from groups

• After Action Review: bringing together a team to discuss a task, event, activity or project, in an open and honest fashion.
• Brainstorming: focusing on a problem and then allowing participants to come up with as many solutions as possible.
• Card Visualization: brainstorming in a group using individual paper cards to express participants thoughts about particular ideas or issues.
• Concept Mapping: showing how different ideas relate to each other – sometimes this is called a mind map or a cluster map.
• Delphi Study: soliciting opinions from groups in an iterative process of answering questions in order to gain a consensus.
• Dotmocracy: collecting and recognizing levels of agreement on written statements among a large number of people.
• Fishbowl Technique: managing group discussion by using a small group of participants to discuss an issue while the rest of the participants observe without interrupting.
• Future Search Conference: identifying a shared vision of the future by conducting a conference with this as its focus.
• Interviews with groups
• Focus Group Discussions: discovering the issues that are of most concern for a community or group when little or no information is available.
• Mural: collecting data from a group of people about a current situation, their experiences using a service, or their perspectives on the outcomes of a project.
• ORID: enabling a focused conversation by allowing participants to consider all that is known (Objective) and their feelings (Reflective) before considering issues (Interpretive) and decisions (Decisional).
• Q-methodology: investigating the different perspectives of participants on an issue by ranking and sorting a series of statements (also known as Q-sort).
• Social mapping: Identifying households using pre-determined indicators that are based on socio-economic factors.
• SWOT Analysis: reflecting on and assessing the Strengths, Weaknesses, Opportunities and Threats of a particular strategy.
• World Cafe: hosting group dialogue in which the power of simple conversation is emphasised in the consideration of relevant questions and themes.
• Writeshop:a writing workshop involving a concentrated ​process of drafting, presenting, reviewing, and revising documentations of practice.

Observation

Gathering information by observing people, places and/ or processes either directly or through still or moving images (photography or video). This cluster of options involves watching and documenting the incidence of objects and/ or the behaviour of people.

These options do not involve gathering data directly from individuals or groups, but rather about observing individuals, groups and things. Evaluators of an education project may observe the physical attributes of a school, the accessibility of the site, the availability of latrines, library, and playground. The evaluator may observe the numbers of boys and girls in a classroom, the teaching techniques used and the types of resources that children use.

• Field Trips: organizing trips where participants visit physical sites.
• Non-participant Observation:observing participants without actively participating.
• Participant Observation: identifying the attitudes and operation of a community by living within its environs.
• Photography/video: discerning changes that have taken place in the environment or activities of a community through the use of images taken over a period of time.
• Transect: gathering spatial data on an area by observing people, surroundings and resources while walking around the area or community.

Physical measurements

Measuring physical changes based on agreed indicators and measurement procedures. Examples include birth weight, nutrition levels, rain levels, and soil fertility.

• Biophysical: measuring physical changes over a period of time related to a specific indicator by using an accepted measurement procedure.
• Geographical:capturing geographic information about persons or objects of interest such as the locations of high prevalence of a disease or the location of service delivery points.

Existing documents and data

Reviewing existing knowledge through project documents, information on related projects, government records and publicly available statistics.

• Big data: Large data sets that cannot be analysed using conventional methods, often produced as byproducts of engagement, such as social media data.
• Logs and Diaries: monitoring tools for recording data over a long period of time.
• Official Statistics:obtaining statistics published by government agencies or other public bodies such as international organizations. These include quantitative or qualitative information on all major areas of citizens’ lives such as economic and social development, living conditions, health, education, the environment.
• Previous Evaluations and Research:using the findings from evaluation and research studies that were previously conducted on the same or closely related areas.
• Project Records:retrieving relevant information from a range of documents related to the management of a project such as the project description, strategic and work plans, budget and procurement documents, official correspondence, minutes of meetings, description and follow-up of project participants, progress reports.
• Reputational Monitoring Dashboard: monitoring and quickly appraising reputational trends at a glance and from a variety of different sources.

Good data management includes developing effective processes for consistently collecting and recording data, storing data securely, backing up data, cleaning data, and modifying data so it can be transferred between different types of software for analysis.

Good data management is inextricably linked to data quality assurance –the processes and procedures that are used to ensure data quality. Using data of unknown or low quality may result in making the wrong decisions about policies and programmes.  Data quality assurance (DQA) should be built into each step in the data cycle − data collection, aggregation and reporting, analysis and use, and dissemination and feedback.

Even when data have been collected using well-defined procedures and standardised tools, they need to be checked for any inaccurate or missing data. This “data cleaning” involves finding and dealing with any errors that occur during writing, reading, storage, transmission, or processing of computerised data.

Ensuring data quality also extends to presenting the data appropriately in the evaluation report so that the findings are clear and conclusions can be substantiated. Often, this involves making the data accessible so that they can be verified by others and/or used for additional purposes such as for synthesising results across different evaluations.

Commonly referred to aspects of data quality are:

• Validity: The degree to which the data measure what they are intended to measure.
• Reliability: Data are collected consistently; definitions and methodologies are the same when doing repeated measurements over time.
• Completeness: Data are complete (i.e., no missing data or data elements).
• Precision: Data have sufficient detail.
• Integrity: Data are protected from deliberate bias or manipulation for political or personal reasons
• Availability: Data are accessible so they can be validated and used for other purposes.
• Timeliness: Data are up-to-date current and available on time.

Options

• Consistent Data Collection and Recording:processes to ensure data are collected consistently across different sites and different data collectors.
• Data Backup:onsite and offsite, automatic and manual processes to guard against the risk of data being lost or corrupted.
• Data Cleaning: detecting and removing (or correcting) errors and inconsistencies in a data set or database due to the corruption or inaccurate entry of the data.
• Effective Data Transfer:processes to move data between systems, including between software packages, to avoid the need to rekey data.
• Secure Data Storage:processes to protect electronic and hard copy data in all forms, including questionnaires, interview tapes and electronic files from being accessed without authority or damaged.
• Archive Data for Future Use:systems to store de-identified data so that they can be accessed for verification purposes or for further analysis and research in the future.

Resources

• Data management: supports the design of quality data management systems. (Food and Agriculture Organization, Fisheries and Aquaculture Department)
• Data Quality Tools and Mechanisms: guides to three tools that can be used to assess the quality of data and reporting systems. (The Global Fund)
• Data Qualitycourse developed by the Global Health Learning Centre. Introduces learners to key concepts surrounding data quality and discusses steps programs can take to ensure quality.

Using a combination of qualitative and quantitative data can improve an evaluation by ensuring that the limitations of one type of data are balanced by the strengths of another. This will ensure that understanding is improved by integrating different ways of knowing. Most evaluations will collect both quantitative data (numbers) and qualitative data (text, images), however it is important to plan in advance how these will be combined.

Options

When data are gathered

• Parallel Data Gathering: gathering qualitative and quantitative data at the same time.
• Sequential Data Gathering(Sequencing): gathering one type of data first and then using this to inform the collection of the other type of data.

When data are combined

• Component design: collecting data independently and then combining at the end for interpretation and conclusions.
• Integrated design: combining different options during the conduct of the evaluation to provide more insightful understandings.

Purpose of combining data:

• Enriching: using qualitative work to identify issues or obtain information on variables not obtained by quantitative surveys.
• Examining: generating hypotheses from qualitative work to be tested through the quantitative approach.
• Explaining: using qualitative data to understand unanticipated results from quantitative data.
• Triangulation (Confirming/reinforcing; Rejecting):verifying or rejecting results from quantitative data using qualitative data (or vice versa)

Resources

Guides

• Introduction To Mixed Methods In Impact Evaluation: This guide, written by Michael Bamberger for InterActionoutlines the main elements of a mixed methods approach with particular reference to how MM can be used in an impact evaluation.
• Conducting Mixed-Method Evaluations:This technical note from the US Agency for International Development (USAID) provides an overview to using a mixed-methods approach for evaluation and outlines some of the important considerations that must be taken into account when using the MM approach.
• Conducting mixed methods research: These YouTubevideos feature Alan Bryman from the University of Leicester School of Management presenting a lecture focused on the use of a mixed methods approach when conducting research.

Data visualisation is the process of representing data graphically in order to identify trends and patterns that would otherwise be unclear or difficult to discern. Data visualisation serves two purposes: to bring clarity during analysis and to communicate.

The choice of what type of graph or visualisation to use depends greatly on the nature of the variables you have, such as relational, comparative, time-based, etc..

That said, sometimes graphing data with an inappropriate visualisation can lead to insights during analysis that would have remained hidden. Experimentation with visualisations during analysis is okay, but when communicating a visualisation, use the graph types listed under the proper options below. Incorrect visualisation leads to confusion, errors, and abandonment among viewers.

The options listed here can support both purposes of analysis and communication. You may want to graph data during analysis to see, for example, spikes in website traffic related to your social media campaigns. Visualisation, in this instance, eases data analysis. When communicating that data, however, the visualisation may need to be simplified and key areas may need emphasis in order to call the attention of readers and stakeholders. See the discussion under Report and Support Use for more information about how you may want to repackage a data visualisation for communication purposes.