The CGIAR research program on Livestock and Fish has just published its Monitoring, Evaluation and Learning (MEL) Framework. The purpose of this Framework is to provide a concise overview of the monitoring and evaluation (M&E) approach that the program will employ for accountability, management decision making and program learning.
Foundational to the Framework is an innovative Theory of Change (ToC) approach and an associated Evidence Base, which overtime will gradually strengthen and validate our ToC as we learn what works and how. In conjunction with our ToC, our Evidence Base will not only focus on the linkages between cause and effect, but it will allow us to provide an evidence-based narrative of our intervention logic. Over time, our Evidence Base will grow into a densely packaged body of knowledge that will allow us to either validate our assumptions or re-formulate them. This process of continually testing and validating our ToC will allow us to better target interventions and make more informed claims about their contribution to impact.
Challenges turned into opportunity for the framework design
Designing the Framework has not been devoid of challenges. The MEL team faced a number of challenges, the most daunting one being the sheer complexity of the Livestock and Fish Program. The causal pathway between research outputs and development impacts involves a complex sequence of linkages that necessarily occur over an extended period of time, involving multiple actors working in multiple scales and geographies.
A second daunting challenge was the fact that there are very few examples of how to develop M&E architecture sufficiently broad to meet all the needs of the program! Tried and proven M&E approaches are fairly established for simple development projects that involve turning development outputs into development impacts, but the M&E terrain for a complex research project like the Livestock and Fish program remains nearly virgin territory.
These challenges required the MEL team to think outside the box of conventional M&E practice and to use considerable ingenuity to overcome. For example, at the program level, operational complexity made “attribution” claims almost impossible, which of course becomes problematic when dealing with donors in today’s “prove it” accountability culture. There is no simple solution to this, so the team chose to employ the concept of “contribution” because it more accurately depicts the confluence of multiple causal factors to a particular change and focuses on the whether or not and how the intervention contributes to the change.
The team also decided to adopt what it calls a ‘track two’ approach. The first track is more appropriate for complex research interventions and falls within a ‘theory-based’ approach to evaluative research; the second track is intended primarily for development projects and is situated firmly in the tradition of results-based project implementation. By identifying how and where the two tracks overlap, the Framework attempts to harmonize conventional M&E methodologies with a more expansive and experimental research program approach.
Different components
Because of the complexity of the program, the MEL Framework is divided into four components, each covering a substantive area of concern to the CGIAR Research program (CRP) on Livestock and Fish
Component One: Learning and Reflection provides an overview of how the CRP intends to learn what science research outputs work (Best Bets) and why? More broadly, it outlines an approach for CRP learning that is designed to show that the correct science is being produced to achieve the desired development outcomes and impacts. This component of the MEL Framework relies heavily on a program theory –based evaluation approach developed substantively by John Mayne and others (see here and here), and constitutes an area of research in its own right, but with obvious implications for how the CRP is managed in general. All of the activities covered in this component of the Framework feed into the MEL Learning Agenda.
Component Two: Program Evaluation outlines the activities designed to help keep the CRP accountable, including CGIAR mandated external evaluations. These evaluations will feed into the Evidence Base outlined in component one, but are also an important accountability mechanism that need to be conducted on a regular basis. The Independent Evaluation Arrangement (IEA) provides detailed guidance on how to conduct different kinds of evaluations.
Component Three: Program Monitoring explains briefly how the CRP will collect Intermediate Development Outcomes (IDO) and medium-term indicator data and how this data will be used. This component of the Framework should be read in conjunction with the CRP Indicator Manual, which is a separate document that is publicly accessible. By comparing our output and impact targets with what we actually achieve, this component of the Framework is designed to answer the question: “Are we achieving our stated goals?”
Regardless of how well individual components of the MEL Framework function they are only as useful as the knowledge management system that links data and analysis with real-life stakeholders. Component Four: Knowledge Management provides an overview of the various information management systems employed by the CRP.
Way forward
Over the coming months, the MEL team will be shifting its attention from developing the conceptual architecture of the M&E structure toward actual implementation. The MEL Framework tells us what we want our M&E house to look like, but it does not give us a step-by-step guide to actually build it; for that the MEL team will be working on tools development over the next few months, with the intention of rolling out its implementation over the first half of 2015 in two value chain countries and two technology flagships.
Read about the program flagships here
CGIAR Research Program on Livestock and Fish 