By: Tessa Ahner- McHaffie, FHI360
Are we learning enough about integrated programs through impact evaluations?
Integrated development raises a lot of questions. Tessa Ahner-McHaffie and colleagues from FHI 360 tried to answer some of these in a Systematic Review published on Gates Open Research and currently being peer reviewed. In this guest blog, she describes the work that her and colleagues have done in exploring the evidence around this area of development work.
As someone working on ‘integration’ at FHI 360, a large non-profit dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions, I often get asked ‘does integration work?’ To answer questions like ‘where does working in multiple sectors add value?’ or ‘what does the evidence say about multi-sector integration in this context?’ we first have to determine what evidence is available, and what research questions are being asked and answered. So, my team and I set out to do that by conducting a systematic review to find impact evaluations on integrated strategies, and see how those evaluations were being designed.
Gathering the evidence
[pullquote]Integration is an umbrella phrase that can describe thousands of different cross-sector approaches — from health and microfinance, to nutrition and education, to conservation and livelihoods.[/pullquote]
Conducting a systematic review on something as broad (and often fluid) as integrated development was, from the beginning, difficult. Conducting keyword searches for “integration” – which we defined as any program working across one of the core development sectors agriculture and food security, economic development, education, environment, governance, health, nutrition, and water, sanitation and hygiene (WASH) – was nearly impossible, since people do not consistently use this phrase in titles or abstracts.
So, we needed to come at it from a different direction. We decided to start from the entire realm of impact evaluations, and manually screened all of the 4,000+ entries in 3ie’s Impact Evaluation Repository, looking for which studies involved an integrated program.
The result? We found a lot more evaluations than we originally expected: 601 peer-reviewed impact evaluations on integrated, multi-sector projects. These studies crossed every sector and region. Due to the variability, we were unable to conduct a meta-analysis on the effectiveness of the projects studied. We were, however, able to find some interesting trends on what is getting studied (and what is not).
What the evidence told us
We found that integrated, multi-sector programs are being evaluated, both rigorously and often. Most report positive findings, although we need to consider publication bias, so we can say little beyond that these programs work sometimes. Yet, the majority of these studies did not consider integration as a component of the program to be evaluated, or an objective to monitor or measure in and of itself.
Only 38 had a partial or full factorial Randomized Controlled Trial (RCT) design, which would be the most straightforward way to look at the possible amplification of outcomes of multi-sector integration, although it exponentially adds to the time, resources, and complexity of the evaluation.
Importantly, there are other possible ways that integration can add value to a program beyond the final quantitative results: by improving the reach of the project to more vulnerable people, improving the satisfaction of the staff or community utilizing the project, creating cost efficiencies, in addition to others. Yet, although 60 assessments used a qualitative component, and 43 used a costing component, most of those were measuring some other angle of the program, and not deliberately unpacking the effect of integration.
[pullquote]Very few of these evaluations consider if this integration is adding to (or detracting from) the aims of the project.[/pullquote]
Moving forward, we know that a large proportion of those programs being studied combine multiple sectors in one package. We need to be more deliberate in our attempt to learn about the added value of combining the interventions as a component of outcome evaluations. Doing so does not necessarily require a complicated or expensive design each time – it can mean creatively using mixed methods and other strategies to explore the targeted effects of integration.
Why publish on Gates Open Research?
One reason we were excited to publish on Gates Open Research was the speed at which we would be able to start a dialogue about what we found. Having an open, thoughtful conversation through peer review that can be viewed by readers is appealing, and we are excited to be a part of it. We hope that funders, researchers, and program designers will use the vast set of evaluations we have gathered to dig into the sectors or models of their interest, use those to further analyze effectiveness, and inform new ideas, decisions, and learning in the future.
Gates Open Research provides all Gates Foundation-funded researchers with a place to rapidly publish any results they think are worth sharing. All articles benefit from immediate publication, transparent refereeing and the inclusion of all source data. If you are a grantee of the Gates Foundation you can find more information about how to publish on Gates Open Research here.