Program Review: A Chance to Learn & Adapt

By Anne Guillen, Rebecca Farmer, and Jimmy Leak, Nuru International

This blog was originally posted on the USAID Learning Lab website on November 7, 2017 at https://usaidlearninglab.org/lab-notes/program-review-chance-learn-adapt.

Setting goals is a key component of project planning. When establishing a new project, it is customary to set the overarching goal and objectives for the project and then work backwards developing timelines, milestones, and indicators to track progress towards this goal. As the project begins, progress is assessed based on whether milestones are reached and mid-term assessments. Eventually, the project comes to an end and the final endline report determines if the goal of the project has been successfully achieved.

At Nuru International, an NGO with an integrated programming model working with farmers and their families in Kenya and Ethiopia, we have a different approach. While our project’s goal remains constant, the method used to achieve that goal is flexible within the timeline of the project. A systematic project design process allows us to create program models that are based on community needs, combining local and expatriate staff input. This process allows a strong starting place for programs based on research and best practices. Instead of stopping there and running stagnant programming until the end of the program or funding cycle, Nuru learns and adapts along the way.

Program Review Process

In summer 2017, Nuru’s Ethiopia project piloted a process called “Program Review” for the first time.  The Program Review concept was developed by Nuru’s senior leadership team after the collection of first year impact evaluation data for all programs. The Program Review process was developed locally to align with Nuru’s philosophy of making data-driven decisions. The Program Review process was a weeklong process where program managers from each of Nuru’s four program areas: Agriculture, Financial Inclusion, Education, and Healthcare could sit with their expatriate counterparts and Nuru’s M&E department to review monitoring and evaluation data. While it was important to see if program activities were tracking towards desired levels of impact, it was more important to have discussions about why and how activities were working (or not) and what processes and systems could be improved. Instead of walking away from the session simply saying, “Let’s work harder to accomplish our goals”, Nuru program managers were given an opportunity to adapt their future program activities based on lessons learned from the data.

Managers’  ability to change their program activities was instrumental. They used data, coupled with input from their international advisors, to inform proposals for changes and adaptations to program activities. This gave program managers a real sense of ownership, while maintaining accountability. Nuru has a mantra of “Fail Fast, Learn Fast,” and lessons are not fully learned until mistakes are corrected and changes are made moving forward.

Program Adaptations

Each of Nuru’s programs experienced successes and identified activities to continue from the data analysis phase of the process.  Additionally, each program identified areas to be improved and proposed adaptations to programming based on research and best practices.

The Agriculture program had success in getting farmers to adopt best agronomic practices, but also dealt with many environmental challenges such as drought and Fall Army Worm infestation, which caused farmers to have poor crop yields and, in turn, lower loan repayment. Nuru’s Financial Inclusion program had a high percentage of women savers weekly, but the money they saved was not enough to cope with shocks. In response to these challenges, Nuru Ethiopia’s Agriculture and Financial Inclusion staff developed a proposal to pilot working more through government for input loans, while strengthening training on best agronomic practices, and scaling up an animal husbandry pilot to help farmers diversify income during times of environmental shocks. By diversifying income streams and providing risk reserves to farmer cooperatives, Nuru can help ensure income and food security.

Nuru’s Education program saw increases in literacy rates through its interventions both in and outside of schools, but experienced challenges in the implementation of its community reading activities through volunteers. Nuru’s Education Program manager suggested piloting reading camp activities that  occurred after school and during school break times through after school tutorial sessions already being held by teachers at local schools that Nuru worked with.  The teachers would be more reliable and better trained facilitators, and the school system would help to ensure sustainability of the activity after Nuru staff are no longer there.

Nuru’s Healthcare Program showed positive results around maternal and child health indicators, with high percentages of mothers attending antenatal care visits and delivering in health centers, however these results seemed to be primarily driven by Nuru staff members. Nuru intended to cascade trainings on maternal and child health down to government health extension workers, but data and observations from the field showed that trainings presented too much information for government health extension workers to take in at one time. Nuru decided to break down and adapt its training curriculum so that there were fewer topics per training to facilitate better understanding of topics and long-term sustainability.

The changes and adaptations to Nuru’s programs are being implemented this year and will be monitored and evaluated in 2018 to determine their effectiveness.

Lessons Learned and the Future

The Program Review process was a great success that enabled Nuru Ethiopia leaders to make data-driven decisions to improve their programs and see themselves as agents of change. As with all of its initiatives, Nuru learned lessons from the Program Review Process.  One of the main takeaways was the need for more time between receiving data, developing new ideas, and making decisions. There were some challenges with data analysis delays, which shortened the timeline for developing new ideas, especially when it came to ideas for more integration between programs. Nuru staff resolved to add more time for data analysis and review, especially so more local stakeholders, such as government officials, could be involved. Nuru staff also learned that while there were many great ideas, not all were feasible, and the synthesis process could be refined to ensure only a few ideas remain. Finally, Nuru developed new processes that it could use to strengthen its regular quarterly reflections and reports on data.

Nuru’s Ethiopia Program Managers are excited to build upon successes and adapt their programs to achieve Nuru’s ultimate goal of lifting communities out of extreme poverty in a sustainable way. Managers now see how data can be used to influence programming and make decisions, not simply evaluate effectiveness.  Nuru will take these lessons as it scales to new areas and hopes to repeat this process every few years to evaluate progress and iterate past challenges.