Coordinating Implementation Research to Communicate Learning and Evidence
USAID Health Research Program, CIRCLE Project
DE is fairly new in global health, and decision makers interested in implementing one are often unsure where to begin. DE involves embedding evaluators in a project over its lifecycle. Building strong relationships and commitment with partners is vital, and this requires more time and attention than in other types of evaluation.
The CIRCLE Project’s process for implementing the DE for Boresha Afya involved seven steps implemented over six months:
1. Build local capacity for Developmental Evaluation:
Identifying a qualified individual to lead a DE is no easy task, particularly in settings with minimal exposure to the approach. In DE, interpersonal skills such as empathy and consensus-building, as well as familiarity with complex systems-thinking and the local context are as important as technical competence with evaluation methods.
The CIRCLE Project brought on Dr. Beati Mboya as the Principal Investigator. As a DE specialist, I was tasked to collaborate with Dr. Mboya on developing the roadmap for the DE. Mboya is a Tanzanian medical doctor with many years of experience managing and evaluating health programs in Tanzania. His extensive knowledge of the national health system and key stakeholder connections added considerable value to this process and accelerated engagement with Boresha Afya implementing partners and the USAID Tanzania mission.
2. DE Orientation:
Projects that intend to use DE can benefit from orienting stakeholders to the approach early on. Formal meetings and presentations, mini-interviews, and even informal coffee chats all provide opportunities for knowledge exchange.
CIRCLE conducted consultative meetings with Boresha Afya stakeholders to lay out the rationale for DE and the principles and practices that differ from traditional monitoring and evaluation. Developing a DE orientation packet to share with stakeholders was invaluable. Similarly, an evaluation reflection exercise, where stakeholders reflected on past experiences with traditional evaluations was useful. Hearing about pain points and missed opportunities with evaluations especially helped to open up dialog with partners and USAID about the value of DE and the priorities and challenges in Boresha Afya.
3. Readiness Assessment:
It is important to note – an interest in DE does not necessarily indicate buy-in from all stakeholders. This is where conducting a readiness assessment plays a critical role prior to starting a DE. The goal of a readiness assessment is to understand any potential obstacles to implementation that relate to stakeholders’ organizational culture. Such obstacles may include rigid decision-making structures, low tolerance of risk and change, strong methodological biases, unpredictable team dynamics, and limited bandwidth.
For Boresha Afya partners, we undertook a qualitative assessment of stakeholders’ perspectives about DE gathered through meetings and correspondence. We also conducted an online survey that quantified three dimensions of readiness: the “developmental features” of their project, adaptive capacity in the organization, and openness to learning. These data helped CIRCLE prepare to support different Boresha Afya partners based on their readiness to engage with DE.
4. Stakeholder Analysis:
Implementation research often uses stakeholder analysis to identify the key actors and their relative position of influence on a project. DE takes stakeholder analysis a step further by also mapping the intended users of the evaluation (see previous DE blog). In other words, we want to determine who are the key change agents who need the evaluation findings, when do they need them, and what are the preferred channels of knowledge sharing.
For Boresha Afya, we used various tools to identify and rank stakeholders in terms of priority for routinely engaging them. Over time, we revisited our mapping to newly capture and prioritize evaluation users.
5. DE Enculturation Workshop:
An enculturation workshop is often the centerpiece of DE start-up, serving as a venue for knowledge sharing and project planning. The structure and content of a workshop can vary widely based on project contexts and stakeholder priorities.
The early DE startup phase kicked into high gear at our DE Enculturation Workshop held in Dar-es-Salaam. For the first time, all implementing partner organizations from across the three project zones of Boresha Afya, as well as representatives from USAID and the Government of Tanzania, came together. The goal was to establish common expectations around implementing DE in the different project zones.
The lynchpin of the workshop was an iterative co-design process to develop theory of change and cross-cutting evaluation questions that broadly defined the focus of the DE. Prior to this, Boresha Afya had no established theory of change of health service integration. The workshop provided the opportunity to collectively identify and test assumptions about key change pathways and expected results. Details of our workshop activities are described in a process report on the USAID Health Research Program website.
6. Evaluation Team Recruited and Embedded:
Depending on its scope and needs, a DE may involve only one evaluator or a team of evaluators. For a project like Boresha Afya, we determined that a small team was required. We embedded three experienced evaluators with the respective implementing partners in each of the Boresha Afya zones. Embedding with partners helped to cultivate partnership and real-time engagement on evidence generation and learning. In addition, a knowledge management team supported evaluators in synthesizing and translating evidence and learning for project stakeholders.
7. Evaluation Protocol Delivered:
In traditional evaluations, the evaluation protocol is often the first item to be developed. However, in DE an important goal is to cultivate ownership of the project among partners being evaluated. The protocol for the Boresha Afya DE benefited from an extensive participatory process that began with the enculturation workshop and continued with follow up planning meetings with partners and the USAID Mission. This approach ensured that all needs and concerns could be considered in the DE framework.
The seven steps described here depict a stepwise process that the CIRCLE Project used to start-up a DE for Boresha Afya. Of course, the timeframe and specific startup activities may vary based on a project’s needs, resources, and operational context. In sum, the steps highlight the importance of strong stakeholder engagement and enculturation, team building, and planning to ensure that the DE can be successfully carried out.
*Ashwin Budden is an independent program evaluator and medical anthropologist. He provides technical assistance on developmental evaluation and implementation research to the CIRCLE Project.