Seven Steps to Setting up a Developmental Evaluation

Related Project:
Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE)
Author(s):
Ashwin Budden*
Organization:
USAID Health Research Program, CIRCLE Project
Our previous blog introduced you to the fundamentals of Developmental Evaluation (DE) and to the USAID Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) Project, which is carrying out a Developmental Evaluation of Boresha Afya, an integrated health services delivery initiative in Tanzania. This installment describes the start-up phase of the CIRCLE DE of Boresha Afya and highlights important steps for setting up a DE that can be applied to most contexts.

DE is fairly new in global health, and decision makers interested in implementing one are often unsure where to begin. DE involves embedding evaluators in a project over its lifecycle. Building strong relationships and commitment with partners is vital, and this requires more time and attention than in other types of evaluation. 

The CIRCLE Project’s process for implementing the DE for Boresha Afya involved seven steps implemented over six months:

Source: CIRCLE Project

1. Build local capacity for Developmental Evaluation:

Identifying a qualified individual to lead a DE is no easy task, particularly in settings with minimal exposure to the approach. In DE, interpersonal skills such as empathy and consensus-building, as well as familiarity with complex systems-thinking and the local context are as important as technical competence with evaluation methods. 

Dr. Beati Mboya (center), with representatives from USAID and Tanzanian Government.
Source: CIRCLE Project

The CIRCLE Project brought on Dr. Beati Mboya as the Principal Investigator. As a DE specialist, I was tasked to collaborate with Dr. Mboya on developing the roadmap for the DE. Mboya is a Tanzanian medical doctor with many years of experience managing and evaluating health programs in Tanzania. His extensive knowledge of the national health system and key stakeholder connections added considerable value to this process and accelerated engagement with Boresha Afya implementing partners and the USAID Tanzania mission. 

2. DE Orientation: 

Projects that intend to use DE can benefit from orienting stakeholders to the approach early on. Formal meetings and presentations, mini-interviews, and even informal coffee chats all provide opportunities for knowledge exchange.

CIRCLE  conducted consultative meetings with  Boresha Afya stakeholders to lay out the rationale for DE and the principles and practices that differ from traditional monitoring and evaluation. Developing a DE orientation packet to share with stakeholders was invaluable. Similarly, an evaluation reflection exercise, where stakeholders reflected on past experiences with traditional evaluations was useful. Hearing about pain points and missed opportunities with evaluations especially helped to open up dialog with partners and USAID about the value of DE and the priorities and challenges in Boresha Afya.

3. Readiness Assessment:

It is important to note – an interest in DE does not necessarily  indicate buy-in from all stakeholders. This is where conducting a readiness assessment plays a critical role prior to starting a DE.  The goal of a readiness assessment is to understand any potential obstacles to implementation that relate to stakeholders’ organizational culture. Such obstacles may include rigid decision-making structures, low tolerance of risk and change, strong methodological biases, unpredictable team dynamics, and limited bandwidth.

For Boresha Afya partners, we undertook a qualitative assessment of stakeholders’ perspectives about DE gathered through meetings and correspondence. We also conducted an online survey that quantified three dimensions of readiness: the “developmental features” of their project, adaptive capacity in the organization, and openness to learning. These data helped CIRCLE prepare to support different Boresha Afya partners based on their readiness to engage with DE.

4. Stakeholder Analysis:

A stakeholder analysis tool
Source: Ashwin Budden, CIRCLE Project

Implementation research often uses stakeholder analysis to identify the key actors and their relative position of influence on a project. DE takes stakeholder analysis a step further by also mapping the intended users of the evaluation (see previous DE blog). In other words, we want to determine who are the key change agents who need the evaluation findings, when do they need them, and what are the preferred channels of knowledge sharing. 

For Boresha Afya, we used various tools to identify and rank stakeholders in terms of priority for routinely engaging them. Over time, we revisited our mapping to newly capture and prioritize evaluation users.

5. DE Enculturation Workshop:

An enculturation workshop is often the centerpiece of DE start-up, serving as a venue for knowledge sharing and project planning. The structure and content of a workshop can vary widely based on project contexts and stakeholder priorities.

DE Enculturation Workshop in progress
Source: Omar Abdalla
DE Enculturation Workshop in progress
Source: Omar Abdalla

The early DE startup phase kicked into high gear at our DE Enculturation Workshop held in Dar-es-Salaam. For the first time, all implementing partner organizations from across the three project zones of Boresha Afya, as well as representatives from USAID and the Government of Tanzania, came together. The goal was to establish common expectations around implementing DE in the different project zones.  

The lynchpin of the workshop was an iterative co-design process to develop theory of change and cross-cutting evaluation questions that broadly defined the focus of the DE. Prior to this, Boresha Afya had no established theory of change of health service integration. The workshop provided the opportunity to collectively identify and test assumptions about key change pathways and expected results. Details of our workshop activities are described in a process report on the USAID Health Research Program website. 

6. Evaluation Team Recruited and Embedded:

Depending on its scope and needs, a DE may involve only one evaluator or a team of evaluators. For a project like Boresha Afya, we determined that a small team was required. We embedded three experienced evaluators with the respective implementing partners in each of the Boresha Afya zones. Embedding with partners helped to cultivate partnership and real-time engagement on evidence generation and learning. In addition, a knowledge management team supported evaluators in synthesizing and translating evidence and learning for project stakeholders.

DE team members on a scoping visit to a health facility
Source: Ashwin Budden, CIRCLE Project
Tanzanian health facility
Source: Ashwin Budden, CIRCLE Project

7. Evaluation Protocol Delivered:

In traditional evaluations, the evaluation protocol is often the first item to be developed. However, in DE an important goal is to cultivate ownership of the project among partners being evaluated.  The protocol for the Boresha Afya DE benefited from an extensive participatory process that began with the enculturation workshop and continued with follow up planning meetings with partners and the USAID Mission. This approach ensured that all needs and concerns could be considered in the DE framework.

The seven steps described here depict a stepwise process that the CIRCLE Project used to start-up a DE for Boresha Afya. Of course, the timeframe and specific startup activities may vary based on a project’s needs, resources, and operational context. In sum, the steps highlight the importance of strong stakeholder engagement and enculturation, team building, and planning to ensure that the DE can be successfully carried out.

*Ashwin Budden is an independent program evaluator and medical anthropologist. He provides technical assistance on developmental evaluation and implementation research to the CIRCLE Project.

Planning to attend the Fifth Global Symposium on Health Systems Research in Liverpool?

Join us for a skills-building session on developmental evaluation!

Skill-building Session #144
Monday, October 8th
8:00am-12:00pm
Exhibition Centre Room 18