Understanding Implementation Research
Definitions and Characteristics
Implementation can be considered the act of carrying out policies, programs or individual practices (collectively called interventions). Implementation research helps to understand what, why and how those interventions work in “real world” settings and tests approaches to improve them.1 Occasionally, implementation research is expanded to Implementation Research and Delivery Science (IRDS) to emphasize both the research and delivery aspects of the process of understanding “what” works and “how” to get what works to the people that need it.
Because it is an approach that builds upon several research traditions, each with their own audiences, core disciplines and typical questions, implementation research is often used interchangeably with several other concepts with slightly different but distinct meanings such as implementation science, operational or operations research, and process evaluation. Certain defining characteristics of implementation research are, however, widely agreed upon:2
- Context specific: Implementation research pays attention to the differential need for and benefit from interventions across sub-groups.
- Relevant and agenda-setting purpose: Implementation research is used to identify and address implementation problems, set priorities and build commitment.
- Methods fit for purpose: Research design is based on answering an implementation problem or question and is sensitive to social stratifiers.
- Demand driven: Research questions are based on needs identified by implementers, intended beneficiaries, policy makers, and research consumers in the health system.
- Multi-stakeholder and multi-disciplinary: Implementers, policy makers, and researchers (and often communities) should co-produce the research, co-create solutions, and use the results together.
- Real world context: Implementation research takes place within the reality of implementing organizations, communities, and health/financing systems.
- Non-linear and iterative process: Evidence is provided through short feedback loops to make real-time improvements or course-corrections.
- Implementation processes and implementation outcomes: Implementation research focuses on how interventions are implemented to assess acceptability, fidelity, adoption, scale-up, and impact.
Asking the Right Questions
Implementation Research can help to answer the following questions:
- Why am I not seeing the results I expected from my program?
- How can I develop solutions to the challenges I’m seeing?
- How can I help my program get better results?
- How can I adapt or improve my program as it is being implemented?
- How can I make sure my program is reaching those that need it?
- How do I make sure my program is sustainable in the long-term?
If any of these questions were answered affirmatively, it might be time to consider implementation research. Here are a few key elements to begin the implementation research process.
To answer these questions, consider certain core components related to the program.
- Intervention: What is the project planning to do (and why)?
- Implementation Strategies: How will the intervention get done?
- Implementation Outcomes: What was actually done and how?
- Systems/Process: What was changed?
- Intervention (service/client) Outcomes (including unintended): Did it make a difference/have the planned effect?
- Contextual Factors: What else affected the implementation and intervention outcomes?
- Stakeholders: Who are the key users and beneficiaries and when and how to engage them?
See below for additional information on how developing a theory of change and utilizing existing conceptual frameworks can help to define these core program elements, understand how they interact, and decide how they can be measured.
Understanding the Theory of Change
A theory is a systematic way of mapping out the relationships between events or behaviors that one wants to understand or change and the concepts that explain or predict them.3 Several well-documented behavioral theories such as Diffusion of Innovations4 or Stages of Change5 provide generic and broadly applicable foundations for development a project/study-specific theory of change .
A theory of change provides a clear explanation of how and why specific relationships lead to specific events – for instance, how a given intervention will effect change.6 A theory of change describes the steps along the pathway toward a goal and the assumptions about why they occur. These steps and assumptions are then the subject of research for verification, testing, or further explanation.
The process of developing a theory of change can help stakeholders agree on the program’s common purpose, break down silos, identify areas of common interest, and attract collaborators working toward the same positive results. Theories of change can vary in complexity, but should communicate what the project is trying to achieve, how to go about it, and assumptions being made that inform the relationships being mapped.
A theory of change is a critical first step in implementation research design and should inform:
- What or who is targeted and when.
- What strategies are implemented to drive change.
- What research methods are necessary.
The Health Research Program’s Ponya Mtoto project is adapting World Health Organization guidelines for managing possible serious bacterial infection (PSBI) in young infants in Kenyan facilities when referral is not feasible. They have adopted the theory of change below:
Meaningful stakeholder engagement is one of the Health Research Program’s guiding principles. Specifically, the Program supports countries on their journey to self-reliance by:
- Jointly identifying problems and solutions among implementers, researchers and decision-makers in a co-design process.
- Actively working to expand the role for partners in low- and middle-income countries partners to jointly design implementation research programs
- Fostering multisectoral, multidisciplinary partnerships.
Care should be taken to identify key stakeholders and establish systematic ways of engaging them throughout the implementation research process.
- Stakeholders can play a critical role in exploration, preparation and adaptation through formative research conducted before implementation starts by validating the acceptability of proposed interventions, identifying appropriate contextual factors for measurement, and other aspects of implementation design.
- During implementation stakeholders can help to interpret findings and solve problems as real-time adjustments are made to improve implementation.
- Stakeholders can play a critical role in disseminating the knowledge gained through the implementation research process and supporting further diffusion and scale-up efforts after the initial implementation research is completed.
Implementation Research Frameworks
One indication of a strong implementation research design is the use of a conceptual framework or model that provides a systematic way to design, implement and evaluate interventions.7 A framework usually denotes a structure, overview, outline, system or plan consisting of various descriptive categories (i.e. concepts, constructs or variables) and the relations between them that are presumed to account for a phenomenon.8
Two of the most commonly utilized frameworks related to implementation research conducted around maternal, newborn and child health are the RE-AIM framework9 and the Consolidated Framework for Implementation Research (CFIR).10
Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM)11,12 are critical elements determining public health impact. The RE-AIM framework has been widely utilized for reporting research results across diverse topics, but is increasingly being used to help plan programs and improve their chances of working in “real-world” settings, and to understand the relative strengths and weaknesses of different approaches to health promotion and disease management. The framework encourages program planners, evaluators, readers of journal articles, funders, and policy-makers to pay more attention to essential program elements including external validity that can improve the sustainable adoption and implementation of effective, generalizable, evidence-based interventions.
Consolidated Framework for Implementation Research (CFIR)
The CFIR13,14 offers a comprehensive structure to systematically evaluate all possible contextual and intervention factors that influence implementation and effectiveness of a program. The CFIR is comprised of the following elements:
- Intervention characteristics – including strength and quality of the evidence;
- Outer setting – including patient needs and resources;
- Inner setting – including organizational culture and leadership engagement;
- Characteristics of the individuals involved – including individual attitudes, belief and capabilities; and
- The process of implementation – including other influential factors related to the process of implementation itself, such as planning evaluation and reflection.
A study assessing optimization of Prevention of Mother-to-Child Transmission (PMTCT) of HIV used the CFIR to guide data collection and analysis.15
Goals: The Systems Analysis and Improvement Approach (SAIA) study was a cluster randomized trial to test whether a package of systems engineering tools could strengthen PMTCT programs. The study sought to (1) define core and adaptable components of the SAIA intervention, and (2) explain the heterogeneity in SAIA’s success between facilities.
Findings: Flow mapping and continuous quality improvement were the core to the SAIA in all settings, whereas the PMTCT cascade analysis tool was the core in high HIV prevalence settings. Five CFIR constructs distinguished strongly between high and low performers: 2 in inner settings (networks and communication, available resources) and 3 in process (external change agents, executing, reflecting and evaluating).
Implementation strategies are the methods or techniques used to enhance adoption, implementation, and sustainability of a program or practice.11,12 Typically, a combination of implementation strategies that are tailored to a particular context is necessary to facilitate the application of evidence into practice.16 A small sample of implementation strategies17 might include:
- training or coaching,
- technical assistance,
- performance monitoring,
- facilitation, etc.
Implementation strategies describe how the intervention will be applied and should be defined in the theory of change. Therefore, capturing and reporting accurate descriptions of implementation strategies is critical to successful implementation research – this includes being clear about who is actually delivering the strategy, what actions or steps they are taking, who or what the strategy is targeting, the order or sequencing of the strategy, its dose or intensity, which implementation outcomes are being targeted, and justification or rationale for the strategies chosen.18 Being clear and specific in defining the implementation strategies is a critical step that facilitates interpretation and use implementation research findings.
Implementation outcome variables are the building blocks of implementation research and serve as indicators of how and why a given implementation is actually working.
More traditional evaluation efforts focus more specifically on intervention outcomes – did the intervention make a difference or have the planned effect – and systems or process level changes that result from implementation. These changes or outcomes may be unintended or as a result of effective implementation. In addition to measuring these more traditional outcomes, implementation research emphasizes measurement of implementation outcomes to describe what was actually done and how it was done.19
The box below provides working definitions for the most commonly tested implementation outcomes.
Implementation Outcome Variables
Perception among stakeholders that intervention is agreeable
Intention, initial decision or action to employ new intervention
Perceived fit of the intervention in a particular setting for a particular audience
Extent to which an intervention can be carried out in a particular setting for a particular audience
Degree to which an intervention was implemented as it was designed in an original protocol, plan or policy
Incremental cost of delivering the strategy
Degree to which an intervention-eligible population actually receives it
Extent to which an intervention is maintained or institutionalized in a setting
Unique to implementation research is the emphasis on defining, understanding and measuring contextual factors that affect the implementation and intervention outcomes. Contextual factors might include:16
- external environmental factors,
- structural/organizational dynamics,
- cultural factors
- contextual elements reflecting collaboration, resources and leadership, such as poverty, geographical remoteness, traditional beliefs, etc.
Frameworks such as the CFIR and RE-AIM are useful tools for identifying the elements that might influence the implementation of an intervention. These contextual elements can be particularly important in planning for implementation scale-up. Identifying and engaging key stakeholders can help to identify the appropriate contextual factors for measurement, and various other aspects of implementation design.