DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
Understanding Why, When, and What it Will Take to do Operations and/or Implementation Research in a Program_ Florence Nyangara
1. Understanding Why, When, and What it will
take to do Operations and/or
Implementation Research in a Program
Experiences from CSHGP and TRAction Projects
Florence Nyangara, PhD, MCHIP/ICF
Pre-CORE Spring Meeting on OR/IR
April 22nd
, 2013
2. Objectives of this OR/IR Workshop
1. Establish a common understanding of terms used in
program-based research (learning by doing) - OR/IR
2. Share knowledge and skills on why and how to
identify & formulate OR/IR research questions, and
decide on research approaches that fit the questions
3. Recognize what it will take to undertake OR/IR within
your program
4. Understand major issues surrounding OR/IR within
community-based health programs i.e. by NGOs
3. Caveat!
The 4-hour workshop is NOT
intended to make you OR/IR
experts!
BUT, we hope you will leave here
feeling that OR/IR can be done in
your program
4. Workshop Format
Interactive and participatory with:
Limited presentations just to overviews on taxonomy
of OR/IR, and about CSHGP & TRAction projectc
Discussions, demonstrations, and illustrative examples
from existing implementations (CSHGP/TRAction)
Open discussions of common issues, unknowns, and
future directions of OR/IR types of research
5. Brainstorming Activity 1
What is Operations Research (OR)?
What is Implementation Research (IR)?
Instructions: Members of each table take 5 minutes to
define what they understand by OR and IR terms;
each group identifies 1 person to present the
definitions to the audience!!
6. Terms Used in Program Research – Are You Confused?
Monitoring and Evaluation - routine
Implementation Research (IR) - Field
Implementation Science (IS)
Process Evaluation
Operations Research (OR)
Impact Evaluation
Within Routine settings: Clinical, community, Policy
Others e.g., Case studies
7. Overlap in Terminologies in Public Health Programs’
Research
Operations Research
OTHER (IS): Health Systems
Research (HSR), QI
Implementation
Research
Im
plem
entation
Science
8. Another Way of Understanding OR versus IR!
Operations Research Implementation Research
Focus Addresses specific
problems within a
program by testing a
feasible solution (problem
must be what managers
can address (control)
More general research inquiry
focusing on "what is happening
and why" in all program aspects
-design, implementation,
administration, operation, services,
& outcomes
Approach Uses scientific techniques
-wide range - descriptive
to experimental for
impact assessments - i.e.
evaluative studies
Uses scientific techniques: -------- -
- Describes program experiences
- Assesses and explains, what/why
- No experimental studies, i.e. not
designed to compare “what is
happening Vs. what would have
happened
Purpose Effectiveness, efficiency, Identify common implementation
9. Present examples on OR and IR
OR – CSHGP – Africare example
IR – TRAction
10. Overview of the Child Survival & Health OR Grants
30 CSHGP/OR studies imbedded within diverse programs implemented
by iNGOs including MNCH, Nutrition, family planning, Malaria, etc to
support national (i.e. MOH) efforts to reduce maternal & child
mortality in low resourced countries
CSHGP mainly focus at community level health systems, remote areas,
and vulnerable populations to address inequities and improve overall
MNCH outcomes but links all levels of the HS (national
community)
CSHGP develops new partnerships with local communities, CBOs,
research institutions, and national stakeholders to design and
implement responsive programs
Strong partnerships between iNGOs, North/South Universities, MOH,
several CBOs, and a few donor agencies i.e. UNICEF and WFP10
11. Key Guiding Principles for Successful OR?
1. Contextual relevance (i.e. aligned with
MOH policy/strategy/plan)
2. Use-focused results (i.e., how results will
inform/influence/change)
3. Diverse and strategic partnerships
4. Stakeholder engagement and learning
12. Examples of Programming Barriers Identified for CSHGP/OR
Inadequate human capacity, i.e. limited availability of
skilled and motivated CHWs
Limited accessibility to services i.e.:
Physical inaccessibility (i.e. distance to facility)
Financial – inability to pay
Cultural, etc
Poor quality of available services i.e. lack of coordination
of services and continuum of care
Poor information and monitoring systems i.e.:
Limited availability and use of evidence-based data for
decision-making
13. Examples of Solutions tested through CSHGP/OR
Maternity Waiting Homes (MWH)
Incentivize, motivate and retain community-based human
capacity e.g. Community Health Workers (CHW)
Integrate service delivery within the health sector and/or
across other relevant sectors (i.e. Agriculture, education)
System Integration or coordination for continuum of care
Strengthen local community structures and build their capacity
(i.e. commitment, resources, skills)
Strengthen Community-based Health Information Systems for
informed decision making
14. Illustrative CSHGP/OR Example
Maternity Waiting Homes (MWH) by
Africare, in Liberia (2010 Cohort)
Key Partners: University of Michigan,
MOHSW, County Health Teams
15. Programming Barrier addressed by Africare/OR
Problem: physical
inaccessibility (long
distance, poor roads to
facility) to get Maternal and
Newborn services (i.e.
Skilled Birth)
Policy: MOH’s national policy
directive promotes only
facility based skilled births
(no home delivery!)
16. Africare, Liberia --------Continued
Solution: establish Maternity Waiting
Homes (MWH) structures near/close
to a health facility for pregnant
women to stay 2-3 weeks before
delivery due date and post-delivery if
they choose to rest/need further
observation)
Relevance: aligned with MOH’s national
policy directive for only facility based
skilled births. Therefore findings likely
to be used to inform the
implementation of this policy
Results: increase facility-based births
and access to other MNC services and
improved outcomes
19. OR Steps and Processes: CSHGP/SCALE
Problem Analysis in
consultation with key
stakeholders
Develop
Solution
(Develop
objectives &
questions)
Conduct
Formative
Research
Refine the
Intervention mix
i.e. concepts,
messages, survey
tools, training
materials, etc
Incorporate findings to
program operations
and monitor
Ongoing Analysis and
Use of Information
Implement the study
design & conduct
ongoing monitoring &
documentation
Evaluative /
Summative Phase
Final Evaluation of Study
- Reflect on Findings with
Stakeholders
Scale-up successful
interventions
20. Problem Analysis and Contextualization
Identify and define the operational problem
(barrier) hindering uptake of interventions
Identify subsystems affected by the problem
Analyze and narrow the problem (i.e. lit. review,
experience, context, observations, etc)
Prioritize the specific aspects of the problem for
OR that if addressed might bring the most
impact
21. Solution/Intervention Development
What are the objectives of proposed solution?
What is the rationale for selecting this solution?
Identify & prioritize specify decision variables i.e.
Training methods, amount of ORS packets
Specify constraints on the solution i.e. SES
Specify the key intervention mix (package)
Specify research questions
Develop a conceptual model -- important
22. Describe the OR Interventions in Detail
Intervention components must be described in
detail for replication or scale-up
Without details we are often left knowing very
little about the details of an intervention or the
functional relationship between the components
of the intervention and outcomes
23. Question?
Instructions: Using the MWH intervention by
AFRicare as an example, please list 3 key
components that must be included in MWH
package?
1.
2.
3.
24. Some Response
Instructions: Using the MWH intervention by
AFRicare as an example, please list 3 key
components that must be included in MWH
package?
1.Structure (building)
2.Community outreach
3.Training (CHWs, facility staff, etc)
25. Objectives and research questions: GOOD
Objectives Research questions
Objective #1: To
develop content of
intervention to deliver
PPFP at household level
Q 1.1: What are main barriers to promotion
and acceptance of PPFP, and how can they
be addressed?
Q 1.2: What is optimal timing and content
of PP home visits by community midwives?
Objectives and Research questions for the
Formative Research concern the specific
intervention content
26. Objectives and research questions: GOOD
Objectives Research questions
Objective #1: To
develop content of
intervention to deliver
PPFP at household level
Q 1.1: What are main barriers to promotion
and acceptance of PPFP, and how can they
be addressed?
Q 1.2: What is optimal timing and content
of PP home visits by community midwives?
Objective #2: To
measure quality and
impact of intervention
Q 2.1: Can adequate supply of
contraceptives be maintained at
community level?
Q 2.2: What is quality of counseling by CM?
Q 2.3: What is effect of intervention on
LAM and modern contraceptive use?
27. Objectives and research questions: BAD
Objectives Research questions
Objective #1: To
conduct formative
research using
focus groups
Q 1.1: Why do people have wrong beliefs about
family planning?
Q 1.2: Why do people think that babies need
extra water in addition to breast milk?
Too specific
Questions anticipate a
certain answer
Assumption that
community is wrong
“To conduct FR” can’t
be an objective
Methods shouldn’t be
in the objectives
28. Objectives and research questions: BAD
Objectives Research questions
Objective #1: To
conduct formative
research using
focus groups
Q 1.1: Why do people have wrong beliefs about
family planning?
Q 1.2: Why do people think that babies need
extra water in addition to breast milk?
Objective #2: To
prove that the
intervention is
highly effective
Q 2.1: What is the contraceptive prevalence rate?
Q 2.2: What is the mean age of women who use
contraceptives?
Q 2.3: How much lower is the maternal mortality
rate in the intervention group compared to the
comparison group, from the household survey?
Q 2.4: What is the cost-effectiveness of
contraceptive distribution by midwives?
29. Objectives and research questions: BAD
Research questions
Q 2.1: What is the contraceptive
prevalence rate?
Q 2.2: What is the mean age of women
who use contraceptives?
Q 2.3: How much lower is the maternal
mortality rate in the intervention group
compared to the comparison group,
from the household survey?
Q 2.4: What is the cost-effectiveness of
contraceptive distribution by midwives?
Questions too
specific
Insufficient
sample size for
maternal
mortality
Impossible to
determine given
the methods
30. Formative Research: Objectives
Define the intervention content
Design and pre-test specific intervention
components:
Communication materials
Counseling guidelines
Packaging/instructions for delivering the services
Pilot test entire package on small scale e.g. a few
villages to refine the intervention
31. Formative Research: Methods
Women Men, mothers-in-
law
Community
midwives
In-depth
interviews or FG
to identify
facilitators
/barriers
Pretesting of
materials and
counseling
guidelines
Pilot test of PP
home visits
32. Evaluative Research: Objectives
Assess the quality of intervention
Measure coverage of intervention
Measure study outcomes
Implement, Evaluate, and Scale-up the Solution
Ongoing evaluation and modifications if needed
Scale-up – Integrate the solution with the larger
system
33. Rigor/Study design: How believable is the evidence?
Non-experimental (Case study)
• Exploratory to see how to implement the intervention
• Ongoing documentation on how bets it can be implemented
• No comparison group
Quasi-experimental (most common for CSHGP/OR)
• Two groups (intervention and comparison)
• Non-random selection to groups
Experimental
• Two groups (intervention and Control)
• Random selection of subjects (1 - CSHGP/OR using clusters)
34.
35. A generic study design diagram
Source: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs
36. A generic diagram: Information to include
Who was your
population? Site and
study participants
How did you
decide who
received the
intervention?
What did the intervention
and comparison groups
receive?
What methods are you
using to compare
outcomes in intervention
and comparison
groups?
What analysis will
you conduct to
compare
outcomes in
intervention and
comparison
groups?
37. Reporting results of a study: What happened?
Source:
http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=53;issue=4;sp
39. CSHGP OR Study Design: Quasi-Experimental
Quasi-experimental (most common for OR)
• Two groups (intervention and comparison)
• Non-random selection to groups
40. Challenges of designing a rigorous OR design
Difficult for program staff to design research study (limited skill &
time)
Non-random placement of program participants to - intervention
areas and control areas often not comparable
Suitable control areas may not exist – other programs in control
areas or cross-over of interventions to control areas
Need to control for other factors beyond the program that might
affect outcomes
Interventions are often multi-faceted – causality difficult to judge
41. Challenges of designing a rigorous OR design
Timing – projects are already underway and it is
hard to incorporate a strong evaluation design
Scale – many projects are too small to expect to be
able to demonstrate impact
Pressure for rapid results before projects ends
Ethical considerations
42. Cross-cutting issues and concluding thoughts
Open discussions of common issues, unknowns,
and future directions of OR/IR types of research
1.What are the common issues?
2. What else should we know – unknowns
3. Suggestions for Future directions of OR/IR types
of research
Instructions: Divide into 3 groups and choose 1, 2,
or 3, discuss among yourselves and with
audience?
Key Workshop Facilitators: Florence Nyangara, PhD: CSHGP/MCHIP/ICF Kristina Gryboski, PhD: CSHGP/USAID Emily Peca, MA, MPH: TRAction/URC Others from the CSHGP and TRACtion projects will be commenting Jim Sherry, M.D., PhD: TRAction/URC
Before we provide our definitions of OR and IR as applied to our programs, first I will like to know what your definitions of these terms are?
Implementation Science encompasses all aspects of research relevant to the scientific study of methods to promote the uptake of research findings into routine settings in clinical, community and policy contexts Implementation science is the scientific study of methods to promote the integration of research findings and evidence-based interventions into healthcare policy and practice and hence, to improve the quality and effectiveness of health services and care
Unfortunately, there continues to be great variation in how these terms are used depending on who is using them and the purpose. However for the purpose of this workshop this is what we mean:
Before we go into discussions on how these OR studies are done, let us hear about IR by TRACtion