1. Understanding Why, When, and What it will
take to do Operations and/or
Implementation Research in a Program
Experiences from CSHGP and TRAction Projects
Florence Nyangara, PhD, MCHIP/ICF
Pre-CORE Spring Meeting on OR/IR
April 22nd
, 2013

2. Objectives of this OR/IR Workshop
1. Establish a common understanding of terms used in
program-based research (learning by doing) - OR/IR
2. Share knowledge and skills on why and how to
identify & formulate OR/IR research questions, and
decide on research approaches that fit the questions
3. Recognize what it will take to undertake OR/IR within
your program
4. Understand major issues surrounding OR/IR within
community-based health programs i.e. by NGOs

3. Caveat!
 The 4-hour workshop is NOT
intended to make you OR/IR
experts!
 BUT, we hope you will leave here
feeling that OR/IR can be done in
your program

4. Workshop Format
Interactive and participatory with:
 Limited presentations just to overviews on taxonomy
of OR/IR, and about CSHGP & TRAction projectc
 Discussions, demonstrations, and illustrative examples
from existing implementations (CSHGP/TRAction)
 Open discussions of common issues, unknowns, and
future directions of OR/IR types of research

5. Brainstorming Activity 1
 What is Operations Research (OR)?
 What is Implementation Research (IR)?
Instructions: Members of each table take 5 minutes to
define what they understand by OR and IR terms;
each group identifies 1 person to present the
definitions to the audience!!

6. Terms Used in Program Research – Are You Confused?
Monitoring and Evaluation - routine
Implementation Research (IR) - Field
Implementation Science (IS)
Process Evaluation
Operations Research (OR)
Impact Evaluation
Within Routine settings: Clinical, community, Policy
Others e.g., Case studies

7. Overlap in Terminologies in Public Health Programs’
Research
Operations Research
OTHER (IS): Health Systems
Research (HSR), QI
Implementation
Research
Im
plem
entation
Science

8. Another Way of Understanding OR versus IR!
Operations Research Implementation Research
Focus Addresses specific
problems within a
program by testing a
feasible solution (problem
must be what managers
can address (control)
More general research inquiry
focusing on "what is happening
and why" in all program aspects
-design, implementation,
administration, operation, services,
& outcomes
Approach Uses scientific techniques
-wide range - descriptive
to experimental for
impact assessments - i.e.
evaluative studies
Uses scientific techniques: -------- -
- Describes program experiences
- Assesses and explains, what/why
- No experimental studies, i.e. not
designed to compare “what is
happening Vs. what would have
happened
Purpose Effectiveness, efficiency, Identify common implementation

9. Present examples on OR and IR
 OR – CSHGP – Africare example
 IR – TRAction

10. Overview of the Child Survival & Health OR Grants
 30 CSHGP/OR studies imbedded within diverse programs implemented
by iNGOs including MNCH, Nutrition, family planning, Malaria, etc to
support national (i.e. MOH) efforts to reduce maternal & child
mortality in low resourced countries
 CSHGP mainly focus at community level health systems, remote areas,
and vulnerable populations to address inequities and improve overall
MNCH outcomes but links all levels of the HS (national
community)
 CSHGP develops new partnerships with local communities, CBOs,
research institutions, and national stakeholders to design and
implement responsive programs
 Strong partnerships between iNGOs, North/South Universities, MOH,
several CBOs, and a few donor agencies i.e. UNICEF and WFP10

11. Key Guiding Principles for Successful OR?
1. Contextual relevance (i.e. aligned with
MOH policy/strategy/plan)
2. Use-focused results (i.e., how results will
inform/influence/change)
3. Diverse and strategic partnerships
4. Stakeholder engagement and learning

12. Examples of Programming Barriers Identified for CSHGP/OR
 Inadequate human capacity, i.e. limited availability of
skilled and motivated CHWs
 Limited accessibility to services i.e.:
 Physical inaccessibility (i.e. distance to facility)
 Financial – inability to pay
 Cultural, etc
 Poor quality of available services i.e. lack of coordination
of services and continuum of care
 Poor information and monitoring systems i.e.:
 Limited availability and use of evidence-based data for
decision-making

13. Examples of Solutions tested through CSHGP/OR
 Maternity Waiting Homes (MWH)
 Incentivize, motivate and retain community-based human
capacity e.g. Community Health Workers (CHW)
 Integrate service delivery within the health sector and/or
across other relevant sectors (i.e. Agriculture, education)
 System Integration or coordination for continuum of care
 Strengthen local community structures and build their capacity
(i.e. commitment, resources, skills)
 Strengthen Community-based Health Information Systems for
informed decision making

14. Illustrative CSHGP/OR Example
Maternity Waiting Homes (MWH) by
Africare, in Liberia (2010 Cohort)
Key Partners: University of Michigan,
MOHSW, County Health Teams

15. Programming Barrier addressed by Africare/OR
Problem: physical
inaccessibility (long
distance, poor roads to
facility) to get Maternal and
Newborn services (i.e.
Skilled Birth)
Policy: MOH’s national policy
directive promotes only
facility based skilled births
(no home delivery!)

16. Africare, Liberia --------Continued
Solution: establish Maternity Waiting
Homes (MWH) structures near/close
to a health facility for pregnant
women to stay 2-3 weeks before
delivery due date and post-delivery if
they choose to rest/need further
observation)
Relevance: aligned with MOH’s national
policy directive for only facility based
skilled births. Therefore findings likely
to be used to inform the
implementation of this policy
Results: increase facility-based births
and access to other MNC services and
improved outcomes

17. IR by TRAction Project

18. How OR is done?

19. OR Steps and Processes: CSHGP/SCALE
Problem Analysis in
consultation with key
stakeholders
Develop
Solution
(Develop
objectives &
questions)
Conduct
Formative
Research
Refine the
Intervention mix
i.e. concepts,
messages, survey
tools, training
materials, etc
Incorporate findings to
program operations
and monitor
Ongoing Analysis and
Use of Information
Implement the study
design & conduct
ongoing monitoring &
documentation
Evaluative /
Summative Phase
Final Evaluation of Study
- Reflect on Findings with
Stakeholders
Scale-up successful
interventions

20. Problem Analysis and Contextualization
 Identify and define the operational problem
(barrier) hindering uptake of interventions
 Identify subsystems affected by the problem
 Analyze and narrow the problem (i.e. lit. review,
experience, context, observations, etc)
 Prioritize the specific aspects of the problem for
OR that if addressed might bring the most
impact

21. Solution/Intervention Development
 What are the objectives of proposed solution?
 What is the rationale for selecting this solution?
 Identify & prioritize specify decision variables i.e.
Training methods, amount of ORS packets
 Specify constraints on the solution i.e. SES
 Specify the key intervention mix (package)
 Specify research questions
 Develop a conceptual model -- important

22. Describe the OR Interventions in Detail
 Intervention components must be described in
detail for replication or scale-up
 Without details we are often left knowing very
little about the details of an intervention or the
functional relationship between the components
of the intervention and outcomes

23. Question?
Instructions: Using the MWH intervention by
AFRicare as an example, please list 3 key
components that must be included in MWH
package?
1.
2.
3.

24. Some Response
Instructions: Using the MWH intervention by
AFRicare as an example, please list 3 key
components that must be included in MWH
package?
1.Structure (building)
2.Community outreach
3.Training (CHWs, facility staff, etc)

25. Objectives and research questions: GOOD
Objectives Research questions
Objective #1: To
develop content of
intervention to deliver
PPFP at household level
Q 1.1: What are main barriers to promotion
and acceptance of PPFP, and how can they
be addressed?
Q 1.2: What is optimal timing and content
of PP home visits by community midwives?
Objectives and Research questions for the
Formative Research concern the specific
intervention content

26. Objectives and research questions: GOOD
Objectives Research questions
Objective #1: To
develop content of
intervention to deliver
PPFP at household level
Q 1.1: What are main barriers to promotion
and acceptance of PPFP, and how can they
be addressed?
Q 1.2: What is optimal timing and content
of PP home visits by community midwives?
Objective #2: To
measure quality and
impact of intervention
Q 2.1: Can adequate supply of
contraceptives be maintained at
community level?
Q 2.2: What is quality of counseling by CM?
Q 2.3: What is effect of intervention on
LAM and modern contraceptive use?

27. Objectives and research questions: BAD
Objectives Research questions
Objective #1: To
conduct formative
research using
focus groups
Q 1.1: Why do people have wrong beliefs about
family planning?
Q 1.2: Why do people think that babies need
extra water in addition to breast milk?
 Too specific
Questions anticipate a
certain answer
Assumption that
community is wrong
 “To conduct FR” can’t
be an objective
 Methods shouldn’t be
in the objectives

28. Objectives and research questions: BAD
Objectives Research questions
Objective #1: To
conduct formative
research using
focus groups
Q 1.1: Why do people have wrong beliefs about
family planning?
Q 1.2: Why do people think that babies need
extra water in addition to breast milk?
Objective #2: To
prove that the
intervention is
highly effective
Q 2.1: What is the contraceptive prevalence rate?
Q 2.2: What is the mean age of women who use
contraceptives?
Q 2.3: How much lower is the maternal mortality
rate in the intervention group compared to the
comparison group, from the household survey?
Q 2.4: What is the cost-effectiveness of
contraceptive distribution by midwives?

29. Objectives and research questions: BAD
Research questions
Q 2.1: What is the contraceptive
prevalence rate?
Q 2.2: What is the mean age of women
who use contraceptives?
Q 2.3: How much lower is the maternal
mortality rate in the intervention group
compared to the comparison group,
from the household survey?
Q 2.4: What is the cost-effectiveness of
contraceptive distribution by midwives?
Questions too
specific
Insufficient
sample size for
maternal
mortality
Impossible to
determine given
the methods

30. Formative Research: Objectives
 Define the intervention content
 Design and pre-test specific intervention
components:
 Communication materials
 Counseling guidelines
 Packaging/instructions for delivering the services
 Pilot test entire package on small scale e.g. a few
villages to refine the intervention

31. Formative Research: Methods
Women Men, mothers-in-
law
Community
midwives
In-depth
interviews or FG
to identify
facilitators
/barriers
Pretesting of
materials and
counseling
guidelines
Pilot test of PP
home visits

32. Evaluative Research: Objectives
 Assess the quality of intervention
 Measure coverage of intervention
 Measure study outcomes
 Implement, Evaluate, and Scale-up the Solution
 Ongoing evaluation and modifications if needed
 Scale-up – Integrate the solution with the larger
system

33. Rigor/Study design: How believable is the evidence?
Non-experimental (Case study)
• Exploratory to see how to implement the intervention
• Ongoing documentation on how bets it can be implemented
• No comparison group
 Quasi-experimental (most common for CSHGP/OR)
• Two groups (intervention and comparison)
• Non-random selection to groups
 Experimental
• Two groups (intervention and Control)
• Random selection of subjects (1 - CSHGP/OR using clusters)

35. A generic study design diagram
Source: http://hsl.lib.umn.edu/biomed/help/understanding-research-study-designs

36. A generic diagram: Information to include
Who was your
population? Site and
study participants
How did you
decide who
received the
intervention?
What did the intervention
and comparison groups
receive?
What methods are you
using to compare
outcomes in intervention
and comparison
groups?
What analysis will
you conduct to
compare
outcomes in
intervention and
comparison
groups?

37. Reporting results of a study: What happened?
Source:
http://www.jpgmonline.com/article.asp?issn=0022-3859;year=2007;volume=53;issue=4;sp

38. Reporting results of a study: What happened?

39. CSHGP OR Study Design: Quasi-Experimental
 Quasi-experimental (most common for OR)
• Two groups (intervention and comparison)
• Non-random selection to groups

40. Challenges of designing a rigorous OR design
 Difficult for program staff to design research study (limited skill &
time)
 Non-random placement of program participants to - intervention
areas and control areas often not comparable
 Suitable control areas may not exist – other programs in control
areas or cross-over of interventions to control areas
 Need to control for other factors beyond the program that might
affect outcomes
 Interventions are often multi-faceted – causality difficult to judge

41. Challenges of designing a rigorous OR design
 Timing – projects are already underway and it is
hard to incorporate a strong evaluation design
 Scale – many projects are too small to expect to be
able to demonstrate impact
 Pressure for rapid results before projects ends
 Ethical considerations

42. Cross-cutting issues and concluding thoughts
Open discussions of common issues, unknowns,
and future directions of OR/IR types of research
1.What are the common issues?
2. What else should we know – unknowns
3. Suggestions for Future directions of OR/IR types
of research
Instructions: Divide into 3 groups and choose 1, 2,
or 3, discuss among yourselves and with
audience?

43. Web-based Tools and Resources
 www.MCHIPNGO.net
 www.coregroup.org
 www.MCHIP.net
43

44. Reference Materials on Operations and
Implementation Research