In this presentation, we explain the process through which a realist evaluation could be conducted on complex interventions through the building and refining of programme theories of these interventions.

1. Using programme theory for
evaluation of complex interventions
Prashanth NS
Faculty & PhD Scholar
Institute of Public Health, Bangalore

2. Outline
• Background
– Interventions, HS interventions and complexity
– Framing evaluation questions? “Did it…”, “What..”,
“Why (or why not?)..” or “How…”
– Multiple paths
• How to use a programme theory approach in a
complex local health systems intervention

3. “Medicine is a social science, and
politics is nothing else but medicine
on a large scale. Medicine, as a
social science, as the science of
human beings, has the obligation to
point out problems and to attempt
their theoretical solution: the
politician, the practical
anthropologist, must find the means
Source: Pd photo/NLM, USA for their actual solution…”

4. Randomise and control
Origins in drug testing for effectiveness
– Outcomes measurable, verifiable; animal models
available. Environmental conditions can be
controlled.
• Linearity (linear and observable transitions from inputs
to outcome; environmental conditions known)
• No ethical issues in setting up control
• Gold-standard for effectiveness

5. Hazardous journeys
Smith, G. C. S., & Pell, J. P. (2003). Parachute use to prevent death
and major trauma related to gravitational challenge: systematic
review of randomised controlled trials. BMJ, 327(7429), 1459–61.
doi:10.1136/bmj.327.7429.1459

6. But health systems are complex…
WHO AHPSR 2008

7. Software neglected

8. Complexity & systems-thinking
• Simple, complicated and complex (Glouberman & Zimmerman)
– Examples: from recipies to rockets to children
• Characteristics
– Multiple interacting elements (within and outside)
– Organisational structure and interactions
– Multiple paths/configurations to same outcome;
same structural configuration but different
outcome (path dependency)
– Unpredictability due to feedback loops

9. Complexity in health systems/interventions
Institute of Public Health, Bangalore

10. Medical research council guidance

11. Effectiveness or Mechanism?
Existing methods in public
health research focus on
effectiveness:
“Did it work? To what
extent”
But, middle-level managers,
decision-makers are looking
for:
“How did it work?”, “Why
did it not work for x or y?”

12. The evaluation question
Why did Arogyashri not benefit
beneficiaries from SC/ST proportional to
Ideally, evaluations their population in AP?
Is there a social phenomenon operating
should be able to here that could help us understand
inform not only if a implementation of financing or other
schemes in general?
given intervention
worked, but also how it
worked, and why it
worked for some (and
not for others)
Fan, V. Y., & Mahal, A. (2011). Learning and getting better : Rigorous evaluation of health
policy in India. National Medical Journal of India, 24(6), 325–327.

13. Answering how questions
Various methods possible
(MRC guidance)
– CrCT with process
evaluations, stepped-
wedge designs
– Modeling: causal
modelling,
mathematical
modelling, TdI,
economic modelling
– Natural experiments &
cohort studies

14. The programme theory approach
A programme theory is way of representing
the expected relationship between the
elements of the intervention and its expected
outcomes.
Consider the pulse polio programme

15. But, for complex interventions…

16. Initial programme assumptions

17. Learning from general theory

18. Reconstruction

19. Empircise

20. Applications of the PT
• Theorising and understanding “how”: in parallel
with RCTs (process evaluation) or preliminary to a
realist evaluation (my study design)
• Informing design/development of future similar
programmes
• Forms a basis for analysing the qualitative data
and for using frameworks (Eg. Using MPF for
understanding organisational performance)

21. Six-step model
Van Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G.
(2010). How to develop a theory-driven evaluation design?
Lessons learned from an adolescent sexual and reproductive
health programme in West Africa. BMC public health, 10,
741. doi:10.1186/1471-2458-10-741

22. A mixed method RE design using PT

23. Addressing complexity by understanding
relationships rather than structural outcomes
Marchal et. al’s representation of Sicotte framework

24. Using PT for evaluation
Examining the data using the
framework helps further
refine the PT. Allows for
identifying positive and
negative configurations of
the context-mechanism-
outcome that can be
emperically verified to
answer the question
“How does capacity-building
work in this intervention?”

25. Key references
Prashanth, N. S., Marchal, B., Hoeree, T., Devadasan, N., Macq, J., Kegels, G., & Criel, B. (2012).
How does capacity building of health managers work? A realist evaluation study protocol. BMJ
open, 2(2), e000882. doi:10.1136/bmjopen-2012-000882
Marchal, B., Van Belle, S., Van Olmen, J., Hoeree, T., & Kegels, G. (2012). Is realist evaluation
keeping its promise? A review of published empirical studies in the field of health systems
research. Evaluation, 18(2), 192–212. doi:10.1177/1356389012442444
Van Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G. (2010). How to develop a theory-driven
evaluation design? Lessons learned from an adolescent sexual and reproductive health
programme in West Africa. BMC public health, 10, 741. doi:10.1186/1471-2458-10-741
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and
evaluating complex interventions: the new Medical Research Council guidance. British Medical
Journal, 337(sep29 1), a1655–a1655. doi:10.1136/bmj.a1655
Marchal, B., Dedzo, M., & Kegels, G. (2010). A realist evaluation of the management of a well-
performing regional hospital in Ghana. BMC Health Services Research, 10(October 2000), 24.
On Mendeley, see my reading list on critical realism and realist evaluation at:
http://www.mendeley.com/groups/535071/critical-realism-and-realist-evaluation/papers/

26. Acknowledgement
Bart Criel, Guy Kegels, Jean Macq, Bruno Marchal & Tom Hoeree at ITM, Antwerp
Upendra Bhojani, Tanya Seshadri, Arima Mishra & N Devadasan for discussions
and re-discussions