1. 27:03:2013
Set-theoretic, diagnostic
and Bayesian approaches
to impact evidence
Barbara Befani
Impact, Innovation and Learning: Towards a Research and Practice Agenda for the Future
Brighton, 26-27 March 2013

2. Outline
Set-theoretic Methods (e.g. QCA) and the new
challenges
– Uncertainty (equifinality)
– Causal contribution (multiple-conjunctural
causality)
– Causal asymmetry (necessity and sufficiency)
Diagnostic and Bayesian approaches
– Uncertainty (can be quantified with probabilities)
– The strength of qualitative evidence can be
measured

3. Defining & explaining events with Set Theory
In uncertain and emergent contexts, we cannot
define “impact” (or success) precisely
An Impact “space” of possible events, all desirable
– All compatible with given values and goals
Success is likely to look like ANY of a number of
events = a LOGICAL UNION
Success looks more like “being on the right track”
than achieving a specific goal
Being “on the right track” means avoiding a number
of pitfalls / dead ends
Sets can be defined as NEGATION of other sets
The three main operators in set theory are
– Negation, union, intersection

6. Causal Asymmetry and Contribution Analysis
What is a sufficient causal package (a branch of
blue nodes)
Principal Contributory Cause = INUS, a necessary
part of the (sufficient) combination (each blue node
of a given branch)
In Set Theory terminology, a causal package is an
INTERSECTION of contributory causes
A combination of necessary causes (necessary
within that causal package)
Set Theory provides the mathematical basis for
1. analyzing causal contribution
2. dealing with uncertainty (particularly Fuzzy Sets)

7. UNION of A and B
A U B
A OR B
A + B
INTERSECTION of A and B
A ∩ B
A AND B
A * B
A A
B B
NEGATION of A
=
~A, NOT A

8. Conf
ID
Cold Chain
Integrity
(necessary)
Vaccine (V)
(intervention)
Health
System
Quality
(HSQ)
Success
(S)
(reduction in
specific
morbidity)
No.
Cases
A 1 1 1 1 3
B 1 1 0 1 1
C 1 0 1 1 2
D 1 0 0 0 2
E 0 1 1 1/0 0
F 0 1 0 1/0 0
G 0 0 1 1/0 0
H 0 0 0 1/0 0
Why are Causal Combinations important?
Impact is contingent on the context
Finding ONE counterfactual is not enough
QCA helps finding many counterfactuals through systematic cross-
case comparison

9. Probability and Diagnosis: what the evidence
says
Realm of “unknown knowns”
General problem of the strength / quality of
evidence: how to assess it?
In clinical practice, physicians use tests
– Specificity
• Probability that absence of the disease will
return negative evidence on that test
– Sensitivity
• Probability that presence of the disease will
return positive evidence on that test
– (Positive) Predictive Power
• Probability that positive evidence signals
presence of the disease

10. When is evidence strong?
When it is sensitive and specific
– Sensitive: P ( Evidence | Impact ) high
– Specific: <= P ( Evidence ) low
• false positives are low
– Predictive: P ( Impact | Evidence ) high
– The latter can be calculated with the Bayes
formula P ( I | E ) = P ( I ) * P ( E | I ) / P ( E )
Two important principles of high-quality evidence
– of all kinds, quali, quanti, etc.
Evidence is strong when:
– The prior probability of observing positive
evidence P ( E ) is LOW (~specificity)
– The probability of observing positive evidence IF
the intervention was successful / had an impact
P ( E | I ) is HIGH (sensitivity)

11. Impact ‘I’
Predictive
Evidence
Weak
Evidence
Strong
Evidence
Sensitive
Evidence
Specific
Evidence
Evidence ‘E’ (of ‘I’)

12. REALITY about success of
intervention
Intervention
Successful (I)
Intervention
Unsuccessful (~I)
EVIDENCE about
success of
intervention
Evidence
Positive (E) True Positive False Positive
Evidence
Negative (~E) False Negative True Negative
Seeking evidence of impact: diagnostic tests

13. REALITY about success of
intervention
Intervention
Successful (I)
Intervention
UNsuccessful
(~I)
EVIDENCE
about success
of intervention
Evidence
Positive (E) True Positive False Positive
Positive
predictive
value =
Σ True Positive
Σ Evidence Posi
tive
Evidence
Negative (~E) False Negative True Negative
Negative
predictive
value =
Σ True
Negative
Σ Evidence Neg
ative
Sensitivity =
Σ True Positive
Σ Intervention
Successful
Specificity =
Σ True
Negative
Σ Intervention
UNsuccessful

14. When articulated ToCs explaining impact are
supported by evidence IT IS STRONG
EVIDENCE OF IMPACT
The prior probability of observing a sophisticated ToC
with several components is LOW, because
The probability of a combination is the product of the
probability of components (a very SMALL #)
P (a, b, c, ... N ) = P (a) * P (b) * P (c) * ... * P (n)
When ToCs with many components are confirmed, it
is strong evidence of impact, because:
– the chances of all components being observed
simultaneously were LOW
• P ( E ) low, specificity high
– If the ToC explaining impact holds true, the
probability of observing evidence of all
components is HIGH
• P ( E | I ) = sensitivity high

15. Conclusion: let’s use other branches of
mathematics
In the form of SET THEORY or PROBABILITY
THEORY (used differently than in frequentist statistics)
They provide new ways of dealing with uncertainty
SET THEORY helps with:
– Defining success in a more flexible, open and
inclusive way (being “on the right track”)
– Explaining success by defining and identifying
contributory causes rigorously through data
analysis (eg. with QCA)
PROBABILITY THEORY helps with:
– Assessing the strength of evidence in terms of
sensitivity, specificity and predictive value
– Qualitative evidence CAN be strong if a number
of conditions are met
– Carefully weigh each piece of evidence as in a court of law,
using conditional and subjective probabilities

16. References
Befani, B. (2013) “Between Complexity and Rigour:
addressing evaluation challenges with QCA” in
Evaluation (forthcoming)
Befani, B. (2013) “What were the chances?
Diagnostic Tests and Bayesian Tools to Assess the
Strength of Evidence in Impact Evaluation”, CDI
Practice Paper (forthcoming)
John Mayne (2013) “Making Causal Claims”
(presentation to this event)
Bruno Marchal (2013) “Conceptual distinctions:
Complexity and Systems – Making sense of
evaluation of complex programmes” (presentation to
this event)