2. What decision-makers want to know
Is the new treatment
better than the
established one ?
If not, is it equally
effective and preferable
for some other reason
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
3. Overview
Definitions and concepts
Challenges of non-inferiority trials
What does this mean for systematic reviews?
Language considerations for comparative
effectiveness reviews
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
4. Proving Efficacy
1. Showing superiority of one
(new) treatment over
another (placebo or active)
2. Showing equivalence or non-
inferiority of a new
intervention relative to an
already existing efficacious
treatment
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
5. The interpretation of non-inferiority and
equivalence can be confusing
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
7. Gerald Gartlehner
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
8. Superiority game: the winner takes it all….
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
9. Equivalence game – equally good or clearly
better
Equivalence margin: tied or less than 1 goal
difference
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
10. Non-inferiority game– at least not substantially
worse….
Non-inferiority margin: can‘t lose with more
than 1 goal difference
Gerald Gartlehner
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
11. Definitions
Superiority trial
Objective: To determine a clinically relevant difference
between two interventions
Equivalence trial
Objective: To determine whether a (new) intervention is
neither worse nor better than another (established)
intervention
Non-inferiority trial
Objective: To determine whether a (new) intervention is not
inferior to another (established) intervention
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
12. Equivalence - Non-Inferiority
The naïve approach:
If a head-to-head trial shows no statistically significant
difference, two interventions are “equivalent”
Problem – underpowered studies or high variance will
create “equivalent” treatments
The statistical approach:
Define a margin of non-inferiority or equivalence
If 95% confidence interval of the difference DOES NOT
cross the margin, the new intervention is non-inferior or
equivalent
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
13. Arthroscopy vs. sham arthoscopy in patients
with knee osteoarthritis
favors sham favors arthroscopy
MID
20 10 0 10 20
Knee Specific Pain Scale: difference in points
Mosley et al. New England J Med, 2002;347:81-88
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
14. Non-inferiority margin (d)
The limit of acceptable inferiority:
Minimal important difference
Clinical judgement
Statistical considerations
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
15. Determining the non-inferiority margin
statistically
Standard vs. placebo
2 points (1.5 to 2.5)
New vs. standard
2 1 0
Maximimum non-inferiority margin: dmax = 1.5
Fractional preservation of treatment effect: f = 0.5
d = dmax x (1 – f)
d = 1.5 x ( 1 – 0.5) = 0.75
16. Peculiar issues of non-inferiority trials: the
backwardness
Null hypothesis and alternative hypothesis are
reversed
Type I and type II errors are reversed
Per-protocol analyses can be more important than
ITT analysis (ITT analyses are biased towards finding
no difference)
P-value is one-sided (0.025)
17. Assay Sensitivity and Constancy Assumption
The ability of a trial to distinguish effective from
ineffective treatments (depends on the effect size
the trial wants to detect).
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
18. Assay Sensitivity
Assumptions in non-inferiority trials:
The efficacy of the active control was preserved in
the non-inferiority study (i.e. that it had assay
sensitivity).
If it was not, equivalence or non-inferiority
conclusions are meaningless (The non-inferior drug
could have no effect at all).
19. Constancy Assumption
Active comparator must be well established and have
predictable and consistent treatment effects
Participants must be similar to those in trial
establishing efficacy
Outcomes must be similar to those in trials
establishing efficacy
20. Biocreep
Effective treatment
Still clinically relevant?
2 1 0
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
21. Key Points for critical appraisal
Was a non-inferiority margin defined based on
clinical considerations and statistical reasoning?
Was it established a priori?
Was the study powered based on the non-inferiority
margin?
Was an ITT and a per-protocol analysis conducted?
Was the trial design (e.g. eligibility criteria)
consistent with placebo controlled trials of the
established treatment?
22. What does this mean for systematic reviews ?
For meta-analyses – data can be used just as
from any superiority trial
For qualitative assessments – language
considerations
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
23. Language considerations
“..there was no statistically significant difference
between A and B..”
“..studies failed to show a difference..”
Can mean:
1) The evidence shows equivalence
2) The evidence is inconclusive (because confidence
intervals are wide-lack of precision)
AHRQ guidance: Assessing equivalence and non-inferiority [draft]
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
24. Language considerations
Better:
“..treatments A and B had similar mortality rates..”
Expressing non-inferiority:
“…treatment A is at least as effective as treatment B for
[Outcome or study objective]…”
AHRQ guidance: Assessing equivalence and non-inferiority [draft]
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems
25. Do we need to establish equivalence
margins for CERs ?
Editor's Notes
For example: The null hypothis becomes the alternative hypothesisType I error becomes type II error and vice versaITT is suddenly not as good as per protocol analysesThere is some backwardness that can be very confusing – so I tried to come up with a simple analogy to capture the basic concepts of superiority-non-inferiority –and equivalence and I ended up with soccer: