SlideShare ist ein Scribd-Unternehmen logo
1 von 40
Downloaden Sie, um offline zu lesen
Absence of a gold standard in diagnostic test
accuracy research

with application in context of childhood TB
Maarten van Smeden, PhD
Post-doctoral researcher Julius Center for Health Sciences and Primary Care
WEON 2017 Pre-conference Accounting for Measurement Error in Epidemiology
Antwerp, June 7, 2017
Outline
• Diagnostic test accuracy
• The problem: absence of a gold standard
• Possible solution: latent class analysis in context of TB
Diagnostic testing
Diagnostic testing
Diagnostic testing
Diagnostic testing
• “New test better than the existing test(s)?”
• “(Where to) add new test to diagnostic pathway?”
• “Recommend new test in practice guidelines?”
Fig from: Bossuyt, BMJ, 2006
Diagnostic test accuracy studies (DTA)
• Evaluation of “new” diagnostic tests (=index test) by
comparison to a “gold standard”
• Misclassification probabilities of index test: sensitivity,
specificity, negative/positive predictive values, etc.
Classical DTA analysis
Subjects undergo the index test (T) and gold standard test (GS)
GS + GS -
T + A C
T - B D
Classical DTA analysis
Sensitivity (Se) = A/(A+B)

Specificity (Sp) = D/(D+C)
GS + GS -
T + A C
T - B D
Reporting guideline: STARD
Reporting guideline: STARD
“.. a gold standard would be an error-free reference standard”
All that glitters is not gold
• Commonly the best available reference standard: Se < 1 and
Sp < 1: not a “gold standard”. 



Because:

detection limits (e.g. culture), infeasible/not ethical to execute
in some patients (e.g. biopsy), observer errors (e.g. MRI), etc.
All that glitters is not gold
• Commonly the best available reference standard: Se < 1 and
Sp < 1: not a “gold standard”. 



-> misclassifications of the target condition by the reference
standard (= measurement error) 

When using imperfect reference standard
Assuming: reference standard Se = 1, index test Sp = Se = 0.7, conditional independence reference standard and index test

0.5 0.6 0.7 0.8 0.9 1.0
Specificity Reference Standard
E[SenstivityIndexTest]
Disease prevalence = 0.05
Disease prevalence = 0.25
Disease prevalence = 0.50
0.3
0.4
0.5
0.6
0.7
When using imperfect reference standard
• Bias, sometimes called “reference standard bias”. Not
necessarily a lower bound of Se/Sp



• Philosophical problems when index test is believed to be
more accurate than the best available reference standard
When using imperfect reference standard
Absence of a gold standard
Misclassifications by the reference standard -> 

no straightforward approaches to estimation of
misclassification probabilities of index tests (that are valid)
Absence of a gold standard in diagnostic test accuracy research
Tuberculosis (TB)
Paulsen, Nature, 2013
■ FIGURE 2.16a
Top causes of death worldwide in 2012.a,b Deaths from TB
among HIV-positive people are shown in grey.c
Road injury
HIV/AIDS
Diabetes mellitus
Diarrheal diseases
Tracheal, bronchus,
lung cancers
TB
Chronic obstructive
pulmonary disease
Lower respiratory
infections
Stroke
Ischaemic heart
disease
0 1 2 3 4 5 6 7
Millions
■ F
Est
20
in g
a This is the latest year for which estimates for all causes are currently
available. See WHO Global Health Observatory data repository,
available at http://apps.who.int/gho/data/node.main.GHECOD
(accessed 27 August 2015).
b For HIV/AIDS, the latest estimates of the number of deaths in 2012
a F
t
o
b
i
b D
d
HIV
WPR 9.2 8.3–10.0 0.29
Global 35.2 30.9–39.4 8.4
WHO Global TB report 2015
Data
• 749 hospitalised children with suspected pulmonary TB in
Cape Town, South Africa
• Study procedures, a number of tests for TB for each subject:
• Microscopy
• Culture
• Xpert (NAAT)
• TST (skin test)
• Radiography
Primary publication
Primary publication
48%: “possible tuberculosis”
Solution?
• The idea:
Simple latent class model
Pr(T = 1) = ⇡Se + (1 ⇡)(1 Sp)
= Pr(D = 1)Pr(T = 1|D = 1)+
Pr(D = 0)Pr(T = 1|D = 0)
• With two conditionally independent binary tests (T0 and T1)
Simple latent class model
Pr(T0 = 1, T1 = 1) = ⇡Se0Se1+
(1 ⇡)(1 Sp0)(1 Sp1)
• With J conditionally independent tests (and bit of algebra):
Simple latent class model
Pr(T1, . . . , TJ ) = ⇡
JY
j=1
Se
Tj
j (1 Sej)1 Tj
+
(1 ⇡)
JY
j=1
Sp
1 Tj
j (1 Spj)Tj
Latent class model estimation
• Maximum likelihood
• Gibbs sampling
Heuristic model for TB data
Heuristic model for TB data
• Conditional independence
between all tests is unlikely
• Conditional dependence
between: Xpert, culture,
microscopy, and TST among TB
diseased due to “bacterial load”
• Bacterial load modelled by a
random effect
Modeling dependence
Pairwise correlation residual (misfit)
Conditional independence model Random effects model
Main results
Conditional independence model Random effects model
Is latent class analysis useful?
• In TB example, I believe: yes
• More realistic than assuming reference standard (culture)
has Se = Sp = 1
• Results ‘robust’ to changing prior distributions and
conditional dependence structure
• Lack of robust alternative approaches for DTA in the
absence of a gold standard
Is latent class analysis useful?
• But:
• Latent class analysis for DTA is still rare
Latent class analysis in diagnostic research
Systematic review from 2014
• 69 theoretical papers
• 64 applied papers in human research + 47 in veterinary sciences
• applications of LCA still not common in human diagnostic research
van Smeden, AJE, 2014
Is latent class analysis useful?
• But:
• Latent class analysis for DTA is still rare
• Robustness to misspecification of the conditional
dependence structure is a concern
Absence of a gold standard in diagnostic test accuracy research
Is latent class analysis useful?
• But:
• Latent class analysis for DTA is still rare
• Robustness to misspecification of the conditional
dependence structure is a concern
• Identifiability requirements
Why Bayesian?
• Practical arguments:
• Model specifications in non-commercial software packages
(e.g. randomLCA vs rjags in R)
• (Weakly) informative prior distributions can solve non-
identifiability problems
• Additional calculations (e.g. positive/negative predictive
values with CrI)
Final remarks
• Misclassification in DTA studies is often both the primary topic
of study (for the index test) and the problem (when occurring
in the reference standard)
• Model based estimation of index test accuracy by latent class
analysis can be useful
• There is some evidence that robustness of the latent class
model can be improved when disease status can be verified
with certainty in a subset
• While the focus of this talk was on DTA, other studies such as
“incremental value” studies suffer from the same problems
Acknowledgements
Thanks to all co-authors in:
Supported by a grant from Canadian Institutes of Health Research (MOP
#89857)

Más contenido relacionado

Was ist angesagt?

Guideline for high-quality diagnostic and prognostic applications of AI in he...
Guideline for high-quality diagnostic and prognostic applications of AI in he...Guideline for high-quality diagnostic and prognostic applications of AI in he...
Guideline for high-quality diagnostic and prognostic applications of AI in he...Maarten van Smeden
 
Machine Learning Pipelines
Machine Learning PipelinesMachine Learning Pipelines
Machine Learning Pipelinesjeykottalam
 
RWE & Patient Analytics Leveraging Databricks – A Use Case
RWE & Patient Analytics Leveraging Databricks – A Use CaseRWE & Patient Analytics Leveraging Databricks – A Use Case
RWE & Patient Analytics Leveraging Databricks – A Use CaseDatabricks
 
To explain or to predict
To explain or to predictTo explain or to predict
To explain or to predictGalit Shmueli
 
Lect6 Association rule & Apriori algorithm
Lect6 Association rule & Apriori algorithmLect6 Association rule & Apriori algorithm
Lect6 Association rule & Apriori algorithmhktripathy
 
Improve Clinical Trial Processes With a Unified Platform
Improve Clinical Trial Processes With a Unified PlatformImprove Clinical Trial Processes With a Unified Platform
Improve Clinical Trial Processes With a Unified PlatformVeeva Systems
 
Metadata and ADaM
Metadata and ADaMMetadata and ADaM
Metadata and ADaMKevin Lee
 
Classification techniques in data mining
Classification techniques in data miningClassification techniques in data mining
Classification techniques in data miningKamal Acharya
 
Regression shrinkage: better answers to causal questions
Regression shrinkage: better answers to causal questionsRegression shrinkage: better answers to causal questions
Regression shrinkage: better answers to causal questionsMaarten van Smeden
 
Solving the CT problem for LBTESTCD and LBTEST values
Solving the CT problem for LBTESTCD and LBTEST valuesSolving the CT problem for LBTESTCD and LBTEST values
Solving the CT problem for LBTESTCD and LBTEST valuesBernard Tchoula
 
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...Melanie Swan
 
Data Mining: Concepts and Techniques (3rd ed.) - Chapter 3 preprocessing
Data Mining:  Concepts and Techniques (3rd ed.)- Chapter 3 preprocessingData Mining:  Concepts and Techniques (3rd ed.)- Chapter 3 preprocessing
Data Mining: Concepts and Techniques (3rd ed.) - Chapter 3 preprocessingSalah Amean
 
Trial Design Domains
Trial Design DomainsTrial Design Domains
Trial Design DomainsAnkur Sharma
 

Was ist angesagt? (20)

Guideline for high-quality diagnostic and prognostic applications of AI in he...
Guideline for high-quality diagnostic and prognostic applications of AI in he...Guideline for high-quality diagnostic and prognostic applications of AI in he...
Guideline for high-quality diagnostic and prognostic applications of AI in he...
 
Machine Learning Pipelines
Machine Learning PipelinesMachine Learning Pipelines
Machine Learning Pipelines
 
RWE & Patient Analytics Leveraging Databricks – A Use Case
RWE & Patient Analytics Leveraging Databricks – A Use CaseRWE & Patient Analytics Leveraging Databricks – A Use Case
RWE & Patient Analytics Leveraging Databricks – A Use Case
 
Bias in AI
Bias in AIBias in AI
Bias in AI
 
Tools for writing Review Article
Tools for writing Review ArticleTools for writing Review Article
Tools for writing Review Article
 
To explain or to predict
To explain or to predictTo explain or to predict
To explain or to predict
 
Lect6 Association rule & Apriori algorithm
Lect6 Association rule & Apriori algorithmLect6 Association rule & Apriori algorithm
Lect6 Association rule & Apriori algorithm
 
Improve Clinical Trial Processes With a Unified Platform
Improve Clinical Trial Processes With a Unified PlatformImprove Clinical Trial Processes With a Unified Platform
Improve Clinical Trial Processes With a Unified Platform
 
Metadata and ADaM
Metadata and ADaMMetadata and ADaM
Metadata and ADaM
 
MLOps.pptx
MLOps.pptxMLOps.pptx
MLOps.pptx
 
Algorithm based medicine
Algorithm based medicineAlgorithm based medicine
Algorithm based medicine
 
Classification techniques in data mining
Classification techniques in data miningClassification techniques in data mining
Classification techniques in data mining
 
Uncertainty in AI
Uncertainty in AIUncertainty in AI
Uncertainty in AI
 
Regression shrinkage: better answers to causal questions
Regression shrinkage: better answers to causal questionsRegression shrinkage: better answers to causal questions
Regression shrinkage: better answers to causal questions
 
UMC Utrecht AI Methods Lab
UMC Utrecht AI Methods LabUMC Utrecht AI Methods Lab
UMC Utrecht AI Methods Lab
 
Solving the CT problem for LBTESTCD and LBTEST values
Solving the CT problem for LBTESTCD and LBTEST valuesSolving the CT problem for LBTESTCD and LBTEST values
Solving the CT problem for LBTESTCD and LBTEST values
 
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
Deep Learning Explained: The future of Artificial Intelligence and Smart Netw...
 
Data management & statistics in clinical trials
Data management & statistics in clinical trialsData management & statistics in clinical trials
Data management & statistics in clinical trials
 
Data Mining: Concepts and Techniques (3rd ed.) - Chapter 3 preprocessing
Data Mining:  Concepts and Techniques (3rd ed.)- Chapter 3 preprocessingData Mining:  Concepts and Techniques (3rd ed.)- Chapter 3 preprocessing
Data Mining: Concepts and Techniques (3rd ed.) - Chapter 3 preprocessing
 
Trial Design Domains
Trial Design DomainsTrial Design Domains
Trial Design Domains
 

Ähnlich wie Absence of a gold standard in diagnostic test accuracy research

Biostatistics in Clinical Research
Biostatistics in Clinical ResearchBiostatistics in Clinical Research
Biostatistics in Clinical ResearchAbhaya Indrayan
 
Analysing & interpreting data.ppt
Analysing & interpreting data.pptAnalysing & interpreting data.ppt
Analysing & interpreting data.pptmanaswidebbarma1
 
Test of significance in Statistics
Test of significance in StatisticsTest of significance in Statistics
Test of significance in StatisticsVikash Keshri
 
Techniques in clinical epidemiology
Techniques in clinical epidemiologyTechniques in clinical epidemiology
Techniques in clinical epidemiologyBhoj Raj Singh
 
Elashoff approach section in grant applications
Elashoff approach section in grant applicationsElashoff approach section in grant applications
Elashoff approach section in grant applicationsUCLA CTSI
 
Evaluation of the clinical value of biomarkers for risk prediction
Evaluation of the clinical value of biomarkers for risk predictionEvaluation of the clinical value of biomarkers for risk prediction
Evaluation of the clinical value of biomarkers for risk predictionEwout Steyerberg
 
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...Pat Barlow
 
1. complete stats notes
1. complete stats notes1. complete stats notes
1. complete stats notesBob Smullen
 
Trends towards significance
Trends towards significanceTrends towards significance
Trends towards significanceStephenSenn2
 
Bio-Statistics in Bio-Medical research
Bio-Statistics in Bio-Medical researchBio-Statistics in Bio-Medical research
Bio-Statistics in Bio-Medical researchShinjan Patra
 
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1Eyenirvaan
 
Intro to tests of significance qualitative
Intro to tests of significance qualitativeIntro to tests of significance qualitative
Intro to tests of significance qualitativePandurangi Raghavendra
 
Meta-Analysis -- Introduction.pptx
Meta-Analysis -- Introduction.pptxMeta-Analysis -- Introduction.pptx
Meta-Analysis -- Introduction.pptxACSRM
 

Ähnlich wie Absence of a gold standard in diagnostic test accuracy research (20)

Stats7.0
Stats7.0Stats7.0
Stats7.0
 
Biostatistics in Clinical Research
Biostatistics in Clinical ResearchBiostatistics in Clinical Research
Biostatistics in Clinical Research
 
8.pdf
8.pdf8.pdf
8.pdf
 
Overview of statistical tests: Data handling and data quality (Part II)
Overview of statistical tests: Data handling and data quality (Part II)Overview of statistical tests: Data handling and data quality (Part II)
Overview of statistical tests: Data handling and data quality (Part II)
 
Analysing & interpreting data.ppt
Analysing & interpreting data.pptAnalysing & interpreting data.ppt
Analysing & interpreting data.ppt
 
Test of significance in Statistics
Test of significance in StatisticsTest of significance in Statistics
Test of significance in Statistics
 
Techniques in clinical epidemiology
Techniques in clinical epidemiologyTechniques in clinical epidemiology
Techniques in clinical epidemiology
 
Elashoff approach section in grant applications
Elashoff approach section in grant applicationsElashoff approach section in grant applications
Elashoff approach section in grant applications
 
Evaluation of the clinical value of biomarkers for risk prediction
Evaluation of the clinical value of biomarkers for risk predictionEvaluation of the clinical value of biomarkers for risk prediction
Evaluation of the clinical value of biomarkers for risk prediction
 
Biostatistics
BiostatisticsBiostatistics
Biostatistics
 
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
 
1. complete stats notes
1. complete stats notes1. complete stats notes
1. complete stats notes
 
Trends towards significance
Trends towards significanceTrends towards significance
Trends towards significance
 
Statistic and orthodontic by almuzian
Statistic and orthodontic by almuzianStatistic and orthodontic by almuzian
Statistic and orthodontic by almuzian
 
05 diagnostic tests cwq
05 diagnostic tests cwq05 diagnostic tests cwq
05 diagnostic tests cwq
 
Bio-Statistics in Bio-Medical research
Bio-Statistics in Bio-Medical researchBio-Statistics in Bio-Medical research
Bio-Statistics in Bio-Medical research
 
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1
Evaluating a diagnostic test presentation www.eyenirvaan.com - part 1
 
Intro to tests of significance qualitative
Intro to tests of significance qualitativeIntro to tests of significance qualitative
Intro to tests of significance qualitative
 
Data in science
Data in science Data in science
Data in science
 
Meta-Analysis -- Introduction.pptx
Meta-Analysis -- Introduction.pptxMeta-Analysis -- Introduction.pptx
Meta-Analysis -- Introduction.pptx
 

Mehr von Maarten van Smeden

A gentle introduction to AI for medicine
A gentle introduction to AI for medicineA gentle introduction to AI for medicine
A gentle introduction to AI for medicineMaarten van Smeden
 
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...Maarten van Smeden
 
Prognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient healthPrognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient healthMaarten van Smeden
 
Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?Maarten van Smeden
 
Clinical prediction models for covid-19: alarming results from a living syste...
Clinical prediction models for covid-19: alarming results from a living syste...Clinical prediction models for covid-19: alarming results from a living syste...
Clinical prediction models for covid-19: alarming results from a living syste...Maarten van Smeden
 
Five questions about artificial intelligence
Five questions about artificial intelligenceFive questions about artificial intelligence
Five questions about artificial intelligenceMaarten van Smeden
 
Development and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutionsDevelopment and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutionsMaarten van Smeden
 
Prediction models for diagnosis and prognosis related to COVID-19
Prediction models for diagnosis and prognosis related to COVID-19Prediction models for diagnosis and prognosis related to COVID-19
Prediction models for diagnosis and prognosis related to COVID-19Maarten van Smeden
 
Clinical prediction models: development, validation and beyond
Clinical prediction models:development, validation and beyondClinical prediction models:development, validation and beyond
Clinical prediction models: development, validation and beyondMaarten van Smeden
 
Correcting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confoundingCorrecting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confoundingMaarten van Smeden
 
Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead Maarten van Smeden
 
Living systematic reviews: now and in the future
Living systematic reviews: now and in the futureLiving systematic reviews: now and in the future
Living systematic reviews: now and in the futureMaarten van Smeden
 
The statistics of the coronavirus
The statistics of the coronavirusThe statistics of the coronavirus
The statistics of the coronavirusMaarten van Smeden
 
COVID-19 related prediction models for diagnosis and prognosis - a living sys...
COVID-19 related prediction models for diagnosis and prognosis - a living sys...COVID-19 related prediction models for diagnosis and prognosis - a living sys...
COVID-19 related prediction models for diagnosis and prognosis - a living sys...Maarten van Smeden
 
Measurement error in medical research
Measurement error in medical researchMeasurement error in medical research
Measurement error in medical researchMaarten van Smeden
 
The basics of prediction modeling
The basics of prediction modeling The basics of prediction modeling
The basics of prediction modeling Maarten van Smeden
 
The absence of a gold standard: a measurement error problem
The absence of a gold standard: a measurement error problemThe absence of a gold standard: a measurement error problem
The absence of a gold standard: a measurement error problemMaarten van Smeden
 

Mehr von Maarten van Smeden (20)

A gentle introduction to AI for medicine
A gentle introduction to AI for medicineA gentle introduction to AI for medicine
A gentle introduction to AI for medicine
 
Associate professor lecture
Associate professor lectureAssociate professor lecture
Associate professor lecture
 
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...
Shrinkage in medical prediction: the poor man’s solution for an inadequate sa...
 
Predictimands
PredictimandsPredictimands
Predictimands
 
Prognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient healthPrognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient health
 
Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?
 
Clinical prediction models for covid-19: alarming results from a living syste...
Clinical prediction models for covid-19: alarming results from a living syste...Clinical prediction models for covid-19: alarming results from a living syste...
Clinical prediction models for covid-19: alarming results from a living syste...
 
Five questions about artificial intelligence
Five questions about artificial intelligenceFive questions about artificial intelligence
Five questions about artificial intelligence
 
Development and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutionsDevelopment and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutions
 
Prediction models for diagnosis and prognosis related to COVID-19
Prediction models for diagnosis and prognosis related to COVID-19Prediction models for diagnosis and prognosis related to COVID-19
Prediction models for diagnosis and prognosis related to COVID-19
 
Clinical prediction models: development, validation and beyond
Clinical prediction models:development, validation and beyondClinical prediction models:development, validation and beyond
Clinical prediction models: development, validation and beyond
 
Correcting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confoundingCorrecting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confounding
 
Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead
 
Living systematic reviews: now and in the future
Living systematic reviews: now and in the futureLiving systematic reviews: now and in the future
Living systematic reviews: now and in the future
 
Voorspelmodellen en COVID-19
Voorspelmodellen en COVID-19Voorspelmodellen en COVID-19
Voorspelmodellen en COVID-19
 
The statistics of the coronavirus
The statistics of the coronavirusThe statistics of the coronavirus
The statistics of the coronavirus
 
COVID-19 related prediction models for diagnosis and prognosis - a living sys...
COVID-19 related prediction models for diagnosis and prognosis - a living sys...COVID-19 related prediction models for diagnosis and prognosis - a living sys...
COVID-19 related prediction models for diagnosis and prognosis - a living sys...
 
Measurement error in medical research
Measurement error in medical researchMeasurement error in medical research
Measurement error in medical research
 
The basics of prediction modeling
The basics of prediction modeling The basics of prediction modeling
The basics of prediction modeling
 
The absence of a gold standard: a measurement error problem
The absence of a gold standard: a measurement error problemThe absence of a gold standard: a measurement error problem
The absence of a gold standard: a measurement error problem
 

Último

Development of a Questionnaire for Identifying Personal Values in Driving
Development of a Questionnaire for Identifying Personal Values in DrivingDevelopment of a Questionnaire for Identifying Personal Values in Driving
Development of a Questionnaire for Identifying Personal Values in Drivingstudiotelon
 
Basics Of Computers | The Computer System
Basics Of Computers | The Computer SystemBasics Of Computers | The Computer System
Basics Of Computers | The Computer SystemNehaRohtagi1
 
Phagocytosis Pinocytosis detail presentation
Phagocytosis Pinocytosis detail presentationPhagocytosis Pinocytosis detail presentation
Phagocytosis Pinocytosis detail presentationdhaduknevil1
 
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITY
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITYRHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITY
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITYDnyandaBopche
 
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...dkNET
 
Geometric New Earth, Solarsystem, projection
Geometric New Earth, Solarsystem, projectionGeometric New Earth, Solarsystem, projection
Geometric New Earth, Solarsystem, projectionWim van Es
 
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERIC
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERICROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERIC
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERICsnehalraut2002
 
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...ORAU
 
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...Naomi Baes
 
Advance pharmacology presentation..............
Advance pharmacology presentation..............Advance pharmacology presentation..............
Advance pharmacology presentation..............SIMRAN VERMA
 
structure of proteins and its type I PPT
structure of proteins and its type I PPTstructure of proteins and its type I PPT
structure of proteins and its type I PPTvishalbhati28
 
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...Ajay kamboj
 
Introduction to Green chemistry ppt.pptx
Introduction to Green chemistry ppt.pptxIntroduction to Green chemistry ppt.pptx
Introduction to Green chemistry ppt.pptxMuskan219429
 
Production of super male Tilapia (Sex reversal techniques).pptx
Production of super male Tilapia (Sex reversal techniques).pptxProduction of super male Tilapia (Sex reversal techniques).pptx
Production of super male Tilapia (Sex reversal techniques).pptxAKSHAY MANDAL
 
Theory of indicators: Ostwald's and Quinonoid theories
Theory of indicators: Ostwald's and Quinonoid theoriesTheory of indicators: Ostwald's and Quinonoid theories
Theory of indicators: Ostwald's and Quinonoid theoriesChimwemweGladysBanda
 
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...klada0003
 
Advances in Oncolytic Virotherapy - Creative Biolabs
Advances in Oncolytic Virotherapy - Creative BiolabsAdvances in Oncolytic Virotherapy - Creative Biolabs
Advances in Oncolytic Virotherapy - Creative BiolabsCreative-Biolabs
 
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...ORAU
 
Project report on Fasciola hepatica.docx
Project report on Fasciola hepatica.docxProject report on Fasciola hepatica.docx
Project report on Fasciola hepatica.docxpriyanshimanchanda4
 
Solid waste management_13_409_U1_2024.pptx
Solid waste management_13_409_U1_2024.pptxSolid waste management_13_409_U1_2024.pptx
Solid waste management_13_409_U1_2024.pptxkrishuchavda31032003
 

Último (20)

Development of a Questionnaire for Identifying Personal Values in Driving
Development of a Questionnaire for Identifying Personal Values in DrivingDevelopment of a Questionnaire for Identifying Personal Values in Driving
Development of a Questionnaire for Identifying Personal Values in Driving
 
Basics Of Computers | The Computer System
Basics Of Computers | The Computer SystemBasics Of Computers | The Computer System
Basics Of Computers | The Computer System
 
Phagocytosis Pinocytosis detail presentation
Phagocytosis Pinocytosis detail presentationPhagocytosis Pinocytosis detail presentation
Phagocytosis Pinocytosis detail presentation
 
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITY
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITYRHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITY
RHEOLOGY MODIFIERS: ENHANCING PERFORMANCE AND FUNCTIONALITY
 
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...
dkNET Webinar "The Multi-Omic Response to Exercise Training Across Rat Tissue...
 
Geometric New Earth, Solarsystem, projection
Geometric New Earth, Solarsystem, projectionGeometric New Earth, Solarsystem, projection
Geometric New Earth, Solarsystem, projection
 
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERIC
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERICROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERIC
ROLE OF HERBS IN COSMETIC SKIN CARE: ALOE AND TURMERIC
 
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...
Non equilibrium Molecular Simulations of Polymers under Flow Saving Energy th...
 
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...
Naomi Baes's PhD Confirmation Presentation: A Multidimensional Framework for ...
 
Advance pharmacology presentation..............
Advance pharmacology presentation..............Advance pharmacology presentation..............
Advance pharmacology presentation..............
 
structure of proteins and its type I PPT
structure of proteins and its type I PPTstructure of proteins and its type I PPT
structure of proteins and its type I PPT
 
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...
INFLUENCE OF PREHARVEST PRACTICES, ENZYMATIC AND TEXTURAL CHANGES, RESPIRATIO...
 
Introduction to Green chemistry ppt.pptx
Introduction to Green chemistry ppt.pptxIntroduction to Green chemistry ppt.pptx
Introduction to Green chemistry ppt.pptx
 
Production of super male Tilapia (Sex reversal techniques).pptx
Production of super male Tilapia (Sex reversal techniques).pptxProduction of super male Tilapia (Sex reversal techniques).pptx
Production of super male Tilapia (Sex reversal techniques).pptx
 
Theory of indicators: Ostwald's and Quinonoid theories
Theory of indicators: Ostwald's and Quinonoid theoriesTheory of indicators: Ostwald's and Quinonoid theories
Theory of indicators: Ostwald's and Quinonoid theories
 
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...
1David Andress - The Oxford Handbook of the French Revolution-Oxford Universi...
 
Advances in Oncolytic Virotherapy - Creative Biolabs
Advances in Oncolytic Virotherapy - Creative BiolabsAdvances in Oncolytic Virotherapy - Creative Biolabs
Advances in Oncolytic Virotherapy - Creative Biolabs
 
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...
Mining Data for Ore Natural Language Processing to Identify Lithium Minerals ...
 
Project report on Fasciola hepatica.docx
Project report on Fasciola hepatica.docxProject report on Fasciola hepatica.docx
Project report on Fasciola hepatica.docx
 
Solid waste management_13_409_U1_2024.pptx
Solid waste management_13_409_U1_2024.pptxSolid waste management_13_409_U1_2024.pptx
Solid waste management_13_409_U1_2024.pptx
 

Absence of a gold standard in diagnostic test accuracy research

  • 1. Absence of a gold standard in diagnostic test accuracy research
 with application in context of childhood TB Maarten van Smeden, PhD Post-doctoral researcher Julius Center for Health Sciences and Primary Care WEON 2017 Pre-conference Accounting for Measurement Error in Epidemiology Antwerp, June 7, 2017
  • 2. Outline • Diagnostic test accuracy • The problem: absence of a gold standard • Possible solution: latent class analysis in context of TB
  • 6. Diagnostic testing • “New test better than the existing test(s)?” • “(Where to) add new test to diagnostic pathway?” • “Recommend new test in practice guidelines?” Fig from: Bossuyt, BMJ, 2006
  • 7. Diagnostic test accuracy studies (DTA) • Evaluation of “new” diagnostic tests (=index test) by comparison to a “gold standard” • Misclassification probabilities of index test: sensitivity, specificity, negative/positive predictive values, etc.
  • 8. Classical DTA analysis Subjects undergo the index test (T) and gold standard test (GS) GS + GS - T + A C T - B D
  • 9. Classical DTA analysis Sensitivity (Se) = A/(A+B)
 Specificity (Sp) = D/(D+C) GS + GS - T + A C T - B D
  • 11. Reporting guideline: STARD “.. a gold standard would be an error-free reference standard”
  • 12. All that glitters is not gold • Commonly the best available reference standard: Se < 1 and Sp < 1: not a “gold standard”. 
 
 Because:
 detection limits (e.g. culture), infeasible/not ethical to execute in some patients (e.g. biopsy), observer errors (e.g. MRI), etc.
  • 13. All that glitters is not gold • Commonly the best available reference standard: Se < 1 and Sp < 1: not a “gold standard”. 
 
 -> misclassifications of the target condition by the reference standard (= measurement error) 

  • 14. When using imperfect reference standard Assuming: reference standard Se = 1, index test Sp = Se = 0.7, conditional independence reference standard and index test
 0.5 0.6 0.7 0.8 0.9 1.0 Specificity Reference Standard E[SenstivityIndexTest] Disease prevalence = 0.05 Disease prevalence = 0.25 Disease prevalence = 0.50 0.3 0.4 0.5 0.6 0.7
  • 15. When using imperfect reference standard • Bias, sometimes called “reference standard bias”. Not necessarily a lower bound of Se/Sp
 
 • Philosophical problems when index test is believed to be more accurate than the best available reference standard
  • 16. When using imperfect reference standard Absence of a gold standard Misclassifications by the reference standard -> 
 no straightforward approaches to estimation of misclassification probabilities of index tests (that are valid)
  • 18. Tuberculosis (TB) Paulsen, Nature, 2013 ■ FIGURE 2.16a Top causes of death worldwide in 2012.a,b Deaths from TB among HIV-positive people are shown in grey.c Road injury HIV/AIDS Diabetes mellitus Diarrheal diseases Tracheal, bronchus, lung cancers TB Chronic obstructive pulmonary disease Lower respiratory infections Stroke Ischaemic heart disease 0 1 2 3 4 5 6 7 Millions ■ F Est 20 in g a This is the latest year for which estimates for all causes are currently available. See WHO Global Health Observatory data repository, available at http://apps.who.int/gho/data/node.main.GHECOD (accessed 27 August 2015). b For HIV/AIDS, the latest estimates of the number of deaths in 2012 a F t o b i b D d HIV WPR 9.2 8.3–10.0 0.29 Global 35.2 30.9–39.4 8.4 WHO Global TB report 2015
  • 19. Data • 749 hospitalised children with suspected pulmonary TB in Cape Town, South Africa • Study procedures, a number of tests for TB for each subject: • Microscopy • Culture • Xpert (NAAT) • TST (skin test) • Radiography
  • 23. • The idea: Simple latent class model Pr(T = 1) = ⇡Se + (1 ⇡)(1 Sp) = Pr(D = 1)Pr(T = 1|D = 1)+ Pr(D = 0)Pr(T = 1|D = 0)
  • 24. • With two conditionally independent binary tests (T0 and T1) Simple latent class model Pr(T0 = 1, T1 = 1) = ⇡Se0Se1+ (1 ⇡)(1 Sp0)(1 Sp1)
  • 25. • With J conditionally independent tests (and bit of algebra): Simple latent class model Pr(T1, . . . , TJ ) = ⇡ JY j=1 Se Tj j (1 Sej)1 Tj + (1 ⇡) JY j=1 Sp 1 Tj j (1 Spj)Tj
  • 26. Latent class model estimation • Maximum likelihood • Gibbs sampling
  • 28. Heuristic model for TB data • Conditional independence between all tests is unlikely • Conditional dependence between: Xpert, culture, microscopy, and TST among TB diseased due to “bacterial load” • Bacterial load modelled by a random effect
  • 30. Pairwise correlation residual (misfit) Conditional independence model Random effects model
  • 31. Main results Conditional independence model Random effects model
  • 32. Is latent class analysis useful? • In TB example, I believe: yes • More realistic than assuming reference standard (culture) has Se = Sp = 1 • Results ‘robust’ to changing prior distributions and conditional dependence structure • Lack of robust alternative approaches for DTA in the absence of a gold standard
  • 33. Is latent class analysis useful? • But: • Latent class analysis for DTA is still rare
  • 34. Latent class analysis in diagnostic research Systematic review from 2014 • 69 theoretical papers • 64 applied papers in human research + 47 in veterinary sciences • applications of LCA still not common in human diagnostic research van Smeden, AJE, 2014
  • 35. Is latent class analysis useful? • But: • Latent class analysis for DTA is still rare • Robustness to misspecification of the conditional dependence structure is a concern
  • 37. Is latent class analysis useful? • But: • Latent class analysis for DTA is still rare • Robustness to misspecification of the conditional dependence structure is a concern • Identifiability requirements
  • 38. Why Bayesian? • Practical arguments: • Model specifications in non-commercial software packages (e.g. randomLCA vs rjags in R) • (Weakly) informative prior distributions can solve non- identifiability problems • Additional calculations (e.g. positive/negative predictive values with CrI)
  • 39. Final remarks • Misclassification in DTA studies is often both the primary topic of study (for the index test) and the problem (when occurring in the reference standard) • Model based estimation of index test accuracy by latent class analysis can be useful • There is some evidence that robustness of the latent class model can be improved when disease status can be verified with certainty in a subset • While the focus of this talk was on DTA, other studies such as “incremental value” studies suffer from the same problems
  • 40. Acknowledgements Thanks to all co-authors in: Supported by a grant from Canadian Institutes of Health Research (MOP #89857)