Journal appraisal as a tool to learn about validation

1. Evaluating a clinical tool
Dr Tan Chai Eng
17 June 2014
PPP Morning read

2. Title of the article
• What does the title tell you?
– Validation study
– Malay version for Malaysian patients
– Berlin questionnaire
– Obstructive Sleep Apnoea

3. Who are the authors?
• Respiratory physician from IPR, MOH
• Biostatistician from CRC
• Epidemiologists from CRC
• Why is this important?
– Credibility of the article
– Appropriate expertise in the study

4. What do you know from the abstract?
• Objective stated clearly
• Justification of study
• Methods used: back-to-back translation, study
population in tertiary respiratory medicine
centre, concurrent validation – AHI as gold
standard, test-retest reliability, internal
consistency of Berlin-M
• Results: sensitivity 92%, specificity 17%.
Cronbach alpha 0.75-0.89
• Conclusion: good screening tool, not
confirmatory

5. Introduction
• Prevalence of OSA
• Comorbidities of OSA – why it’s important to
diagnose OSA
• Gold standard for diagnosing OSA
• Screening tools for OSA
• Doesn’t explain adequately why they want to
validate BQ instead of ESS
• Doesn’t describe limitations of ESS
• Doesn’t explain why need Malay tool

6. Materials and Methods
• Phase 1 (translation process)
Parallel forward translation
Clinician (M1), certified translator (M2)
Independently back-translated (E1, E2)
Expert panel review – harmonised version
• Phase 2 (pre-test)
7 subjects for pre-test – How many required?
Given questionnaires, followed by focus group
discussion
Clear, relevant and comprehensible
Linguistic
validation

7. Linguistic validation
• Need to produce a tool in local language with
semantic and conceptual equivalence.
• E.g. I tend to drop off to sleep while driving
 Saya cenderung untuk jatuh ke tidur semasa
memandu
Saya senang tertidur semasa memandu
Important to ensure cross-cultural equivalence!

8. Linguistic validation
• Important to get the comments from
participants regarding comprehensibility of
the Malay version
• Can be affected by the variety of patients
approached for pre-test (literacy level, cultural
background)
• If any further ammendments made, should
repeat the pre-testing

9. Materials and methods
• Phase 3 – validation
Study population 150 patients in IPR who were
referred for overnight PSG
• How will this affect the results?
• How does the inclusion and exclusion criteria affect the
results?
Convenience sampling
• Does this affect the results?
Type of validation – predictive validity

10. Reliability
• Test-retest reliability
– Check the results for same patient at different
point of time
• Internal consistency reliability
– Cronbach alpha to see the internal consistency of
each scale
– Category 1: Items 2,3 and 5
– Category 2: Items 6, 7, 9

13. Criterion validity
• Predictive validity – how well does the tool
predict the results of another tool?
Sensitivity and specificity based on comparison with
another gold-standard tool
Overnight
polysomnography
AHI >5
Berlin-M

14. Sensitivity
The sensitivity of a test in the ability of the test to identify
correctly affected individuals
Proportion of persons testing positive among affected individuals
Affected persons
(Positive by gold standard)
Persons testing positive
(True positives)
Persons testing negative
(False negatives)
Sensitivity = True positives / Affected persons
Estimate the 95% confidence interval
Slide from WHO Laboratory Training for Field Epidemiologists

15. Specificity
The specificity of a test in the ability of the test to identify
correctly non-affected individuals
Proportion of person testing negative among non affected individuals
Non-affected persons
(Negative by gold standard)
Persons testing negative
(True negatives)
Persons testing positive
(False positives)
Specificity = True negatives / Non-affected persons
Estimate the 95% confidence interval
Slide from WHO Laboratory Training for Field Epidemiologists

16. So how good is this tool?
Sensitivity 0.92
Specificity 0.17
Positive predictive value 0.97
Negative predictive value 0.29
Not
affected by
prevalence Affected by
prevalence,
higher in
high
prevalence
population

17. What are the limitations of this study?
• High number of patients with OSA based on
AHI – good for calculating sensitivity, but not
so good for specificity
• Excluded patients with conditions that have
similar symptoms to OSA, may affect the
sensitivity
• May not be good for those with poor literacy
of Malay language

18. Conclusion: Would you use this tool on
your patients?
• For screening to determine the need for
referral for PSG
• Malay speaking patients
• Not for diagnosis of OSA because of poor
specificity

19. Thank you