Decision-making tools for clinicians

1. Clinical Prediction Rules
Jen-Hsiang Chuang, MD, MS, PhD
Centers for Disease Control Taiwan
jhchuang@cdc.gov.tw
1

2. Clinical Prediction Rules (CPRs)
• Synonym: clinical decision rules
• Definition: decision-making tools for
clinicians including 3 or more variables
– Provide the probability of an outcome
– Suggest a diagnostic or therapeutic course of
action
Laupacis A, et al. Clinical prediction rules. JAMA 1997;277:488-494. 2

3. Clinical Prediction Rules Vs.
Clinical Practice Guidelines
• Clinical prediction rules
– Derived from original research involving many
patients and mathematical analysis
• Clinical practice guidelines
– Consensus among experts
– GOBSAT (Good Old Boys Sat At Table)
(Miller J, et al. Lancet 2000;355:82-3)
– But can include CPRs
3

4. Functions of CPRs
• CPRs help clinicians cope with uncertainty
and improve efficiency
– Cope with uncertainty
• Community-acquired pneumonia (Fine MJ,
et al. NEJM 1997;336:243-250)
– Improve efficiency
• Ottawa Ankle Rules for the use of
radiography (Stiell IG, et al. Ann Emerg
Med 1992;21:384-90)
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 4

5. Prototype of a CPR for
Predicting Death
Predictor variables Score
Age > 75 yr 6
Severe pain 10
Emergency 5
Total points 0-21
Interpretation of the score
High risk: > 6 points (30% deaths) -> aggressive Tx
Low risk: ≤ 6 points (3% deaths) -> conservative Tx
Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 5

6. Three Stages in the
Evaluation of a CPR
1. Development of a CPR
2. Prospective validation of a CPR
3. Impact analysis of a CPR
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 6

7. So What?
• Q: “Give me the reasons why I need to stay
here to listen your presentation?”
• A: a medical informatician may play two
roles
– Reader role
– Developer role
7

8. Development of a CPR
8

9. Checklist of Standards for
Development of a CPR
1. Definition of outcome
2. Definition of predictor variables
3. Reliability of predictor variables
4. Selection of subjects
5. Sample size
6. Mathematical techniques
7. Sensibility of CPR
8. Accuracy
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 9

10. 1. Definition of Outcome
• Clearly defined and clinically important
– Explicit criteria for diagnosis
– Biologic better than behavioral outcome
• Blind assessment of outcome
– More important for a “soft” outcome
– Less important for a “hard” outcome
10

11. 2. Definition of Predictor Variables
• Clearly defined
– Best: collected prospectively, specifically
– Less good: collected prospectively as part of
another study
– Worst: collected from retrospective review of
records
• Blind assessment of predictor variables
11

12. 3. Reliability of Predictor Variables
• Only reliable variables be included
– Intraobserver reliability
– Interobserver reliability
• Measurement of reliability
– Dichotomous or nominal data: κ
– Ordinal data: weighted κ
– Continuous data: intraclass correlation
coefficient
http://www.dmi.columbia.edu/homepages/chuangj/kappa/
12

13. 4. Selection of Subjects
• Patient characteristics stated
– Inclusion and exclusion criteria
– Method of selection
– Clinical and demographic characteristics
• Study site described
– Type of institution (primary, secondary,
tertiary)
– Setting (clinic, ER, hospital ward)
– Teaching or non-teaching
13

14. 5. Sample Size
• Overfitting problem
– Too few outcome events per predictor variable
• Appropriate sample size
– Rule of thumb: at least 10 outcome events per
independent variable
– e.g., 3 findings to predict death => at least 30
patients died
14

15. 6. Mathematical Techniques
• Mathematical methods adequately described and justified
– Multivariate analysis
• Logistic regression
• Discriminant analysis
– Machine learning
• Recursive partitioning (including decision tree
learning)
• Neural networks
– Survival analysis (survival data only)
• Cox's Proportional Hazard Model
15

16. Multivariate Analysis
• General model
• Logistic regression
– Where P is probability of outcome; G is log odds of
outcome
• Discriminant analysis
– Compute cutoff (C)
– Assign patient to class 1 if G < C; otherwise assign
patient to class 2
nn xbxbxbbG ++++= 22110
P
P
G
−
=
1
ln
16

17. Logistic Regression Vs.
Discriminant Analysis
• Logistic regression is much more popular than
discriminant analysis (Concato et, al. 1993)
– Logistic regression
• Binary outcome
• Estimate individual risk and odds ratios
– Discriminant analysis
• Categorical outcome
• Optimal performance requires many predictor
variables as continuous data
• Famous application: diagnosis alcoholism
17

18. Recursive Partitioning
• Principle
– Build an empirical tree diagram by repetitively
splitting patient population into smaller and
smaller categories
Yes No
4
5Employed
2
1
Yes
3
No
Age>30
18

19. Recursive Partitioning Vs.
Multivariate Analysis
• Recursive partitioning provides a simpler
classification rule
• Recursive partitioning may identify
nonlinear relationships with outcome event
• Recursive partitioning need greater sample
size
• Logistic regression can estimate individual
risk and odds ratios
Cook EF, et al. J Chron Dis 1984;37:721-31. 19

20. Cross SS, et al. Introduction to neural network. Lancet 1995;346:1075-9.
Neural Networks
20

21. Clinical Applications of
Neural Networks
• Diagnosis
– AMI, Appendicitis, back pain, dementia, STD
• Imaging
– Radiographs, PET, NMR, perfusion scans
• Analysis of wave forms
– ECGs, EEGs
• Outcome prediction
– Recovery from surgery, cancer, liver transplantation
• Identification of pathological specimens
• Genomics
Baxt WG. Application of ANN to clinical medicine. Lancet 1995;346:1135-8.
21

22. Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.
Advantages of Neural Networks
•Multiple partitioning
•Nonlinear partitioning
22

23. Disadvantages of Neural
Networks
• Slow to train
• “Black-boxes”
Kattan MW, et al. ANN for medical classification decisions. Arch Pathol Lab Med 1995;119:672-7.23

24. 7. Sensibility of CPR
• “Sensibility”
– clinically reasonable, easy to use, course of action
described
– judgment
• Clinically reasonable
– Content validity
• Easy to use
– Length of time needed to apply
– Simplicity of interpretation
• Course of action described
24

25. 8. Accuracy of CPR
• Rationale
• Measurement of accuracy
– 2x2 table with sensitivity, specificity, with
respective 95% CIs
– Receiver operator characteristic (ROC) curves
• Statistical validation
– Cross-validation: Training set vs. test set
Wasson JH, et al. Clinical prediction rules. NEJM 1985;313:793-9. 25

26. Classification
Performance of a CPR
Predicted
Outcome
Actual Outcome
Disease No Disease
Disease 74 244
No Disease 0 247
Sensitivity (95% CI): 1.0 (0.95-1.0)
Specificity (95% CI): 0.50 (0.46-0.55)
Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 26

27. Kuo HS, Chuang JH, Tang GJ, et al. Chin Med J (Taipei) 1999;62:673-681.
27

28. Example: Development of a CPR
Ottawa Ankle Rules
Stiell IG, et al. Ann Emerg Med 1992;21:384-90
28

29. Ottawa Ankle Rules
29

30. The Need for an Ankle Rule
• Blunt ankle trauma
– One of the most common injuries in ER
– Less than 15% of patients have fractures
– Physicians used to order radiography for all
ankle injury patients
– 85% negative for fracture
– $500 M annually in North America
– No widely accepted guideline
Stiell IG, et al. Implementation of the Ottawa Ankle Rules. JAMA 1994;271:827-32. 30

31. Study Design
• Objective
– Develop CPR with 100% sensitivity
• Design
– Prospective survey of ED patients over 5 months
• Patient population:
– Setting: Two university hospital EDs in Ottawa
– Inclusion: All acute blunt injuries of ankle
– Exclusion: < 18 y/o, pregnant, referral, etc
• Data collection
– 32 clinical variables collected by 21 trained physicians
before radiography
– 100 patients examined by a 2nd
physician 31

32. Study Design (Cont.)
• Measurements of outcomes
– Radiography interpreted by a radiologist blinded to the
contents of data collection sheets
• No fracture or insignificant fracture
• Clinically significant fracture
• Data analysis
– Variables found to be both strongly associated with a
significant fracture (P < 0.05) and reliable (κ > 0.6)
were analyzed by logistic regression and recursive
partitioning
32

33. Results
• 70 (10.2%) significant malleolar fractures in
689 ankle injury patients
• Univariate analysis: 17 variables were
significantly associated with fractures
• 9 non-reliable variables were further
eliminated
• Logistic regression: Sen: 1.0, Spe: .29
• Recursive partitioning: Sen: 1.0, Spe: .40
33

34. 689
561128
70#
39#
Yes No
31#
67
Yes
12# 494
No
19#
21118#
Yes
283
No
1#
35 248
0#
NoYes
1#
441
High Risk 248 Low Risk
A
B
C
D
LEGEND
# Fracture
A Unable to bear weight
immediately and in ED
B Age 55 or greater
C Bone tenderness B4 or B5
D Bone tenderness B8 or B9
Recursive Partitioning of 689 Cases
34

35. Prospective Validation
of a CPR
35

36. Problems of CPRs With
Statistical Validation Only
• Many statistically derived rules fail to
perform well when tested in a new
population
– Overfitting or instability in the original derived
model
– Differences in prevalence of disease
– Differences in severity of cases
– Differences in how the CPR is applied
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 36

37. Prospective Validation of a CPR
• Validation
– Its repeated application leads to the same
results
• Types of validation
– Narrow validation: application of rule in a
similar setting and population
– Broad validation: application of rule in multiple
clinical settings with varying prevalence and
outcomes of disease
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 37

38. Development of a Clinical
Prediction Rule
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 38

39. Methodological Standards for
Validation of a CPR
• Unbiased, wide spectrum patient population
• Blinded assessment of outcomes and
predictor variables
• Careful follow-up of predicted normal
patients
• Training for correctly applying rules
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 39

40. Results of Validation Studies of
Ottawa Ankle Rules
Markert RJ, et al. Am J Emerg Med 1998;16:564-7.
Country
(Year)
# of
Subjects
Sensitivity
(%)
Specificity
(%)
CA (1993) 1032 100 39
US (1994) 71 100 19
NZ (1994) 350 93 11
US (1994) 631 100 19
US (1995) 422 95 16
40

41. Impact Analysis of a CPR
41

42. Reasons for No Impact
of an Accurate CPR
• Clinician’s intuition may be as good as the
CPR
• Calculations involved may be cumbersome
• Practical barriers to acting on the results of
CPR
– Medical liability risk
– Patient demand factor
McGinn TG, et al. Users’ guide to the medical literature. JAMA 2000;284:79-84. 42

43. Methodological Standards for
Impact Analysis of a CPR
• Study design
– Cluster-based randomized control trial
– Before-after study
• Effect on use
– e.g., ordering of radiography
• Accuracy of rule
• Acceptability of physicians & patients
Stiell IG, et al. Annals Emergency Med 1999;33:437-47. 43

44. Impact Analysis of Ottawa
Ankle Rules in France
• Randomized 5 EDs to use or not use CPR
• 2 in intervention group (906 patients)
– Meeting, pocket cards, posters, and data collection
forms
• 3 in usual care group (1005 patients)
– data collection forms only
• Results: (unit of analysis was physician)
– ordering of radiography: I: 79%; C: 99% (P=.03)
– I: 3/112 missed fractures (incomplete data forms: 2,
rule interpretation error by physician: 1)
Auleley GR, et al. JAMA 1997;277:1935-9. 44

45. Summary
• Development of an effective prediction rule
is a long, rigorous, and expensive process
• Properly developed and validated prediction
rules can influence clinical practice
45

46. Performance Evaluation
• Discrimination
– Ability of a prediction model to separate those
who experience events from those who do not
– Area under a ROC curve (c statistic)
• Calibration
– Measures how closely predicted outcomes
agree with actual outcomes
– Hosmer-Lemeshow goodness-of-fit test
46

47. Hosmer-Lemeshow
Goodness-of-fit Test
47

48. 48

49. Thank You!
49