Screening

1. Validity and Reliability of
Diagnostic and Screening Tests
EPI 504D
Week 8
1

2. Screening vs. diagnostic tests
! Diagnos(c
test
establishes
or
rules
out
disease
in
individuals
with
abnormal
signs
or
symptoms.
! Screening
test
establishes
or
rules
out
disease
in
individuals
that
appear
healthy,
but
may
be
at
risk
! Usually
persons
with
abnormal
screening
test
would
then
undergo
diagnos:c
tes:ng
2

3. Two-Step Process
3
Screening test
Positive
Negative
Diagnostic
test
+
-
Disease
No disease

4. Part
I:
Accuracy
of
screening
and
diagnos(c
tests
4

5. Fundamental truth about tests
Error
is
expected
5

6. Four possible outcomes of a test
! True positive (TP)
! True negative (TN)
! False negative (FN)
! False positive (FP)
6

7. 7

8. 8

9. Sensi:vity
(Se)
=
a/a+c
=
TP/(TP+FN)
Specificity
(Sp)
=
d/b+d
=
TN/(FP+TN)
=
Posi:ve
Predic:ve
Value
(PPV)
=
a/a+b
=
TP/(TP+FP)
Nega:ve
Predic:ve
Value
(NPV)
=
d/c+d
=
TN/(TN+FN)
9

10. 10

11. 11

12. 12

13. 13

14. 14

15. 15

16. Trade-offs between Se and Sp
Example: diabetes screening
16*two
hours
aJer
a
meal

17. • Se
can
be
increased
at
the
expense
of
Sp
• If
cut-­‐off
is
180
we
will
miss
61.4%
of
the
cases
(high
Sp
at
the
expense
of
Se)
• Using
70
as
the
cut-­‐off
we
will
iden:fy
virtually
all
cases
(Se=
98.6%)
but
will
end
up
with
a
lot
of
false
posi:ves
(Sp
=8.8%)
17
Trade-offs between Se and Sp
Example: diabetes screening
*two
hours
aJer
a
meal

18. Receiver
opera:ng
characteris:cs
(ROC)
curve
18Adapted from : gim.unmc.edu/dxtests/roc3.htm

19. Diabetes screening example: ROC curve
19
*two
hours
aJer
a
meal
1-Specificity
Sensitivity

20. ! Sensi:vity
and
Specificity
are
aZributes
of
test
accuracy
! PPV
and
NPV
are
aZributes
of
both
test
accuracy
and
disease
prevalence
20

21. PPV
(%)
by
Prevalence
at
Selected
Levels
of
Se
and
Sp
Disease
Prevalence
(%)
Se
Sp
0.5
1
2
5
10
50
50
0.5
1
2
5
10
50
90
2
5
9
21
36
75
50
0.7
1
3
7
14
90
95
8
15
27
49
67
21

22. Part
II:
Reliability
of
screening
and
diagnos:c
tests
22

23. Two
radiologists
are
evalua:ng
200
x-­‐rays
looking
for
evidence
of
pneumoconiosis
(chronic
occupa:onal
lung
disease)
23
Normal lungs Mild disease Severe disease
In some cases both radiologists diagnosed pneumoconiosis
In some cases both radiologists found no evidence of disease
In some cases radiologist #1 diagnosed disease but radiologist #2 did not
In some cases radiologist #2 diagnosed disease but radiologist #1 did not
How do we measure agreement between these two observers?

24. Method
1:
Percent
Agreement
24
Percent
agreement
=
(a
+
d)
/
(a+b+c+d)

25. 25
Percent
agreement
=
??

26. Percent
Agreement
• Problem:
Two
observers
will
agree
on
a
certain
propor:on
of
subjects
simply
by
coincidence
(chance)
• Ques:on:
To
what
extent
would
two
observers
agree
beyond
what
is
expected
by
chance?
26

27. Kappa
Sta:s:c
“Kappa
expresses
the
extent
to
which
the
observed
agreement
exceeds
that
which
would
be
expected
by
chance
alone
(numerator)
rela:ve
to
the
most
that
the
observers
could
hope
to
improve
their
agreement
(denominator)”
(Gordis,
4th
ed.
P.
104)
27

28. 28
Observed
agreement
=
150/200=75%
Agreement
expected
by
chance
=
??
NOTE: Expected numbers in each cell of a 2×2 table are obtained as

29. 29
Observed
agreement
=
150/200=75%
Agreement
expected
by
chance
=
??
KAPPA
=
??

30. Interpreta:on
of
Kappa*
>
0.75
“Excellent”
agreement
0.40
–
0.75
“Intermediate”
to
“Good”
<
0.40
“Poor”
agreement
*note:
other
interpreta:ons
exist
30

31. Part
III:
Evalua:on
of
Screening
Programs:
Methodological
Issues
31

32. Length
bias
• Slowly
progressing
diseases
(e.g.
certain
cancers)
have
more
opportunity
to
be
detected
by
screening,
AND
• Slowly
progressing
diseases
take
longer
to
lead
to
death
than
faster
ones
• BeZer
survival
of
people
with
screen-­‐detected
disease
will
be
ar:ficially
increased
32

33. 33

34. Lead
:me
bias
• Without
screening
survival
is
the
:me
between
clinical
diagnosis
and
death
• If
screened,
survival
is
the
:me
between
posi:ve
screening
test
and
death
• If
screening
results
in
earlier
detec:on
but
not
in
decreased
mortality,
there
may
be
false
appearance
of
increased
survival
34

35. Lead time bias
35

36. Overdiagnosis
• Of
par:cular
interest
in
cancer
screening
• Screening
detects
lesions
that
would
not
become
clinically
apparent
otherwise
• Large
number
of
prevalent
cancers
at
the
start
of
screening
program
• Classic
example:
PSA
test
for
prostate
cancer
36

37. 37

38. Solu:on
against
various
methodological
problems:
randomized
trial
38
Enrolled patients
Randomization
Screened Not screened
Endpoint (e.g., mortality) Endpoint (e.g., mortality)

39. When
to
screen?
– When
disease
is
an
important
problem
– When
natural
history
of
disease
presents
a
window
of
opportunity
for
early
detec:on
AND
treatment
– When
cost
of
screening,
diagnosis
and
treatment
is
favorably
weighed
against
treatment
costs
at
usual
:me
of
diagnosis
– When
a
suitable
screening
test
is
available
39

40. What
is
a
suitable
screening
test?
A
test
that
is
– Accurate
(Se,
Sp,
PPV)
– Acceptable
to
the
popula:on
– Safe
– Inexpensive
– Iden:fies
cases
that
would
benefit
from
early
detec:on
and
treatment
40

41. Ethical
considera:ons
• Informed
consent
for
tes:ng
and
diagnos:c
follow
up
• Considera:on
of
risks
of
screening
–
physical
and
psychological
• Availability
of
the
screening
test
to
all
those
in
the
target
popula:on
• Ensuring
adequate
diagnos:c
resources
to
evaluate
all
posi:ves
in
a
:mely
manner
41

42. Summary
• Validity
of
diagnos:c
and
screening
tests
can
be
measured
by
sensi:vity,
specificity,
PPV
and
NPV
– These
measures
are
inter-­‐related
– Disease
prevalence
will
affect
predic:ve
value
• Reliability
of
tests
can
be
assessed
by
percent
agreement
or
kappa
• Screening
is
not
always
appropriate,
nor
is
it
always
beneficial
42