Statistics False Positive

1. False Positives in Statistics
by Professor Haller

2. Reading of Statistical Results Arise - Puzzles and
Paradoxes are Common
• We observe incorrectly
• We misinterpret
• We misapply
Accurate statistical processing of any data benefits from fellow statisticians to
• Validate
• Verify
• Vindicate

3. Interpreting
1 Chance in 10 of
truly being
infected. Get
retested!
Results
AIDS Test Low
Risk Population
Test
1 in 10,000
Test Positive
10 Might Have AIDS
True Positive
1 Truly is Infected
False Positive
9 of 10 Are Not
Infected
Test Negative
9990
True Negative
9990 Are Not
Infected
False Negative
0 Had a False
Reading and are
infected
The Drunkard’s Walk by Leonard
Mlodinow pages 114-116

4. The Drunkard’s
Walk by Leonard
Mlodinow pages
114-116
Test
1 in 10,000
• Test Positive
10 Might Have AIDS
• True Positive
1 Truly is Infected
• False Positive
9 of 10 Are Not
Infected
Not Infected
9 Chance Out of
10
90% Chance
Not Infected

5. Personal Application
Should I seek a
Retesting
when a test
gives bad
results?
YES!

7. DNA TESTING

8. Prosecutors Fallacy – Test A Large
Enough Population
Interpreting
Results
DNA Testing
Test A Very Large
Population 500,000
Test Positive
10
True Positive
1 True Match
False Positive
9 Match Due to Lab
Error - Intentional or
Unintentional
Test Negative
499,990
True Negative
499,989 Fail to Match
False Negative
1 Had a False Reading
and are a true reading
or else was never in the
database

9. How to Interpret
• Out of a population of 500,000 we have narrowed the pool of possible
candidates to:
– 1 True Positive
– 9 False Positives ( Due to contaminated sample or lab personnel misreads
unintentionally or intentionally)
– 1 False Negative ( Lab or test misses a match or DNA of true culprit not in the
database)
• Total candidates 11
• Based on DNA alone what is the probability the prosecutor has the true
suspect = 1 chance out of 11 or about 10%.
• Conclusion: One cannot rely on DNA testing alone. Use double blind
testing in case one lab gets it wrong either unintentional or intentionally.

11. Sometimes Labs Get It Wrong Due to Human Error or Even Sadly Intent

12. Trial by
Jury
Null Hypothesis is True Ho =
Person is Not Guilty
Null Hypothesis is false Ha
= the Person is Guilty
False Positive - Type I Error is the
mistake of rejecting the null
hypothesis when it is true. A truly
not guilty person is sent to prison
True Negative: Correct to
reject the null hypothesis.
Person is deemed guilty and is
sent to prison.
True Positive: Correct to not reject
the null hypothesis. Person is
deemed Not Guilty and is released
False Negative - Type II Error is
the mistake of failing to reject
the null hypothesis when it is
false.
A truly guilty person is released
declared not guilty.
The aim is to avoid a
truly not guilty
person going to
prison

13. Misapplication of Statistics in Law
• OJ Simpson Trial
– How many woman report battering by a husband or spouse each
year? 4,000,000.
– How many homicides were of women during the year? 1600.
– Conclusion given wife abuse the chance of a homicide is
1600/4,000,000 = .0004 or .04% Right? No wrong framing of the
question!
• Correct question
– Given a woman is a victim of a homicide what is the likelihood it done
by the spouse or boyfriend? ≈ 90%
– Wrong Conclusion: Avoid having a boy friend or husband if you wish to
have a greater chance of a longer life?
– Right Conclusion: Take a very close look at her boy friend or husband.
If no evidence, then look for a socio pathetic stranger on the loose.
This proved true in the Michael Morton case.

14. Mis Application of Statistics in Law
• Sally Clark case – Sudden Infant Death Syndrome of two infants
– Both cases random. One in 8500 newborns dies of SIDS. Twice would be one chance in 73
million.
– One could be random and one infanticide.
– Both infanticide
• Wrong assumptions led to her being accused of a double homicide and conviction.
– It wrongly assumed independent events. SIDS is more common to be experienced in other
future births. There is a correlation. There are conditional events. It was not compared against
all the variety of infant deaths attributed to parental causes such as suffocation as to a
percentage.
• Later statistical studies indicated successive accidents are between 4.5 and 9 times
more likely than are successive murders, so that the a priori odds of Clark's guilt
were between 4.5 to 1 and 9 to 1 against. This freed Ms Clark. By then she was so
distraught she drank too much and passed away due to her alcoholism.

15. Problems Associated with Being an Expert
• People who are incompetent make frequent,
largely unimportant errors, and that is
understandable. They are largely unimportant
errors because people who are incompetent
rarely get into positions of power.
• Those who are highly competent are susceptible
to rare, but hugely significant errors.
• Misscalibration errors are especially difficult to
catch because they are made by domain experts
who are hard to argue with.
Malcolm Gladwell noted Writer and Speaker

16. Examples in History
• General Joe Hooker overconfident at Chancellorsville
• Robert Lee overconfident at Gettysburg
• Hitler overconfident in invasion of Russia
• Goldman Sacs 1929 prior to the crash overconfident. Their
last lawsuit settled in 1969.
• Bears Sterns and Lehman Brothers collapse in 2007 led by
CEOs with immense hubris and overconfidence.
• The owners believed they had perfect information and
were in control of their environment. They were blinded to
the existence of outliers. They got overconfident.
• Bush’s security team overconfident about Weapons of
Mass Destruction

17. Examples of Miscalibration
Story of LTCM and most recently MF Global
Sudden Wealth Destruction
Long Term Capital Management 1998 MF Global 2011

18. Miscalibration
• Self confidence runs way ahead ahead of
information at hand and possible alternatives like
unknown unknowns.
• Given more information how much better does
expert judgment get? It rises only marginally.
• The user over values the extra research. Result
self delusion.
• Calibration a difference between how good you
think you are (confident in perception and
judgment) and what you really are (reality or
truth or fact).

19. 0
1
2
3
4
5
6
7
8
More
Data
>
Confidence
Accuracy
Experts suffer a gap known as miscalibration. Their confidence out runs
the amount the data they have been given. They lose a perspective they
could be wrong.
This over confidence leads to not exiting the market when prudent. It leads
to assuming too much as a general. More data and more data does not
necessarily improve judgment. In fact it can lead to impaired judgment.
Miscalibration

20. Application
• How much should I trust an expert?
• Are they ever 100% accurate?
• If they get it wrong will there be unlivable
or terrible consequences to deal with?

21. False Positives in Statistics
by Professor Haller