Stan Young, PhD,. slides from Phil 6334 guest presentation.

1. 1
Abstract
Claims coming from human medical observational studies, when tested rigorously,
most often fail to replicate. Whereas randomized clinical trials replicate over 80%
of the time, medical observational studies replicate only 10 to 20% of the time. Multiple
re-test studies reported JAMA failed to replicate. For example in the early 1990s,
Vitamin E was reported to protect against heart attacks. Large, well-conducted
randomized clinical trials did not replicate this claim. The claim that Type A Personality
leads to heart attacks failed to replicate in two separate studies, yet the myth still lives.
Clearly, there are systematic problems with how observational studies are conducted
and analyzed that need to be identified and fixed. Edwards Deming, the most famous
quality expert ever, says that any problem with a failed process is not the fault of the
workers, scientists conducting observational studies, but of management. Funding
agencies and journal editors need to fix a clearly broken process. Technical problems
are identified. Tough management solution are proposed. A simple statistical analysis
strategy is presented. Many human health problems can only be examined using
observational data. Our proposals, technical and managerial, should lead to more
reliable claims along with fair ways to judge their reliability.
NISS

2. 22
Contact
Information
Stan Young
National Institute of Statistical Sciences
www.niss.org
young@niss.org
919 685 9328
NISS

3. NISS 3
Hayek, 1974, Nobel Lecture
It is often difficult enough for the expert,
and certainly in many instances impossible
for the layman, to distinguish between
legitimate and illegitimate claims advanced
in the name of science.
It is often difficult enough for the expert,
and certainly in many instances impossible
for the layman, to distinguish between
legitimate and illegitimate claims advanced
in the name of science.

4. NISS 4
Hayek (2)
...much effort will have to be directed toward debunking
such arrogations*, some of which have by now become
the vested interests of established university departments.
*Claims without proper foundation

5. 5
Reliability of Literature Claims
S. Stanley Young
National Institute of Statistical Sciences
Young@niss.org, 919 685 9328
VIP Lecture
NISS

6. 66
Science point of view
What is the meaning of life?
What is real?
What is reproducible?
Fooled (fooling) by randomness?
NISS

7. 77
The Players
1. The workers – scientists
2. The communicators –
a. PR people
b. Bloggers
c. Reporters
d. Science writers
3. The consumers – public, regulatory
agencies, trial lawyers
4. The management – funding agencies,
journal editors
NISS

8. 88
The Worker is not the Problem.
W. Edwards Deming,
the most visionary innovator ever on quality control, said
The worker is not the problem.
The problem is at the top! Management!
To Deming, blaming the workers—individual researchers—
is as incorrect as it is useless.
Bringing the system under control is the responsibility
of those managing it.
NISS

9. 9
Problems with observational studies
“Everything is dangerous”
1. Data staging
2. No written analysis protocol
3. Multiple testing
4. Multiple modeling
5. Uncorrected bias
6. Self-serving paper writing
7. Self-serving press release
8. Actually believe the claims
9NISS

10. Assertion : Every study is positive
Data Staging
 Bias
Multiple testing
Multiple model searching
Any or all will lead to essentially all
observational studies being positive!
10NISS

11. 11
First, data staging
Stan:
Why do you think data staging is a big issue?
Because it can be done in myriad ways, is
rarely documented, and is usually not
reproducible?
David Madigan
11NISS

12. 12
Multiple Testing: P-value, t-test
Population,
real or theoretical
Two samples,
random
NISS

13. 10-sided dice experiment
12/25/12
NISS

14. 14
How do you get a “p < 0.05”?
Answer: Ask lots of questions.
61 questions
95% chance of
a positive study!
NISS

15. 1515
Let’s run an epidemiology study!
10-sided dice simulation:
Coffee causes X.
NISS

16. 1616
P-value plot – 60 p-values.
NISS

17. 17
Cereal determines human gender
Really?????
17NISS

18. NISS

19. 19
2 Cancer types, 48 pesticies, 96 questions
Three claims made, only one appears valid.

20. NISS
Multiple Modeling/Bias
Take a simple difference

21. Paper, data, claim
American Cancer Society Cancer Prevention Study II
No association with CV deaths, corrected for PM2.5.
Ozone associated with respiratory deaths.
21

22. Jerrett et al. Large Search Space
22

23. Covariate Adjustment
27
= 128
23

24. Large search space
32 x 128 = 4,096
The data used in this paper is not available.
We are asked to trust that analysis decisions
were good and claims are robust.
Any adjustment for multiple testing
and/or multiple modeling renders p-values NS.
24

25. 2525
Crisis in science? 2011, 2012
Nature, 2012
Significance, 2011
NISS

26. 26
Claims from observational studies tested in RCTs

27. 27
What can funding agencies do?
Fund data generation and analysis
separately.
Fund replication studies.
Require data used in publication be
posted on publication.

28. 28
What can journal editors do?
Quality by inspection, p-value < 0.05, is not working.
(Many workers are gaming the system.)
Management needs to re-design the system to build
quality into the product.
Papers following good manufacturing procedures and
addressing important questions, should be accepted
without regard to statistical significance.
Require data used in publication be posted on
publication.

29. 29
What can you, the consumer, do?
(not much)
1. Be skeptical of observational study claims.
2. Read the actual paper.
3. Count the claims under consideration.
4. Ask for the data set.
5. Letter to editor : voodoo stats and trust me
science. (Educate editors.)
6. Write to funding agency.
7. Write to congressman.
29NISS

30. 30
“New” p-value plot, - log10
(p-value), Dmitri Zaykin

31. 31
Conclusions
Most science claims do not replicate.
Deming: Don't blame the worker
(or expect them to adopt different methods).
Funding agencies and journal editors have been AWOL.
Require data to be placed in depository on publication.

32. 32
One irate study evaluator, 2012
Mens Sana Monograph, 2012

33. 3333
Contact
Information
Stan Young
National Institute of Statistical Sciences
www.niss.org
young@niss.org
919 685 9328
NISS

34. Ozone/PM2.5 Acute Deaths LA
34

35. 3535
Suggestions for effective management
of observational studies
No funding / publication without:
1. Public posting protocol before study initiation.
2. Public posting of data set on publication.
3. Clear statement of questions under consideration.
4. Conform to “Reproducible Research” guidelines.
5. Any claims must be independently replicated.
NISS

36. 3636
Congressional Management:
True Science Transparency Act
Any federal agency proposing rule-making or legislation
shall specifically name each document used to support
the proposed rule-making or legislation and provide
all data used in said document for viewing by the public.
See also OSTP memorandum, 22Feb2013.
NISS