Presentation at the University of Alabama, Birmingham, June 14, 2013. Contains some slides from previous talks.

1. Can You Trust What You Read In
(Scientific and News)papers?
Ivan Oransky, MD
Executive Editor, Reuters Health
Co-Founder, Retraction Watch
@ivanoransky
University of Alabama, Birmingham
June 14, 2013

3. Put Down That Coffee!

4. Put Down That Coffee!
“Although the statistical association does not prove
that coffee causes cancer, Dr. Brian MacMahon of
Harvard, leader of the research group, said he stopped
drinking coffee a few months ago when the results of
the study became clear. In a telephone interview, he
said that he would not presume to advise others.”
STUDY LINKS COFFEE USE TO PANCREAS CANCER
New York Times, March 12, 1981

5. Or Get A Refill

6. Or Get A Refill
''This otherwise excellent paper may be flawed in one
critical way,'' said a letter from Dr. Steven Shedlofsky of
the Veterans Administration Hospital in White River
Junction, Vt. He questioned the comparison of
pancreatic cancer patients with persons hospitalized
for noncancerous diseases of the digestive system.
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981

7. Or Get A Refill
“Such patients, he noted, might be expected to give up
coffee drinking because of their illness. This, he argued,
would tilt the proportion of coffee drinkers away from
the ''control'' group who were being compared with
the cancer patients. Amplifying the letter in an
interview, Dr. Shedlofsky said many patients with
digestive diseases give up coffee because they believe
it aggravates their discomfort, and others do so
because their doctors have advised them to.
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981

8. We Cured Cancer 15 Years Ago

9. We Cured Cancer 15 Years Ago
“Within a year, if all goes well, the first cancer patient will
be injected with two new drugs that can eradicate any
type of cancer, with no obvious side effects and no drug
resistance -- in mice.”
…
''Judah is going to cure cancer in two years,'' said Dr.
James D. Watson, a Nobel laureate who directs the Cold
Spring Harbor Laboratory, a cancer research center on
Long Island. Dr. Watson said Dr. Folkman would be
remembered along with scientists like Charles Darwin as
someone who permanently altered civilization.”
The New York Times, May 3, 1998

10. Or Maybe We Didn’t. Here’s Why.
The New York Times,
February 11, 2013

11. Or Maybe We Didn’t. Here’s Why.
“The study’s findings do not mean that mice are useless
models for all human diseases. But, its authors said,
they do raise troubling questions about diseases like
the ones in the study that involve the immune system,
including cancer and heart disease.”
The New York Times, February 11, 2013

12. How Are The Media Doing?

13. How Are The Media Doing?
Schwitzer G. How do U.S. journalists cover treatments, tests, products,
and procedures? An evaluation of 500 stories. PLoS Medicine 2008
doi:10.1371/journal.pmed.0050095

14. 207 stories
• 83 (40%) did not report benefits quantitatively
• 124 did, but
- 103 (83%) reported relative benefits only,
- 3 (2%) absolute benefits only,
- 18 (15%) both absolute and relative benefits
• 98 (47%) mentioned potential harm to patients
• 63 (30%) mentioned costs
• 170 stories cited an expert or a scientific study
- 85 (50%) cited at least one source with disclosed financial ties
- 33 (39%) disclosed these ties
Moynihan R et al. Coverage by the news media of the
benefits and risks of medications. N Engl J Med 2000;
342:1645-1650
How Are The Media Doing?

15. 193 articles reporting at leastone benefit or harm of a drug
100% mentioned at least one benefit
132 (68%) did not mentionside effects or harms
119 (62%) did not quantify benefits or harms
– Of 510 mentions of benefits and harms, only 120 (24%) gave quantitative
information
– In 26% (31/120) the magnitude was presented in relative terms
37(19%) articles reported only surrogate benefits
7 (4%) mentioned contraindications
61 (32%) mentioned drug costs,
89 (46%) mentioned drug alternatives
30 (16%) mentioned nondrug options (such as exerciseor diet)
Cassels A. Drugs in the news: an analysis of Canadian
newspaper coverage of new prescription drugs. CMAJ,
April 29, 2003; 168 (9)
How Are The Media Doing?

16. 193 articles reporting at leastone benefit or harm of a drug
120 (62%) quotedat least one interviewee.
After exclusion of industry and governmentspokespeople, potential
financial conflicts of interest were reported for only 5 of 164 interviewees
(3%)
Of 57 articles covering studies, only 15 (26%) includedinformation on
study funding
Cassels A. Drugs in the news: an analysis of Canadian
newspaper coverage of new prescription drugs. CMAJ,
April 29, 2003; 168 (9)
How Are The Media Doing?

17. Wells: “Newspapers over-represent support for screening
mammography for ages 40 to 49”
Reports would have been improved by
• Identification of all sources for information cited
• Less reliance on relatively few sources
• Discussion of benefits in absolute terms
Medical journalism may need standards similar to those used for
reporting medical research
Wells J. Newspaper reporting of screening
mammography. Ann Intern Med 2001;35:1029-
1037.
How Are The Media Doing?

18. FDA message: Use of pediatric antidepressants is linked to a
risk of suicidality (as opposed to suicide itself)
• Reported correctly in the vast majorityof news stories, BUT
• Other key health messages in FDA warningoften missing
• News stories morelikely to include anecdotes of children harmed
versus childrenhelped by antidepressants
• Quoted experts more likely to emphasize benefits over risks
Coverage grew increasingly neutralover time, conveyingneither the
impression that the risks outweighed the benefits nor that benefits
outweighed the risks
Barry CL, Busch SH. News coverage of FDA
warnings on pediatric antidepressant use and
suicidality. Pediatrics 2010; 125:88-95.
How Are The Media Doing?

19. Why Is It So Bad?

20. In a national survey of U.S. health and medical journalists:
• Nearly 70% had at least a bachelor’s degree
• 19% reported having a master’s degree;
• 4.5% had a doctorate; about 3% were M.D.s
• Almost half had a degree in journalism
• 13% had a degree in communications
• 8% were ‘‘life sciences’’ majors
Viswanath K et al: Occupational practices and the making
of health news: A national survey of U.S. health and
medical science journalists. Journal of Health
Communication 2008; 13:759–777.
Why Is It So Bad?

21. The Problem
Fewer reporters are doing more stories, broadcasts, and
blog posts
• Sites chasing a smaller number of advertising dollars
• Pressure to cover more and more, which places heavy
reliance on journals and meetings

22. The Problem
Many health reporters feel it’s hard to find
independent experts willing to assist journalists
They think editors need education in critical
appraisal of medical news
Larrson A. Medical messages in the media--barriers and
solutions to improving medical journalism. Health
Expectations 2003;6:323-31.

23. The Problem
Barriers to improving medical journalism
• Lack of time, space and knowledge (the most common
obstacles)
• Competition for space and audience
• Difficulties with terminology
• Problems finding and using sources
• Problems with editors and commercialism
Larrson A. Medical messages in the media--barriers and
solutions to improving medical journalism. Health
Expectations 2003;6:323-31.

24. But It’s Not Just Journalists
Academic medical centers issue a mean of 49 press releases/year
Among 200 randomly selected releases
– 87 (44%) promoted animal or laboratory research, of which 64
(74%) explicitly claimed relevance to human health
– Among 95 releases about clinical research, 22 (23%) omitted
study size and 32 (34%) failed to quantify results
– 113 releases promoted human research
• 17% promoted randomized trials or meta-analyses
• 40% reported on uncontrolled interventions, small samples
(<30 participants), surrogate primary outcomes, or
unpublished data—yet 58% lacked the relevant cautions
Woloshin S et al. Press releases by academic
medical centers: not so academic? Ann Intern
Med 2009;150:613-618

25. How Often Are Studies Wrong?

26. Retractions on the Rise
-The Wall Street Journal

27. How Often Are Medical Studies Wrong?

28. Positive Publication Bias

29. Positive Publication Bias
“The overall frequency of positive supports
has grown by over 22% between 1990 and
2007, with significant differences between
disciplines and countries.”
“…the strongest increase in positive results
was observed in disciplines—like Clinical
Medicine, Pharmacology & Toxicology,
Molecular Biology”
Fanelli, Scientometrics 2012.

30. How Often Are Studies Wrong?
Ioannidis JPA. PLoS Med 2005; 2(8): e124

31. Let’s Work to Avoid This

32. Acknowledgement/Contact
• Nancy Lapid, Reuters Health
ivan-oransky@erols.com
Twitter: @ivanoransky