We Are All Gatekeepers Now: Scrutiny of Science Goes Public
Â
Covering Medical Studies: How Not to Get It Wrong
1. Covering Medical Studies:
How Not to Get It Wrong
AHCJ
Boston, March 2013
Ivan Oransky, MD
Executive Editor, Reuters Health
Co-Founder, Retraction Watch
@ivanoransky
4. Put Down That Coffee!
STUDY LINKS COFFEE USE TO PANCREAS CANCER
New York Times, March 12, 1981
âAlthough the statistical association does not prove
that coffee causes cancer, Dr. Brian MacMahon of
Harvard, leader of the research group, said he stopped
drinking coffee a few months ago when the results of
the study became clear. In a telephone interview, he
said that he would not presume to advise others.â
6. Or Get A Refill
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981
''This otherwise excellent paper may be flawed in one
critical way,'' said a letter from Dr. Steven Shedlofsky of
the Veterans Administration Hospital in White River
Junction, Vt. He questioned the comparison of
pancreatic cancer patients with persons hospitalized
for noncancerous diseases of the digestive system.
7. Or Get A Refill
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981
âSuch patients, he noted, might be expected to give up
coffee drinking because of their illness. This, he argued,
would tilt the proportion of coffee drinkers away from
the ''control'' group who were being compared with
the cancer patients. Amplifying the letter in an
interview, Dr. Shedlofsky said many patients with
digestive diseases give up coffee because they believe
it aggravates their discomfort, and others do so
because their doctors have advised them to.
9. We Cured Cancer 15 Years Ago
The New York Times, May 3, 1998
âWithin a year, if all goes well, the first cancer patient will
be injected with two new drugs that can eradicate any
type of cancer, with no obvious side effects and no drug
resistance -- in mice.â
âŚ
''Judah is going to cure cancer in two years,'' said Dr.
James D. Watson, a Nobel laureate who directs the Cold
Spring Harbor Laboratory, a cancer research center on
Long Island. Dr. Watson said Dr. Folkman would be
remembered along with scientists like Charles Darwin as
someone who permanently altered civilization.â
10. Or Maybe We Didnât. Hereâs Why.
The New York Times,
February 11, 2013
11. Or Maybe We Didnât. Hereâs Why.
The New York Times, February 11, 2013
âThe studyâs findings do not mean that mice are useless
models for all human diseases. But, its authors said,
they do raise troubling questions about diseases like
the ones in the study that involve the immune system,
including cancer and heart disease.â
12. Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
13. Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
5
How many of those made it to FDA approval?
14. Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
5
How many of those made it to FDA approval?
1
Source: http://www.phrma.org/issues/intellectual-property (PhRMA)
15. How to Get It Wrong
⢠Write about compounds in pre-clinical trials as
if they were about to be on pharmacy shelves
⢠Write about every drug in phase I and phase II
trials as if it would definitely be approved
16. How Often Are Studies Wrong?
Ioannidis JPA. PLoS Med 2005; 2(8): e124
22. Always Read the Study
Get the full study and read it â
âI think itâs journalistic malpractice to not have the full
study in front of you when youâre reporting,â Oransky
says.
23. How to Get Studies
⢠www.EurekAlert.org for embargoed material
⢠AHCJ membership includes access to Cochrane
Library, Health Affairs, JAMA, and many other
journals www.healthjournalism.org
⢠ScienceDirect (Elsevier) gives reporters free access
to hundreds of journals www.sciencedirect.com
⢠Open access journals (e.g., Public Library of Science
www.plos.org)
⢠Ask press officers, or the authors
24. Ask âDumbâ Questions
If you lack experience dealing with scientific material,
donât be afraid to ask for definitions of jargon and
scientific terms. This is no time to pretend you
understand everything. Oransky says the science and
medical industries are full of jargon that mask important
details. âYouâll get off the phone and have a notebook
full of gibberish and jargon,â he says. âYou canât be
afraid of asking a dumb question.â
25. Ask Smart Questions
⢠Was it:
â Peer-reviewed?
â Published? Where? Not all journals are
created equal.
âDr. X said they published in Y rather than a
clinical journal because the paper was too
long for the word limits in the clinical
journals. I'm not sure where a detail like that
would goâŚbut he was impressed with my
question.â
26. Ask Smart Questions
⢠Was it in humans?
â Itâs remarkable there are any mice left with
cancer, depression, or restless leg syndrome
27. Ask Smart Questions
⢠Size matters
Look for the power calculation, and ask if you
donât see one
28. Ask Smart
Questions
⢠Was it well-designed?
From Covering Medical Research, Schwitzer/AHCJ
29. Ask Smart Questions
⢠âWere those your primary endpoints?â
⢠âLooks as though that endpoint reached statistical
significance. Is that difference clinically
significant?â
30. Read the Discussion
Good journals will insist that authors include
limitations.
Read accompanying editorials, too.
31. Whatâs Your Angle?
⢠Are you trying to help readers, listeners, and
viewers make better health care decisions?
⢠Covering a study because it has a good business
angle, or itâs about a local project, is perfectly OK,
but it doesnât mean readers deserve less
evidence and skepticism
32. Who Could Benefit?
⢠How many people have the disease?
⢠Keep potential disease-mongering in mind
33. How Effective is the Treatment?
⢠Clinically significant endpoints, or surrogates â
does this matter?
⢠Preventing complications? How many?
⢠Always remember to quantify results, not just
âpatients improvedâ
34. What Are The Side Effects?
⢠Every treatment has them
⢠Where to look:
â Go beyond press releases and abstracts
â Look at tables, charts, and results sections
35. Who Dropped Out?
⢠Why did they leave the trial?
⢠Intention to treat analysis
36. How Much Does it Cost?
⢠If itâs ready to be the subject of a story,
someone has projected the likely cost and
market.
â At least ask.
37. Who Has an Interest?
⢠Disclose conflicts
⢠PharmedOut.org
⢠Dollars For Docs series
http://projects.propublica.org/docdollars/
38. Are There Alternatives?
⢠Did the study compare the new treatment to
existing alternatives, or to placebo?
⢠What are the advantages and disadvantages
(and costs) of those existing alternatives?
⢠Consider alternative explanations. Remember
coffee and pancreatic cancer?
39. Donât Rely Only on Study Authors
⢠Find outside sources. Hereâs how:
40. Use Anecdotes Carefully
⢠Is the story representative?
⢠Does the source of the story have any conflicts?
41. Watch Your Language
⢠Lifestyle/diet â are they randomized controlled
trials, or just observational?
⢠If observational, make the language fit the
evidence:
â YES: âtied,â âlinkedâ
â NO: âreduces,â âcausesâ
42. A Dirty Little Secret
Keep a biostatistician in your back pocket
Photo by Peyri Herrera, on Flickr