1. What are your criteria in
reporting on health care
research?
Gary
Schwitzer, Publisher, HealthNewsReview.org
2. Our plan for next 2 hours:
Background on HealthNewsReview.org, what we do, how we
do it, why, and what we’ve found
Recurring pitfalls we’ve seen in 6 years of daily monitoring
An editor’s perspective on addressing these issues
We’ll review a story – together – using
HealthNewsReview.org criteria.
Plenty of time for Q & A – tapping Ivan’s editor expertise
Handouts:
Summary of our 6-year, 1,700 story experience
Resources for reporting on studies
Story to review & scoresheet
4. We review stories that
include claims about…
Medical treatments
Tests
Products
Procedures
6. Site stats
28 reviewers: 19 clinicians or researchers, 7
journalists, 2 breast cancer survivors
Site launched April 17, 2006 – 6th anniversary week!
>1,700 stories reviewed
Blog launched in 2004 - >2,500 posts
Sole funding from the Informed Medical Decisions
Foundation, founded by Jack Wennberg et al
8. Global Reach for such efforts
= HealthNewsReview.org (U.S.), Media Doctor sites in
Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in
UK and Austria
= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto
Rico only in two years’ time
9. Our criteria: Does the story explain…
What’s the total cost?
How often do benefits occur?
How often do harms occur?
How strong is the evidence?
Is the condition exaggerated?
Is this really a new approach?
Is it available?
Are there alternative choices?
Who’s promoting this?
Do they have a financial conflict of interest?
10. After 6 years and 1,700 stories
70% fail to adequately discuss costs.
66% fail to quantify benefit – often exaggerating
potential benefit
65% fail to quantify harm - often minimizing
potential harm
62% fail to evaluate the quality of the evidence
57% fail to compare new idea with existing options
12. Recurring themes seen after 6
years of daily monitoring
Failure to explain limitations of observational
studies. Stories that conflate association and
causation.
Framing numbers to exaggerate
benefit, minimize harms
Stories about screening tests that emphasize
only benefits, minimizing or ignoring potential
harms
15. Be skeptical about observational studies
Remind readers that the findings may be more
about the people being studied than the "exposure".
They can point to a strong statistical
association, but they cannot prove cause and effect.
16. White rice increases risk of type 2 diabetes
Thursday, March 15, 2012
The risk of type 2 diabetes is significantly increased if white rice is eaten
regularly, claims a study published today on bmj.com.
More than 400 words in the news release but NONE about the limitations of
such an observational study.
19. Following slides courtesy
Drs. Steve Woloshin & Lisa Schwartz
Dartmouth Medical School
White River Junction, Vermont VA
Part of their syllabus at:
NIH Medicine in the Media
MIT Medical Evidence Boot Camp
21. For the first time, there is a clinically proven
way for many women at high risk of
developing breast cancer to significantly reduce
that risk.
The proof? In a landmark study…women who
took Nolvadex had 48% fewer breast cancers
than women taking sugar pills.
22. Women who took Nolvadex had 48% fewer breast
cancers….
48% of what ???
23. It's like a coupon…
Extremely Fancy Store What if selected items were..
TV’s, washing machines?
48
On selected items!
%
OFF save $100s
Things like a pack of gum?
save pennies
―48% of what‖ matters!
Know the REGULAR price!
24. What is the
"Breast cancer risk"
48% off Chance of getting breast
coupon really
cancer
worth?
Placebo NOLVADEX
??? ???
How much do you save?
25. The chance of getting breast cancer over 6
years
in the PLACEBO group
PLACEBO
3.3%
The REGULAR price!
The "base rate"
Absolute risk in the placebo group
Event rate in the placebo group
26. The chance of getting breast cancer over 6
years
in the INTERVENTION group
PLACEBO NOLVADEX
3.3% 1.7%
The REGULAR price! The SALES price!
27. What is the effect of Nolvadex?
How good is the sale?
The REGULAR price!
PLACEBO The SALES price!
NOLVADEX
3.3% 1.7%
How much do you save?
Absolute risk reduction Regular price – Sales price = 1.6%
Savings = 3.3% 1.7%
Chance of women breast cancer (over 6 years)placebo for 6
If 100 getting took NOLVADEX instead of with NOLVADEX
years, there would be about 2 fewer cases placebo cancer.
was 1.6 % points lower than with of breast
28. What is the effect of Nolvadex?
How good is the sale?
PLACEBO NOLVADEX
3.3% 1.7%
Chance of outcome (intervention)
Relative Risk =
Chance of outcome (control)
29. Describing the effect of NOLVADEX
So finally....this is how you get to
PLACEBO NOLVADEX
the48% off sale!
3.3% 1.7%
It's the relative risk reduction
1.7%
???
Relative Risk = = 0.52
???
3.3%
"% Lower" format =1 - RR =1 - .52 =.48
At 6 years, the chance of breast cancer for women taking
NOLVADEX was 48 % lower than that of women taking placebo.
30. Two ways of saying the same thing:
the benefit of NOLVADEX
One feels big
Extremely Fancy Store
One feels small
Extremely Fancy Store
4
On selected items!
%
OFF
1.6%
matters!
SAVINGS
How you say it On selected items!
"Framing"
31. Relative vs. absolute"savings"
"% off" risk
reductions
Chance of death at 1 Risk reduction
year
Relative Absolute
Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)
30% 10% 67%
?? 20%
??
3% 1% 67% 2%
0.003% 0.001% 67% 0.002%
32. The proof? In a landmark
Benefit study…women who took Nolvadex had
48% fewer breast cancers than women
taking sugar pills.
Harm
"Nolvadex isn't for every woman…In the study women
taking Nolvadex were 2 to 3 times more likely to develop
uterine cancer or blood clots in the lung and legs, although
each occurred in less than 1% of women".
‖Strokes, cataracts more common with
No numbers
Nolvadex. Most women experience some
level of hot flashes and vaginal discharge".
34. The proof? In a landmark
study…women who took Nolvadex had
48% fewer breast cancers than women
taking sugar pills.
210% more uterine cancer and potentially
210% more
life threatening blood clots in the lung and legs.
35. Harm
6-yr chance of venous thromboembolic event:
PLACEBO NOLVADEX
0.5% 1.0%
Like inflation…doubled the price!
36. Over the next 6 years, what happened when women…
PLACEBO NOLVADEX
Benefits: Nolvadex lowered chance
Getting breast cancer 3.3% 1.7%
Harms: Nolvadex increased chance
Having a serious blood clot 0.5% 1.0%
Getting uterine cancer 0.5% 1.1%
Net effect of Nolvadex for every 1000 women:
16 fewer women get breast cancer
5 more women get serious blood clots
6 more get uterine cancer
38. Take home messages
Extrapolate with caution! Don't tell people what to worry about – or do –
based on very preliminary animal / lab science.
Recognize pseudo-evidence Publication in a medical journal - even "the
New England Journal of Medicine" - does not guarantee the findings are
true (or even important).
Be wary of inherently weak science Without a comparison group, it is
impossible to be sure if the drug was responsible for the findings.
Pay attention to the outcome Surrogate outcomes (like tumor shrinkage)
do not reliably translate into clinically meaningful outcomes (longer life).
Be skeptical about observational studies Remind readers that the
findings may be more about the people being studied than the "exposure".
Avoid exaggerated numbers Use absolute risks for both benefits and
harms.
39. Pitfalls of a steady diet of
journal stories
PLoS Med 2005; 2(8): e124
40. The problem begins with the public’s rising expectations of
science. Being human, scientists are tempted to show that
they know more than they do. The number of investigators—
and the number of experiments, observations and analyses
they produce—has also increased exponentially in many
fields, but adequate safeguards against bias are lacking.
Research is fragmented, competition is fierce and emphasis is
often given to single studies instead of the big picture. Much
research is conducted for reasons other than the pursuit of
truth. Conflicts of interest abound, and they influence
outcomes.
42. Spinning results of
randomized clinical trials
Boultron et al, JAMA May 26, 2010, 303 (20): 2058
In this representative sample of RCTs published in
2006 with statistically non-significant primary
outcomes, the reporting and interpretation of findings
was frequently inconsistent with the results.
43. Reporting bias in medical research
- a narrative review
McGauran et al. Trials 2010, 11:37
We identified reporting bias in 40 indications
comprising around 50 different
pharmacological, diagnostic, and preventive
interventions. Many cases involved the withholding
of study data by manufacturers and regulatory
agencies or the active attempt by manufacturers to
suppress publication. The ascertained effects of
reporting bias included the overestimation of
efficacy and the underestimation of safety risks of
interventions.
44. BE AWARE OF…..
• What Ivan’s project has shown about
retractions, research
fraud, fabrication, falsification of data
• Unpublished data (BMJ recently published 8
articles in one edition on “the extent, causes and
consequences of unpublished evidence”)
• Conflicts of interest – guideline setting
45. • Commercialization of research: contract
research organizations, commercial IRBs or
institutional review boards, medical education
and communication companies (Carl Elliott)
• Ghostwriting of journal articles (The Public
Library of Science hosts a “Ghostwriting
Collection” on its website.)
• The focus on surrogate markers in many
studies may be hurting patient care. (primer in
HealthNewsReview.org online toolkit)
46. News coverage & poor public discussion of screening
tests is one of the most concerning public policy issues.
Worst, most biased coverage I’ve seen in 37 years
47. What the Task Force actually wrote:
“The decision to start regular, biennial screening
mammography before the age of 50 years
should be an individual one and take patient
context into account, including the patient's
values regarding specific benefits and harms.”
48. As 37-year ChiTrib & NYT vet John
Crewdson wrote in The Atlantic…
“There are multiple reasons women are ill-
informed about breast cancer. The fault lies
primarily with their physicians, the cancer
establishment, and the news media--especially the
news media. Until coverage of breast cancer rises
above the level of scary warnings mixed with
heartwarming stories of cancer survivors, women
are likely to go on being perplexed."
49. Other examples:
Chicago Sun-Times
Wall Street Journal
Washington Post
New York Daily News
Minneapolis Star Tribune
All TV networks
51. HealthDay wire service didn’t challenge
researcher promoting universal pancreatic
cancer screening for everyone over 50.
After study of tissue from 7 people!
52. We could have given the finger
to this story
"Relative finger length
could be used as a
simple test for prostate
cancer risk, particularly
in men aged under 60,"
said one of the
researchers.
54. Why don’t we deliver this message?
“All screening tests
cause harm;
some may do good.”
But much health journalism consistently emphasizes
benefits & minimizes harms
55. A form of disease-mongering
Selling sickness
Selling the search for weapons of mass destruction inside
everyone
Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK
―Screening the apparently healthy potentially saves a few lives. But it
definitely drags many others into the system needlessly — into needless
appointments, needless tests, needless drugs and needless operations. This
process doesn’t promote health; it promotes disease. People suffer from
more anxiety about their health, from drug side effects, from complications
of surgery. A few die. And remember: these people felt fine when they
entered the health care system.‖
Communication of the evidence is a key health policy issue
56. ―I honestly believe it is better to know nothing
than to know what ain’t so.‖
Josh Billings (pen name of humorist
Henry Wheeler Shaw, 1818 – 1885)