SlideShare ist ein Scribd-Unternehmen logo
1 von 57
Downloaden Sie, um offline zu lesen
What are your criteria in
reporting on health care
research?
                                       Gary
  Schwitzer, Publisher, HealthNewsReview.org
Our plan for next 2 hours:

  Background on HealthNewsReview.org, what we do, how we
   do it, why, and what we’ve found
  Recurring pitfalls we’ve seen in 6 years of daily monitoring
  An editor’s perspective on addressing these issues
  We’ll review a story – together – using
   HealthNewsReview.org criteria.
  Plenty of time for Q & A – tapping Ivan’s editor expertise
  Handouts:
    Summary of our 6-year, 1,700 story experience
    Resources for reporting on studies
    Story to review & scoresheet
AHCJ 2012 Atlanta conf. talk
We review stories that
include claims about…
  Medical treatments
  Tests
  Products
  Procedures
AHCJ 2012 Atlanta conf. talk
Site stats

 28 reviewers: 19 clinicians or researchers, 7
  journalists, 2 breast cancer survivors

 Site launched April 17, 2006 – 6th anniversary week!

 >1,700 stories reviewed

 Blog launched in 2004 - >2,500 posts

 Sole funding from the Informed Medical Decisions
  Foundation, founded by Jack Wennberg et al
AHCJ 2012 Atlanta conf. talk
Global Reach for such efforts

= HealthNewsReview.org (U.S.), Media Doctor sites in
Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in
UK and Austria

= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto
Rico only in two years’ time
Our criteria: Does the story explain…

 What’s the total cost?
 How often do benefits occur?
 How often do harms occur?
 How strong is the evidence?
 Is the condition exaggerated?
 Is this really a new approach?
 Is it available?
 Are there alternative choices?
 Who’s promoting this?
 Do they have a financial conflict of interest?
After 6 years and 1,700 stories


 70% fail to adequately discuss costs.

 66% fail to quantify benefit – often exaggerating
  potential benefit

 65% fail to quantify harm - often minimizing
  potential harm

 62% fail to evaluate the quality of the evidence

 57% fail to compare new idea with existing options
Kid-in-candy-store picture of
       U.S. health care




 Everything   is terrific
 Nothing   is risky
 No   price tags
Recurring themes seen after 6
  years of daily monitoring
 Failure to explain limitations of observational
  studies. Stories that conflate association and
  causation.
 Framing numbers to exaggerate
  benefit, minimize harms
 Stories about screening tests that emphasize
  only benefits, minimizing or ignoring potential
  harms
AHCJ 2012 Atlanta conf. talk
AHCJ 2012 Atlanta conf. talk
Be skeptical about observational studies
Remind readers that the findings may be more
about the people being studied than the "exposure".
They can point to a strong statistical
association, but they cannot prove cause and effect.
White rice increases risk of type 2 diabetes

Thursday, March 15, 2012

The risk of type 2 diabetes is significantly increased if white rice is eaten
regularly, claims a study published today on bmj.com.

More than 400 words in the news release but NONE about the limitations of
such an observational study.
Does your language fit the evidence of observational studies?
Absolute versus relative risk
Following slides courtesy
Drs. Steve Woloshin & Lisa Schwartz
     Dartmouth Medical School
 White River Junction, Vermont VA

     Part of their syllabus at:
   NIH Medicine in the Media
 MIT Medical Evidence Boot Camp
Nolvadex (tamoxifen)
Reducing breast cancer risk by 50
                   .
For the first time, there is a clinically proven
way for many women at high risk of
developing breast cancer to significantly reduce
that risk.
 The proof? In a landmark study…women who
 took Nolvadex had 48% fewer breast cancers
 than women taking sugar pills.
Women who took Nolvadex had 48% fewer breast
cancers….




              48% of what ???
It's like a coupon…
   Extremely Fancy Store    What if selected items were..

                            TV’s, washing machines?



48
 On selected items!
                      %
                      OFF       save $100s

                            Things like a pack of gum?
                                save pennies

                            ―48% of what‖ matters!

                            Know the REGULAR price!
What is the
                      "Breast cancer risk"
48% off            Chance of getting breast
coupon really
                   cancer
worth?

    Placebo     NOLVADEX
      ???          ???




     How much do you save?
The chance of getting breast cancer over 6
                  years
        in the PLACEBO group
          PLACEBO
            3.3%




   The REGULAR price!
         The "base rate"
Absolute risk in the placebo group
 Event rate in the placebo group
The chance of getting breast cancer over 6
                  years
     in the INTERVENTION group
       PLACEBO             NOLVADEX
        3.3%                 1.7%




   The REGULAR price!    The SALES price!
What is the effect of Nolvadex?
How good is the sale?

     The REGULAR price!
         PLACEBO                   The SALES price!
                                     NOLVADEX
           3.3%                        1.7%

                 How much do you save?

Absolute risk reduction Regular price – Sales price = 1.6%
            Savings =      3.3%            1.7%

Chance of women breast cancer (over 6 years)placebo for 6
   If 100 getting took NOLVADEX instead of with NOLVADEX
 years, there would be about 2 fewer cases placebo cancer.
          was 1.6 % points lower than with of breast
What is the effect of Nolvadex?
How good is the sale?

          PLACEBO                    NOLVADEX
            3.3%                        1.7%


                    Chance of outcome (intervention)
 Relative Risk =
                    Chance of outcome (control)
Describing the effect of NOLVADEX

   So finally....this is how you get to
          PLACEBO                      NOLVADEX
   the48% off sale!
          3.3%             1.7%
   It's the relative risk reduction
                        1.7%
                        ???
 Relative Risk =                =    0.52
                        ???
                        3.3%

 "% Lower" format =1 - RR             =1 - .52 =.48
At 6 years, the chance of breast cancer for women taking
 NOLVADEX was 48 % lower than that of women taking placebo.
Two ways of saying the same thing:
   the benefit of NOLVADEX
    One feels big
   Extremely Fancy Store
                                   One feels small
                                    Extremely Fancy Store




4
On selected items!
                       %
                      OFF
                                 1.6%
                            matters!
                                       SAVINGS

             How you say it On selected items!
                  "Framing"
Relative vs. absolute"savings"
              "% off"     risk
           reductions
Chance of death at 1           Risk reduction
year
                              Relative   Absolute

Placebo       DRUG     (1-[DRUG/Placebo]) (Placebo-DRUG)


30%           10%           67%
                            ??           20%
                                          ??

  3%            1%          67%           2%

0.003%        0.001%        67%          0.002%
The proof? In a landmark
Benefit        study…women who took Nolvadex had
               48% fewer breast cancers than women
               taking sugar pills.
Harm
  "Nolvadex isn't for every woman…In the study women
  taking Nolvadex were 2 to 3 times more likely to develop
  uterine cancer or blood clots in the lung and legs, although
  each occurred in less than 1% of women".




 ‖Strokes, cataracts more common with
                                                  No numbers
 Nolvadex. Most women experience some
 level of hot flashes and vaginal discharge".
Present benefits and harms the same way
The proof? In a landmark
              study…women who took Nolvadex had
              48% fewer breast cancers than women
              taking sugar pills.


210% more uterine cancer and potentially
  210% more
   life threatening blood clots in the lung and legs.
Harm
6-yr chance of venous thromboembolic event:

  PLACEBO                NOLVADEX
   0.5%                     1.0%




         Like inflation…doubled the price!
Over the next 6 years, what happened when women…


                                PLACEBO    NOLVADEX

Benefits: Nolvadex lowered chance
 Getting breast cancer              3.3%      1.7%

Harms: Nolvadex increased chance
 Having a serious blood clot        0.5%      1.0%
Getting uterine cancer              0.5%      1.1%
        Net effect of Nolvadex for every 1000 women:
           16 fewer women get breast cancer
            5 more women get serious blood clots
            6 more get uterine cancer
AHCJ 2012 Atlanta conf. talk
Take home messages
Extrapolate with caution! Don't tell people what to worry about – or do –
   based on very preliminary animal / lab science.
Recognize pseudo-evidence Publication in a medical journal - even "the
   New England Journal of Medicine" - does not guarantee the findings are
   true (or even important).
Be wary of inherently weak science Without a comparison group, it is
  impossible to be sure if the drug was responsible for the findings.
Pay attention to the outcome Surrogate outcomes (like tumor shrinkage)
  do not reliably translate into clinically meaningful outcomes (longer life).
Be skeptical about observational studies Remind readers that the
  findings may be more about the people being studied than the "exposure".
Avoid exaggerated numbers Use absolute risks for both benefits and
  harms.
Pitfalls of a steady diet of
      journal stories




                     PLoS Med 2005; 2(8): e124
The problem begins with the public’s rising expectations of
science. Being human, scientists are tempted to show that
they know more than they do. The number of investigators—
and the number of experiments, observations and analyses
they produce—has also increased exponentially in many
fields, but adequate safeguards against bias are lacking.
Research is fragmented, competition is fierce and emphasis is
often given to single studies instead of the big picture. Much
research is conducted for reasons other than the pursuit of
truth. Conflicts of interest abound, and they influence
outcomes.
AHCJ 2012 Atlanta conf. talk
Spinning results of
  randomized clinical trials
Boultron et al, JAMA May 26, 2010, 303 (20): 2058



In this representative sample of RCTs published in
2006 with statistically non-significant primary
outcomes, the reporting and interpretation of findings
was frequently inconsistent with the results.
Reporting bias in medical research
       - a narrative review

 McGauran et al. Trials 2010, 11:37

 We identified reporting bias in 40 indications
  comprising around 50 different
  pharmacological, diagnostic, and preventive
  interventions. Many cases involved the withholding
  of study data by manufacturers and regulatory
  agencies or the active attempt by manufacturers to
  suppress publication. The ascertained effects of
  reporting bias included the overestimation of
  efficacy and the underestimation of safety risks of
  interventions.
BE AWARE OF…..

• What Ivan’s project has shown about
  retractions, research
  fraud, fabrication, falsification of data

• Unpublished data (BMJ recently published 8
  articles in one edition on “the extent, causes and
  consequences of unpublished evidence”)


• Conflicts of interest – guideline setting
• Commercialization of research: contract
  research organizations, commercial IRBs or
  institutional review boards, medical education
  and communication companies (Carl Elliott)

• Ghostwriting of journal articles (The Public
  Library of Science hosts a “Ghostwriting
  Collection” on its website.)

• The focus on surrogate markers in many
  studies may be hurting patient care. (primer in
  HealthNewsReview.org online toolkit)
News coverage & poor public discussion of screening
tests is one of the most concerning public policy issues.




Worst, most biased coverage I’ve seen in 37 years
What the Task Force actually wrote:



“The decision to start regular, biennial screening
mammography before the age of 50 years
should be an individual one and take patient
context into account, including the patient's
values regarding specific benefits and harms.”
As 37-year ChiTrib & NYT vet John
 Crewdson wrote in The Atlantic…

 “There are multiple reasons women are ill-
  informed about breast cancer. The fault lies
  primarily with their physicians, the cancer
  establishment, and the news media--especially the
  news media. Until coverage of breast cancer rises
  above the level of scary warnings mixed with
  heartwarming stories of cancer survivors, women
  are likely to go on being perplexed."
Other examples:

   Chicago Sun-Times

   Wall Street Journal

   Washington Post

   New York Daily News

   Minneapolis Star Tribune

   All TV networks
The ink
isn’t even
  dry on
  studies
  before
marketing
 begins
HealthDay wire service didn’t challenge
researcher promoting universal pancreatic
cancer screening for everyone over 50.
     After study of tissue from 7 people!
We could have given the finger
          to this story



 "Relative finger length
  could be used as a
  simple test for prostate
  cancer risk, particularly
  in men aged under 60,"
  said one of the
  researchers.
Screening Madness

 Crusading
 one-sided
 advocacy
Why don’t we deliver this message?



  “All screening tests
  cause harm;
  some may do good.”
But much health journalism consistently emphasizes
benefits & minimizes harms
A form of disease-mongering

 Selling sickness

 Selling the search for weapons of mass destruction inside
  everyone

 Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK
    ―Screening the apparently healthy potentially saves a few lives. But it
    definitely drags many others into the system needlessly — into needless
    appointments, needless tests, needless drugs and needless operations. This
    process doesn’t promote health; it promotes disease. People suffer from
    more anxiety about their health, from drug side effects, from complications
    of surgery. A few die. And remember: these people felt fine when they
    entered the health care system.‖

 Communication of the evidence is a key health policy issue
―I honestly believe it is better to know nothing
than to know what ain’t so.‖

           Josh Billings (pen name of humorist
             Henry Wheeler Shaw, 1818 – 1885)
Thank you




Gary@HealthNewsReview.org

Weitere ähnliche Inhalte

Ähnlich wie AHCJ 2012 Atlanta conf. talk

Cervical Cancer Educational Presentation
Cervical Cancer Educational PresentationCervical Cancer Educational Presentation
Cervical Cancer Educational Presentationrinki singh
 
Cancer and screening - A presentation by Cancer Rose
Cancer and screening - A presentation by Cancer RoseCancer and screening - A presentation by Cancer Rose
Cancer and screening - A presentation by Cancer RoseCancer Rose
 
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with Medicine
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with MedicineGlobal Medical Cures™ | BREAST CANCER- Reducing the Risk with Medicine
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with MedicineGlobal Medical Cures™
 
Health conditions_cervical_cancer_facts
 Health conditions_cervical_cancer_facts Health conditions_cervical_cancer_facts
Health conditions_cervical_cancer_factsgregoriop
 
Interstitial Therapy (ICRT)
Interstitial Therapy (ICRT)Interstitial Therapy (ICRT)
Interstitial Therapy (ICRT)Karen Thompson
 
Hpv Vs Cervical Cancer
Hpv Vs Cervical CancerHpv Vs Cervical Cancer
Hpv Vs Cervical CancerAbby Johnson
 
Management and prevention of cervical cancer.pptx
Management and prevention of cervical cancer.pptxManagement and prevention of cervical cancer.pptx
Management and prevention of cervical cancer.pptxAmin Badamosi
 
Cervical Cancer Awarness!!!!
Cervical Cancer Awarness!!!!Cervical Cancer Awarness!!!!
Cervical Cancer Awarness!!!!dharsubaby
 
Cervicalcancer 180428125921-converted - copy final
Cervicalcancer 180428125921-converted - copy finalCervicalcancer 180428125921-converted - copy final
Cervicalcancer 180428125921-converted - copy finalmadurai
 
Cervical Cancer Awareness
Cervical Cancer AwarenessCervical Cancer Awareness
Cervical Cancer Awarenessdharshinee-shri
 
Your Pap Smear Was Abnormal: Now What?
Your Pap Smear Was Abnormal: Now What?Your Pap Smear Was Abnormal: Now What?
Your Pap Smear Was Abnormal: Now What?bkling
 
High Chektogenic Diets
High Chektogenic DietsHigh Chektogenic Diets
High Chektogenic DietsBeth Johnson
 
Risky Business: Risk communicat ion in the provider-patient encounter
Risky Business: Risk communicat ion in the provider-patient encounterRisky Business: Risk communicat ion in the provider-patient encounter
Risky Business: Risk communicat ion in the provider-patient encounterZackary Berger
 

Ähnlich wie AHCJ 2012 Atlanta conf. talk (20)

Birth Control Pills
Birth Control PillsBirth Control Pills
Birth Control Pills
 
Cervical Cancer 101
Cervical Cancer 101Cervical Cancer 101
Cervical Cancer 101
 
Cervical Cancer Educational Presentation
Cervical Cancer Educational PresentationCervical Cancer Educational Presentation
Cervical Cancer Educational Presentation
 
Cancer and screening - A presentation by Cancer Rose
Cancer and screening - A presentation by Cancer RoseCancer and screening - A presentation by Cancer Rose
Cancer and screening - A presentation by Cancer Rose
 
Cancer Research Paper
Cancer Research PaperCancer Research Paper
Cancer Research Paper
 
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with Medicine
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with MedicineGlobal Medical Cures™ | BREAST CANCER- Reducing the Risk with Medicine
Global Medical Cures™ | BREAST CANCER- Reducing the Risk with Medicine
 
Health conditions_cervical_cancer_facts
 Health conditions_cervical_cancer_facts Health conditions_cervical_cancer_facts
Health conditions_cervical_cancer_facts
 
Cervical ca
Cervical caCervical ca
Cervical ca
 
Interstitial Therapy (ICRT)
Interstitial Therapy (ICRT)Interstitial Therapy (ICRT)
Interstitial Therapy (ICRT)
 
Hpv Vs Cervical Cancer
Hpv Vs Cervical CancerHpv Vs Cervical Cancer
Hpv Vs Cervical Cancer
 
Management and prevention of cervical cancer.pptx
Management and prevention of cervical cancer.pptxManagement and prevention of cervical cancer.pptx
Management and prevention of cervical cancer.pptx
 
Cervical Cancer Awarness!!!!
Cervical Cancer Awarness!!!!Cervical Cancer Awarness!!!!
Cervical Cancer Awarness!!!!
 
Cervicalcancer 180428125921-converted - copy final
Cervicalcancer 180428125921-converted - copy finalCervicalcancer 180428125921-converted - copy final
Cervicalcancer 180428125921-converted - copy final
 
Surgical Oncologist
Surgical OncologistSurgical Oncologist
Surgical Oncologist
 
Speech outline
Speech outlineSpeech outline
Speech outline
 
Cervical Cancer Awareness
Cervical Cancer AwarenessCervical Cancer Awareness
Cervical Cancer Awareness
 
Your Pap Smear Was Abnormal: Now What?
Your Pap Smear Was Abnormal: Now What?Your Pap Smear Was Abnormal: Now What?
Your Pap Smear Was Abnormal: Now What?
 
Cervical cancer
Cervical cancerCervical cancer
Cervical cancer
 
High Chektogenic Diets
High Chektogenic DietsHigh Chektogenic Diets
High Chektogenic Diets
 
Risky Business: Risk communicat ion in the provider-patient encounter
Risky Business: Risk communicat ion in the provider-patient encounterRisky Business: Risk communicat ion in the provider-patient encounter
Risky Business: Risk communicat ion in the provider-patient encounter
 

Mehr von Gary Schwitzer

How journalists (and other mass media) often promote too much medicine
How journalists (and other mass media)  often promote too much medicineHow journalists (and other mass media)  often promote too much medicine
How journalists (and other mass media) often promote too much medicineGary Schwitzer
 
Preventing Overdiagnosis 2016 talk
Preventing Overdiagnosis 2016 talkPreventing Overdiagnosis 2016 talk
Preventing Overdiagnosis 2016 talkGary Schwitzer
 
Role of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisRole of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisGary Schwitzer
 
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Gary Schwitzer
 
McGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationMcGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationGary Schwitzer
 
Lessons from 1,889 story reviews
Lessons from 1,889 story reviewsLessons from 1,889 story reviews
Lessons from 1,889 story reviewsGary Schwitzer
 
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Gary Schwitzer
 
Schwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruSchwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruGary Schwitzer
 
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Gary Schwitzer
 
My talk to University of Wisconsin event, "Science Writing in Age of Denial"
My talk to University of Wisconsin event, "Science Writing in Age of Denial"My talk to University of Wisconsin event, "Science Writing in Age of Denial"
My talk to University of Wisconsin event, "Science Writing in Age of Denial"Gary Schwitzer
 
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Gary Schwitzer
 
Applying HealthNewsReview.org criteria
Applying HealthNewsReview.org criteria Applying HealthNewsReview.org criteria
Applying HealthNewsReview.org criteria Gary Schwitzer
 
Covering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgCovering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgGary Schwitzer
 

Mehr von Gary Schwitzer (13)

How journalists (and other mass media) often promote too much medicine
How journalists (and other mass media)  often promote too much medicineHow journalists (and other mass media)  often promote too much medicine
How journalists (and other mass media) often promote too much medicine
 
Preventing Overdiagnosis 2016 talk
Preventing Overdiagnosis 2016 talkPreventing Overdiagnosis 2016 talk
Preventing Overdiagnosis 2016 talk
 
Role of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisRole of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosis
 
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
 
McGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationMcGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers Association
 
Lessons from 1,889 story reviews
Lessons from 1,889 story reviewsLessons from 1,889 story reviews
Lessons from 1,889 story reviews
 
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
 
Schwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruSchwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, Peru
 
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
 
My talk to University of Wisconsin event, "Science Writing in Age of Denial"
My talk to University of Wisconsin event, "Science Writing in Age of Denial"My talk to University of Wisconsin event, "Science Writing in Age of Denial"
My talk to University of Wisconsin event, "Science Writing in Age of Denial"
 
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
 
Applying HealthNewsReview.org criteria
Applying HealthNewsReview.org criteria Applying HealthNewsReview.org criteria
Applying HealthNewsReview.org criteria
 
Covering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgCovering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.org
 

Kürzlich hochgeladen

Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...
Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...
Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...virgfern3011
 
Anantkumar Hegde
Anantkumar Hegde  Anantkumar Hegde
Anantkumar Hegde NewsFeed1
 
One India vs United India by Dream Tamilnadu
One India vs United India by Dream TamilnaduOne India vs United India by Dream Tamilnadu
One India vs United India by Dream TamilnaduDreamTamilnadu
 
EED - Profits uncontained. An analysis of container shipping ETS surcharges
EED - Profits uncontained. An analysis of container shipping ETS surchargesEED - Profits uncontained. An analysis of container shipping ETS surcharges
EED - Profits uncontained. An analysis of container shipping ETS surchargesEl Estrecho Digital
 
Por estos dos motivos, defensa de JOH solicita repetir juicio
Por estos dos motivos, defensa de JOH solicita repetir juicioPor estos dos motivos, defensa de JOH solicita repetir juicio
Por estos dos motivos, defensa de JOH solicita repetir juicioAlexisTorres963861
 
Another Day, Another Default Judgment Against Gabe Whitley
Another Day, Another Default Judgment Against Gabe WhitleyAnother Day, Another Default Judgment Against Gabe Whitley
Another Day, Another Default Judgment Against Gabe WhitleyAbdul-Hakim Shabazz
 
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...Light Rail in Canberra: Too much, too little, too late: Is the price worth th...
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...University of Canberra
 
Right to Privacy Case Studies PDF …………..
Right to Privacy Case Studies PDF …………..Right to Privacy Case Studies PDF …………..
Right to Privacy Case Studies PDF …………..mathiasanna557
 

Kürzlich hochgeladen (8)

Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...
Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...
Green Aesthetic Ripped Paper Thesis Defense Presentation_20240311_111012_0000...
 
Anantkumar Hegde
Anantkumar Hegde  Anantkumar Hegde
Anantkumar Hegde
 
One India vs United India by Dream Tamilnadu
One India vs United India by Dream TamilnaduOne India vs United India by Dream Tamilnadu
One India vs United India by Dream Tamilnadu
 
EED - Profits uncontained. An analysis of container shipping ETS surcharges
EED - Profits uncontained. An analysis of container shipping ETS surchargesEED - Profits uncontained. An analysis of container shipping ETS surcharges
EED - Profits uncontained. An analysis of container shipping ETS surcharges
 
Por estos dos motivos, defensa de JOH solicita repetir juicio
Por estos dos motivos, defensa de JOH solicita repetir juicioPor estos dos motivos, defensa de JOH solicita repetir juicio
Por estos dos motivos, defensa de JOH solicita repetir juicio
 
Another Day, Another Default Judgment Against Gabe Whitley
Another Day, Another Default Judgment Against Gabe WhitleyAnother Day, Another Default Judgment Against Gabe Whitley
Another Day, Another Default Judgment Against Gabe Whitley
 
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...Light Rail in Canberra: Too much, too little, too late: Is the price worth th...
Light Rail in Canberra: Too much, too little, too late: Is the price worth th...
 
Right to Privacy Case Studies PDF …………..
Right to Privacy Case Studies PDF …………..Right to Privacy Case Studies PDF …………..
Right to Privacy Case Studies PDF …………..
 

AHCJ 2012 Atlanta conf. talk

  • 1. What are your criteria in reporting on health care research? Gary Schwitzer, Publisher, HealthNewsReview.org
  • 2. Our plan for next 2 hours:  Background on HealthNewsReview.org, what we do, how we do it, why, and what we’ve found  Recurring pitfalls we’ve seen in 6 years of daily monitoring  An editor’s perspective on addressing these issues  We’ll review a story – together – using HealthNewsReview.org criteria.  Plenty of time for Q & A – tapping Ivan’s editor expertise  Handouts:  Summary of our 6-year, 1,700 story experience  Resources for reporting on studies  Story to review & scoresheet
  • 4. We review stories that include claims about…  Medical treatments  Tests  Products  Procedures
  • 6. Site stats  28 reviewers: 19 clinicians or researchers, 7 journalists, 2 breast cancer survivors  Site launched April 17, 2006 – 6th anniversary week!  >1,700 stories reviewed  Blog launched in 2004 - >2,500 posts  Sole funding from the Informed Medical Decisions Foundation, founded by Jack Wennberg et al
  • 8. Global Reach for such efforts = HealthNewsReview.org (U.S.), Media Doctor sites in Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in UK and Austria = talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto Rico only in two years’ time
  • 9. Our criteria: Does the story explain…  What’s the total cost?  How often do benefits occur?  How often do harms occur?  How strong is the evidence?  Is the condition exaggerated?  Is this really a new approach?  Is it available?  Are there alternative choices?  Who’s promoting this?  Do they have a financial conflict of interest?
  • 10. After 6 years and 1,700 stories  70% fail to adequately discuss costs.  66% fail to quantify benefit – often exaggerating potential benefit  65% fail to quantify harm - often minimizing potential harm  62% fail to evaluate the quality of the evidence  57% fail to compare new idea with existing options
  • 11. Kid-in-candy-store picture of U.S. health care  Everything is terrific  Nothing is risky  No price tags
  • 12. Recurring themes seen after 6 years of daily monitoring  Failure to explain limitations of observational studies. Stories that conflate association and causation.  Framing numbers to exaggerate benefit, minimize harms  Stories about screening tests that emphasize only benefits, minimizing or ignoring potential harms
  • 15. Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure". They can point to a strong statistical association, but they cannot prove cause and effect.
  • 16. White rice increases risk of type 2 diabetes Thursday, March 15, 2012 The risk of type 2 diabetes is significantly increased if white rice is eaten regularly, claims a study published today on bmj.com. More than 400 words in the news release but NONE about the limitations of such an observational study.
  • 17. Does your language fit the evidence of observational studies?
  • 19. Following slides courtesy Drs. Steve Woloshin & Lisa Schwartz Dartmouth Medical School White River Junction, Vermont VA Part of their syllabus at: NIH Medicine in the Media MIT Medical Evidence Boot Camp
  • 21. For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
  • 22. Women who took Nolvadex had 48% fewer breast cancers…. 48% of what ???
  • 23. It's like a coupon… Extremely Fancy Store What if selected items were.. TV’s, washing machines? 48 On selected items! % OFF save $100s Things like a pack of gum? save pennies ―48% of what‖ matters! Know the REGULAR price!
  • 24. What is the "Breast cancer risk" 48% off Chance of getting breast coupon really cancer worth? Placebo NOLVADEX ??? ??? How much do you save?
  • 25. The chance of getting breast cancer over 6 years in the PLACEBO group PLACEBO 3.3% The REGULAR price! The "base rate" Absolute risk in the placebo group Event rate in the placebo group
  • 26. The chance of getting breast cancer over 6 years in the INTERVENTION group PLACEBO NOLVADEX 3.3% 1.7% The REGULAR price! The SALES price!
  • 27. What is the effect of Nolvadex? How good is the sale? The REGULAR price! PLACEBO The SALES price! NOLVADEX 3.3% 1.7% How much do you save? Absolute risk reduction Regular price – Sales price = 1.6% Savings = 3.3% 1.7% Chance of women breast cancer (over 6 years)placebo for 6 If 100 getting took NOLVADEX instead of with NOLVADEX years, there would be about 2 fewer cases placebo cancer. was 1.6 % points lower than with of breast
  • 28. What is the effect of Nolvadex? How good is the sale? PLACEBO NOLVADEX 3.3% 1.7% Chance of outcome (intervention) Relative Risk = Chance of outcome (control)
  • 29. Describing the effect of NOLVADEX So finally....this is how you get to PLACEBO NOLVADEX the48% off sale! 3.3% 1.7% It's the relative risk reduction 1.7% ??? Relative Risk = = 0.52 ??? 3.3% "% Lower" format =1 - RR =1 - .52 =.48 At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
  • 30. Two ways of saying the same thing: the benefit of NOLVADEX One feels big Extremely Fancy Store One feels small Extremely Fancy Store 4 On selected items! % OFF 1.6% matters! SAVINGS How you say it On selected items! "Framing"
  • 31. Relative vs. absolute"savings" "% off" risk reductions Chance of death at 1 Risk reduction year Relative Absolute Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG) 30% 10% 67% ?? 20% ?? 3% 1% 67% 2% 0.003% 0.001% 67% 0.002%
  • 32. The proof? In a landmark Benefit study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. Harm "Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women". ‖Strokes, cataracts more common with No numbers Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
  • 33. Present benefits and harms the same way
  • 34. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. 210% more uterine cancer and potentially 210% more life threatening blood clots in the lung and legs.
  • 35. Harm 6-yr chance of venous thromboembolic event: PLACEBO NOLVADEX 0.5% 1.0% Like inflation…doubled the price!
  • 36. Over the next 6 years, what happened when women… PLACEBO NOLVADEX Benefits: Nolvadex lowered chance Getting breast cancer 3.3% 1.7% Harms: Nolvadex increased chance Having a serious blood clot 0.5% 1.0% Getting uterine cancer 0.5% 1.1% Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
  • 38. Take home messages Extrapolate with caution! Don't tell people what to worry about – or do – based on very preliminary animal / lab science. Recognize pseudo-evidence Publication in a medical journal - even "the New England Journal of Medicine" - does not guarantee the findings are true (or even important). Be wary of inherently weak science Without a comparison group, it is impossible to be sure if the drug was responsible for the findings. Pay attention to the outcome Surrogate outcomes (like tumor shrinkage) do not reliably translate into clinically meaningful outcomes (longer life). Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure". Avoid exaggerated numbers Use absolute risks for both benefits and harms.
  • 39. Pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124
  • 40. The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators— and the number of experiments, observations and analyses they produce—has also increased exponentially in many fields, but adequate safeguards against bias are lacking. Research is fragmented, competition is fierce and emphasis is often given to single studies instead of the big picture. Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes.
  • 42. Spinning results of randomized clinical trials Boultron et al, JAMA May 26, 2010, 303 (20): 2058 In this representative sample of RCTs published in 2006 with statistically non-significant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.
  • 43. Reporting bias in medical research - a narrative review  McGauran et al. Trials 2010, 11:37  We identified reporting bias in 40 indications comprising around 50 different pharmacological, diagnostic, and preventive interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.
  • 44. BE AWARE OF….. • What Ivan’s project has shown about retractions, research fraud, fabrication, falsification of data • Unpublished data (BMJ recently published 8 articles in one edition on “the extent, causes and consequences of unpublished evidence”) • Conflicts of interest – guideline setting
  • 45. • Commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies (Carl Elliott) • Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.) • The focus on surrogate markers in many studies may be hurting patient care. (primer in HealthNewsReview.org online toolkit)
  • 46. News coverage & poor public discussion of screening tests is one of the most concerning public policy issues. Worst, most biased coverage I’ve seen in 37 years
  • 47. What the Task Force actually wrote: “The decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take patient context into account, including the patient's values regarding specific benefits and harms.”
  • 48. As 37-year ChiTrib & NYT vet John Crewdson wrote in The Atlantic…  “There are multiple reasons women are ill- informed about breast cancer. The fault lies primarily with their physicians, the cancer establishment, and the news media--especially the news media. Until coverage of breast cancer rises above the level of scary warnings mixed with heartwarming stories of cancer survivors, women are likely to go on being perplexed."
  • 49. Other examples:  Chicago Sun-Times  Wall Street Journal  Washington Post  New York Daily News  Minneapolis Star Tribune  All TV networks
  • 50. The ink isn’t even dry on studies before marketing begins
  • 51. HealthDay wire service didn’t challenge researcher promoting universal pancreatic cancer screening for everyone over 50.  After study of tissue from 7 people!
  • 52. We could have given the finger to this story  "Relative finger length could be used as a simple test for prostate cancer risk, particularly in men aged under 60," said one of the researchers.
  • 53. Screening Madness  Crusading  one-sided  advocacy
  • 54. Why don’t we deliver this message? “All screening tests cause harm; some may do good.” But much health journalism consistently emphasizes benefits & minimizes harms
  • 55. A form of disease-mongering  Selling sickness  Selling the search for weapons of mass destruction inside everyone  Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK ―Screening the apparently healthy potentially saves a few lives. But it definitely drags many others into the system needlessly — into needless appointments, needless tests, needless drugs and needless operations. This process doesn’t promote health; it promotes disease. People suffer from more anxiety about their health, from drug side effects, from complications of surgery. A few die. And remember: these people felt fine when they entered the health care system.‖  Communication of the evidence is a key health policy issue
  • 56. ―I honestly believe it is better to know nothing than to know what ain’t so.‖ Josh Billings (pen name of humorist Henry Wheeler Shaw, 1818 – 1885)