This presentation was made by Joseph WHITE, United States, at the 5th Meeting of the joint OECD DELSA/GOV Network on Fiscal Sustainability of Health Systems held on 4-5 February 2016 at the OECD Conference Centre in Paris.

1. Performance Information, Transparency, and
Accountability in the Health Sector:
Experience in the United States (and a bit beyond)
Joseph White, Ph.D.
Case Western Reserve University
Prepared for 5th DELSA GOV meeting on Sustainability of
Health Systems
5 February 2016
OECD Conference Center
1

2. “Paying for Performance” (P4P) in Health Sector and
“Performance Budgeting”
• Common Aspirations –
• Efficiency or Value for Money, Making Economy More Acceptable
• Common Problems –
• Difficulties in Measuring Performance, and Agreeing on Measures
• Both technique and source must be credible.
• Strategic Responses (e.g. “Gaming,” “Teaching to the Test”)
• Using the Results
• One Possible Advantage in Health if Not Budgeting
• Do Not Have to Compare the Incomparable - Mostly
• Huge Problems if “Performance Budgeting for Health”
• Voters Want Rescue and Care, Analysts Look at Health Statistics
• In U.S. (and other countries) Institutional Separation of Programs 2

3. Logical Questions (or choices)
• Information About What?
• Activity vs. Outcomes or “Value”
• Provided to Whom?
• Citizens? Consumers? Patients?
• Payers? Governments? Managers?
• So “Transparent” to Whom? And Why?
• “Legitimacy” or Control?
• Accountability for What?
• Quality? Spending? Compliance?
• Applied to What Parts of the System?
• Hospitals, physicians, medical groups, “health plans”
3

4. Measuring Activity – Usually to Increase Activity
• To Reduce Waiting Times
• NHS “Targets and Terror”
• To Induce Greater Productivity Through Payment
• DRGs outside the United States: “Activity-Based Payment”
• To Improve Management
• “Knowing what you are buying.” DRGs again
• To Encourage Specific Services – Often “Prevention”
• “Pay for Performance” for NHS GPs, U.S. Primary Care Groups
• To Punish Specific Activities
• Readmission rates in United States
4

5. Measuring Quality, Outcomes or “Value”
• Results of Treatment
• New York Cardiac Surgery Reporting System
• Medicare ACOs patient-reported experience of care and health status
• Overall Rankings or Ratings by “Experts”
• Intended to stand in for quality, but usually not based on strong measures.
Many versions in U.S.; NHS Star Ratings
• Various Theories of How These Would Work
• Change: Providers act to improve when see comparisons, even if not public
• Selection: Providers respond to threat of patients selecting other providers
• Reputation: “Naming and Shaming.” e.g. media coverage
• Individual self-interest, e.g. managers could be fired, provider income
affected, budgets could be raised or cut, autonomy “earned.”
• Voltaire on British Navy: “Ici on tue de temps en temps un amiral pour encourager les
autres”
5

6. Measurement Focuses Attention on Some Things, Not Others
6
• Source: larrycuban.wordpress.com

7. “Core Assumptions” in Using Measurement to Inform
“One is that measurement problems are unimportant, that the part on which
performance is measured can adequately represent performance of the
whole, and that distribution of performance does not matter. The other is
that this method of governance is not vulnerable to gaming by agents.”
Danger of “hitting the target and missing the point” (Bevan and Hood, 2006)
7

8. Common Measurement Problems
• Statistical Validity
• Too few cases per provider for specific measures. Bigger problem in U.S.
where providers face multiple payers. U.S. IHA vs. English QOF
• If providers collect the data, may be gamed; if outsiders do, may not be
trusted by providers (and in U.S., often rightly so – IHA).
• Only Measure What Can Observe (see previous slide)
• Black Boxes – What is Being Measured isn’t Transparent
• Aggregate ratings of the “same thing” may be very different
• Documentation May Not Equal Performance
• Either because measures provoke documenting activity that already
occurred, or documentation isn’t accurate (gaming)
• Risk Adjustment is Very Difficult and Contentious
• Measuring “Improvement” Requires Accurate Baselines
8

9. Common Implementation Issues
• Cost in Time and Money
• Cost in Diverted Attention from Other Activities
• Cost in Inverted Incentives if Linked to Sanctions or Rewards
• Ariely et al on theory of motivation. Bevan and Hood on behavior changing
as people are punished for not achieving something they could not achieve.
• But: providers happier to act to “improve quality” if sweetened with $ € £
• Public Rarely Uses “Value” Information
• Does not have same doubts about quality as the experts do.
• Not likely to notice information; if do will ignore highly detailed
presentations; if presentation is simple, may be misleading.
• Governments and other payers may not want bad news on quality
• Payers tend to care much more about cost, do not want evidence they
should spend more.
9

10. “Accountability” Based on Measures Requires
Conditions That May be Hard to Meet
• The Message is Clear and Targeted Properly
• Measures are Perceived as Accurate
• Measures are Perceived as Fair and Legitimate
• (or incentives really strong anyway, but then cheating becomes more likely)
• This result may be related as much to organizational context as to the
measures themselves. U.S. physicians in IPAs were in very different situations
than English GP practices
• Targets of Incentive Are Able to Respond Effectively
• The desired result is potentially subject to significant influence by the
recipient of the incentive (technical capacity)
• Organizational setting provides institutional capacity to respond
• Targets Don’t Have Political Capacity to Change the Rules
10

11. “Accountability” and “Pay for Performance”
• English QOF certainly led to higher recorded “performance,” but at
very high monetary cost.
• English “Targets and Terror” likely did improve target performance
beyond gaming effects. But at some costs on unmeasured
performance dimensions
• U.S. “P4P” experience shows generally modest or null effects.
• “largely disappointing results” (Markowitz and Ryan 2016)
• For physicians, “among the strongest studies, there were no or relatively
small improvements in performance.” In the hospital setting, “all of the
studies… found modest but often statistically insignificant effects” (Damberg et
al. 2014)
• Some acceleration of improvement in early years of Premier demonstration.
• Performance on some Hospital Inpatient Quality Reporting Program
measures is now so high they are being eliminated. But they are very basic
measures.
• Risk adjustment concerns, especially for hospitals with low SES patients
11

12. Some Other Tentative or Ironic Results
• When processes are measured, improved performance is rarely
associated with better final outcomes
• Long or weak causal chains between measured feature and outcome
• Bad (or good) outcomes may be rare. Or outcomes may emerge only after many years
• Comparisons over time are difficult because governments and
other payers keep changing the measures
• U.S. Policy-Makers Want to Use “Transparency” to Reduce
Prices.
• The new-new thing. Not so necessary anywhere else. A whole other topic…
• “Transparency” is sometimes associated with avoiding sicker
patients
• Percutaneous revascularization is less common, and that is associated with
higher mortality, in states that report “quality” of AMI care at the physician
level and those that do not.
12

13. Final Thoughts (for the moment)
• Some use of measures for management “is a form of indirect control
necessary for the governance of any complex system” (Bevan and Hood 2006)
• Measurement results from lack of trust in providers, is likely to
decrease trust, but will work best in the presence of trust.
• If measures are inaccurate they cannot increase “transparency”
• Hardly anybody likes the current U.S. measurement regime:
• “The quality measurement enterprise in U.S. health care is troubled. Physicians, hospitals,
and health plans view measurement as burdensome, expensive, inaccurate, and indifferent
to the complexity of care delivery. Patients and their caregivers believe that performance
reporting misses what matters most to them and fails to deliver the information they need
to make good decisions. In an attempt to overcome these troubles, measure developers are
creating ever more measures, and payers are requiring their use in more settings and tying
larger financial rewards or penalties to performance. We believe that doing more of the
same is misguided: the time has come to reimagine quality measurement.” – (McGlynn et al,
NEJM, 2014.)
• But reform ideas require even more organizational capacity…
13

14. Improvement Through Performance Management
Would Be a Beautiful Thing
• “Unicorn in Captivity,” Property of The Cloisters, Metropolitan Museum of Art. One of the great treasures of New York City

15. Some Sources Reviewed for This Presentation (1)
• Austin, J. M., A. K. Jha, P. S. Romano, S. J. Singer, T. J. Vogue, R. M. Wachter, P. Pronovost. 2015. National Hospital Rating Systems Share Few Common Scores and
May Generate Confusion Instead of Clarity. Health Affairs 34(3): 423-430.
• Bevan, G. and C. Hood. 2006. What’s Measured is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration 84(3): 517-
538.
• Bevan, G. and R. Hamblin. 2009. Hitting and missing targets for ambulance services for emergency calls: effects of different systems of performance measurement
within the UK. Journal of the Royal Statistical Society 172(1): 161-190
• Hood, C. and R. Dixon. 2015. A Government that Worked Better and Cost Less? Evaluating Three Decades of Reform and Change in UK Central Government.
Oxford, UK: Oxford University Press.
• Markowitz, A. A. and A. M. Ryan. 2016. Pay-for-Performance: Disappointing Results or Masked Heterogeneity? Medical Care Research and Review 71 (online first)
1-76.
• Damberg, C. L., M. E. Sorbero, S. L. Lovejoy, G. Martsolf, L. Raaen, D. Mandel. 2014. Measuring Success in Health Care Value-Based Purchasing Programs. RAND
Corporation Research Report sponsored by the Office of the Assistant Secretary for Planning and Evaluation, Department of Health and Human Services
• Friedberg, M. W. and C. L. Damberg. 2012. A Five-Point Checklist To Help Performance Reports Incentivize Improvement and Effectively Guide Patients. Health
Affairs 31(3): 612-618.
• Hibbard, J. H. 2008. Editorial: What Can We Say About the Impact of Public Reporting? Inconsistent Execution Yields Variable Results. Annals of Internal Medicine
148(2): 160-161.
• Hibbard, J.H., J. Greene, S. Sofaer, K. Firminger, J. Hirsh. 2012. An Experiment Shows That A Well-Designed Report On Costs And Quality Can Help Consumers
Choose High-Value Health Care. Health Affairs 31(3): 560-67.
• Kahn, C. N., T. Ault, L. Potetz, T. Walke, J. H. Chambers, S. Burch. 2015. Assessing Medicare’s Hospital Pay-For-Performance Programs And Whether They Are
Achieving Their Goals. Health Affairs 34(8): 1281-1288.
• Laverty, A. A., P. C. Smith, U. J. Pape, A. Mears, R. M. Wachter, C. Millett. 2012. High-Profile Investigations Into Hospital Safety Problems in England Did Not Prompt
Patients To Switch Providers. Health Affairs 31(3): 593-601.
• Lawson, E. H., D. S. Zingmond, B. L. Hall, R. Louie, R. H. Brook, C. Y. Ko. 2015. Comparison Between Clinical Registry and Medicare Claims Data on the Classification
of Hospital Quality of Surgical Care. Annals of Surgery 261(2): 290 – 296
• McDonald, R., J. White, T. R. Marmor. 2009. Paying for Performance in Primary Medical Care: Learning about and Learning from “Success” and “Failure” in England
and California. Journal of Health Politics, Policy and Law 34(5): 747-776.
• McGlynn, E. A., E. C. Schneider and E. A. Kerr. 2014. Reimagining Quality Measurement. New England Journal of Medicine 371(23): 2150-2153
15

16. Some Sources Reviewed for This Presentation (2)
• Parast, L., B. Doyle, C. L. Damberg, K. Shetty, D. A. Ganz, N. S. Wenger, P. G. Shekelle. 2015. Perspective: Challenges in Assessing the Process-Outcome Link in
Practice. Journal of General Internal Medicine 30(3): 359-64.
• Rosenbaum, L. 2015. Scoring No Goal: Further Adventures in Transparency. New England Journal of Medicine 373(15): 1385-1388.
• Ryan, A. M., B. K. Nallamothu, J. B. Dimick. 2012. Medicare’s Public Reporting Initiative on Hospital Quality Had Modest Or No Impact on Mortality From Three Key
Conditions. Health Affairs 31(3): 585-592.
• Smith, M.A., A. Wright, C. Queram, G. C. Lamb. 2012. Public Reporting Helped Drive Quality Improvement In Outpatient Diabetes Care Among Wisconsin Physician
Groups. Health Affairs 31(3): 570-577.
• Tanenbaum, S. J. 2009. Pay for Performance in Medicare: Evidentiary Irony and the Politics of Value. Journal of Health Politics, Policy and Law 34(5): 717-746.
• Waldo, S. W., J. M. McCabe, C. O’Brien, K. Kennedy, K. E. Joynt, R. W. Yeh. 2015. Association Between Public Reporting of Outcomes With Procedural Management
and Mortality for Patients With Acute Myocardial Infarction. Journal of the American College of Cardiology 65(11): 1119-1126.
• Werner, R. M., J. T. Kolstad, E. A. Stuart, D. Polsky. 2011. The Effect Of Pay-For-Performance In Hospitals: Lessons For Quality Improvement. Health Affairs 30(4):
690-698.
• Woolhandler, S., D. Ariely, D. Himmelstein. 2012. Will Pay for Performance Backfire? Insights From Behavioral Economics. Health Affairs blog,
http://healthaffairs.org/blog/2012/10/11/will-pay-for-performance-backfire-insights-from-behavioral-economics/
16