Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Hier ansehen

1 von 33 Anzeige

Weitere Verwandte Inhalte

Ähnlich wie Pic talk (20)


Aktuellste (20)

Pic talk

  1. 1. Performance Information Use: Cognition & Design Donald Moynihan Washington DC, May 15, 2015
  2. 2. Research & Practice • We are surrounded by data – what do we do with it? • What’s the role of research? – Practice and reforms outpace research – We “do” before we “know” – Research is better at explaining what is, than what can be
  3. 3. Two factors that matter • Cognition: how individuals process performance information – Strong evidentiary basis on biases & preferences – Cannot control, but can be aware of, compensate for • Design: how organizations structure how information is received and used – Weak evidentiary basis – case studies & surveys – Can more directly control – Some cognitive insights can inform design
  4. 4. COGNITION: BIASES, HEURISTICS AND PREFERENCES It aint what you don’t know that gets you into trouble, its what you know that just ain’t so - Mark Twain
  5. 5. Negativity bias • Loss aversion – People are more motivated by loss than equivalent gain – We see this in how media chooses to cover public sector
  6. 6. Negativity bias • Citizens evaluate services more negatively if the same performance data is presented in a negative rather than a positive terms – if individuals are told that 10% of patients are dissatisfied, they are more likely to be critical of a program than if told that 90% are satisfied (Olsen 2015)
  7. 7. Negativity bias • Public more critical of UK political incumbents when performance data is labeled as poor, but do not provide equivalent credit when performance is good (James 2011). • Less votes for incumbent local govt officials with low performance scores, but positive performance scores are not equivalently rewarded (Boyne et al 2009).
  8. 8. Negativity bias in resource decisions • Limited effect of Bush-era program performance scores concentrated on poor performers (Gilmour and Lewis 2006; Rhee 2014) • Punishing poor performers might be counterproductive: – Low NCLB scores lead to lower success in referenda on school levies, which in turn lower performance (Kogan, Lavertu and Peskowitz 2015)
  9. 9. Partisanship • Conservative elected officials more likely to believe in logic of personalized responsibility – that leaders control outcomes (Nielsen and Moynihan 2015A) • Ideological distance affects how performance data is used: – Agencies defined as liberal tended to receive lower performance scores, more scrutiny, more improvement plans than conservative peers (Lavertu, Lewis & Moynihan 2013)
  10. 10. Advocates can mute negativity bias • Advocacy can be successful… – The County Sheriffs Office is responsible for a wide variety of law enforcement activities. The Narcotics Task Force focuses on illegal drug use. A number of media reports in the last year have focused on an increase in drug-related crime, and have been critical of local law enforcement. Treatment: The County Sheriff acknowledges that drug use and related crime may be up, but says “We usually see this sort of thing spike when the economy is in recession. This means that the Narcotics Task Force needs more money to tackle this problem.” (Moynihan 2015) • …but might depend on audience – Comments by teachers unions criticizing test scores reduced tendency to hold school principals personally accountable, but only for liberal elected officials (Nielsen and Moynihan 2015b)
  11. 11. Expectancy disconfirmation • Expectations about appropriate level of performance affect how actual performance interpreted • Targets anchor expectations about what constitutes good or bad performance • Experiment: – Control: job training program showing good performance – Treatment: Addition of performance targets, which are higher than actual performance
  12. 12. Result: Missed targets (even unrealistic ones) weakens support for programs (Moynihan 2015)
  13. 13. Leadership attribution bias • Organizations are complex systems, outsiders need simple explanations • Leader a personal heuristic – We tend to find personalized explanations more salient than situational ones – Leaders are easily accessible explanation – The organization becomes understood through a person • Need for explanation increases in context of unusual outcomes
  14. 14. Elected officials more likely to believe school principals cause performance when performance high or low (Nielsen and Moynihan 2015a)5.8 6 -.5 0 .5-.4 -.3 -.2 -.1 .1 .2 .3 .4 Performance Level Predicted values of leadership attribution L e a d e r s h i p a t t r i b u t i o n
  15. 15. Data sources may affect credibility • Public find data showing very good performance more credible when it comes from independent source, rather than agency itself (James and Van Ryzin 2015)
  16. 16. DESIGN Any jackass can kick down a barn, but it takes a good carpenter to build one - Sam Rayburn
  17. 17. From case & survey-based studies • Performance information use higher: – Culture that values mission or innovation – Leader is committed to performance, articulates compelling vision – Goal clarity (Moynihan & Lavertu 2012)
  18. 18. • Problem with governmentwide reforms: can’t use experimental methods • Can we find reasonable “treatment” for reform? Effect of governmentwide performance reforms
  19. 19. Findings • Involvement in Government Performance and Results Act, or Program Assessment Rating Tool not associated with purposeful use • Involvement is associated with passive use
  20. 20. Learning forums • Governments built performance systems on routines of measuring and disseminating data • Routines of performance information use an afterthought • Case studies: Organizations that use performance data a lot tend to have learning forums – Routine discussions of data focused on goal achievement, mixes data and experiential knowledge (Moynihan 2008;Hatry and Davies 2011; Behn 2014)
  21. 21. Learning routines: quarterly reviews • Is exposure to quarterly reviews mandated by Modernization Act associated with learning? • Evidence from GAO (2012-2013) survey • Exposure to quarterly reviews associated with higher performance information use (Moynihan and Kroll 2014)
  22. 22. A well-designed learning forum • Meetings take place on a routine basis • Focus on important goals • Agency leaders are involved and seen as committed • Multiple level of employees facilitate learning and problem solving • Need appropriate and timely information • Need staff and technological capacity to analyze data • Quality data (reliable, accurate, valid, disaggregated to the right level, comparative) • facilitates analysis • Follow-up on issues raised in prior meetings • Positive reinforcement • Constructive feedback • Reviews establish process of analysis
  23. 23. Learning forums: quarterly reviews • When quarterly reviews are well run, association with performance information use even stronger (Moynihan and Kroll 2014)
  24. 24. Managerial autonomy and performance information use • Basic claim: autonomy increases incentive to use performance data • Experimental test from Denmark: some school principals given more autonomy on hiring than others • Those granted more autonomy more likely to download performance data seven months later
  25. 25. Motivating effects of comparison • Provision of comparative data (how you ranked compared to other schools) made principals more likely to download data
  27. 27. Guard against inherent tendency to… • Use measures punitively • Overvalue importance of leaders • Be more critical in use of data for programs we disagree with
  28. 28. Design Principles • Present performance in terms of achievement not failure • No unrealistic targets • Look to work with independent sources to communicate performance • Advocate for your program, explain outcomes • Construct learning routines & run them well • Build in autonomy • Compare with others
  29. 29. Conclusion Welcome your feedback and questions Google: Performance Information Project dmoynihan@lafollette.wisc.edu @donmoyn
  30. 30. References Boyne, George, Oliver James, Peter John, Nicolai Petrovsky. 2009. “Democracy and Government Performance: Holding Incumbents Accountable in English Local Governments.” Journal of Politics 71: 1273–1284. http://bit.ly/1tOHsFj Calmar Andersen, Simon and Donald Moynihan. 2015. Bureaucratic Investments in Expertise: Evidence from a Randomized Controlled http://bit.ly/1L45Eax Gilmour, John B., and David E. Lewis. "Assessing performance budgeting at OMB: The influence of politics, performance, and program size." Journal of Public Administration Research and Theory 16.2 (2006): 169-186. http://bit.ly/1HjkziQ Lavertu, Stephane, David Lewis and Donald P. Moynihan. 2013. “Administrative Reform, Ideology, and Bureaucratic Effort: Performance Management in the Bush Era.” Public Administration Review 73(6): 845-856. http://bit.ly/1AFfgVU James, Oliver. 2011. “Managing Citizens’ Expectations of Public Service Performance: Evidence from Observation and Experimentation in Local Government.” Public Administration 89(4): 1419–1435. http://onlinelibrary.wiley.com/doi/10.1111/j.1467- 9299.2011.01962.x/abstract Kogan, Vladimir, Stéphane Lavertu, and Zachary Peskowitz. 2015. Performance Federalism and Local Democracy: Theory and Evidence from School Tax Referenda American Journal of Political Science bit.ly/1K5URiB
  31. 31. Moynihan, Donald P. “Uncovering the Circumstances of Performance Information Use: Findings from an Experiment.” Public Performance and Management Review. http://bit.ly/1A2IXTw Moynihan, Donald P. and Stéphane Lavertu. 2012. “Does Involvement in Performance Reforms Encourage Performance Information Use? Evaluating GPRA and PART.” Public Administration Review 72: 592-602 http://bit.ly/1H5Fa4J Moynihan, Donald P. and Alex Kroll. 2014. Performance Management Routines that Work: AN Early Assessment of the GPRA Modernization Act. La Follette School Working Paper No. 2014-005 http://bit.ly/1LFfoK4 Olsen, Asmus. 2015. Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study. Public Administration Review 75(3): 469-478 http://onlinelibrary.wiley.com/doi/10.1111/puar.12337/abstract Nielsen, Poual Aees and Donald Moynihan. 2015. Elected Officials’ Assignment of Bureaucratic Responsibility for Performance: A Romance of Leadership? http://bit.ly/1EI0MCJ Nielsen, Poual Aees and Donald Moynihan. 2015b. Biases and Governing: Experimental Evidence from Elected Officials Use of Performance Data. Paper to be presented at 2015 Public Management Research Conference. Rhee, Dong-Young. "The Impact of Performance Information on Congressional Appropriations." Public Performance & Management Review 38.1 (2014): 100-124 http://www.tandfonline.com/doi/abs/10.2753/PMR1530-9576380105. Van Ryzin, Greg and Oliver James. 2015. Incredibly Good Performance: . American Review of Public Administration An Experimental Study of Source and Experimental Effects on the Credibility of Good Government http://bit.ly/1EIshvN

Hinweis der Redaktion

  • Reforms based on “administrative doctrine” – not strong evidence
    Supply side logic: if you build data, people will use it
    Core public services: requirements

    Drawing research from field of public management, mostly experimental designs, so high causal validity. Most evidence on performance management is really bad, but that is changing
  • Individuals
    Most of us not naturally number-crunchers
    Data does not talk for itself; has to be interpreted, shaped by individual preferences and biases
    Experimental designs starting to help us understand how this works

    Design is harder!
    We have strong evidentiary basis on cognitive limitations to draw on (psychology & behavioral econ), easy to test via survey experiments
    Less strong evidence and harder to test design issues
    Govts rarely implement real changes using experimental design, left with observational studies
  • To test whether the significant squared performance term was an artificial result of the model specification, we tested whether additionally introducing cubic and quadratic performance terms changed the findings. Adding the cubic term made no changes to the coefficient or significance of the squared term, and additionally adding the quadratic term only slightly increased the p-value of the squared term, though remaining significant at the 0.10 level, most likely as a result of multicollinearity. Neither the cubic nor quadratic performance terms had significant effects. As an additional robustness test of the curvilinear relationship making no assumption about the functional relationship, we regressed leadership attribution on different sets of raw dummies for different performance intervals, again producing results that closely resemble those presented here.
    To test whether the curvilinear pattern is a more general phenomenon in elected officials’ attitudes to school management and organization (and thus possibly unrelated to a RoL effect), we examined a number of other variables measuring elected officials attitudes to school resources, goal-setting autonomy, and different aspects of operational autonomy (Verhoest et al. 2004), and the findings indicated no signs of other curvilinear relationship with performance. This evidence is consistent with Meindl et al. (1985) who used an experimental design to show that individual beliefs about other organizational factors did not demonstrate the same curvilinear relationship with performance as did leadership performance. We discuss the implications of our findings in greater depth in the next section.
  • The principals with high level of additional discretion download on average 34% more reports than the control group.