Presentation at CES Toronto 2013 Evaluation Conference by Robert Lahey

1. Rethinking the Relationship between
Evaluation and Performance
Measurement/Monitoring – and RBM
Robert Lahey
Presentation to the Canadian Evaluation Society
Annual Conference
Toronto: June 10, 2013

2. RELahey@rogers.com - CES Conference 2013 2
Talking Points
Complementarity? – the theory vs the practice
Some observations – Canada; International
experience
Some considerations for RBM

3. RELahey@rogers.com - CES Conference 2013 3
Two Tools to Measure ‘Performance’
E – Evaluation (Evaluators)
M – Performance Measurement/Monitoring
(Program Managers)
Continuum for measuring ‘performance’
(results chain)

4. RELahey@rogers.com - CES Conference 2013 4
The Theory
 M supports E
 E supports M
 Various notions of ‘complementarity’
* Informational * Sequential
* Organizational * Methodical
* Hierarchical
 Reference: New Directions for Evaluation, No. 137,
Spring 2013

5. RELahey@rogers.com - CES Conference 2013 5
The Practice
 Complementarity? Taken advantage of?
 Can and do organizations (and governments) use the
M&E information in a coherent system?
 Observations: from Canada; Internationally
 Experience to date? - good, bad & ugly

6. RELahey@rogers.com - CES Conference 2013 6
The Good
 E supporting M – derivation of performance
frameworks, relevant indicators
 Moving the focus up from activities to include
‘results’
 A more systematic, structured & results-
oriented approach to understanding program,
theory & articulating expected results
 ‘Methodical complementarity’

7. RELahey@rogers.com - CES Conference 2013 7
The Bad
 M not supporting E to the level expected (by
central authorities & senior officials)
 To a large extent ‘results’ still not being
measured by M – for a variety of reasons:
* lack of data to populate indicators
* methodological issues re measuring outcomes
* Managers not equipped to carry out M (resource,
skill & time constraints)

8. RELahey@rogers.com - CES Conference 2013 8
The Ugly
 Cases where E being ignored as an important
tool to measure & understand performance
 Unrealistic expectations re the ability of M to
deliver cost-effective approach to measuring
outcomes
 Dumbing down of performance reporting
* Observations vs understanding

9. Some Conclusions
 Some level of complementarity
(opportunities)
 But, limits to this – much relates to practical
implementation issues
 Extent that M can support E is probably
overstated
 Importance of informing/educating senior
officials – in terms meaningful to them
RELahey@rogers.com - CES Conference 2013 9

10. RELahey@rogers.com - CES Conference 2013 10
Some Considerations for the Governance
Model that M&E Supports
 Both M and E - key tools to generate
performance information to support RBM
 ‘Results’ information - various uses & users:
* Learning/Knowledge * Internal Needs
* External Needs
* Accountability * Internal Needs
* External Needs

11. RELahey@rogers.com - CES Conference 2013 11
Potential Uses/Users for M&E Information
M E
Learning – Internal Use Learning – Internal Use
Learning – External Use Learning – External Use
Accountability – Internal Accountability – Internal
Accountability - External Accountability - External

12. The Practice – M, E and RBM
 Is there a coordination of M and E to support
RBM?
 Some Differences:
* Different players in their production
* Different timelines
* (Potentially) serving different purposes
* Operational disconnect between the two?
RELahey@rogers.com - CES Conference 2013 12

13. RELahey@rogers.com - CES Conference 2013 13
Focus of M and E – largely on
‘Accountability’ for External Audiences
M E
Learning – Internal Use Learning – Internal Use
Learning – External Use Learning – External Use
Accountability – Internal Accountability – Internal
Accountability - External Accountability - External

14. RELahey@rogers.com - CES Conference 2013 14
Rethinking the Relationship between M, E
and RBM – Measurement Considerations
 How should E support M? M support E?
 Appropriate role for Evaluators? Program
Managers?
 Is something missing within organizations to
deliver on the measurement needs of RBM?
 Are organizations/governments willing to
resource to the level needed?
 Move from silos to ‘knowledge strategy’

15. RELahey@rogers.com - CES Conference 2013 15
Rethinking the Relationship between M, E
and RBM – Governance Model
 What should be the appropriate balance for
both M and E re:
* Uses: a focus on ‘accountability’ vs ‘knowledge’?
* Users: Internal vs External?
 More clarity likely needed around ‘uses’ within
organizations
 Capacity building of ‘users’

16. RELahey@rogers.com - CES Conference 2013 16
Contact Coordinates
Robert Lahey
REL Solutions Inc.
Ottawa, Canada
Tel.: (613) 728-4272
E-mail: RELahey@rogers.com