This is a paper presentated to CES Toronto 2013. It describes a case study of how a management process is being used to embed effective design of regulatory compliance approaches into practise, and how it makes space for monitoring and evaluation. It highlights the importance of team formation in bringing expertise toegether from across the regulatory system (standard setting, monitoring and behaviour change). It also highlights some of the lessons from the initial application of the framework

1. Risk Based Compliance Framework as an
Approach to Engage Practitioners in
Evaluation
Canadian Evaluation Society Conference 2013
Mark White
Ontario Ministry of the Environment

2. Purpose
• Describe the Risk Based Compliance Framework
• A case study in the use of a management process to
engage practitioners in evaluation
• Highlight our learning relevant to the themes of
engagement across boundaries and embedding
evaluation in the organization
• It has to be needed... now!
• Familiarity breeds... engagement
• It helps to be good enough, not perfect
• Adapt to the environment
• Providing a trigger for evaluation
2 Ministry of the Environment

3. 3
Risk Based Compliance Framework: Overview
Context
•A healthy environment for a strong Ontario
•Compliance approach = a combination of
tools to monitor compliance and to influence
behaviour
Aim
•To develop compliance approaches that
ensure efficient and effective delivery of
regulatory objectives
Key Features
•Integrates expertise across organizational
boundaries
•Systematically considers influences
Benefits
•Focus on evidence based decisions
•Focus on communicating the benefits we
deliver
3 Ministry of the Environment

4. Risk based Compliance Framework: Steps
4

5. The Risk based Compliance Framework and Evaluation
5 Ministry of the Environment

6. Performance Monitoring
6 Ministry of the Environment
• Performance Monitoring
underpinned by logic model
 A conceptual framework forming
the basis for a management
scorecard
 Intuitive way of linking activities
to regulatory objectives
 Tailored for each compliance
approach
• Challenges
 Outcome measures
 Limited evidence base
 Cost effectiveness

7. Engaging Practitioners in Evaluation
7 Ministry of the Environment

8. Theme 1: It has to be needed… now!
8 Ministry of the Environment

9. Theme 2: Familiarity breeds… Engagement
•Relationship between evaluative
process and context is important
for engagement
 Familiarity with concepts
 Familiarity with tasks
 Fit with process
•Engagement on easier tasks
provides basis for more complex
evaluative tasks
9 Ministry of the Environment

10. Theme 3: Good enough, not perfect
10
*
* Haynes, L. et al. (2012). Test, Learn, Adapt: Developing Public Policy
with Randomised Controlled Trials, Cabinet Office. Behavioural Insights
• Working together to improve
the framework
 Improves the product
 Improves engagement
• Pilot project
 Test on a ‘Real’ problem
 Engages project team
 Engages management
• Awareness raising as an
opportunity to learn and
adapt

11. Theme 4: Adapt to the Environment
11 Ministry of the Environment
Darwin’s finches*
*Wikimedia Commons (Public
Domain)
• Adapting the framework to
meet the needs of the
specific circumstances
 Subject matter
 Tailoring the framework to context
 Importance of understanding
interdependencies between
compliance tools
 Limitations in evidence base
 Users
 Integration of different subject
matter expertise
 Decision required
 Most effective & efficient
combination of approaches across
compliance program

12. Theme 5: Providing a ‘Trigger’ for Evaluation
12 Ministry of the Environment
• ‘Trigger’ points for
evaluation are important
 Using a management
process to ensure evaluation
is considered in a timely
fashion
 Consideration of what is good
practice
 Development of Monitoring and
Evaluation Plan prior to
approval

13. Conclusion
 Evaluation across Boundaries
– Functional & Professional
 Engaging Practitioners in Evaluation
– Management process:
o Driven by need to promote effectiveness
o Approach tailored to the circumstances
– Evaluation:
o Design Effectiveness
o Implementation Effectiveness
o Outcome Effectiveness
13 Ministry of the Environment
Questions?
mark.white@ontario.ca