Presentation by Céline Kauffmann and Rebecca Schultz, Regulatory Policy Division, OECD, at the 9th Conference on Measuring Regulatory Performance - Closing the Regulatory Cycle: Effective ex post Evaluation for Improved Policy Outcomes which took place in Lisbon on 20-21 June 2017. Further information is available at www.oecd.org/gov/regulatory-policy/measuring-regulatory-performance.htm.

1. Breakout session 1
INSTITUTIONALISING EX POST EVALUATION TO
FEED BACK INTO THE REGULATORY PROCESS
9th OECD Conference on Measuring Regulatory Performance
Lisbon, 20-21 June 2017
Processes and institutions for effective ex post evaluation
Céline KAUFFMANN, OECD Regulatory Policy Division
Rebecca SCHULTZ, OECD Regulatory Policy Division

2. Stock management
reviews
• Regulator-based
strategies
• Stock-flow rules
• Red tape
reduction targets
Programmed
mechanisms
• Sun-setting
• Ex-post review
requirements in
new regulation
• Post
implementation
reviews
Ad-hoc/special
purpose reviews
• Stocktakes of
burden
• Principles-based
• Benchmarking
• In-depth reviews
The heterogeneity in Ex post evaluation approaches
Ongoing As neededAt a set time
What does it mean in terms of oversight arrangements?

3. • 20 /35 countries have formal requirements for ex post evaluation
• Do regulations include automatic evaluation requirements?
– Primary laws: 3 for all; 3 for major; 12 for some. 14 NEVER
– Subordinate regulations: 5 for all; 2 for major; 11 for some.
• Do regulations include ‘sunsetting’ clauses?
– Primary laws: 1 for major; 17 for some 15 NEVER
– Subordinate regulations: 2 for all; 1 for major; 14 for some.
• When developing regulation, are regulators required to identify a
process for assessing progress in achieving a regulation’s goals?
– Primary laws: 8 for all; 5 for major; 3 for some. 17 NEVER
– Subordinate regulations: 9 for all; 3 for major; 4 for some.
Evidence on ex post evaluation requirements

4. Who conducts ex post evaluation?
Source: OECD (2015), OECD Indicators of Regulatory Policy and Governance (iREG), www.oecd.org/gov/regulatory-
policy/indicators-regulatory-policy-and-governance.htm.
16 12 6 13 7 6
0
5
10
15
20
25
30
35
Government The body itself
Is there a standing
body that regularly
undertakes
reviews of existing
regulations?
If yes, does it have
a degree of
independence
from government?
Who is responsible for deciding what
the body reviews?
Has this body
conducted any "in-
depth reviews" of
specific regulatory
areas in the last 3
years?
If yes, did this
body report its
findings publicly?
Numberofjurisdictions

5. Oversight and quality control mechanisms are
weak particularly for ex post evaluation
Source: Arndt, C., S. Hermanutz, C. Kauffmann and R. Schultz (2016), “Building regulatory policy systems in
OECD countries”, OECD Working Paper on Regulatory Policy, No. 5.
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Systematic
adoption
Methodology Transparency Oversight and
quality control
Stakeholder engagement Regulatory impact assessment Ex post evaluation

6. What do we know of quality control of ex post
evaluation?
Source: OECD (2015), OECD (2015), OECD Indicators of Regulatory Policy and Governance (iREG),
www.oecd.org/gov/regulatory-policy/indicators-regulatory-policy-and-governance.htm.
13
6
1
4
0
5
10
15
20
25
30
35
Is there a quality control system for ex-
post evaluations?
The ex-post evaluations are reviewed by
an independent body who is responsible
for ensuring each type of assessment is
completed
Numberofjurisdictions
For some primary laws
For major primary laws
For all primary laws
Yes

7. • Built-in checks and balances to resist inevitable political pressure
while staying connected to political decision-making
• Create linkages between ex ante and ex post evaluation
• Collect relevant data and build internal capacities for data analysis
• Take into account the views of the regulated
• Make it a good policy making and not another « procedure »
• Promote ownership of regulatory stock within the administration
• Transparency about oversight activities
Critical elements for an effective evaluation
system identified at Sydney conference

8. Body in charge of regulatory
oversight
Merits Challenges
Line ministry  performs RIA for a particular
type of impact
 no integrated approach to
regulatory oversight
 may lack authority/credibility
Central government  can take on a bird’s eye view
 access to information from
different parts of government
 may face more pressure for
exemptions
Independent body  provides an independent view
to parliament and the general
public
 may be too far from decision-
making processes
Parliament  independent from government  actor that is involved in the
development of regulations at a
late stage
 not all regulations pass through
parliament
Merits and challenges of different oversight
models for ex ante and ex post evaluation

9. What metrics to assess evaluation bodies?
Institutions i) Analytical capability? ii) Linkages? iii) Transparency? iv) Independence?
Units within Ministries ✔✔ ✔✔✔ ✔ ✔
Private consultants ✔✔ ✔ ✔ ✔✔
Government research
and advisory bodies ✔✔✔ ✔✔ ✔✔✔ ✔✔✔
i) Analytical capability (including quantitative)
ii) Direct linkage to regulation-making processes
iii) Ability to test work with stakeholders
iv) Ability to avoid ‘undue influence’
Source: Gary Banks, Presentation to OECD’s 8th Conference on Measuring Regulatory Performance, 15 June 2016,
Sydney, Australia

10. • What oversight arrangements and allocation of institutional
responsibility are needed to support high-quality ex post evaluation?
How can the contribution of institutions for evaluation at arm’s
length from government be maximised?
• How can effective oversight mechanisms support a feedback loop to
ensure insights from evaluation lead to better policy outcomes?
• How can ex ante and ex post evaluation be linked more effectively to
avoid regulation being adopted without a prior analysis of the
existing regulatory framework in a given sector?
Focus for the breakout session