Paper by Florentin Blanc & Giuseppa Ottimofiore, prepared for the 6th Expert Meeting on Measuring Regulatory Performance: Evaluating Stakeholder Engagement in Regulatory Policy, Breakout Session 3, The Hague, 16-18 June 2014. Further information is available at http://www.oecd.org/gov/regulatory-policy/

1. 1
Academic pre-conference – The Hague, June 16, 2014
Florentin Blanc & Giuseppa Ottimofiore
Specific Challenges of Stakeholder Engagement Seen from the Experience of Reform
Practice
(both in OECD countries e.g. France, Slovenia, Italy or the UK – and outside of OECD in
emerging/transition economies)
Introduction
While willing to improve transparency, in particular through stakeholder engagement,
governments may be confronted with a number of issues presented below.
By way of introduction, two particular issues related to stakeholder engagement can be
highlighted:
- In certain circumstances, stakeholder engagement can be asked by authorities genuinely
looking for input into a still open policy; in other cases, such engagement will be aiming at
building buy-in and support for an already (mostly) decided policy. If the second sort is
done in a way that is perceived as really “fake” (i.e. nothing is really open for discussion at
all) it typically fails. However, it is important to underline that most situations are on a
continuum between these two extremes.
- Stakeholder engagement is characterized by its cost in financial terms, but also in terms of
duration –adequate resources are required, as the process is by its very nature expensive
and time-consuming (it requires time, but at the time, if the process is too slow, it may
generate other problems, such as: the fact that support for change drops and adverse
effects of status quo are not being addressed). Nevertheless such costs represent a major
investment –provided that it is correctly done– of both resources.
I. Preliminary remarks: definitions and context
(a) About stakeholders
Stakeholders need to reflect not only businesses and citizens/consumers/non-governmental
organization (that is to say representatives of civil society) but also officials/inspectors.
Concerning the legal nature of stakeholders:
- Representatives of civil society (business and citizens/consumers/NGOs): they are now
often an integral part of a decision-making process, in particular of a legislative process
(European Union –“A model for stakeholder dialogue and consultation Charter action
line”: http://ec.europa.eu/enterprise/policies/sme/best-practices
/database/SBA/index.cfm?fuseaction=practice.detail&gp_pk=107&). In fact, this type of
consultation is beginning to be seen as the willingness of the legislator to broaden the

2. 2
legislative process through an “extended impact assessment” –in order to improve the
quality of the proposals and simplify the regulatory environment. The democratic
legitimacy of the process can be just improved. Representatives of civil society are directly
affected by regulation, since they are subject to and/or beneficiaries of regulation.
- Officials/inspectors: as “representatives” of the administrative bodies, they are also part
of the executive power. They have an extended experience –and skills– of the practice of
law and their engagement can therefore add a particular value to the process. Moreover,
they may have a direct interest in getting a satisfactory outcome of the process. Let it be
recollected that officials/inspectors are those implementing the regulations (which can be
positive or negative).
(b) About stakeholder engagement
- Possible/desired effects of stakeholder are:
- an increased transparency,
- an increased effectiveness (by addressing shortcomings in regulation and regulatory
delivery),
- a decreased burden (by targeting problems flagged by users/regulatees, etc.)
- and an increased legitimacy of governments and regulations –both because of the process
itself (increased transparency as a procedural justice effect) and because of increased
effectiveness and reduced burden. In fact, legitimacy is both a product of other effects and
a driver of effectiveness.
- Stakeholder engagement can only have these effects if it fulfills a certain number of conditions –
none of which are easy to meet:
- Representation of stakeholders has to be large/comprehensive enough –and felt to be
really representative. Should this not be the case, the procedural justice effect will fail to
materialize (as will not feel themselves duly represented/consulted, the legitimacy of the
process will not increase). A biased perspective also means that policymakers and officials
will not be getting really representative insights (and hence are likely to make wrong
decisions on what needs to be done to improve regulations, processes, etc.).
- Stakeholders’ feedback has to be informed and meaningful.
o Informed: means that stakeholders are given sufficient information to put their
current situation in perspective and/or that the questions/feedback mechanisms
are structured so as to ask stakeholders from a perspective that already
incorporates some knowledge about “benchmarking” (what could possibly be done
better based on other experiences).
o Meaningful: means that questions are asked/structured in a way that does not lead
respondents/participants to just voice whatever their latest “mood” is (to this end,
subjective questions or questions without background leading to emotional
responses must be avoided). Questions should be presented in a way that leads to
precise answers based on facts, and as much as possible give precise
information/feedback and suggestions on concrete issues.

3. 3
- Feedback/participation/input has to be free (unconstrained): to achieve this,
stakeholders need to be sure that there is no risk of reprisals (most common would be
businesses afraid to criticize regulators –and workers afraid to criticize employers).
- Correctly structured engagement/feedback should help avoid “Risk Regulation Reflex”
kind of loops. These are based on what could be called “the wrong kind of stakeholders’
feedback” (biased, unrepresentative, uninformed, etc.).
- “Engagement” is a term that needs to be specified: what is targeted here are the following
forms/mechanisms of stakeholders engagement:
- Consultations: they must exist before the adoption of new regulations (or changing
institutions, practices, etc.). They can also be foreseen in order to assess implementation
and challenges; in this case, consultations may take the form of roundtables, workshops,
etc. (with stakeholders at grassroots level or with representatives of business
associations, trade unions, NGOs, etc.). NGOs have less risk to be afraid, but have their own
agenda/bias.
- Surveys: direct questions are sent to a sample of stakeholders –statistically representative
(in theory/preferably).
- Feedback mechanisms (typically online/application/hotline) where stakeholders
voluntarily give their input based e.g. on their most recent experience with a regulator and
so on.
Certainly, each of these tools has specific limitations, issues and strengths.
Our focus concerns here specifically:
- Specific stakeholders: mostly businesses, giving feedback/engaging on regulatory and
regulatory delivery issues, particularly on business inspections/regulatory enforcement
and inspections.
- Mechanisms: all of the above-mentioned. We will also take a look at the issues of access to
information and transparency requirements (in particular, proactive transparency, which
is the pre-condition for feedback/engagement –if you don’t know what is the
process/what are the rules, how can you engage/give feedback meaningfully?).
- Stakeholder engagement/feedback must be in relation to regulatory enforcement and
inspections.
II. Challenges due to lack of experience, reference points, objectivity and assessments
(a) The "risk" for stakeholders to be too open and thus jeopardize their relations with
regulators
This phenomenon is seen everywhere around the world –surveys recently conducted by the WBG

4. 4
in Kyrgyzstan saw a high number of people refusing to answer because they were afraid of
potential negative consequences. In the UK some time ago inspection officials sometimes were
handing out satisfaction survey forms at the end of visit –obviously resulting in somewhat biased
answers. The fear of potential negative consequences is universal and requires proper tools
(trust in confidentiality for “grassroots” feedback –reliance on representative associations when
discussions are held) but also a change in regulatory enforcement behaviour (stakeholders need
to trust regulators that they really want to get open feedback).
Over several surveys in Tajikistan, readiness to respond to corruption questions (“objective”
questions i.e. “did you or others have to pay/make gifts” and not “subjective” questions such as
“is corruption a problem”) decreased massively. In 2003, more than 90% reported paying bribes
during various administrative procedures. In 2006, this decreased so brutally that the WBG team
had to try and ask the question in different ways (for example: ‘did you give gifts’, ‘use friends’
‘not only during the procedure but before/after’) and also look at “refused to respond” and other
‘hints’ of what the real level could be. Even when guaranteed confidentiality, many people will be
cautious (probably based on well-grounded concerns). Hence, it is difficult to get feedback
precisely when the problems are worst.
(b) The difficulty often for stakeholders to perceive an issue as being a problem
(business inspections, by way of example)
This can happen because of the fact that they lack a reference point to another type of practice in
another country and thus think this is the "natural" state of things.
Examples (from World Bank Enterprise Surveys):
- In practical terms, transportation in Tajikistan is a major issue (lack of paved roads
and/or horrendous conditions of said roads, very limited rail links to the outside, rare
international flights etc.), but only 2 to 5% of respondents in 2002-2005 rated it as a
“major constraint”. While real improvements (however limited) took place after 2005, the
percentage suddenly jumped by a factor of 10 to nearly 23%. This suggests that people
respond to such questions based on the salience of various issues for them at a given time,
their expectations, their points of comparison (or lack thereof), possibly the order of
questions, etc. A roughly similar comment can be made for data on Kyrgyzstan and
Armenia. In the same vein, in 2005, nearly 11% of respondents in Spain rated
transportation a major constraint –far more than in landlocked Tajikistan and Kyrgyzstan,
more even than in Armenia. While this may correspond to different propensities to trade
beyond an enterprise’s home city (if no trade is undertaken beyond a few kilometers,
transportation may indeed not be a “constraint”), it is nonetheless clear that such data is
quite simply irrelevant when one tries to determine the “picture” for a given country in
terms of constraints for economic development.
- Similarly over several surveys the percentage of respondents judging corruption a “major
problem/constraint” in Tajikistan has decreased, even though this is clearly not reflecting
reality, but rather changed perceptions.
III. Challenges related to feedback

5. 5
(a) Scepticism on stakeholders' side as to whether their feedback will have any impact
In Germany it was reported that it is highly difficult to engage SMEs in regulatory impact
discussions –essentially they do not see the interest or have other things to do. The same
phenomenon is seen in many developing/emerging economies, as well as in OECD countries, etc.:
businesses have no time and/or do not think that there will be an impact. To some extent, low
income countries represent an exception, but there the businesses have usually a much weaker
position –their participation is mostly mediated through international projects/institutions– and
businesses are still sceptical about the outcomes.
It seems that business engagement is improving e.g. in the United Kingdom –BRDO experience
(LEPs, national fora, etc.).
(b) Provisions to engage stakeholders in order to get feedback on how regulation is
implemented, delivered and enforced are not much/always used
Examples on hotlines and similar services: in Kyrgyzstan around 10% of businesses inspected by
ecological inspectorate use the hotline –though their answers on other questions suggest that
problems arise in far more than 10% of cases. Similarly, in Tajikistan, in spite of more than 50%
of businesses reporting major problems in surveys hotline setup on inspections a few years ago
(2007-2011) receive nearly no calls.
The situation is not always better in more developed countries –in fact, as shown in Latvia
inspections reform in 1998-2001, complaints actually increased after the reform because
entrepreneurs now trusted the system and thought their complaints could have results.
(c) Relatedly, “transparency” provisions do not always work
Slovenia has a “best practice” law on transparency according to which all government documents
that impact on third parties should be made public upon request (or, whenever possible, even
before). However, in practice, when one asks, for instance, inspectorates about the guidelines for
inspectors –which have a direct impact on what happens during inspections– and about strategic
planning (which has a direct effect on their effectiveness and therefore is of public interest), the
response is “these are confidential documents”.
Apparently there is a strong transparency ‘watchdog’ in Slovenia but: maybe, on the one hand, no
business/association asks for these –because they don’t know this is important, don’t know this
is public information in some other countries, etc.– and/or on the other hand, they are afraid of
conflict (they may believe that the inspectorate will not like the request and therefore will like
even less if businesses/associations appeal to the watchdog upon the inspectorate’s refusal).
(d) When stakeholders do make their voices heard on regulatory implementation
delivery, enforcement and inspections, it is mostly in response to some adverse
event and in a "risk regulation reflex" style
This case concerns mostly “non business” stakeholders, but in fact not only: businesses, in
particular SMEs, tend to complain loudly about adverse events that affect them and ask then for
more regulation and stricter enforcement (especially if not directly affecting them). At least this
is the way the “unorganised, uninformed” risk-regulation reflex feedback is perceived (that is to
say that these are the voices which are being amplified through the media/political process).

6. 6
IV. Lessons learned and possible solutions
(a) The difficulty to get "smaller" actors engaged
The greatest difficulties are the lack of time, lack of awareness and lack of trust.
It is therefore of paramount importance to devise specific tools, approaches, fora, etc.
- Examples:
- LBRO (predecessor of BRDO) published a report called “From the Business End of
the Telescope” some years back, which set out a prefiguration of much of the
subsequent work.
- Business Reference Panel: this group of around 60 organisations and trade
associations, representing a wider network of around 750,000 businesses, helps
BRDO to better understand the views of the business community with regard to
regulation. It explores issues, identifies potential solutions and gives feedback on
BRDO’s ideas and proposed work programmes to ensure that issues of real
importance to a broad base of the business community are addressed:
https://www.gov.uk/government/publications/business-reference-panel-terms-
of-reference
BRDO maintains a close relationship with nine core members:
 Association of Convenience Stores
 British Chambers of Commerce
 British Retail Consortium
 Confederation of British Industry
 Federation of Small Businesses
 Food and Drink Federation
 Forum of Private Business
 Institute of Directors
 National Federation of Retail Newsagents
These tools have significant costs in terms of time and resources.
(b) Managing contradictory feedback and weighting different stakeholders
The existence of a plurality of groups of stakeholders imply different perceptions of problems
and different solutions.
This can be true even within a group –of course because of sub-groups (different industries have
different interests and problems, etc.)–, but also within the same industry between incumbents
and newcomers –or “would-be newcomers” who are not let in. This is the case see in many
countries, where large, established biz (particularly monopolists/oligopolists) in a sector can be
strong opponents of regulatory reform. For example, in Ukraine some of these were opposing
reductions in the number of goods/sectors subject to mandatory certification, because this
process reduced innovation and thus ability for competitors to differentiate their products.
There is a need for the authorities to ensure that they engage them all –and have a way to make

7. 7
sense of the input.
- Examples:
- BRDO is also looking at the citizens/consumers side of stakeholders (though this is
more recent): the “causal pathway” methodology gives an understanding of what
drives desired outcomes and which are the players, including consumers, citizens,
etc. and involving them accordingly. However, this is only one aspect of things;
fundamental would also be broader citizens’ engagement/consultation, partly done
by regulators themselves though (e.g. through management board of HSE,
Environmental Agency, etc.), on the agenda for more development.
- Regulatory Excellence Forum: this group comprises 18 professional and
representative bodies, including national regulators. It was established in 2008 to
simplify the complex regulatory system and develop the right conditions for
effective local regulation. It provides an opportunity to share views, approaches
and best practice at a national level. It supports common approaches to regulatory
challenges, shaping BRDO’s on-going projects in areas such as professional
competency, data returns and risk assessment:
https://www.gov.uk/government/publications/regulatory-excellence-forum-
terms-of-reference
- Business Reference Panel (see above).
- Local Authority Reference Panel: this group of 25 senior local authority
environmental health, fire, licensing, port health and trading standards officers is
drawn from across England with representatives from Wales and Scotland. It
provides an opportunity to share practitioner experience of better regulation
initiatives in practice and a forum for identifying expertise to shape BRDO’s work
from a range of front line perspectives:
https://www.gov.uk/government/publications/local-authority-reference-panel-
terms-of-reference.
- Better Business for All/Local Enterprise Partnerships:
https://www.gov.uk/government/publications/business-regulation-better-
business-for-all.
BRDO is working with LEPs on the Better Business for All programme, a
partnership approach to addressing local regulatory issues. As example of a LEP,
see http://www.llep.org.uk/about_us/.
- “Trading Places” is another interesting experiment, by which inspectors spend a
couple days working “on the business side” to understand the other’s perspective.
(c) The need to link such feedback to a structured process including in terms of
performance measurement
The consequences resulting from the feedback are fundamental.
The typical issue is that of measuring the performance of regulatory delivery agencies
(inspectorates) in “volume” of activity (inspection visits) and volume of sanctions. The idea is
that “more is better”.
But measuring such performance purely in terms of quantity raises a problem: it is meaningless,
since the objective should be the efficient use of resources (not “Spending as much resources as

8. 8
possible”) and to ensure the highest possible compliance (on the contrary, a high volume of
sanctions highlights that compliance is low). Consequently, another performance management is
needed.
One side is impact (level of safety, health, revenue, etc.) – but another side is feedback:
 from businesses: this is not the only/primary performance indicator, but it counts. The
goal is to obtain a high level of satisfaction (i.e. the process must be optimised for a given
level of regulation);
 from other stakeholders: this is a proxy for impact, to some extent;
 in both cases, how the authorities collect/inform about/mediate feedback is important to
avoid “receiving” only the ingrained bias of different sides.
(d) Some interesting examples
- Lithuania
- Call centres: by law (and additional CabMin decree) all inspectorates must have a
centralized call centre with answers on interpretation of the regulations that are
binding for the inspectorate (i.e. a business cannot be sanctioned for following
them; if the inspectorate finds out later on that the reply was wrong, it can be
corrected –the business will have to follow the new guidance, but will not be
‘punished’ for doing what the inspectorate told to do). By itself, this is not an
engagement/feedback tool, but it shows a radical turn in interactions with
businesses. At the same time, most Lithuanian inspectorates also implement the
“first year of business” declaration –whereby during the first year of operations of
a business, if they find problems, they advice and instruct how to correct them, but
do not issue sanctions (barring extreme cases). These transformations in practices
are interesting in that they change the context of interactions by building
trust/confidence. They are also conducive to start a dialogue.
- Reform scorecard: i.e. checking the status of reform implementation by each
inspectorate – based as much as possible on indicators that are “objectively
verifiable” (for example: adoption of secondary legislation, institutional changes,
etc.). In addition, feedback from stakeholders (mostly businesses) is used to assess
the level of implementation of changes in practice. The direct link between the
feedback and the ministerial action is intended to improve the effectiveness and
the trust in the system.
- The next step will consist of introducing a “feedback” mechanism, different options
being considered/tested. Surveys –carried out on an annual basis– exist already.
Random surveys of the whole enterprise population illustrate how many were
inspected and how many not, but there are limits on the ability to capture the
“quality” of inspections because the survey casts a broad net. The aim is to have a
mechanism that businesses really trust so they use it and effectively report it if
there are issues. Now the question going forward is: as “inspection experience”
improves, will businesses still take time to give feedback and will drive for further
improvements stall?
- United Kingdom BRDO
- Business surveys and business engagement mechanisms, etc.:
https://www.gov.uk/government/collections/brdo-reference-panels-terms-of-

9. 9
reference
See other parts of website:
https://www.gov.uk/government/organisations/better-regulation-delivery-
office/about.
- Regular business surveys gathering feedback from businesses about the
“regulatory experience”:
https://www.gov.uk/government/publications/local-regulation-perceptions-
survey.
Conclusion
Event though the issues presented above affect both OECD and non-OECD countries (emerging
markets, developing economies, etc.), they are possibly more intense in the latter. In this context,
challenges are greater (e.g. having stakeholders not fear the consequences and speak up, or
having people believe that something can change, etc.).
In addition, one more important question warrants consideration: has stakeholder
engagement/feedback become a buzzword of sorts?
Finally, the most significant challenge consists in the difficulty of making the system work.

10. 10
Some bibliographic information
- Ayres, I. and Braithwaite, J. (ed.) (1992), Responsive Regulation: Transcending the
Deregulation Debate, Oxford University Press, Oxford
- Blanc, F. (2011), Reforming Inspections, Measuring Success - Challenges in Former Soviet
Republics and their Neighbours, Paper presented to the ECPR General Conference,
Reykjavik,
http://www.ecprnet.eu/MyECPR/proposals/reykjavik/uploads/papers/2136.pdf
- Diver, C. (1983), The optimal precision of administrative rules, Yale Law Journal 93, pp.
65-109
- Elffers, H., Verboon, P. and Huisman, W. (ed.) (2006), Managing and Maintaining
Compliance, Boom Legal Publishers, Den Haag
- Gunningham, N. and Grabosky, P. (1998), Smart Regulation: Designing Environmental
Policy, Oxford University Press, Oxford
- HM Revenue and Customs (2010), Delivering a new relationship with business - Reducing
burdens and helping businesses get it right, UK HMRC, London,
www.hmrc.gov.uk/budget2010/new-rel-paper-1340.pdf
- Helsloot, I. and Schmidt, A. (2012), The Intractable Citizen and the Single-Minded Risk
Expert – Mechanisms Causing the Risk Regulation Reflex Pointed Out in the Dutch Risk
and Responsibility Programme, in: European Journal of Risk Regulation 3/2012, pp. 305-
312, Lexxion, Berlin
- Hough, M. (et al.) (2010), Procedural Justice, Trust, and Institutional Legitimacy, in:
Policing Vol. 4, Issue 3, pp. 203-2010
- Macrory, R. (2006), Regulatory Justice: Making Sanctions Effective, Better Regulation
Executive, Department for Business, Innovation and Skills of the United Kingdom, London,
www.bis.gov.uk/files/file44593.pdf
- Morgan, B. and Yeung, K. (2007), An Introduction to Law and Regulation – Text and
Materials, Cambridge University Press, Cambridge
- Ogus, A. (2004), Regulation: Legal Form and Economic Theory, Hart Publishing, Oxford and
Portland
- Radaelli, C. M. (2004), The diffusion of regulatory impact analysis – Best practice or
lesson-drawing?, in: European Journal of Political Research Vol. 43, Issue 5, pp. 723-747
- Scholz, J. T. (1991), Cooperative Regulatory Enforcement and the Politics of
Administrative Effectiveness, in: The American Political Science Review Vol. 85, No. 1, pp.
115-136
- Sunshine, J. and Tyler, T. R. (2003), The Role of Procedural Justice and Legitimacy in
Shaping Public Support for Policing, in: Law & Society Review Vol. 37, Issue 3, pp. 513-548.
- Tyler T. R., (2003), Procedural Justice, Legitimacy, and the Effective Rule of Law, in: Crime
and Justice Vol. 30, pp. 283-357
- Van Tol, J. (2011), Summary Analysis of the Risk-Regulation Reflex Entrenched Beliefs and
Six Possible Avenues for Solutions, OECD, Paris, www.oecd.org/regreform/regula
torypolicy/48654345.pdf