Dr Paul Dawson, Mayor's Office for Policing and Crime The Police Foundation's Annual Conference 2018 Innovation and learning in policing

1. Evaluation Tales: Practical to
Organisational Insights
Police Foundation Annual Conference
24-Oct-18
Dr. Paul Dawson CPsychol AFBPsS
Mayor’s Office for Policing And Crime
paul.dawson@mopac.london.gov.uk

2. Mayor’s Office for Policing And Crime (MOPAC)
Headed up by the Mayor of London
who is responsible for policing in
London
Led by Sophie Linden – Deputy Mayor
for Policing and Crime (DMPC)
MOPAC sets the direction and budget of the
Metropolitan Police on behalf of the Mayor

3. Who are MOPAC Evidence & Insight...
• E&I is the largest dedicated civilian research unit in UK policing or PCC in the country.
• Approximately (22) specialist staff, such as criminologists & psychologists.
• Based within the MPS for many years, moved to MOPAC with wider scope April 2014.
• Our work is anchored to the Police and Crime Plan - the strategic core of the organisation...
• Working to drive change and improvements that benefit London (within policing, crime and justice…)
from the inside (A unique and privileged position).
• Four main areas of expertise:
1) Performance analytics and data visualisation (setting performance frameworks, removing targets,
performance dashboards – linked heavily to oversight and transparency).
2) Survey design and conduction (oversee 2 large scale pan-London surveys).
3) Social research/evaluation (15 current evaluations ongoing - wide range).
4) Liaison with academic organisations.
Our website: https://www.london.gov.uk/what-we-do/mayors-office-policing-and-crime-mopac/data-
and-statistics

4. A focus upon evaluation – innovation and learning
Genuine honour to be here ...
a. Our general approach to evaluation
b. Some practical examples (and reflections)
c. How we embed this learning within MOPAC
• How can organisations foster and deliver learning through evaluation?
• Evidence based policing and toolkits
• Which initiatives should be evaluated and which should not?
• To present in-depth learning upon the concept and decision making around 'evaluation' and what is
good enough evidence for Policing;
• To better understand formal and informal aspects of research /analytics in the police, whilst also
acknowledging the real-world circumstances in which any tactics or subsequent research/analytics are
conducted...

5. a. Our General
Approach To Evidence
Base and Evaluation
“Evaluation is an objective process of understanding
how a policy or other intervention was implemented,
what effects it had, for whom, how and why”
[Magenta book]

6. Evidence Based Policing and policy making...
• Much written about Policing Culture (Macho, pride, risk averse, craft based, tactical, reactive,
not receptive to evidence, closed to outsiders) ... Somewhat true – but too simple.
• Many officers ‘get it’ ; Many officers want to get it, many policy officers get it...
• EBP won’t gain traction from the periphery. Not from individual officers. Not from external
universities... All these are very welcome – but change needs to be driven from inside.
• Needs to be a far more open and nuanced conversation (i.e., performance pressures, doomed to
succeed, training, short-term appointments, information flow, the ‘time to read’, access to
journals, practical support for officers, learning cultures...).
• Cannot make all decisions based on evidence … where to start (i.e., high risk areas…)
• Find your data and evidence allies and work with those that get it.
• We have had our fair share of victories and losses... Change can take years

7. Can we advance evidence based thinking...
The Evidence based
Pyramid: evidence
based hierarchy

8. • It can be used as a tool for internal and external assurance, as well as reassuring external
stakeholders of the merits of the service change, identify training gaps, value for money,… all the
way to democratic accountability.
• Lots of types and methodologies… no right answer as to which to use… but there are wrong ways.
• ‘Evaluation’ and ‘performance ‘ are closely related. They go together very, very well…
• ‘Good evaluations’ are more than methodology – innovation, pragmatism, statistics, usefulness,
integrity in reporting … and should always be proportionate in design and cost to the project.
• Pragmatic and timely – i.e., use of ‘charge’ rather than ‘proven re-offending’ to gain insights.
• They may not always need to be done!
• Aspects around set-up and design in the beginning are just as important (or more so) than any
potential evaluation. Schemes that are based upon evidence, have a clear problem, clear aims,
effective design, strong implementation – are more likely to achieve expected results.
Our starting points around evaluation…

9. Our broad menu of evaluation
• Once the initiative has been designed and set-up effectively – this opens up
discussions around evaluation methodology...
Performance monitoring
And other analytics
Process evaluation Impact evaluation Economic Analysis
Interviews Randomisation
Focus Groups
Diary Studies
Observations Quasi-Experimental
Predicted/Actual
Not Below this
Break-Even
Cost Benefit
Throughput (referrals,
cases, attrition)
Literature reviews, problem
profiles, Dashboards

10. b. Examples of
MOPAC evaluations
(and reflections)
“E&I have either completed or are conducting
approximately 30 evaluations since 2014...
How to pick 3-4 to talk through...?“

11. 1. Alcohol Abstinence Requirement Order
As part of his 2012 manifesto pledge to introduce ‘compulsory sobriety for drunken
offenders’, the Mayor of London successfully lobbied for new legislation...
From July 2014 (MOPAC) conducted a 12 month proof of concept pilot in four boroughs
Croydon, Lambeth, Southwark and Sutton) the South London Local Justice Area
Small numbers (less than 120 offenders over the year) – more suitable for process learning...
• Training/awareness raising feedback survey (55 stakeholders)
• Stakeholder surveys (x3 waves, n=58)
• MOPAC and stakeholder interviews (35 semi-structured interviews)
• Offender surveys (44 / 113 offenders responded)
• Performance monitoring data (Crime, alcohol related crime, Public Views, technical data
on the tag, criminal backgrounds of those to receive the AAMR)
https://www.london.gov.uk/sites/default/files/aamr_final.pdf

12. The 12 month pilot in numbers...
113
AAMRs
ordered (on
111
individuals)
Average
length of
orders
75 days
Compliance
rate
92%
Only 9 people
breached
Over 6500
monitored
days. Avg of
45 per day...
298,000
alcohol
readings
taken
63% offences
drink driving
or violence
Only 25% linked to
nighttime economy
73% AAMRs
from
Croydon
courts
33
Average age
of AAMR
cohort
(avg. first
sanction 25)
In 34%, the
AAMR
offence was
their first
Avg. sanctions
(inc. AAMR)
8
Majority (77%) low
risk (some med (18%)
high (5%)
Avg. sanctions
(inc. AAMR)
8
Majority (77%) low
risk (some med (18%)
high (5%)
Constantly fed into the governance
to aid decision making

13. AAMR Process learning on the scheme…
• Positive in terms of programme implementation, effectiveness of the technology, and a
strong understanding of the aims.
•Pilot roll out supported by a dedicated MOPAC team, including a project manager with ‘in
the field’ experience.
• Challenges identified and pilot model refined accordingly. Welcomed by delivery
stakeholders as an additional community sentence option that offers an innovative and
tailored response to alcohol related offending.
• Offender concerns re ‘wearability’ – but positive they could complete the order.
• Mixed views on staff workload, but not onerous and largely similar to other requirements.
• Main challenge related to the sustainability of the programme at a wider level.
• The project was viewed as success. Practitioners liked the innovation, no apparent negative
impact on offenders, technology worked as intended, low breach…

14. Inspired by ‘Teach First’, 67 graduates receive a specially designed 6-week training period (300 hours of
learning) before being placed in the most disadvantaged wards in London, followed by 28 day immersion
period on borough and then bespoke support.
2. Police Now Graduate Leadership Development Programme
Two Year Evaluation ...
The first cohort of Police Now participants highly satisfied with the level
and quality of training they received as part of the Summer Academy
What worked? The syndicate structure
emerged as an important feature, along
with the positive learning environment and
the quality of the visiting fellows and guest
lecturers
What could be improved?
Participants thought the training could
have had a larger focus on the practical
aspects of the job and longer shifts on
field training.
A lack of robust evidence in the ‘impact posters’ - only 6
incorporated any previous learning from the literature and
only a third sought to empirically demonstrate success; of
these, only 4 used a comparison group.
Police Now participants demonstrated a strong orientation
toward serving the public and neighbourhood policing.
Officer Surveys...
- x4 Tracker surveys of Police Now officers’ views over the 2 years;
-Wider surveys (MPS staff and recruits - 75 respondents);
-Use of ‘Police Now's own internal surveys.
Interviews...
-x18 with Police Now team and ‘syndicate leads’;
-x22 with Police Now participants, and 25 line managers;
-x13 with non-police now officers entering policing.
Public surveys...
Using the MOPAC Public Attitude Survey
Matched wards on Vulnerable Localities Index
Compared 12 months before to 12 after on various measures
Police Now wards (n= 2732 respondents)
Comparison wards (n= 4218 respondents)
Coding for evidence base in the ‘impact library’
70 ‘impact’ assessments
i.e., quality of literature review, clear aims, conclusions based on
data....
Police Crime data
https://www.london.gov.uk/sites/default/files/mopac_police_now_evaluation_report.pdf

15. Police Now - community perceptions
Overall, Police Now wards showed no change
in public perceptions in the year subsequent to
the programme compared to a matched set of
wards (includes full PAS before and after –
includes over 6000 respondents).
Looking deeper, in the 16-24 group, there were
inconsistent results. Significant improvements
were observed in 'how good a job‘for Police
Now, but opposite patterns elsewhere (listen to
concerns or deal with things that matter).
These ‘youth’ findings have far smaller numbers
n=206 and 202 (year before and year post) and
preclude firm impact statements.

16. Reflections on AAMR and Police Now...
• Ability to push back and rather than do a flawed ‘Impact’ study with a small sample, promote the
performance and process angles. Sometimes a process study is exactly what is needed!
• AAMR the best implemented initiative I have ever come across in the last 10 years! Clear
operational model, documented tools, 6-9 months run-up to go ‘live’, effective communications,
clear roles for staff, dedicated project manager with on-field experience... This all paid off!
• The AAMR process told us the experience and learning. The performance told us the technology
worked as intended. As a result, the scheme was rolled out across London. We are now evaluating
‘impact’ (along with performance and process) at the London level.
• Genuine learning for Police Now – routinely fed into the project on an ongoing basis. Very positive.
• Upholding integrity within reporting. Research/evaluation can be a risk - Thick Skin.
• How does an organisation learn from challenging findings? How to promote a learning culture?
• We made our name in challenging from within... Being independent.

17. Approximately 450 available cameras split across
response teams within 10 boroughs
Emergency response teams selected for the pilot
as they enable BWV to be deployed in a wide
variety of situations and will make the admin and
back office functions more standardised/cheaper.
10 Boroughs identified through analysis, those
with high rates of our primary outcomes.
Within these, 2 teams per borough to receive
cameras and 3 teams in the control group. Teams
randomly selected.
MOPAC, MPS and the College of Policing (CoP) conducted (until recently) the largest urban Randomised
Control Trial (RCT) of Body Worn Video (BWV) to date. Previously promising results from other police forces
– albeit much smaller.
Findings ...
• Can reduce complaints
• Reduces allegations against police by 33%
• No difference on Stop and Search
• No difference on how the police interact with
the public
• No difference in likelihood of an arrest for
violent crime
• Officers positive on the technology
• Public opinion very positive to the innovation
Report https://www.london.gov.uk/sites/default/files/bwv_report_nov_2015.pdf
3a. Our RCT on Body Worn video

18. Received core outcome measures
in a SURVEY BEFORE their training
Mental Health Awareness and Safeguarding Training (MAST) – train staff in issues such as
mental health, gangs, support… RANDOM ALLOCATION to receive core outcome survey
questions either BEFORE or AFTER their training. Based on Skogan (2014).
Core measures focussed upon:
• Mental health knowledge
• Gang membership
• Understanding stereotypes and mental
health
• Service delivery (e..g, making referrals)
• Confidence in working with such cases
Significant improvements on these
issues after training, less positive on the
training itself…
Everyone registered
for MAST training
(n=1020)
Received core measures in a survey
AFTER their training
We can attribute any differences between the groups to the MAST training
3b. RCTs can be quick and easy
Everyone received
general feedback
survey at end
https://www.london.gov.uk/sites/default/files/mopac_mast_report_march_2016.pdf

19. Our RCT reflections
• Gold standard – highest quality 'impact' Cause and effect - 'No Brainer????'
• The BWV RCT demonstrated how challenging this design can be. The
captured data did not enable smooth research (i.e., had to link officers to their
S&S, arrests etc...), was very complex and was years in the making...
• We can add ingredients to the RCT to make it better – within the BWV study
I wanted to pull in public opinion (from our large surveys) to complement
learning.
• Randomised controls trials – can be easy (MAST!) (with the right conditions)
• No complacency just because it’s a RCT… (RCTs can be bad! – poorly
implemented, handling withdrawals and drop outs, poor randomisaton, poor
statistics etc..)
• Ongoing debate - what is good enough for policing?
• A very useful tool in our toolbox.

20. 4. Wrestling with programme integrity
• Largest ‘issue’ within my evaluations over the last 10 years is the issue of programme integrity.
• I use the term ‘implementation’ (i.e., documented problem, evidence based learning, underlying theory,
clear logic model, well-designed, implemented well, documented tools, data capture...)
• An issue in most evaluations, lots of different guises:
Diamond Initiative (too quick), Pathways (too slow), Project Shield (deviated from the logic model)... And
many, many others.... Too many instances not to be something deeply rooted...
•Are staff spending as much time on designing and ensuring effective implementation vs. methodology?
• Most debates focus on encouraging Gold Standard research and training staff in research methods...
• Wrong focus – danger of ending up with good evaluation on poorly developed and implemented
schemes. Harmful. Requires fundamental change to the craft of policing and related professionals
•More insightful questions –e.g., how far can an evidence based model be bent until it breaks?

21. c. How we embed this
learning within MOPAC
“E&I have either completed or are conducting
approximately 30 evaluations since 2014...
... This must have a wider applications right?“

22. • MOPAC has strong role as a commissioner of services. This has increased over recent
years. In 2012/13 MOPAC commissioning budget was £23.6m; this has grown
substantially to a budget of £45.7m in 2018/19.
• Includes many varied projects: new large scale initiatives (i.e., Child House); provision of
funds to boroughs (i.e., London Crime Prevention Fund); all the way to community
projects to tackle Knife Crime (i.e., Community Seed funding).
• My team conducts many evaluations - but not possible to ‘evaluate’ all of these services.
• I’m not clear if all of them would be ‘evaluatable’ (i.e., clear problem, based on
evidence); or, if policy staff have the right training to support effective implementation...
• Formal training around evaluations generally discuss ‘gold standards’ and methods. Not
helpful in this case for my policy colleagues.
• Comes all the way back to democratic accountability of money!
• PCC have asked numerous questions around this topic (quite rightly).
MOPAC has a desire to be evidence based...

23. A new systematic process around ‘MOPAC commissioning’
• Conducting an audit of all of our evaluations over the last 4 years (i.e., type, quality,
publication, cost, how the findings were used)...
• Similar exercise for all MOPAC commissions over recent years (i.e., was it evidence based,
clear aims, logic model, implemented well, was it evaluated, cost...).
• And then carrying this forward on all new commissions going forward.
• Development of range of training and tools to help MOPAC staff through this journey, but
also those groups bidding and receiving money! (logic models, formal and informal
evaluation, templates).
• Designing methods to report back to various governance structures and PCC.
• Enable MOPAC to have a stronger grip (and reporting) on all commissions and spend.
• Allows me more control over what we do or do not evaluate.
• Will be able to track evidence based progress (i.e., how many MOPAC commissions had
‘good logic model’ 3 years ago compared to now, what about in to 12 months time)...
• Increased transparency. To help make MOPAC more effective.

24. MOPAC Evaluation Menu – in realityTimeandresourcesincrease
All contracts/initiatives should have a minimum
ask – that of basic data capture – in terms of
documenting deliverables (inputs, activities, spend,
outcomes, key dates…).
Given the challenge in conducting robust
impact/VFM analytics – this will be the
smallest number.
Has to be implemented well to get here
Most active research will
likely focus on performance/
results analysis and/or
understanding of process.
What initiatives/contracts should be
prioritised for this most robust analytics?
Decision making criteria (cost, innovation,
vulnerability).

26. Thank you...
• Wider reflections on role – previously six + years within the MPS before the move to MOPAC. Our
influence has increased so much!
• Direct links into policy & programme design.
• Touched upon our evaluation experience today.
• We get to raise questions and that challenge our organisations ways of thinking. This does not
always make us popular. I often talk about ‘Thick Skins’ being a requisite.
• Most of my ‘evaluation’ conversations are around design/set-up and implementation – not method.
• Moving from many individual evaluations, to an organisational approach to improve evidence base.
•A very exciting place to be…
paul.dawson@mopac.london.gov.uk