Presented during the Strategic Measurement Workshop at Computers in Libraries, 2015.

1. Ontario Library Association
Super Conference, January 29
Dr. Bill Irwin, Huron University College
Current research in library evaluation:
A review in 3 parts

2. Today’s Presentation (in 3 parts)
Current Research in Library Evaluation2
1. 2014 research on the introduction of new evaluation practice in public libraries and the
impact of organizational culture Irwin. B. & St. Pierre, P. (2014) Creating a culture of
meaningful evaluation in public libraries: Moving beyond qualitative metrics, Sage
On-Line: http://sgo.sagepub.com/content/4/4/2158244014561214
2. Early findings from 2014 survey to Ontario Municipal councillors on their expectations
and considerations on the state of public library evaluation and reporting: “Making
informed public decisions for public libraries” Irwin. B. & St. Pierre, P. In design and
delivery partnership with Counting Opinions
3. Introduction of a future research initiative: Building an L-Value Index for Canadian
Public Libraries: Cavanagh, M., Silk, K., & Irwin, B.

3. Project’s Goal
Current Research in Library Evaluation3
Outcome models assist policy makers, both
inside and outside of organizations, to
address fundamental questions of resource
allocation, delivery methods, and agency
design and purpose.

4. Project’s Goal
Current Research in Library Evaluation4
The key goal of our research is to examine
existing evaluation systems of public libraries,
in an effort to replace the current reliance on a
predominantly output based model of
performance measure, with new models of
evaluation based on capturing outcomes.

5. Defining Outcome Evaluation
Current Research in Library Evaluation5

6. Project Objectives
Current Research in Library Evaluation6
 exploring the limitations and challenges inherent
in the current qualitative evaluation system
 creating a better understanding of the cultural
impacts and policy implications of a sociological
performance framework, on library organizations,
their constituencies, and their stakeholders in the
establishment
 capture changes in the Practice-Program-Policy
continuum and the downstream implications for
how libraries are viewed and valued upon the
introduction of a new system of performance
measurement

7. OPLA Child and Youth Services Committee
Teen Services Benchmarks and Statistical
Report 2013
Current Research in Library Evaluation7
Does your library
measure outcomes
or impacts of teen
programming?
% Yes
42.9%
All 46.7%
100,000+ 38.1%
50,000<100,000 40.0%
15,000<50,000 46.4%
5,000<15,000 75.0%
<5,000 42.9%
Less than 50% of the
libraries reported
measuring outcomes or
impacts of teen
programming, although
several libraries did note
that they are planning to
implement outcome-
based measurement in
the future.

8. Q: Are you satisfied with the methods used by your
library to evaluate teen programs?
Current Research in Library Evaluation8
• It would be helpful to have a
clearer idea of what a youth
program should be i.e. what is
the purpose, what is the
desire outcome.
• It is not consistent
• We do not evaluate anything
past attendance.
• I would like to find out what
the participant got out of
program
• Difficult to measure anecdotal
reports
• Evaluation is based only on
circulation and attendance
performance indicators

9. Q: Are outcomes/impacts measured in other parts
of your library?
Current Research in Library Evaluation9
• Lack of staff capacity
• Do not have anyone trained in
outcome/impact measurement
• Probably the best answer for
why it hasn't happened yet is
"too time consuming."
• Lack of time.
• Not currently considered a
priority by our governing body
• Assessing outcomes falls short
of our priority list.
• No formal criteria which is
implemented system-wide

10. Q: You indicated that you do not assess
outcomes/impacts of teen programs. Why not?
Check all that apply.
Current Research in Library Evaluation10

11. Q: Do the agencies that fund your teen programs
require that you report outcome/impact measures?
Current Research in Library Evaluation11

12. Organizational Culture and Impact
Current Research in Library Evaluation12
Sub-cultures are currently
the greatest inculcator of
values (Schein, 1992)
regarding:
1. the worth and purpose
of evaluations,
2. postulation of
negative influence,
3. evaluation seen as
“busy work”

13. Sub-Culture as Inhibitor
Current Research in Library Evaluation13
Common Responses from Follow-
up Interviews
 Seen as “busy-work”
 Rely on informal feedback
 Some staff value the process,
some question it, seen as
additional work
 Against staff comfort level
 Comments are filtered –each
manager chooses
 Staff wouldn’t want it (new system)
to be more onerous on them
 Staff question why, “Are you
evaluating me?”

14. Inhibitors
Current Research in Library Evaluation14
 Lack of education, not able to produce meaningful
results and change
 Lack of inclusion: “Big picture” relevance
 Required training in best practices
 Librarians feel skill set is inadequate to the task, so
they are reluctant to engage – (Similar circumstances
to the introduction of technology in mid to late 1990s
in public libraries (Author’s observation))
 Technological impasse
 Consider the law of unintended consequences (how
can/does it apply here?)

15. Sustainable change requires
cultural focusing (1)
Current Research in Library Evaluation15
 "Cultural change is one of the most important factors to consider but
one of the hardest to implement" (Preston, 2004).
 “In general, we found a number of library staff skeptical of
quantitative or qualitative data from customers, preferring instead to
rely on their own assumptions and past practices to make decisions.”
(Hiller et al, 2008)
 “Due to the ‘everyone does the same thing’ culture and operational
model it was impossible to make a change to workflow in one area
without it directly impacting other areas - therefore systemic change
was necessary.” (Nussbaumer & Merkley, 2012)

16. Sustainable change requires
cultural focusing (2)
Current Research in Library Evaluation16

17. Integration of Culture & Evaluation Model (1)
Current Research in Library Evaluation17
 The following model was developed to demonstrate the
steps an organization goes through in regards to moving
from an input based to an outcome based evaluation
system.
 It highlights the operational and cultural transformation
process.
 This model can be used to determine where an
organization is situated (at what stage) in terms of its use
of outcome evaluation.
 The model is designed to serve as a “roadmap”, to assist
organizations in their successful movement through a
series of steps, by infusing a culture of outcome
evaluation.
 Once an organization has achieved stage 4, evaluation
then serves as a tool for: realizing organizational priorities,
educating stakeholders and funders of the range of
quantitative and qualitative program and service impacts,

18. Integration of Culture & Evaluation Model (2)
18
Dimension
Stage 0 Stage 1 Stage 2 Stage 3 Stage 4
Purpose Complacency Justification Self Awareness Alignment Actualization
Motivation Inertia Fear & Survival
Toward sense
of self-efficacy
Enlightenment
Success,
Internalization
Organization
al Impact
Tactical /
Strategic
Resistance
Evaluation is
‘busy work’
Tactical: Short-term Strategic: Long-term
Buy-in
Shift from
systems to
individual to
patron/staff focus
Habituation Inculcation
Organization
al Impact
Internal /
External
Closed system Internal (staff)
Toward an
open system
Broad
stakeholder
inclusion
Infusion
(beware of
gaming)
Inhibitors &
Enablers
Lack of
education, lack
of meaningful
results, inertia
Lack of
personal
involvement.
Buy-in not
enough
Cautious
engagement
Enthusiasm;
Coordinating
many voices
Unintended
consequences
(positive or
negative)
Implications
Diminishing
budgets, failure
Awakening
Organizational
learning
Shared
leadership
Assessment is
continual,
naturally
occurring
Creating an outcome evaluation culture
Current Research in Library Evaluation

19. Integration of Culture & Evaluation Model (3)
Current Research in Library Evaluation19
 Kotter (2008) argues that getting buy-in is not enough because
it only engages the head, not the heart.
 Moving through the stages is akin to Kotter’s step 6, creating
short term wins, building staff confidence, an incremental
approach
 Somewhere in the transition between step 1 to 2 need to
create and articulate a vision/mission statement. Why do we
evaluate? What is it all about?
 Building cultural assessment seen as a strategic priority –
otherwise over shadowed by more “important work”
 “When faculty and staff perceive that the administrators are
only motivated to create a culture of assessment for
accreditation, cynicism and low-motivation will likely result”
(Lakos and Phipps, 2004).

20. Next Steps
Current Research in Library Evaluation20
 Formalize the Integration of Culture & Evaluation
Model
 Create an organizational evaluation audit to:
 Assess the stage of the model an organization is at
 Develop strategies to assist moving it forward
 Create a sustainability approach
 Design and deliver the next phase of the
research

21. Next Steps
Q: Do the agencies that fund your teen programs
require that you report outcome/impact measures?
Current Research in Library Evaluation21
Ontario-wide survey to elected
municipal officials focused on
understanding what information they
consider when in their deliberations
on financial support to libraries’
budgets.
In part this research will provide
insight for libraries in how to structure,
and communication the results of,
their institutional evaluations.

22. 49
45
32
10
15
13
1
1
2,501 to 5,000
5,001 to 15,000
15,001 to 30,000
30,001 to 50,000
50,001 to 100,000
100,001 to 250,000
Over 250,000
NR
Number of Responses
Responses by Community
Population
6.9%
27.3%
23.4%
5.7%
4.5%
6.2%
0.4%
2,501 to 5,000
5,001 to 15,000
15,001 to 30,000
30,001 to 50,000
50,001 to 100,000
100,001 to 250,000
Over 250,000
% Response by Population
Response Rate
Total responses:
n=166, 8.4%

36. Building an L-Value Index for
Canadian Public Libraries
Current Research in Library Evaluation36
This project will develop a new, theoretically sound and
comprehensive framework for library evaluation. The
project will be approached via a three-stage strategy:
1.) Identify the limitations inherent in existing library
evaluation systems (the present situation);
2.) Develop a multi-faceted theoretical framework for
evaluation - the "L-Index" -- which analyses outcomes and
impact along four dimensions: economic, social, cultural,
and lifelong learning. These constitute both qualitative and
quantitative outcomes (an analytical tool);
3.) Assist libraries in deploying the L-Index, through training
workshops, conference presentation, and publications,
thereby helping them strategically focus their mission and
vision, resources, operations,and service cultures
(facilitate change).

37. Building an L-Value Index for
Canadian Public Libraries
Current Research in Library Evaluation37
 A key goal of this research is to generate an
inclusive, reliable, outcome-based performance
value framework (PVF) that communities can
adapt to local conditions, and that can also be
used at the sector level.
 The L-Index will include performance measures
that facilitate the creation of benchmarks and
quality indicators.

38. Building an L-Value Index for
Canadian Public Libraries
Current Research in Library Evaluation38
This research proposes to investigate the
potential for establishing model evaluation
systems for public libraries that are based on
these sociological frames of reference; a model
which will examine long term outcomes and
impacts (sometimes referred to as “quality of life
issues”) for public libraries – in a similar fashion
that qualitative measures have been developed
and implemented for public health or recreation
programs and services.

39. Building an L-Value Index for
Canadian Public Libraries – Research
Methods
Current Research in Library Evaluation39
One: Identify outcome/impact measures in use in
libraries, other industries and organisational types,
through a thorough review of the organizational
assessment, evaluation, and public
administration literature. Of particular interest are
qualitative measures in use, such as "social return
on investment.“
Two: Conduct a national survey of public libraries. A
survey will include identification of current
evaluation methodologies, their particular contexts,
and the utility (real or perceived) of
their results, including both advantages and
limitations.

40. Building an L-Value Index for
Canadian Public Libraries – Research
Methods
Current Research in Library Evaluation40
Three: Using results from the first two project
phases, we will develop a provisional L-Index, a
multi-dimensional framework that will measure
the economic, social, cultural, and educational
impact a library has on its community.
Four: Test the validity, reliability, and utility of the L-
index with partners in Alberta and Ontario,
including First Nations communities, small,
medium, and large urban and rural systems.
Five: Promote the L-index via a toolkit to be made
freely available nationally for adoption by
Canadian libraries.

41. Building an L-Value Index for
Canadian Public Libraries – Partners
Current Research in Library Evaluation41
 Administrators of Medium Sized Public
Libraries of Ontario
 Edmonton Public Library
 Calgary Public Library
 Calgary Public Library Foundation
 The Federation of Ontario Public Libraries
 Library Association of Alberta
 Ontario Library Association
 The Ontario Public Library Association