1. Defining New Metrics For Libraries Success
iSchool, University ofToronto Symposium
April 23, 2014
Dr. Bill Irwin, Huron University College
Paul St-Pierre, PHD Candidate, FIMS, UWO
In
Insights: Organizational cultures in public
libraries: Gatekeepers to effective outcome
evaluation?

2. Project’s Goal
Organizational Culture & Evaluation2
The key goal of our research is to examine existing
evaluation systems of public libraries, in an effort to
replace the current reliance on a predominantly output
based model of performance measure, with new models
of evaluation based on capturing outcomes.

3. Project’s Goal
Organizational Culture & Evaluation3
Outcome models assist policy makers, both inside and
outside of organizations, to address fundamental
questions of resource allocation, delivery methods, and
agency design and purpose.

4. Defining Outcome Evaluation
Organizational Culture & Evaluation4

5. Project Objectives
Organizational Culture & Evaluation5
 exploring the limitations and challenges inherent in the
current qualitative evaluation system
 creating a better understanding of the cultural impacts and
policy implications of a sociological performance framework,
on library organizations, their constituencies, and their
stakeholders in the establishment
 capture changes in the Practice-Program-Policy continuum
and the downstream implications for how libraries are
viewed and valued upon the introduction of a new system of
performance measurement

6. OPLA Child and Youth Services Committee Teen
Services Benchmarks and Statistical Report 2013
Organizational Culture & Evaluation6
Does your library
measure outcomes or
impacts of teen
programming?
%Yes
42.9%
All 46.7%
100,000+ 38.1%
50,000<100,000 40.0%
15,000<50,000 46.4%
5,000<15,000 75.0%
<5,000 42.9%
Less than 50% of the libraries
reported measuring outcomes
or impacts of teen
programming, although
several libraries did note that
they are planning to implement
outcome-based measurement in
the future.

7. Q: Are you satisfied with the methods used by your library
to evaluate teen programs?
Organizational Culture & Evaluation7
• It would be helpful to have a clearer
idea of what a youth program should
be i.e. what is the purpose, what is
the desire outcome.
• It is not consistent
• We do not evaluate anything past
attendance.
• I would like to find out what the
participant got out of program
• Difficult to measure anecdotal
reports
• Evaluation is based only on
circulation and attendance
performance indicators

8. Q: Are outcomes/impacts measured in other parts of your
library?
Organizational Culture & Evaluation8
• Lack of staff capacity
• Do not have anyone trained in
outcome/impact measurement
• Probably the best answer for why it
hasn't happened yet is "too time
consuming."
• Lack of time.
• Not currently considered a priority by
our governing body
• Assessing outcomes falls short of our
priority list.
• No formal criteria which is
implemented system-wide

9. Q: You indicated that you do not assess outcomes/impacts of
teen programs. Why not? Check all that apply.
Organizational Culture & Evaluation9

10. Q: Do the agencies that fund your teen programs require
that you report outcome/impact measures?
Organizational Culture & Evaluation10

11. Organizational Culture and Impact
Organizational Culture & Evaluation11
Sub-cultures are currently the
greatest inculcator of values
(Schein, 1992) regarding:
1. the worth and purpose of
evaluations,
2. postulation of negative
influence,
3. evaluation seen as “busy
work”

12. Sub-Culture as Inhibitor
Organizational Culture & Evaluation12
Common Responses from Follow-up
Interviews
 Seen as “busy-work”
 Rely on informal feedback
 Some staff value the process, some question
it, seen as additional work
 Against staff comfort level
 Comments are filtered –each manager
chooses
 Staff wouldn’t want it (new system) to be
more onerous on them
 Staff question why,“Are you evaluating
me?”

13. Inhibitors
Organizational Culture & Evaluation13
 Lack of education, not able to produce meaningful results and
change
 Lack of inclusion:“Big picture” relevance
 Required training in best practices
 Librarians feel skill set is inadequate to the task, so they are
reluctant to engage – (Similar circumstances to the introduction of
technology in mid to late 1990s in public libraries (Author’s
observation))
 Technological impasse
 Consider the law of unintended consequences (how can/does it
apply here?)

14. Sustainable change requires cultural
focusing (1)
Organizational Culture & Evaluation14
 "Cultural change is one of the most important factors to consider but one of the
hardest to implement" (Preston, 2004).
 “In general, we found a number of library staff skeptical of quantitative or qualitative
data from customers, preferring instead to rely on their own assumptions and past
practices to make decisions.” (Hiller et al, 2008)
 “Due to the‘everyone does the same thing’ culture and operational model it was
impossible to make a change to workflow in one area without it directly impacting
other areas - therefore systemic change was necessary.” (Nussbaumer & Merkley,
2012)

15. Sustainable change requires cultural
focusing (2)
Organizational Culture & Evaluation15

16. Integration of Culture & Evaluation Model (1)
Organizational Culture & Evaluation16
 The following model was developed to demonstrate the steps an organization
goes through in regards to moving from an input based to an outcome based
evaluation system.
 It highlights the operational and cultural transformation process.
 This model can be used to determine where an organization is situated (at what
stage) in terms of its use of outcome evaluation.
 The model is designed to serve as a “roadmap”, to assist organizations in their
successful movement through a series of steps, by infusing a culture of outcome
evaluation.
 Once an organization has achieved stage 4, evaluation then serves as a tool for:
realizing organizational priorities, educating stakeholders and funders of the
range of quantitative and qualitative program and service impacts, and
identifying operational issues and challenges.

17. Integration of Culture & Evaluation Model (2)
17
Dimension
Stage 0 Stage 1 Stage 2 Stage 3 Stage 4
Purpose Complacency Justification SelfAwareness Alignment Actualization
Motivation Inertia Fear & Survival
Toward sense of
self-efficacy
Enlightenment
Success,
Internalization
Organizationa
l Impact
Tactical /
Strategic
Resistance
Evaluation is‘busy
work’
Tactical: Short-term Strategic: Long-term
Buy-in
Shift from systems to
individual to
patron/staff focus
Habituation Inculcation
Organizationa
l Impact
Internal /
External
Closed system Internal (staff)
Toward an
open system
Broad
stakeholder
inclusion
Infusion
(beware of
gaming)
Inhibitors &
Enablers
Lack of education,
lack of meaningful
results, inertia
Lack of personal
involvement. Buy-
in not enough
Cautious
engagement
Enthusiasm;
Coordinating many
voices
Unintended
consequences
(positive or
negative)
Implications
Diminishing
budgets, failure
Awakening
Organizational
learning
Shared leadership
Assessment is
continual,
naturally
occurring
Creating an outcome evaluation culture

18. Integration of Culture & Evaluation Model (3)
Organizational Culture & Evaluation18
 Kotter (2008) argues that getting buy-in is not enough because it only
engages the head, not the heart.
 Moving through the stages is akin to Kotter’s step 6, creating short term wins,
building staff confidence, an incremental approach
 Somewhere in the transition between step 1 to 2 need to create and
articulate a vision/mission statement.Why do we evaluate?What is it all
about?
 Building cultural assessment seen as a strategic priority – otherwise over
shadowed by more “important work”
 “When faculty and staff perceive that the administrators are only motivated
to create a culture of assessment for accreditation, cynicism and low-
motivation will likely result” (Lakos and Phipps, 2004).

19. Next Steps
Organizational Culture & Evaluation19
 Formalize the Integration of Culture & Evaluation Model
 Create an organizational evaluation audit to:
 Assess the stage of the model an organization is at
 Develop strategies to assist moving it forward
 Create a sustainability approach
 Design and deliver the next phase of the research

20. Next Steps
Q: Do the agencies that fund your teen programs require that
you report outcome/impact measures?
Organizational Culture & Evaluation20
Ontario-wide survey to elected municipal
officials focused on understanding what
information they consider when in their
deliberations on financial support to libraries’
budgets.
In part this research will provide insight for
libraries in how to structure, and
communication the results of, their
institutional evaluations.