Presentation to Pratt Institute, Information and Library School, Human Information Behavior Class

1. LIBRARY
ASSESSMENT
Jennifer Rutner, Senior Analyst, Ithaka S+R
Pratt Institute
Human Information Behavior
September 22, 2011

2. Senior Analyst, Ithaka S+R
School:
BA in Religious Studies, 2002
MLIS from Pratt, 2005
Enrolled in Quantitative Methods for
the Social Sciences at Columbia,
2010-?
Work:
Assessment & Planning Librarian,
Columbia University Libraries,
2006-2011
Chair, ACRL Assessment Committee,
2011

3. ASSESSMENT LIBRARIANSHIP

4. Assessment
“To assess, in general, is to determine the importance, size,
or value of; to evaluate. Library staff assess operations
by collecting, interpreting, and using data to make
decisions and improve customer service.”
ARL Spec Kit #303, Library Assessment, December 2007

5. What we talk about when we talk
about “assessment”
Quality
assurance
User
ROI/Value
research
Evaluation assessment Impact

6. Culture of Assessment
A Culture of Assessment is an organizational
environment in which decisions are based on
facts, research and analysis, and where
services are planned and delivered in ways
that maximize positive outcomes and impacts
for library clients.
A Culture of Assessment exists in organizations
where staff care to know what results they
produce and how those results relate to
customer expectations.
Amos Lakos: www.usc.edu/.../locations/leavey/news/conference/presentations/presentations_9-16/Assessment/UCLA_Lakos.ppt

7. Why assess?

8. Why assess?

9. Why assess?
Stuff I know Stuff I know I
don't know
Stuff I don't
know I don't
know

10. How is data used in libraries?
• Strategic planning +
management
• Decision making
• Program evaluation
• Advocacy
• Budgeting
• Regular service
improvements

12. Assessment Mission (CUL)
“Serve library users and staff through
the gathering, analysis, and
application of high-quality,
actionable information to guide
library decision making.”

13. The Research Process
Establish Gather Information
Environmental
research available Needs
scan
questions information (unknowns)*
Assign Establish
Test tools Develop tools
methodology priorities
Conduct Decision
IRB Analysis
assessment making

14. The Assessment Process
Brainstorming Information Priority Methodology Formal Outcome
Need Question
“I want to
know…”

15. RESEARCH METHODS

16. Surveys
Use random sampling to generalize to the broader
population.
Set questions, with no opportunity for follow-up.
Allows for statistical analysis.

17. Questionnaires
Evaluation tool: How was it?
Exploratory tool: What do you do?
Preferences tool: What do you want?
Reporting tool: What did you do?
Analysis stops at descriptive statistics.

18. Response Rate vs. Representativeness
“It ain’t response rate.”
70.00%
60.00%
50.00%
40.00%
30.00%
20.00% % of population
% of respondents
10.00%
0.00%

19. Focus Groups
“Researchers attempt to capture peoples explanations”
• Answering “how?” “why?” and “what?” questions – open
ended question
• Look at a topic up-close, rather than get a panoramic view
of the entire issue
Focus groups can uncover insights and
perspectives that are not retrievable
by other methods, from a small group
of participants.

20. Focus Groups
Standard
– Uses a rigid set of questions
Guided
– Uses a set of topics to explore, but the question
wording is flexible
Exploratory
– The most informal, questions arise through the
course of the conversation about the topic

21. Ethnography
Methodologies
– Work study
– Photo essays
– Dream-catcher workshops
– Mapping diaries
– Pilot projects
– Interviews

22. Observation
Wait, watch, write. Head Count+
• Sections of the library
• Type of seating
• Group vs. individual
• Food/drink
• Technology

23. Way Finding: 3D Usability
How do users navigate space?
– Provide task
– Record steps to completion
– Record completion rate
– Record challenges

24. Learning Outcomes
• Rubrics What are we teaching?
• Pre- and post-tests
• Minute Papers
– What’s one useful thing What are they learning?
you learned today?
– What’s one thing you’re
still confused about?
How does this impact
their academic success?

25. Usability Studies
Methods Evaluates
• Card sort • Ease of use
• Heuristiv evaluation • Efficiency of use
• Paper prototyping • Memorability
• Personas • Error frequency and
• Task analysis severity
• Work-study • Subjective satisfaction
www.usability.gov

26. ROI
“For every dollar invested in
the library, the
college/university/school/
community/business
received X dollars in
return.”
“For every dollar invested in
the library, the library
produces X services,
which can be valued at Y.”

27. Institutional Review Board (IRB)
• Protect human subjects. Protocols include:
• Ensure research is • Research questions and
ethical. hypothesis
• Ensure research • Subject profiles
complies with federal • Study procedures
and state laws. • Recruitment materials
• Report on findings

28. LIBQUAL+

29. LibQual+ Survey 2009
www.libqual.org
“22 questions and a box”
Affect of Service
Information Control
Library as Place
Comments?

30. LibQual+ Items
Affect of Service Information Control
AS-1 Employees who instill confidence in users IC-1 Making electronic resources accessible from my
AS-2 Giving users individual attention home or office
AS-3 Employees who are consistently courteous IC-2 A library Web site enabling me to locate information
AS-4 Readiness to respond to users' questions on my own
AS-5 Employees who have the knowledge to answer user IC-3 The printed library materials I need for my work
questions IC-4 The electronic information resources I need
AS-6 Employees who deal with users in a caring fashion IC-5 Modern equipment that lets me easily access
AS-7 Employees who understand the needs of their users needed information
AS-8 Willingness to help users IC-6 Easy-to-use access tools that allow me to find things
on my own
AS-9 Dependability in handling users' service problems IC-7 Making information easily accessible for
independent use
Library as Place IC-8 Print and/or electronic journal collections I require
LP-1 Library space that inspires study and learning for my work
LP-2 Quiet space for individual activities
LP-3 A comfortable and inviting location Local Questions
LP-4 A getaway for study, learning, or research Providing help when and where I need it
LP-5 Community space for group learning and group Making me aware of library services
study Availability of subject assistance
Ability to navigate library Web pages
Access to archives, special collections

31. Response: Representativeness
Response by status across
Status % of % of the University matches the
responses population
population distribution
Undergraduates 40.03% 32.38%
very closely.
Graduates 53.21% 55.21%
Faculty 6.78% 12.40%
Greatest difference: 8%
This is representative data!
http://www.libqual.org/documents/admin/Representativeness.pdf

32. Response: Representativeness
Response by discipline
across the University
matches the population
distribution nearly
perfectly.
E.g. We’re not missing
anyone!
http://www.libqual.org/documents/admin/Representativeness.pdf

33. Reading LibQual+ Charts
Desired
Superiority
Gap
Perceived/Reality Zone of
Tolerance
Adequacy
Gap
Minimum

34. Undergraduate Students
9
8
7
6
5
4
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

35. Graduate Students
9
8
7
6
5
4
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

36. Faculty Responses
9
8
7
6
5
4
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

37. Library Staff
9
8
7
6
5
4
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

38. Following up…
FACULTY PERCEPTION OF
JOURNAL COLLECTIONS

39. The Infamous IC-8
“The print and/or electronic journal collections I
require for my work.”

40. ARL Libraries 2009
Figure 3: LibQUAL+ 2009, ARL Libraries Faculty
10
9
8
7
6
5
4
AS - 1 AS - 2 AS - 3 AS - 4 AS - 5 AS - 6 AS - 7 AS - 8 AS - 9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP - 1 LP - 2 LP- 3 LP - 4 LP - 5

41. ARL Libraries 2009: IC8

42. Have perceptions changed?
Figure 5: LibQUAL+ 2006-09, Faculty Ratings of Journal Collections, ARL Libraries
9
M=8.61, SD=0.07 M=8.57, SD=0.13 M=8.56, SD=0.10
M=8.58, SD=0.10
8
M=7.52, SD=0.15 M=7.51, SD=0.21
M=7.44, SD=0.17 M=7.52, SD=0.14
7
M=7.15, SD=0.39
M=6.77, SD=0.40 M=7.01, SD=0.25
M=6.96, SD=0.35
6
5
4
2006 2007 2008 2009

43. Is there a difference in scores from year to year?
(ANOVA)
• 2006-2009 adequacy gaps from each ARL
institution.
• P-value = 0.119, which is not deemed
statistically significant.
Faculty were no more or less dissatisfied with
journal collections in 2009.

44. Journals and Overall Satisfaction
Figure 8: LibQUAL+ 2009, Correlation of Faculty Satisfaction with Journal Collections (IC-8) and Overall Library
Service, 21 Libraries
9
r =0.71
8
7
6
5
-1.2 -1 -0.8 -0.6 -0.4 -0.2 0

45. IC Item 2006 2007 2008 2009 Standard Mean
Deviation (by
(by question) question)
IC-1 0.44 0.61 0.44 0.55 0.08 0.51
IC-2 0.71 0.55 0.61 0.42 0.12 0.57
IC-3 0.86 0.42 0.31 0.73 0.26 0.58
IC-4 0.71 0.67 0.4 0.61 0.14 0.60
IC-5 0.72 0.49 0.27 0.45 0.19 0.48
IC-6 0.73 0.58 0.42 0.12 0.26 0.46
IC-7 0.81 0.46 0.40 0.67 0.19 0.59
IC-8 0.80 0.60 0.55 0.71 0.11 0.67

46. What else should we be watching?
Figure 6: LibQUAL+ 2006-09, Information Control Adequacy Gaps Over Time
0.2
0.1
0
IC1 IC2 IC3 IC4 IC5 IC6 IC7 IC8 2006 (n=37 )
-0.1 2007 (n=19)
-0.2 2008 (n=14)
2009 (n=21)
-0.3
-0.4
-0.5
-0.6
-0.7

47. What do our faculty say?
Support
“What would be great for faculty would be if
when things are not available, there was one
source in the library, extraordinarily skilled at
tracking down items. […] These people would
be specialists in working the electronic and
journal capabilities.”

48. What do our faculty say?
Search and Online Access
“I think just having free text search, like Google
book search, would be something that would
be very, very useful to have. I still feel like we
are living 20 years behind where the rest of
the world is in terms of being able to search
these databases and large collections of books
that we have.”

49. What do our faculty say?
Work-Arounds
“I just buy them individually from my research
funds, so it’s coming out of my research
money.”

50. What do our faculty say?
Quick List
“If I was to give a suggestion, maybe to have
discipline-specific pointers that could help
each discipline find things. […] It’s more of an
interface issue than a collections issue.”

51. What do our faculty say?
Resources
“The size of the collection is not as important as
getting the current collection working as
smooth as possible. Before, when we used to
go to the library, we got service.”

52. What do our faculty say?
Print vs. Electronic
“A few years ago, I wouldn’t have said that. But, I
guess things have changed.”

53. 2CUL HUMANITIES PHD STUDENTS

54. The 2CUL Project
“Columbia and Cornell University Libraries are pleased
to join forces in a transformative and enduring
partnership between our two great library systems
that enables us to pool resources to provide content,
expertise, and services that are impossible to
accomplish acting alone.”
http://2cul.org/

55. Research Procedures
Spring 2010
• Ethnographic Training
• Focus groups (5 total)
Summer + Fall 2010
• Interviews (45 total)
– 90 minutes, individual
• Post-questionnaire (paper)
Winter 2010-11
$
• Analysis and reporting

56. 870 pages of transcripts.

57. Code Tree
Code Level
The
Research + The First
The Student Institution + The Library
Writing Profession
Department
Institutional Space Coursework
Personal Space Librarians
Funding Exams + Preparation
Job Search
Previous Experience Collections
Dept. Requirements +
Expectations
Prospectus +
Preparation
Second
Personal Expectations Culture + Community Discovery Services
Writing Process +
Advising
Revision
Self-determination Library Space
Publishing
Teaching Defense + Preparation
Personal Life Information Technology
Attrition
Management
Challenges
Third
Successes
Opportunities Fourth

58. Time from BA graduation Through Expected PhD
Completion
Graduation from undergrad to start of PhD Start of PhD to candidacy Candidacy to PhD expected graduation

59. Humanities PhD Student Study Questionaire
Please rate your overall satisfaction with the following at CU:
Library Spaces
3 6 8 5 1 1
(Columbia only)
Library Collections 24 16 2 3
Library Services 21 19 3 2 Very Satisfied
Satisfied
Neutral
Dissatisfied
Advising Very Dissatisfied
10 8 2 1 3
(Columbia only)
n/a
Funding 14 22 2 2 5
Grad Program 14 24 2 3 2
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

60. Time Spent in the Library by Frequency and Duration
10
9
8
7
6
No. of Students
5
4
3
2
1
0
1< 1 -2 hrs 2 - 4 hrs 4 - 6 hrs 6 - 8 hrs 8+
Daily Weekly Monthly

61. Activities in the Library by Status
Post-Exam Pre-Exam
other
meet colleagues
consult librarian
computers
research w. non-library…
research w. library…
office hours
read
write
browse
0% 20% 40% 60% 80% 100%
Percentage of Students

62. Humanities PhD Student Study Questionnaire
Have you visited any non-CU libraries to use their collections
for dissertation research?
5
10
Yes No n/a
30

63. Humanities PhD Student Study Questionnaire
Do you receive financial support from CU for the academic
year?
4
2
Yes No n/a
39

64. Humanities PhD Student Study Questionnaire
Do you have an outside job that provides income?
5 6
Yes No n/a
34

65. Findings: Provide Space
"The thing that has been the best for me
is having a space to work. I got more
done last year when I had my locked
carrel than I had gotten done in years
before or since, because it was a
dedicated space in which I could keep
all of my sources [...]."

66. Findings: Foster Community
“It’s having community. Belonging to
your community, having friends that
are doing this and feeling that you
have something worthwhile to say that
other people are recognizing it.”

67. Findings: Provide Access to
Deep Research Collections
“I have to say that I have had every resource
that I have needed from the library. I really
can’t say, ‘Here I am in the sixth year
because you didn’t buy that set of resources
for me and I don’t have the materials to
work with, so how can you expect me
produce work?’”

68. Findings: Provide Research, Information
Management, and Teaching Expertise Assistance
“[…] maybe sitting down with an advanced
research reference librarian . . . might be in
my best interest as I go into the writing
stage of my paper, just so that I can make
sure I am not saying something that has
already been said or duplicating research, or
that I am not missing something that is
cutting-edge and that's really important to
my argument.”

69. Findings: Developing Scholarly Identity
“I had to tell my committee in an email, I plan
on having a draft of the first chapter to you
by June. If I don't, please get on my case…
So, I actually found that I needed to make
deadlines for myself and then tell them so
that they knew, and even though they
wouldn't care, my pride was at stake at that
point.”

70. ITHAKA S+R

71. Mission
ITHAKA is a not-for-profit organization that helps the academic
community use digital technologies to preserve the scholarly record and
to advance research and teaching in sustainable ways.
We pursue this mission by providing innovative services that aid in the
adoption of these technologies and that create lasting impact.
.

72. Our Services
• Ithaka S+R is a strategic consulting and research service that
focuses on the transformation of scholarship and teaching in
an online environment, with the goal of identifying the critical
issues facing our community and acting as a catalyst for
change.
• JSTOR is a research platform that enables discovery, access,
and preservation of scholarly content.
• Portico is a digital preservation service for e-journals, e-books,
and other scholarly e-content.

73. Ithaka S+R Surveys
2009 Faculty Survey 2010 Library Director Survey
3,025 Faculty responses 239 Library Directors responded
8.6% of population 13% of population
Focus on research institutions 9 Carnegie Classification Levels:
– 79 responses are doctoral
Conducted 2000, 2003, 2006, – 66 are master’s
2009, 2012
– 94 are baccalaureate
Ithaka S+R Staff: Roger Schonfeld, Ross Housewright, Matt Long

74. The Role of the Library: Comparison with Faculty
How important to you is it that your college or university library provide each
of the functions below? (Percentage answering very important.)
Teaching Facilitator
Undergraduate Information Literacy Teacher
Research Supporter
Buyer
Archive
Gateway
0% 20% 40% 60% 80% 100%
Faculty Members Library Directors
Note: Faculty member data are from Ithaka S+R Faculty Survey 2009.

75. Digital vs. Print Spending
“What percentage of your library’s materials
budget is spent on the following items?”
Now
Five years from Now
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Journals Books All other items

76. Digital vs. Print Spending
Journals Books
Directors predict a Directors predict a
106% 46%
drop in spending on drop in spending on
print journals in the print books in the
next 5 years… next 5 years.
…bringing budget shares to: …bringing budget shares to:
12%
Print / 88%
Digital
54%
Print / 46%
Digital

77. Print to Electronic Transition: Publishing
“I am completely comfortable with journals (that I use
regularly/ that my library subscribes to) ceasing their print
versions and publishing in electronic-only form.”
Library Directors Faculty Members
Agree 70% 39% Agree
Neither agree Neither agree
nor disagree 22% 30% nor disagree
Disagree 8% 32% Disagree

78. Print to Electronic Transition: Existing Collections
“Within the next five years, the use of (online or digitized
journals/ electronic versions of scholarly monographs) will
be so prevalent among faculty and students that it will not
be necessary to maintain library collections of hard-copy…
… Journals.” … Books.”
Agree 54% 7% Agree
Disagree 13% 59% Disagree

79. Library Spending Priorities
If you received a 10% increase in your library's
budget next year in addition to the funds you
already expect to receive, in which of the
following areas would you allocate the money?
(Please check up to three areas in the following list
that you would invest in.)

80. Library Spending Priorities
Online or digital journals
Tools for discovery (OPACs, indices,…
Staff for reference and user services/…
Facilities expansions and renovations
Other digital resources
Electronic versions of scholarly…
Staff in management/administration of…
0% 20% 40% 60% 80% 100%
If you received a 10% increase in your library's budget next year in addition to the funds
you already expect to receive, in which of the following areas would you allocate the
money?

81. Library Staffing Priorities
Supporting faculty
instruction and student…
Providing reference Percentage
services ranking this
Purchasing/ licensing item as 1
digital resources
Percentage
Building or maintaining ranking this
local discovery resources item as 2
Supporting the research
projects of faculty…
Developing and
maintaining special…
0% 20% 40% 60% 80% 100%
Ideally, how would you prioritize your staff resources in the following
areas? Please rank the items by order of importance.

82. Discovery: A Declining Role for the Library?
80%
70%
60%
50%
40% Library Directors
Faculty Members
30%
20%
10%
0%
2006 "Now" "5 years from now"
Percentage answering that it is very important that the library serve as a "gateway"

83. User Needs Assessment
In the past 2 years, has your library regularly solicited feedback
about services or collections from library users in any of the
following ways? (Please check all that apply.)
94% Informal discussions with faculty and students or emails soliciting feedback
71% Locally developed polls or surveys
49% Focus groups or test sessions
37% Cross-institutional polls or survey (such as Libqual+)
16% Structured Interviews
10% Ethnographic studies
8% With the help of outside consultants
6% Other
Only 13% do not have a formal means to assess user needs.

84. Library Strategies
35% of respondents agreed that “My
library has a well-developed strategy to
meet changing user needs and research
habits.”

85. CHALLENGES

86. challenges
• Lack tradition of using data for improvement
• No assessment advocate within organization
• Library staff lack research methodology abilities
• Weak analysis and presentation of data
• Inability to identify actionable data
• Library “culture” is skeptical of data
• Leadership does not view as priority/provide resources
• Library organizational structure is “silo-based”
• Staff do not have sufficient time
Turning Results into Action: Using Assessment Information to Improve Library Performance, Steve Hiller (University of Washington)
, Stephanie Wright (University of Washington)

87. challenges

88. jobs

89. Jennifer.rutner@ithaka.org
THANK YOU

90. References
About Assessment
• “Value of Academic Libraries: A Comprehensive Review and Report,” Megan Oakleaf:
http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf
• Focus Groups: A Practical Guide for Applied Research by Richard Krueger and Mary Anne Casey
• ARL SPEC Kit #303 on Library Assessment, December 2007
• Keys to Effective, Sustainable, and Practical Assessment
Steve Hiller, Martha Kyrillidou, and Jim Self http://www.arl.org/arldocs/stats/statsevents/laconf/2006/HillerSelf.ppt
• www.libraryassessment.info (blog)
• Columbia Assessment Program: https://www1.columbia.edu/sec/cu/libraries/bts/assessment/index.html
Interesting Studies
• Studying Students, The Undergraduate Research Project at the University of Rochester
http://www.ala.org/ala/mgrps/divs/acrl/publications/digital/Foster-Gibbons_cmpd.pdf
• For fun: http://www.coolinfographics.com/
• Cornell University Libraries ROI: http://research.library.cornell.edu/value
• Ellsevier/UIUC ROI: http://www.slidefinder.net/t/library_strategic_investment/illinois_roi_study/1313459
• University of Arizona), Learning in an Online Environment: Assessment of an Online Information Literacy Credit
• Course, Yvonne Mery, Jill Newby, Ke Peng: http://libraryassessment.org/bm~doc/2010_lac_poster.pdf
• University of Chicago: Wayfinding Revisited: Improved Techniques for Solving Usability Problems in Physical Spaces
Agnes Tatarka, David K. Larsen
• LibQual+ Survey Literature: www.libqual.org/Publications
• 2CUL Humanities PhD Study:
Conferences
• Library Assessment Conference Proceedings: http://libraryassessment.org
• Northumbria Conference Proceedings: http://www.lib.sun.ac.za/Northumbria7/Programme.htm
Ithaka S+R
• http://www.ithaka.org/ithaka-s-r/research/faculty-surveys-2000-2009
• http://www.ithaka.org/ithaka-s-r/research/ithaka-s-r-library-survey-2010