Where Do We Go From Here? Assessing the Value and Impact of Discovery Systems Michael Levine-Clark, Professor / Associate Dean for Scholarly Communication and Collections Services, University of Denver Libraries Jason S Price, PhD, Director of Licensing Operations, SCELC Library Consortium

1. Where Do We Go From Here?
Assessing the Value and
Impact of Discovery Systems
NISO Forum:
The Future of Library Resource Discovery
Baltimore, MD
October 5, 2015
Michael Levine-Clark
University of Denver
Jason Price
SCELC Library Consortium

2. Why do libraries implement a discovery system?
Discovery System Goals
• To improve the user experience
• To provide a Google-like experience
• One-stop shopping - too many resources, too hard to keep them straight
o for undergrads or beginners / for everyone
o for particular disciplines / for every discipline / for cross-discipline needs
o primarily for articles / for articles and books / for all library resources
• One starting point (when you don’t know where to search)
• To replace the catalog
• To reduce the number of individual A&I databases (to reduce costs?)
• To increase the number of users starting with the library / Compete w/
Google

3. Components of Value
Usage - (searches or full text downloads)
• Perfect example of variation in stakeholder perspectives
• Publishers want their usage to go up
• Librarians want users to find relevant content efficiently
o could lead to decreased usage
o in tension with usage-based resource evaluation (i.e. ⬆ CPU)
• Discovery vendors need to respond to both of those needs
JSTOR as example

4. Components of Value
Effectiveness - accuracy
Efficiency - speed
Comprehensiveness
Integration with other library tools
Vary across stakeholders
Others?

5. Components of Impact
Usage
• Goes up or down overall
• Proportion coming through library (vs Google)
• Proportion coming thru discovery system vs other library sources
• Publisher-hosted vs Aggregator-hosted content
• Proportion in different library resource formats
o Print books vs ebooks vs articles
o Increased usage of previously siloed content?
 e.g. Digital library, special collections, or IR results?

6. Components of Impact
Other
• Bringing users (back) to library resources
o Directly or Indirectly
• Reduction in number of individual databases?
• Fewer questions about basic searching from users
o Users don’t have to wonder where/how to search - it’s obvious
• Reference/Instruction librarians freed up to focus on higher-level
consultations/instruction
• Others?

7. Impact of Discovery Systems on Journal Usage
Our study compared usage before and after 2011 implementations at the
journal level, which is only one way of measuring impact.
Levine-Clark, McDonald, & Price (2014) “Discovery or Displacement? A
Large-Scale Longitudinal Study of the Effect of Discovery Systems on
Online Journal Usage,” UKSG Insights 27(3): 249-256.

8. Study Results
1.All Discovery systems (DS) increased usage relative to control, and
the degree of increase varied from one system to another
2.There is as much variation within DS as across them
3.Discovery systems may sometimes shift usage toward the versions of the
content on their aggregated platform
4.DS impact is statistically significant, but explains only a small portion of the
variance

9. 1. All Discovery Systems increased usage vs control group
Control DS1 DS 2 DS3 DS4

10. Study Results
1.All Discovery systems (DS) increased usage relative to control, and
the degree of increase varied from one system to another
2.There is as much variation within DS as across them
3.Discovery systems may sometimes shift usage toward the versions of
the content on their aggregated platform
4.DS impact is statistically significant, but explains only a small portion
of the variance

11. 2. There is as much variation within DS as across them
Control DS1 DS 2 DS3 DS4

12. Study Results
1.All Discovery systems (DS) increased usage relative to control, and
the degree of increase varied from one system to another
2.There is as much variation within DS as across them
3.Discovery systems may sometimes shift usage toward the
versions of the content on their aggregated platform
4.DS impact is statistically significant, but explains only a small portion
of the variance

13. DS may sometimes shift usage toward their
aggregated platform (unpublished data)
● 2 DS Vendors that also have aggregated journal collections
● Numbers indicate the number of journals that were hosted and used on both
the publisher & aggregator platform during 2013-14
● Different publishers for each DS, but same pattern
● Unlikely to be intentional; can be configured by library

14. Study Results
1.All Discovery systems (DS) increased usage relative to control, and
the degree of increase varied from one system to another
2.There is as much variation within DS as across them
3.Discovery systems may sometimes shift usage toward the versions of
the content on their aggregated platform
4.DS impact is statistically significant, but explains only a small
portion of the variance (η2 < 1% -- .0086)

15. Where do we go from here?
Call to engagement
Ideal Questions -- Participation by multiple stakeholders with an
answer everyone wants to know
Two categories of questions
Broader Synthetic Questions
• How much of an impact to discovery services have?
Narrower Analytic Questions
• [maybe only meaningful if DS impact is compelling?]
• Address the why behind some aspects of impact

16. Synthetic - Do library discovery systems really make a difference?
Single publisher site referring URL data
But what about referrals that lead to Full Text use – aren’t these more important?
• Are users finding/accessing more relevant content?
• Are users who’d previously given up on the library convinced to come back and
use a Discovery System?

17. Synthetic - Is Google Scholar a viable
alternative to a library discovery system?

18. Analytic - To what extent does library e-resource management
and linking configuration limit DS effectiveness?
Why to study
• Wide variation across libraries with same system in our study. This could
be one reason why
• If discovery implementation is only as effective as these other decisions,
important to know
How to study
• Compare libraries with same discovery system and different
configuration options
• Compare same library before/after changing configuration

19. Analytic - Does discovery system configuration affect
the user experience?
Why to study
• Wide variation in our study
• Many libraries appear not to think too carefully about this. An
afterthought or trial and error
How to study
• Compare libraries with different configurations
• Compare before/after in library after changing configurations
• Compare before/after across discovery system when systemwide
change made
Difficulty: Libraries often reconfigure gradually. Hard to link
specific configuration choice to impact.

20. Is there a future (in search & content) for
libraries?
• The even broader question that we rarely ask
• A variation on this question is, “what happens if we cede
discovery to Google Scholar and its ILK?”
• The promise (and threat) of Open Access
• Influence/Change User Behavior
o retain or grow dependence on library resources
o awareness of their use of library resources

21. Thank You!
Michael Levine-Clark
University of Denver
michael.levine-clark @ du.edu
Jason Price
SCELC Library Consortium
jason @ scelc.org