Levine-Clark, Michael, John McDonald, and Jason Price, "Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage," Charleston Conference, November 7, 2013. Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.

1. Discovery
or
Displacement?
A
Large
Scale
Longitudinal
Study
of
the
Effect
of
Discovery
Systems
on
Online
Journal
Usage
Charleston
Conference
November
7,
2013
Michael
Levine-­‐Clark,
University
of
Denver
John
McDonald,
University
of
Southern
California
Jason
Price,
SCELC
ConsorJum

2. “…a steep increase in full text
downloads and link resolver click‐
throughs suggests Summon had a
dramatic impact on user behavior and
the use of library collections during
this time period.”
The Impact of Web-scale Discovery on the Use of a
Library Collection
Doug Way (2010) http://scholarworks.gvsu.edu/
library_sp/9/

3. Vendor
marke5ng
h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html

4. Does implementation of a discovery
service impact journal usage?

6. Web-­‐scale
discovery
services
• Single
source
for
finding
informaJon
– Books
– ArJcles
– Local
content
• Metadata
and/or
full
text
• Content
is
pre-­‐indexed
and/or
pre-­‐harvested
• Single
fast
search
ILS
Publisher
Metadata
MLA
Bibliography
InsJtuJonal
Repository
HathiTrust
Discovery
Service

7. An
assump5on
• At
any
given
insJtuJon,
given
a
relaJvely
stable
user
base,
the
total
search
effort
will
remain
roughly
the
same.
– X
students
will
have
Y
assignments
and
Z
hours
per
day
to
search
– X
faculty
will
publish
Y
papers
and
have
Z
hours
per
day
to
search

8. Discovery
services
§ Will
take
up
an
increasing
amount
of
a
finite
Jme
for
searching
§ Will
draw
users
from
other
(more
or
less
efficient)
search
tools
§ Will
alter
the
overall
producJvity
of
searches
(users
will
find
more
or
less)
§ Will
alter
the
overall
efficiency
of
users
(users
will
access
more
or
less
full-­‐text)

9. Prior
studies
• Some
studies
have
indicated
substanJal
increases
in
usage
a]er
Discovery
implementaJon
– DescripJve
staJsJcs
only
– Single
insJtuJon
studies
only
• Some
publishers
report
decreased
usage
of
content
– Anecdotal,
may
affect
some
and
not
others

10. Data
collec5on
• List
of
libraries
with
discovery
services
> Searched
on
lib-­‐web-­‐cats
• Surveyed
Libraries
> Discovery
service
Implemented
> ImplementaJon
Date
(month/year)
> Search
box
locaJon
> MarkeJng
effort
• 149
Libraries
Gave
Approval
> 24
libraries
selected
for
this
phase
> 6
for
each
of
the
4
major
discovery
services

11. Library
demographics
• 20
US,
1
each
from
UK,
AUS,
NZ,
CA
• 10
ARL
Libraries
included
• WorldCat
book
holdings
> Average:
1,114,193
> Median:
1,044,153
> High:
2,665,796
> Low:
298,365
• ImplementaJon
dates:
> 2010
(3),
2011
(19),
2012
(2)

13. Dataset
• 24
Libraries
• 4
Discovery
services
• 6
Publishers
• 9,206
Journals
• 159,278
ObservaJons
• 141,048
Usable
ObservaJons

14. Methodology
Compared
COUNTER
JR1
total
full
text
arJcle
views
for
the
Included
implementaJon
month
in
Year
1
to
ensure
that
both
periods
included
an
enJre
academic
year
End
Year
2
May
2012
Year
1
ImplementaJon
May
2011
Start
June
2010
12
months
before
vs
12
months
a]er
implementaJon
date

15. Collec5ons
notes
o Excluded
journals
that
did
not
have
24
months
of
COUNTER
reporJng
o Limited
ability
to
control
for
changes
in
aggregator,
backfile
access,
or
expanded
holdings
o Outliers
removed
from
analysis

22. General
trends
• VariaJon
by
insJtuJon
within
each
discovery
service
• VariaJon
by
publisher
within
each
discovery
service
• Some
publishers
saw
overall
net
increase,
while
some
experienced
a
decrease
in
usage

24. Goals of our inferential statistics
Determine whether observed differences are
significant or resulted from chance effects
Determine which of the three factors
(i.e. library, publisher, discovery service)
contribute to determining differences in
usage change at the journal level
Start with an exploratory analysis and end with
a comprehensive model

25. ANOVA - Analyzing the data
Observation
=
Fit
+ Residual
Change
In = Library x + Publisher y + Disc Svc z + Residual Err
usage
+17
= (+2)
+
(-3)
+ (+10)
+
(+8)
After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1

26. ANOVA – F Ratio
Tests whether the means for levels within a
factor are distinguishable from each other
Average variability due to the factor
F-ratio = --------------------------------------------------Average variability due to chance error
So, when F ≈ 1, means are not distinguishable,
when F is > 1, there are real differences among
some means

27. Does usage change vary across libraries?
Overall Average = 8.5
Institution (sorted by Mean Change)

28. Does usage change vary across libraries?
Overall Average = 8.5
Institution (sorted by Mean Change)

29. Does usage change vary across publishers?
Overall Average = 8.9
Publisher (sorted by Mean Change)

30. Does usage change vary across discovery services?
Overall Average = 8.9

31. Does the affect of discovery service differ across publishers?

32. Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)

33. Do the discovery service means differ in the 2 way model?
15.0
12.3
4.5
3.7
Publishers (distinguished by color)

34. 15.0
12.3
4.5
3.7
Publishers (distinguished by color)

35. Do publisher means differ significantly in the two way model?
23.8
6.8-9.5
-3.9
Publishers (distinguished by color)

36. Do publisher means differ significantly in the two way model?
23.8
6.8-9.5
-3.9
Publishers (distinguished by color)

37. Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)

38. Publishers (distinguished by color)

39. Full Model –
including disc srvc, publisher, and library

40. ANOVA Model including all three factors

41. Results - Can we detect differences between
Discovery Services, Publishers, and/or
Libraries and/or their interactions?
Discovery Service – Yes
Publisher – No
Library – Yes
Differential discovery service effect by
publisher – Yes
Differential library effect by publisher -- Yes

42. Interpreta5ons
&
Conclusions
>
>
>
>
>
Analyzing
usage
is
a
complex
task
No
discovery
service
increased
or
decreased
usage
across
all
libraries
and/or
all
publishers
Discovery
service
and
publisher
as
variables
on
their
own
were
significant
predictors
of
usage
change
InteracJon
of
Discovery
service
&
Publisher
was
significant
Some
control
needed
for
no
discovery
service
and
for
size
of
insJtuJon.

43. A
plethora
of
pending
possible
pursuits
• Design
&
test
for
effects
of:
–
–
–
–
–
–
Aggregator
full
text
availability
InsJtuJon
Size
/
Enrollment
Profile
Publisher
Size
Journal
Subject
Overall
usage
trends
(Requires
Disc
Srvc
‘control’)
ConfiguraJon
opJons
in
Discovery
services
• Follow-­‐up
presenta5on
at
UKSG
(April
2014)
– Including
Control
group
&
AddiJonal
libraries
– Add
AddiJonal
variables
&
further
analysis

44. Ques5ons
michael.levine-­‐clark@du.edu
|
johndmcd@usc.edu
|
jason@scelc.org