This document summarizes a large scale longitudinal study examining the effect of discovery systems on online journal usage. The study analyzed journal usage data from 24 academic libraries before and after implementing a discovery service. It found that while some publishers saw an increase in usage and some saw a decrease, the change in usage varied significantly depending on both the discovery service and the individual library. The study utilized ANOVA models to determine that discovery service, library, and the interaction between discovery service and publisher were significant predictors of changes in journal usage. The results indicate the effect of discovery systems on usage is complex and depends on multiple factors.
1. Discovery or Displacement?
A Large Scale Longitudinal Study of the
Effect of Discovery Systems on Online
Journal Usage
Charleston Conference
November 7, 2013
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium
2. “…a steep increase in full text
downloads and link resolver click‐
throughs suggests Summon had a
dramatic impact on user behavior and
the use of library collections during
this time period.”
The Impact of Web-scale Discovery on the Use of a Library
Collection
Doug Way (2010)
http://scholarworks.gvsu.edu/library_sp/9/
6. Web-scale discovery services
• Single source for
finding information
– Books
– Articles
– Local content
• Metadata and/or full
text
• Content is pre-indexed
and/or pre-harvested
• Single fast search
ILS
ILS
Publisher
Publisher
Metadata
Metadata
MLA
MLA
Bibliography
Bibliography
Institutional
Institutional
Repository
Repository
HathiTrust
HathiTrust
Discovery Service
Discovery Service
7. An assumption
• At any given institution, given a relatively
stable user base, the total search effort will
remain roughly the same.
– X students will have Y assignments and Z hours
per day to search
– X faculty will publish Y papers and have Z hours
per day to search
8. Discovery services
Will take up an increasing amount of a finite
time for searching
Will draw users from other (more or less
efficient) search tools
Will alter the overall productivity of searches
(users will find more or less)
Will alter the overall efficiency of users (users
will access more or less full-text)
9. Prior studies
• Some studies have indicated substantial
increases in usage after Discovery
implementation
– Descriptive statistics only
– Single institution studies only
• Some publishers report decreased usage of
content
– Anecdotal, may affect some and not others
10. Data collection
• List of libraries with discovery services
> Searched on lib-web-cats
• Surveyed Libraries
> Discovery service Implemented
> Implementation Date (month/year)
> Search box location
> Marketing effort
• 149 Libraries Gave Approval
> 24 libraries selected for this phase
> 6 for each of the 4 major discovery services
11. Library demographics
• 20 US, 1 each from UK, AUS, NZ, CA
• 10 ARL Libraries included
• WorldCat book holdings
> Average: 1,114,193
> Median: 1,044,153
> High: 2,665,796
> Low: 298,365
• Implementation dates:
> 2010 (3), 2011 (19), 2012 (2)
14. Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
Year 1
Year 2
Included implementation month in Year 1 to ensure that
both periods included an entire academic year
15. Collections notes
o Excluded journals that did not have 24
months of COUNTER reporting
o Limited ability to control for changes in
aggregator, backfile access, or expanded
holdings
o Outliers removed from analysis
16.
17.
18.
19.
20.
21.
22. General trends
• Variation by institution within each
discovery service
• Variation by publisher within each
discovery service
• Some publishers saw overall net
increase, while some experienced a
decrease in usage
23.
24. Goals of our inferential statistics
Determine whether observed differences are
significant or resulted from chance effects
Determine which of the three factors
(i.e. library, publisher, discovery service)
contribute to determining differences in usage
change at the journal level
Start with an exploratory analysis and end with
a comprehensive model
25. ANOVA - Analyzing the data
Observation
=
Fit
+ Residual
Change
In = Library x + Publisher y + Disc Svc z + Residual Err
usage
+17
= (+2)
+
(-3)
+ (+10)
+
(+8)
After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
26. ANOVA – F Ratio
Tests whether the means for levels within a
factor are distinguishable from each other
Average variability due to the factor
F-ratio = --------------------------------------------------Average variability due to chance error
So, when F ≈ 1, means are not distinguishable,
when F is > 1, there are real differences among
some means
27. Does usage change vary across libraries?
Overall Average = 8.5
Institution (sorted by Mean Change)
28. Does usage change vary across libraries?
Overall Average = 8.5
Institution (sorted by Mean Change)
29. Does usage change vary across publishers?
Overall Average = 8.9
Publisher (sorted by Mean Change)
30. Does usage change vary across discovery services?
Overall Average = 8.9
31. Does the affect of discovery service differ across publishers?
32. Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)
33. Do the discovery service means differ in the 2 way model?
15.0
12.3
4.5
3.7
Publishers (distinguished by color)
41. Results - Can we detect differences between
Discovery Services, Publishers, and/or
Libraries and/or their interactions?
Discovery Service – Yes
Publisher – No
Library – Yes
Differential discovery service effect by
publisher – Yes
Differential library effect by publisher -- Yes
42. Interpretations & Conclusions
Analyzing usage is a complex task
No discovery service increased or decreased
usage across all libraries and/or all publishers
> Discovery service and publisher as variables on
their own were significant predictors of usage
change
> Interaction of Discovery service & Publisher was
significant
> Some control needed for no discovery service
and for size of institution.
>
>
43. A plethora of pending possible pursuits
• Design & test for effects of:
–Aggregator full text availability
–Institution Size / Enrollment Profile
–Publisher Size
–Journal Subject
–Overall usage trends (Requires Disc Srvc ‘control’)
–Configuration options in Discovery services
• Follow-up presentation at UKSG (April 2014)
–Including Control group & Additional libraries
–Add Additional variables & further analysis
Selecting the libraries:
Maximized number of libraries balancing the number and size of libraries representing each discovery service and keeping implementation dates within a year or two of each other
Mention that there were only 3 institutions + publisher groupings that we had to leave out
Usable observations = journals at institutions that had counter report values for individ
Explain concept of outliers and why they are removed.
Z-Score on the Y axis is calculated as the number of standard deviations from the mean.
Of the 141,048 obs, we eliminated only ~100 outliers.
Although we were asked to keep the identity of the publishers confidential, we have used a consistent color scheme to identify results that pertain to each publisher
Summary of all journals from a publisher to the libraries having that discovery service. Many of the bars include total journal observations in excess of 10,000 while some of those, particularly for the smallest two publishers (in green an orange) include between 500-1000 observations.
Analysis of Variance (or ANOVA ) allows us to determine whether the dffferences we observed are significant
The analysis Breaks down the influences on each observation into the effect of levels of the factor of interest (I.e. Lib, Pub, Disc) and error
The numbers show an analysis of one observation – each value expresses the difference from the mean of 1 level (Lib x, Pub Y, and Disc Srvc Z) as well as the portion of the value that is not determined by these factors (i.e. the residual error)
If we imagine the hundreds of observations for each combination, we can understand that the size of the residual error relative to the value determines whether the effects are significant predictors of the observed values
Determines whether means (or averages) of each level within a factor are distinguishable from each other
or put another way it assesses the likelihood that journal change values sampled from different levels of a factor (say libraries) are actually estimating the same population…
or statistically different populations
So, in response to our first question: Does usage change vary across libraries?
We see the 24 libraries sorted by mean change along the x axis and
mean change in usage plus or minus two standard errors on the y axis
Standard errors are a measure of the variability around each individual library
In general, when 2xSE bars overlap, those means are not distinguishable
The overall average change was 8.5….
And our F-ratio of about 32 tells us that institution alone is a significant predictor of mean change in usage after discovery service implementation.
Whenever the p value (shown in the significance column) is less than 0.05, it indicates that we can reject the null hypothesis that there is no differences among levels of the factor (in this case, libraries)
But for our single factor anovas, this ignores the impact of different discovery services and journal publishers on mean change in usage
Click The grand mean for change in usage across publishers is 8.9
One publisher appears to have a mean change that isnt distinguishable from zero, whereas…
And the significant F value and non overlapping error bars suggest that the mean change varied across publishers, BUT
As we will see, when we add the other factors we’ll find that these differences are actually explained by discovery service and institution effects rather than publisher differences
This data shows that the mean usage increase was positive for all discovery services, ….
Although we cant distinguish these from industry wide increases since we didn’t examine usage change in libraries that did NOT implement a discovery service
So when we just look at discovery srvc & ignore the variation due to publisher or library we do see differences
When we include both Discovery Service and Publisher in a Two way model,
We can ask whether we can detect differences when we take both Discovery Means & Publisher means into account
In addition to asking whether we can detect a difference across discovery service alone and publisher alone
The two way model addresses whether the impact of discovery service is equivalent for each publisher
We can think of it as asking whether these lines are parallel (statistically) or whether they cross
These are the same data in the previous slide separated into panels by discovery service.
The following slides will step through testing for an effect of discovery service, publisher, and their interaction
Do the discovery service means we see here differ significantly in the two way model?
Do the discovery service means we see here differ significantly in the two way model?
No– they do not.
Do the publisher means we see here differ significantly in the two way model?
Do the publisher means we see here differ significantly in the two way model?
No–they do not.
Does the affect of discovery service differ across publishers?
Does the affect of discovery service differ across publishers?
Yes, it does. Statistically these lines are not parallel.
Mixed Nested Parially-crossed model
Acknowledges that each library can only implement one discovery tool
Takes all three variables into account in the same test.