The document describes a methodology for evaluating the cost-effectiveness of journal packages. It involves collecting usage and cost data, analyzing it using metrics like cost-per-use, and identifying titles that could be renewed or cancelled based on thresholds like cost-per-interlibrary-loan. The methodology reveals usage of electronic resources and value for money but does not show user experience or why low usage may occur. It provides necessary but insufficient data to make strategic decisions about journal packages.

1. ER&L 2010 February 1-3, 2010 Austin, Texas
Developing a Methodology in Evaluating Cost
Effectiveness of Journal Packages
Nisa Bakkalbasi
Head, Electronic Collections
Yale University Library
1

2. ER&L 2010 February 1-3, 2010 Austin, Texas
About today’s talk
Since 1990's many academic libraries signed multi-year
contracts to subscribe to electronic journal packages from
large publishers.
With recent collection budget cuts, many academic
libraries are having second thoughts about journal package
arrangements, which force them to spend too much money
on journals they don't need and which make it difficult to
pay for journals from smaller publishers and scholarly
monographs.
In this presentation, I will share a methodology we have
developed for evaluating the cost effectiveness of journal
packages.
2

3. ER&L 2010 February 1-3, 2010 Austin, Texas
Outline of methodology
 Identify the issue.
 Collect and analyze data for decision making.
 Identify and review options: pros and cons.
 Discuss decision time-line and implementation.
3

4. ER&L 2010 February 1-3, 2010 Austin, Texas
To identify the issue, we need:
 An overview of the product
 How funding was established?
4

5. ER&L 2010 February 1-3, 2010 Austin, Texas
To collect and analyze data, we use:
 Data source:
 COUNTER-compliant usage reports
 Accessible titles list from consortium or publisher
 Publisher’s title-by-title list price
 Subject categories assigned by the publisher or locally
 Data analysis technique
 Exploratory data analysis using quantitative and qualitative
variables
 Tool/software
 Excel PivotTable Report
5

6. ER&L 2010 February 1-3, 2010 Austin, Texas
To review options and discuss pros
and cons, we need:
 Pricing model or the sales model
 Base-value
 Subscribed/non-subscribed titles, if relevant
 Cancellation allowance
 Price cap
 DDP rate
 Perpetual (or post-cancellation) access rights
 Duration of the contract
 Transfer titles
 Third-party titles
6

7. ER&L 2010 February 1-3, 2010 Austin, Texas
Let’s get started!
Using a case study, we will walk through a step-by-step
evaluation of a journal package!
7

8. ER&L 2010 February 1-3, 2010 Austin, Texas
Issue
 An Academic Library has been subscribing to the ABC
Publishers Complete Journals Collections since 2005. The initial
base-value of the journal package arrangement was
established based on historical print spend in 2004.
 The staff wants to gain an understanding of whether the
package is worth retaining when it is up for renewal -- i.e. are
we paying less in the package than we would if we broke the
package and had to pay title-by-title for the journals we need
to retain access to.
 Reductions in collection budget and different usage patterns
among subjects are reasons to consider dissolving the journal
package arrangement.
8

9. Product Overview
 ABC Publishers Complete e-Journal Collection:
 Publishes academic and research journals.
 Publishes in partnership with learned societies.
 Publishes in physical and life sciences, medicine, social
sciences, humanities, law, and mathematics.
 Publishes 200.
ER&L 2010 February 1-3, 2010 Austin, Texas
9

10. ER&L 2010 February 1-3, 2010 Austin, Texas
Data Collection
 Download COUNTER Journal Report 1: Number of Successful
Full-Text Article Requests by Month and Journal.
 Exclude all titles that are based on separate pricing models.
 Exclude journal archives usage if backfile purchase was a
separate acquisition.
 Obtain a list of “Accessible Titles” list, which contains a list of all
titles (subscribed and non-subscribed) included in the package
arrangement.
 Obtain a list of e-only list prices.
 Aggregate all data in a single spreadsheet.
10

11. ER&L 2010 February 1-3, 2010 Austin, Texas
Producing data
 Each record should contain the following variables:
 Journal title
 ISSNs
 Usage count (YTD Total from the COUNTER JR1 report)
 List price
 Categories for subscribed and non-subscribed variables
 Categories for subjects
11

12. ER&L 2010 February 1-3, 2010 Austin, Texas
Enhance data to facilitate decisions
 Calculate cost-per-use (CPU)
CPU= e-only price for the title /YTD Total (per year)
 Using ILL cost as a threshold, create two new categorical
variables to filter for possible renewals and cancellations
Let’s take a look at an anonymized data!
12

13. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Title count
Subscribed vs. non-subscribed title counts
Subject Cluster NO YES Grand Total
Humanities 9 38 47
Law 4 21 25
Social Sciences 11 26 37
Science & Medicine 32 41 73
Mathematics 10 8 18
Grand Total 66 134 200
13

14. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Usage
Subscribed vs. non-subscribed titles usage counts
Subject Cluster NO YES Grand Total
Humanities 1,057 7,775 8,832
Law 257 1,821 2,078
Social Sciences 1,450 4,314 5,764
Science & Medicine 10,544 60,246 70,790
Mathematics 204 1,384 1,588
Grand Total 13,512 75,540 89,052
14

15. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Usage
Cumulative Relative Frequency (%)
10% 20% 50% 80% 90% No use
1% (2) 2% (4) 8% (15) 24% (48) 40% (80) 0
10 % of use comes from two titles, 20 % of use comes from four
titles, 50 % of use comes from fifteen titles, and so on.
15

16. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Cost
Subscribed vs. non-subscribed titles cost
Subject Cluster NO YES Grand Total
Humanities $2,074 $10,725 $12,799
Law $1,666 $9,346 $11,012
Social Sciences $4,511 $10,363 $14,874
Science & Medicine $24,764 $42,071 $66,835
Mathematics $6,402 $9,198 $15,600
Grand Total $39,417 $81,703 $121,120
16

17. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Candidates for renewal
cancellation (based on $10/ILL)
Count of possible candidates for
renewal/cancellation
Subject CPU>= $10 CPU<= $10 Grand Total
Humanities 7 40 47
Law 13 12 25
Social Sciences 6 31 37
Science &
Medicine 8 65 73
Mathematics 16 2 18
Grand Total 50 150 200
17

18. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Candidates for renewal &
cancellation (based on $25/ILL)
Count of possible candidates for
renewal/cancellation
Subject CPU>= $25 CPU <= $25 Grand Total
Humanities 3 44 47
Law 6 19 25
Social Sciences 2 35 37
Science & Medicine 3 70 73
Mathematics 10 8 18
Grand Total 24 176 200
18

19. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Candidates for renewal &
cancellation (based on $10/ILL)
Cost of possible candidates for
renewal/cancellation
Subject CPU>= $10 CPU<= $10 Grand Total
Humanities $1,782 $11,017 $12,799
Law $6,738 $4,274 $11,012
Social Sciences $2,581 $12,293 $14,874
Science & Medicine $7,745 $59,090 $66,835
Mathematics $15,176 $424 $15,600
Grand Total $34,022 $87,098 $121,120
19

20. ER&L 2010 February 1-3, 2010 Austin, Texas
Analysis: Candidates for renewal &
cancellation (based on $25/ILL)
Cost of possible candidates for
renewal/cancellation
Subject CPU>=$25 CPU<=$25 Grand Total
Humanities $588 $12,211 $12,799
Law $3,832 $7,180 $11,012
Social Sciences $649 $14,225 $14,874
Science &
Medicine $1,329 $65,506 $66,835
Mathematics $9,110 $6,490 $15,600
Grand Total $15,508 $105,612 $121,120
20

21. ER&L 2010 February 1-3, 2010 Austin, Texas
Methodology reveals
 proof that users are using electronic resources.
 value for money.
 necessary but insufficient data to make strategic
decisions.
21

22. ER&L 2010 February 1-3, 2010 Austin, Texas
Methodology does not reveal
 The users’ experience or perception of the utility or value
of a collection or service.
 Low use can occur because the product’s user interface
is difficult to use or because users are unaware that the
product is available.
22

23. ER&L 2010 February 1-3, 2010 Austin, Texas
Questions?
23

24. ER&L 2010 February 1-3, 2010 Austin, Texas
Thank you for coming!
24