Today’s library discovery services are primarily based upon indexes derived from journals, e-books and other electronic information of a scholarly nature. The content comes from a range of information providers and products--commercial, open access, institutional, etc. By indexing the content in advance, discovery services have the ability to deliver more sophisticated services with instant performance, compared to the federated search techniques used previously. Libraries increasingly rely on index-based discovery services as their strategic interfaces through which their patrons gain access to the rapidly growing breadth of information that may be available to them. This webinar will discuss the challenges of operating a centralized index-based discovery system. Learn about their strengths, and their weaknesses, the needs for standards and best practices in this arena, how libraries and providers can assess the usage, and how libraries can satisfy audiences with different needs--ranging from undergraduates to faculty across every discipline.

1. [insert web address for NISO webinar page]
Discovery and Delivery: Innovations and
Challenges
September 26, 2012
Speakers: Lucy Harrison,
Timothy Babbitt, David Bietila

2. Introducing the
Open Discovery Initiative
NISO Webinar: Discovery and Delivery:
Innovations and Challenges
Lucy Harrison, Florida Virtual Campus, D2D Liaison
September 26, 2012
2

3. Topics
• What are centralized indexes?
• What are their strengths and
weaknesses?
• What is the NISO ODI initiative?
• How will it help improve the discovery
landscape?
3

4. Evolution of Library Search
• Card Catalogs
• Online Catalogs
• Federated search tools
• Next-generation library catalogs (discovery
interfaces)
• Index-based discovery services
4

5. Discovery Interfaces ILS Data
Digital
Search: Local Collections
Index
ProQuest
Search Results
EBSCOhost
Federated
Search
Engine
…
MLA
Bibliography
ABC-CLIO
Real-time query
and responses

6. Index-based Discovery ILS Data
Digital
Search: Collections
Consolidated Index
ProQuest
Search Results EBSCOhost
…
MLA
Bibliography
ABC-CLIO
Harvesting and indexing performed in advance

7. Strengths of Index-based
Discovery
• Fast response time (vs. federated search)
• Structured metadata:
– Improves search & retrieval
– Faceted navigation
– Improves integration of search results
• Indexing full-text of content amplifies
access
7

8. Issues with Index-based
Discovery
• Important to understand depth of indexing
– Currency, dates covered, full-text or citation,
quality of metadata
• Uneven participation diminishes impact
• Ecosystem dominated by private
agreements
• Complexity and uncertainty poses barriers
for participation

9. Need to Bring Order to Chaos
• Libraries need the ability to understand
and evaluate tools, content, providers
• Information providers need the confidence
that their content is being treated fairly
• Service providers need the ability to more
efficiently integrate content
9

10. Key Areas for Libraries
• Strategic investments (in subscriptions and
discovery solutions)
• Expect comprehensive representation of
resources in discovery indexes
• Need to be able to evaluate the depth and
quality of these index-based discovery
products
• Usage reporting

11. Collection Coverage Questions
• How well does the index cover the body of scholarly
content?
• Why do some publishers not participate?
• How can libraries understand the differences in coverage
among competing services?
• How are your library’s content packages represented by
the discovery service?
• Which resources are not represented in index?
• Is content indexed at the citation or full-text level?
• What is the quality of the metadata?
• What are the restrictions for non-authenticated users?

12. Key Areas for Service Providers
• Encourage information providers to
participate
• Lower thresholds of technical involvement
• Clarify the business rules associated with
involvement
• Common industry standards and
definitions
• Usage reporting

13. Key Areas for Information
Providers
• Discovery brings uncertainty
• Want to expose content widely (increase
usage), but
• There are trust issues
– With Access / Authentication
– With ―Fair‖ Linking
• Private agreements
• Usage reporting

14. Need healthy ecosystem
among discovery service
providers, libraries and
information providers

15. OPEN DISCOVERY INITIATIVE
Promoting Transparency in Discovery
15

16. ODI Pre-History
• June 26, 2011: Exploratory meeting @
ALA Annual
• July 2011: NISO expresses interest
• Aug 7, 2011: Proposal drafted by
participants submitted to NISO
16

17. ODI Proposal
Define standards and/or best practices for
index-based discovery services
– Evaluate the breadth and depth of content
– Evaluate availability of content to different
institutions and to different users
– Streamline workflows
– Define models for fair linking
– Determine what usage statistics should be
collected and disseminated
17

18. ODI Pre-History
• June 26, 2011: Exploratory meeting @
ALA Annual
• July 2011: NISO expresses interest
• Aug 7, 2011: Proposal drafted by
participants submitted to NISO
• Aug 2011: Proposal accepted by D2D
• Vote of approval by NISO membership
• Oct 2011: ODI launched
• Feb 2012: ODI Workgroup Formed 18

19. ODI Charge and Objectives
• Improve information services to end users
as mediated through index-based
discovery services
• Create an environment that broadens
stakeholder participation and ensures
confidence
• Foster development of best practices and
effective means of assessment
19

20. Specific Benefits
Librarians
– Can offer their users as wide a range of
content as possible via their discovery
service of choice
– Can better evaluate discovery services
to address their needs
20

21. Specific Benefits
Information providers
– Have the confidence that the discovery
service providers are handling their
content in an appropriate manner
– Are therefore encouraged to make
available the widest range of content—
in terms of breadth and depth – for
indexing by the discovery service
providers
21

22. Specific Benefits
Discovery service providers
– Receive more standardized and efficient
integration with the information
providers through common industry
definitions and communications
22

23. Balance of Constituents
Libraries
Marshall Breeding, Co-Chair Michele Newberry, Florida Virtual Campus
Jamene Brooks-Kieffer, Kansas State University Sara Brownmiller, University of Oregon
Laura Morse, Harvard University Lucy Harrison, Florida Virtual Campus (D2D
Ken Varnum, University of Michigan liaison/observer)
Information Providers
Lettie Conrad, SAGE Publications Linda Beebe, American Psychological Assoc
Roger Schonfeld, ITHAKA/JSTOR/Portico Aaron Wood, Alexander Street Press
Jeff Lang, Thomson Reuters Peter Noerr, MuseGlobal
Service Providers
Jenny Walker, Ex Libris Group (Co-Chair) David Lindahl, University of Rochester (XC)
John Law, Serials Solutions Jeff Penka, OCLC (D2D liaison/observer)
Michael Gorrell, EBSCO Information Services
23

24. Organization
• Reports in NISO through Document to
Delivery topic committee (D2D)
• Staff support from NISO (Nettie Lagace)
• Co-Chairs
– Jenny Walker (Ex Libris)
– Marshall Breeding (Library Consultant)
• D2D Observers:
– Jeff Penka (OCLC), Lucy Harrison (FLVC)
24

25. ODI Project Goals
1. Identify, possibly through surveys or other
questionnaires, the needs and requirements of
the three stakeholder groups in this area of
work
• Created subgroups for information gathering:
– Level of Indexing
– Library Rights
– Technical formats
– Usage Statistics
– Fair Linking
25

26. ODI Project Goals
1. Identify, possibly through surveys or other
questionnaires, the needs and requirements of
the three stakeholder groups in this area of
work
• Created subgroups for information gathering
• Conducted interviews with stakeholders
• Created survey with input from all sub groups
• Survey is currently live (closes October 4)
26

27. https://www.surveymonkey.com/s/QBXZXSB
27

28. ODI Project Goals
1. Identify, possibly through surveys or other
questionnaires, the needs and requirements of
the three stakeholder groups in this area of
work
• Created subgroups for information gathering
• Conducted interviews with stakeholders
• Created survey with input from all sub groups
• Survey is currently live (closes October 4)
• Analyze results as input to Goal 2
28

29. ODI Project Goals:
2. Create recommendations and tools to
streamline the process by which information
providers, discovery service providers, and
librarians work together to better serve libraries
and their users

30. ODI Project Goals:
3. Provide effective means for librarians to assess
the level of participation by information
providers in discovery services, to evaluate the
breadth and depth of content indexed and the
degree to which this content is made available
to the user

31. Specific deliverables
• Standard Vocabulary
• NISO Recommended Practice:
– Data format & transfer
– Communicating content rights
– Levels of indexing, content availability
– Linking to content
– Usage statistics
– Evaluate compliance
• Inform and Promote Adoption
31

32. Timeline
Milestone Target Date Status
Appointment of working group December 2011
Approval of charge and initial work plan March 2012
Agreement on process and tools June 2012
Survey completed Oct 4, 2012
Completion of information gathering October 2012
Completion of initial draft January 2013
Completion of final draft May 2013
32

33. Connect with ODI
• ODI Project website:
http://www.niso.org/workrooms/odi/
• Interest group mailing list:
http://www.niso.org/lists/opendiscovery/
• Email ODI:
odi@niso.org
33

34. Seeing Discovery Through User Colored Glasses
NISO Webinar: Discovery and Delivery: Innovations and Challenges
Timothy Babbitt, Senior Vice President, Platform Management, ProQuest

35. Understanding What Is Valuable to Users
Foundation of Librarianship
S.R. Ranganathan’s The Five Laws of Library
Science (Madras India: Madras Library Association, 1931)
1. Books are for use.
2. Every reader his [or her] book.
3. Every book its reader.
4. Save the time of the reader.
5. The library is a growing organism.
Even at a time when the emphasis was entirely on
physical media the focus was on the individual goals
and needs each type of user (the reader)

36. The Five Laws Updated
Ranganathan Updated Laws
1. Books are for use. 1. Information in all of its forms is for
2. Every reader his [or her] book. use.
2. Every researcher their
3. Every book its reader.
information.
4. Save the time of the reader. 3. Every medium and delivery
5. The library is a growing platform its user.
organism. 4. Enable efficient discovery by the
user.
5. The library is part of an evolving
research ecosystem.
The proliferation of web based information tools allows us to track users and
their behavior with increasing precision but understanding the unique needs and
behaviors of different researchers requires deep analysis and interpretation of a
variety of data – traditional usage data does not tell the whole story.

37. How do we measure value in the evolving
research landscape?
Traditional Model
 Usage data to measure value (searches and retrievals) –
more usage = more value
Growing Trend
 Web based analytics – behavioral and attitudinal dimensions
segmented by types of users
Key Question
 How can we combine both approaches and what can they provide
that traditional usage data cannot?
 Usage Data = what they did
 Other Analytics = what they were trying to do; did they succeed; what
was the context; and who was doing it?

38. Value Differs by User
Who is doing the research matters
Librarians
 Making their patrons successful
 Promoting services of library to students and faculty with
confidence
 Delivering services that are convenient for their users
Faculty
 Extending their influence and reach in their discipline
through published research
 Efficiently directing students to materials that meet
learning objectives
 Obtaining research grants
Students
 Completing coursework in accordance with faculty
directives
 Remaining in compliance with source attribution policies
 Accessing information conveniently

39. Conventional Wisdom Might Tell Us
 High use = high value
 High satisfaction = high value
 Use (search and retrievals) is homogenous
Including Attitudinal and Behavioral dimensions to
traditional usage data allows us to challenge many long
held assumptions

40. Three Dimensions of Analysis
Focus of Analysis:
The discovery and
consumption of
documents (COUNTER
Usage Reporting e.g. Searches
and Retrieves) etc.
Focus of Analysis: Focus of Analysis:
What is the user’s
What is the user Attitudinal Behavioral context?
satisfaction? (Page Views per
visit, Time on Site,
and Clickstream
etc.)

41. Three Dimensions of Analysis
Focus of Analysis:
The discovery and
consumption of
documents (COUNTER
Usage Reporting e.g. Searches
and Retrieves) etc.
Attitudinal Behavioral

42. Usage in the new age of discovery
A case study
Earlier this year, a library contacted us about a large
increase in search usage in one of their databases:
Month 11-Jan 11-Feb 11-Mar 11-Apr …
Searches Run 50 250 43 199
Month 12-Jan 12-Feb 12-Mar 12-Apr …
Searches Run 265 616 1176 847

43. Web scale discovery and workflow
By default, the user runs an all database search
(thus showing as a ―hit‖ against all DB’s in the library’s subscription.)
In this scenario, the user starts their search for ―space tourism‖ via a discovery service such
as Summon. Finding a document they like, they click through to the ProQuest platform and
land at the document level. Then decide to run another search, which ―hits‖ all databases.

44. Usage in the new age of discovery
 The cause? Heavily utilization of a new web scale discovery
service.
 Users now used our website differently than users who started
at the library ―Databases A-Z‖ page. The change in how their
users were coming to our site led to a change in usage.
 Usage data showed the effect of the change of their patrons’
research environment. Determining the cause required looking
beyond the usage data.
For more details see the white paper ―Usage in the New Age of Discovery‖

45. Three Dimensions of Analysis
Usage
Focus of Analysis:
What is the user’s
Attitudinal Behavioral context?
(Page Views per
visit, Time on Site,
and Clickstream
etc.)

46. Behavioral Analysis:
Precise Measure of Visits

47. Behavioral Analytics
Pages Viewed/ Visits

48. Three Dimensions of Analysis
Usage
Focus of Analysis:
What is the user Attitudinal Behavioral
satisfaction?

49. Attitudinal – Satisfaction Comparison
Survey Methodology
 Over 19,500 Surveys
 Data from November 2011 through September 2012
 Continuous monitoring
 Predictive modeling
Librarians Faculty Students

50. Satisfaction Comparison: Role

51. Satisfaction Comparison: Role
Number of Respondents

52. Attitudinal Analysis:
Student Satisfaction Trend-line

53. Student Satisfaction: By Week

54. Satisfaction Comparison: Primary Purpose

55. Satisfaction Comparison: Primary Purpose
Number of Respondents

56. Attitudinal Analysis
Satisfaction Comparison: School Type

57. Attitudinal Analysis
Satisfaction Comparison: School Type

58. Cross Tab:
Primary Purpose and School Type
Data from November 8, 2011 through August 12, 2012

59. “Deeper Dive” with Segment Analysis:
Data: Nov. 8, 2011 through Aug 7, 2012

60. Seeing Value from a User
Perspective
 We need the triangulation of
 Usage data
 Behavioral data
 Attitudinal data
 Drive understanding of
 What did the users do?
 What was the context of use?
 What were they trying to do and were they successful?
 Important because the research ecosystem is
changing
 i.e. Web scale discovery

61. Toward understanding what is
valuable to users
Updated Laws
1. Information in all of its forms is for use.
2. Every researcher their information.
3. Every medium and delivery platform its
user.
4. Enable efficient discovery by the user.
5. The library is part of an evolving
research ecosystem.

62. Next steps
 Complete integration and analysis of
 Usage data
 Behavioral data
 Attitudinal data
 Give libraries a deep understanding of value from
user segments
 Next stop…content and search analysis!

63. Collecting Patron Perspectives on
Discovery Tools
NISO Webinar
Discovery and Delivery: Innovations and
Challenges
September 26, 2012
David Bietila
Web Program Director
The University of Chicago Library

65. Data Sources
Finding
Aids
Databases Digital
& Indexes Collections
Discovery
Catalog Tool
Web Pages

69. Assessment Questions
 Product Characteristics
 Technical
 Functional
 Interface
 Relevance to Users
 Define use case
 Novice users of databases and electronic resources who need to
find articles on a topic.
 Assess product’s applicability to this use case

70. 1. User Comments
Topic User attitudes toward the product, and specific aspects valued by
users.
Method Posted link to a comment form in the header.
Set up an info table for three days.
Required minimal effort to collect data.
Sample Questions What type of resource were you looking for with the Articles Plus?
Please share any positive or negative comments about your
experience.
Results Unprecedented proportion of positive comments.
Users valued the speed of this search, and the ability to search
both books and articles in one place.
Functional & Interface Assessment

71. Resource sought Journal article
Successful? Yes
Affiliation Graduate or Professional School Student
Comment This was a quick search and gave me exactly what I needed.
Resource sought Academic journal articles
Successful? No
Affiliation Graduate or Professional School Student
Record included books, needed a way to filter this out
Comment
results from Time magazine, not what I was looking for at all
Both books and articles on certain topics (ones related to philosophy, psychology and
Resource sought
literature).
Successful? Partial
Affiliation Graduate or Professional School Student
It's wonderful to have ONE place where you can search for both articles and books! However, it
seems like more books should show up because some books relevant to my search showed up
Comment
in lens but not in Articles Plus. If you don't choose this search tool, please do adopt some search
tool that allows comprehensive searching of both books and articles!

73. 2. Usage Statistics
Topic Usage relative to major databases
Method Examined data presented in the discovery tool’s admin interface.
Compared usage with that of high traffic databases (Web of
Science, JSTOR, Academic Search Premier).
Results Observed increasing usage, reaching parity in the initial year with
Academic Search Premier.
Verified that users were finding and using the tool.
Interface Assessment

74. Searches per Month - Articles Plus and Major Databases
50000
45000
40000
EDS Foundation Index
35000
(ArticlesPlus)
30000 Web of Science
25000
JSTOR
20000
15000
Academic Search
10000 Premier
5000
0

75. 3. Usability Testing
Topic Clarity of search interface, including functional elements and
labels.
User ability to complete article discovery tasks.
Method Assigned representative article searching tasks.
Recording screen activity and spoken comments for analysis.
Sample Questions Can you locate full text of the following article? Mary L.
Dudziak, “Desegregation as a Cold War Imperative,” 41 Stanford
Law Review 61 (1988)
You are working to prepare a summary of developments in the
field of astronomy. Can you locate five articles on astronomy that
were published in Nature in 2010?
Functional & Interface Assessment

76. Task
Participant Participant
Session
Status Department
1 2 3 4 5 6 7 8 9 10
1 Graduate Public Policy         X 
2 Graduate CMES ~  X   X  X X 
3 Undergrad Undeclared     ~  X  X 
4 Undergrad Anthro       ~   
5 Graduate MAPH          
6 Undergrad Physics ~     ~    
7 Undergrad English      ~   X ~
8 Graduate MAPSS ~      ~   ~

77. 3. Usability Testing
 Results
• Cut or relabeled certain facets
• Made heavily used features more prominent.
• Removed unused or confusing features
• Determined collections to be retained or removed from coverage.
Functional & Interface Assessment

81. Iterative Evaluation
Refine
Evaluate
Use
Tool
Case

82. Other Use Cases
 Experienced researchers
 Searching interdisciplinary topics
 Searching outside their primary area
 Searching within their primary area

83. 4. Subject Assessments
Topic Applicability of the discovery tool for use by advanced researchers
in disciplines with a variety of requirements.
Method Enlisted bibliographers and subject specialists as proxies for
different user constituencies.
Created an evaluation rubric to ensure comparable results across
over 40 disciplines assessed.
Results Determined which disciplines would be best served by the tool.
Identified strength of coverage in numerous disciplines.
Functional Assessment

84. Subject Evaluation Form

86. Compared to Compared to
Subjects – Social
First 25 Results Core Database JSTOR/ Google
Sciences Results Scholar Results
Anthropology/Geography/Maps ~ ~ ~
Gender Studies ~ ~ -
History + + +
International Political Economy ~ - ~
Psychology + - ~
Sexuality Studies + + +

87. Outcomes & Future Assessment
 Recommendation to purchase
 Improvements
 Local configuration
 Enhancement requests
 Marketing & Communication
 Benefit of Iterative Assessment
 Future assessment
 Applicability of new coverage
 Revisiting usability
 Overlap with other search tools

88. THANK YOU
Thank you for joining us today.
Please take a moment to fill out the brief online survey.
We look forward to hearing from you!