Acrl2015 presentation gotime
•Als PPTX, PDF herunterladen•
2 gefällt mir•841 views
How successful are discovery tools in providing relevant results? Can these single search boxes really displace/replace the library catalog? This presentation presents the results of an analysis of user queries sampled from the University of Southern California's discovery tool and how results using these same queries compare when entered into Google and Google Scholar in terms of relevancy. This session will provide insight into user search behavior and examine if discovery tools are actually delivering on their promise to deliver relevant and expected results.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Acrl2015 presentation gotime
Ähnlich wie Acrl2015 presentation gotime (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Acrl2015 presentation gotime
- 1. Measuring Our Relevancy: Comparing Results in a Web-Scale Discovery Tool, Google & Google Scholar Elizabeth Namei, University of Southern California Libraries, namei@usc.edu and Christal Young, University of Southern California Libraries, youngc@usc.edu #summonrelevancy ACRL 2015 Conference
- 3. “I could throw an author and title in [Summon] but I know that works very erratically….I often get inundated by irrelevant things…” “[Summon] just doesn’t generally turn up reliable results for me. I’ll search a very obvious keyword or a very specific keyword and it won’t turn up the most relevant results first even though I know the highly relevant results are in there...” English professor Senior music major
- 4. In search of evidence
- 5. methods
- 6. We pulled out a random sample of 384 queries There were 1.2 million searches entered in Summon in Fall 2014.
- 7. From the 384 queries we eliminated: ● 63 queries for known items that USC did not own ● 22 queries that had unrecognizable or foreign characters ● 21 queries that were for “non-scholarly” formats Our final sample size = 278
- 8. defining relevance 1. Relevant: a match for the known item shows up 1st or 2nd in the list of results. 2. Partially Relevant: a match for the known item is found 3rd- 10th in the list of results. 3. Not relevant: known item is not listed in the first 10 results or no results were returned.
- 9. results
- 10. 76% 91% 79%
- 11. 24%
- 12. 100,000
- 16. Partial Citation queries (with 2 metadata elements) that did not find relevant results in Summon and then there were none agatha christie palladio four books on architecture showed up 74th shows up 75th
- 17. Quotes were used in only 6% of the queries (16/278) but…… 100% of these queries returned relevant results
- 19. Formatted citations (with 3 or more metadata elements) that did not find relevant results in Summon Block G Hartman AM Dresser CM Carroll MD Gannon J Gardner L. A data-based approach to diet questionnaire design and testing. Am J Epidemiol 1986; 124:453-469. does not show up in first 10 results (6 metadata elements) no results returned (9 metadata elements) Brooks M. K. (2010). Hospice services: The dilemmas of technology at the end of life. In T. S. Kerson J. L. M. McCoyd & Associates (Eds.) Social work in health settings: Practice in context (3rd ed. pp. 235-246). New York NY: Routledge.
- 20. if fewer metadata elements had been used... Block A data-based approach to diet questionnaire design and testing. shows up 1st (2 metadata elements) Shows up 1st (5 metadata elements) Brooks M. K. (2010). Hospice services: The dilemmas of technology at the end of life. In T. S. Kerson J. L. M. McCoyd & Associates (Eds.) Social work in health settings: Practice in context
- 22. Continue teaching users to be more strategic searchers ● Explain the whys of strategic searching and not just the hows ● Explain the organization of information systems and the benefits field searching (especially for one word/common word titles) ● Provide troubleshooting tips for how to deal with the inevitable failed searches
- 23. follow the user
- 24. Push for In-house solutions ● Using APIs ● Developing programs to “clean” problematic queries Lean on vendors • to improve relevancy • to improve “Did you mean” suggestions • To provide query reformulation suggestions Create Smarter Library Systems
- 25. personalization
- 27. Image credits Slide 1 image https://flic.kr/p/9pfXxq Slide 2 image: http://bit.ly/1NkcLNJ Slide 4 image: http://bit.ly/1G0s5fY Slide 5 image: http://bit.ly/1DQRREd Slide 6 image: http://thevisualharvest.com/lib/img/elements/to-the-drawing-board-graphic.svg Slide 8 image: http://relevantinfo.org/wp-content/uploads/2013/09/good-o-meter.jpg Slide 9 images: ● Jeopardy: http://www.ibm.com/smarterplanet/us/en/ibmwatson/assets/img/tech/watson_on_jeopardy.jpg ● Summon: http://www.almanhal.com/ResourceImages/Publisher/StartigicPublisherLogos/ProQuest.png ● Google: http://bit.ly/1I0b0RZ ● Google Scholar: http://www.redcad.org/members/hadjkacemm/images/GS2.png Slide 12 image: http://wishflowers.tumblr.com/post/10244886760 Slide 16 image: http://bit.ly/1EdFthI Slide 21 image: http://www.healthcarereformmagazine.com/voluntary-benefits/where-do-we-go-from-here-voluntary/ Slide 22 image: http://samanthashorey.blogspot.com/2011_02_01_archive.html Slide 23 image: http://blog.cityspoon.com/2012/02/08/gathering-followers/ Slide 24: Image: http://www.csfieldguide.org.nz/ArtificialIntelligence.html Slide 25: image: http://bloomreach.com/snap Slide 26: image: http://www.gtcw.org.uk/gtcw/index.php?lang=cy
Hinweis der Redaktion
- Get Catalog usage numbers for 2014 as a point of comparison?
- The problem: we were hearing from our users they they were getting “unexpected” results when searching USC’s Summon instance. Google has set people’s expectations, and so have our experiences figuring out how to use our complex systems, but it also due to the fact that some advanced/power users had grown accustomed to the precision searching available in OPACs and library databases Mention the data about users not looking beyond the first page of results
- We’ve had a wide variety of users express frustration with Summon’s results. These are just 2 examples (gathered when we did usability testing of our website in Fall 2013?) This problem of getting “unexpected” or irrlevant results is significant because research has found that if users have a negative experience with an online search system they will quickly abandon it. (footnote 20) Recently more and more librarians and faculty are commenting about discovery layers ability to deal with known item searches the current landscape - do we go with discovery tools or not? Better understand the cause of unexpected results “One additional challenge lies in the ability of discovery services to find known items. Especially when searching for resources with one-word titles or common words, such as Nature or Time, relevancy-based retrieval may not always return the expected results. Each of the discovery services has improved its handling of known-item searching, but this continues as a point of criticism of performance.” Marshall Breeding, “The Future of Library Resource Discovery” (February 2015) These search tools are pervasive on libraries websites, often as the default search option for discovering or accessing materials, and some libraries have made the shift to replacing their OPACs with them.
- MOTIVATION OF THE STUDY was to get a clearer picture of why users were getting unexpected results. Highlight our role as reference/instruction librarian Why does this matter to libraries? - discovery services are becoming the ubiquitous default search option HYPOTHESIS We wanted to know: When and why USC’s discovery service (Summon) returned “unexpected” or irrelevant results when users search for known items. How Summon’s results compare to Google and Google Scholar in terms of relevancy of known item queries? since We wanted to know: Are discovery services, and Summon in particular, up to the task? When and why USC’s discovery service (Summon) returning “unexpected” or irrelevant results, especially for known items queries. Because Google has set user expectations for online searching experience, we wanted to compare the relevancy of results in Summon to Google and Google Scholar. Lastly we wanted to gain insight into how users are searching for known items in our discovery system.
- Transaction log analysis Random sample of 384, Ended up with 278 since we eliminated queries with unrecognizable characters, and non-scholarly and non-USC owned items. Recreated searches and
- In Fall 2014 there were close to 1.2 million searches entered into our Summon instance. Of these, 433,863 were unique searches. We pulled out a random sample of 384 search queries, in order to give us a 95% confidence level, 5% margin of error
- For Google & GS the locating the full text of an item was not required for the results to be deemed relevant.
- Thanks Christal This slide hints at our results of our study with Watson, IBM’s supercomputer (standing in for Google), and crushing the competition
- This chart show how the 278 known queries faired across all 3 search tools. As you can see, Google lives up to its reputation for providing overwhelmingly relevant results. What did surprise us was Google Scholar’s middling performance compared to its big brother. Our findings also confirm our hypothesis that Summon would underperform compared to the two Googles. Even though 76% is not a stellar we were still surprised at how well Summon did at finding relevant results for 3/4 of the queries. Summon: 212/278 = 76% Google: 252/278 = 91% of all scholarly known item queries found relevant results in Google GS: 220/278 = 79% ----- Meeting Notes (3/28/15 06:35) -----
- We were wanted to better understand why these 66 (24%) searches did not lead to what we categorized as relevant results. We have informally labeled these 66 searches as “FAILED” searches since users have an expectation of finding the known item at the top of the list, showing up 3rd-10th is often comparable to showing up 40th To translate/contextualize this for you → next slide 47/278 = 17% of searches totally failed (these are for things we own 66/278 = 24% (¼ of searches failed in Summon) (compared to 6/278 = 2%; 26/278 = 9% failed searches) 48/278 totally failed = 17% 58/278 failed in GS = 21% HOW MANY OF THESE “RELEVANT RESULTS” had LINKING PROBLEMS? What proportion of these failed searches were caused by user errors? (next slide have new pie chart with the numbers w/out errors??) 66 irrelevant queries Partial Citation = 36 - 55% Title search = 18 - 27% Advanced = 1 - 2% Numeric = 2 - 3% Formatted Citation= 9 - 14%
- From the 1.2 million searches entered into Summon in the Fall semester 35% are known items = (420,000) 24% of these = OVER 100,000 searches resulted in users not finding the known item they were searching for That is 1 out of every 4 known item search 36% of 1.2 million =, then 24% of that number did not find the known item being searched for (should we also account for the number of known items we don’t own?) - THIS SLIDE MIGHT WORK BEFORE LAST SLIDE
- As instruction librarians we are particularly interested in how user search behavior impacts the relevancy of results in our discovery system. So for this paper we looked at the types of searches users were entering when looking for known items – We found that there were five categories. As you can see, the majority of queries only included the title. the 2nd most frequently used search type were partial citations w/2 metadata elements Taken together these two search types comprise 91% of all searches Only 3% (9/278) the searches used the Advanced search form Partial Citations - 63/69 were author+title 36 - author’s full name + full title 27 = author’s last name + title The rest: 2 = author + date 1 = title + series title 1 = title + url 1 = authors + date 1 = title + journal title
- Here is a side by side comparison of the percentage of relevant results returned by all three search tools according to search type. Because of time constraints we will only focus on the two worst performing ones
- Partial citation searches, which comprise the 2nd most common search type in Summon were among the worst at providing relevant results, which was somewhat surprising/counterintiutive sinc e you would think that by providing more information (but not too much) the query would be more successful . (the most common are author title, but there were also title+date, and even one title+URL) 52% (22+____/69) of partial citation searches did not find relevant results 21/22 (95%) of the failed searches were author title combinations (there was one failed search that included the title+URL) 91% (63/69) of the partial citation queries were made up of a variation of author+title 32% (22/69) of Partial Citations failed to find relevant results (doesn’t include partially relevant) -
- Here are 2 examples: of partial citation searches that experienced catastrophic failure ASK AUDIENCE: WHAT ADVICE WOULD YOU GIVE TO IMPROVE THESE SEARCHES? Summon’s Product Manager in an email: “many problems can be solved by using facets” “The challenge…is what fields to use to trigger this known item searching behavior. Exact matches are the easiest to solve for, especially if you have an exact match across two different fields (such as title and author). The logic around queries that aren’t exact matches (misspellings, omitted words, terms that fall into different fields) is more challenging.”
- We re-ran the failed title and partial citation searches but this time with quotes to see how they would fair. This matches findings from other studies. We found that quotes improved the relevancy of partial citation queries 74% of the time! Quotes would have corrected 74% (20/27) partial citation searches with two metadata elements that did not find relevant results (this would give Summon 232/278 relevant results - 84% Search types for queries w/quotes: 9/16 - (56%) were title searches 5/16 - (31%) was a partial citation (2 elements) 1/16 - (6%) was an advanced/field search 1/16 - (6%) was a copy/pasted citation (it didn’t have formatting though, like many of the other queries in this category) a manually typed book chapter: Pine Lisa. "The Jewish Family." Nazi Family Policy. (quotes were used one other time, for a video/non-scholarly source, but it was a one word title, so was incorrectly used and didn’t help or hurt the search)
- The worst performing search type are ones that we labeled “formatted citation searches.” These often included abbreviations and formatting and most appeared to be copy and pasted from course syllabi or reference lists Although these types of searches comprised among the smallest of our sample (11), they were THE MOST likely to lead to a failed search with no results returned, and this is true ACROSS all 3 search tools. Many studies have reported an uptick in these types of searches. There were only 11 queries that had 3 or more metadata elements, most often copy and pasted and so included formatting. Of these 2 were successful in finding relevant results, 1 was partially successful and 8 did not find relevant results. 82% that did not find relevant results
- Here are 2 examples from our sample, the first is a journal article that, the 2nd is a book chapter, and as you can see they did not fair well. (I gave up looking for the first query… In Google, the Brooks chapter fails Block article shows up first in Google, but does not show up in first 10 in GS
- We experimented with removing some of the metadata elements (while leaving some formatting) and we found that in most cases this improved the relevancy, leading us to believe that the number of metadata elements entered clearly has an impact on the relevancy of known item searches In Google, the Brooks chapter also fails Block article shows up first in Google, but does not show up in first 10 in GS
- So, what are the implications of our findings and our recommendations. First, we want to make it clear that there are many more questions to be addressed and that there are certainly other contributors to poor relevancy besides user search behavior. 4 main causes of not getting relevant results, our paper only focused on how user search behavior influenced the relevancy of results, user errors (typos)/poor Did you mean suggestions (##??) users entering too much information (formatted citations, urls, full names etc.) (###?) - But Summon should be able to do something about these (clearn them) bad relevancy (when quotes work to make the results show up higher bad metatdata or indexing problems (###??) - Metadata issue on the listserv: - a discovery tool is only as good as the underlying indexing and metadata
- Obviously Christal and I believe in the importance and value of teaching users to develop search strategies and to understand how information systems are organized. EXPLAIN WHY copying and pasting an entire formatted citation is not likely to work in any search tool, which also means explaining why all search systems/search boxes are NOT THE SAME and so must be approached differently. This will in turn give users more sophisticated mental models for online searching, they will in turn learn how to troubleshoot failed searches (facets, removing formatting and abbreviations). But, the answers to improving the outcome of a search cannot ONLY/ALL be found in teaching users to more strategic searhers. Compared to the open web (the fields, full text vs. A&I, naming conventions – abbreviated names and titles, etc.). “Explain underlying metadata structures and how they can be leveraged to improve search efficacy” (Townsend) Mental models of search: help users “conceive of information [in library databases] as something with an organization and underlying system, rather than a mysterious cloud of data” (Townsend) Google is a success engine, it works for ages 4-104. Academic search tools are more sophisticated and allow for more control and precision and don’t offer (as much) fuzzy searching (the training wheels are off); Framework language: “understand how information systems (i.e., collections of recorded information) are organized in order to search for and access relevant information”; understand the potential of each type of strategy; Find Metaliteracy article about teaching Focusing on teaching the user to be smarter searches or……..Don’t fight user behavior, it’s an unwinnable battle ----- Meeting Notes (3/28/15 08:04) ----- FIX THIS
- We want to propose that instead of complaining about the lazy search habits undergraduates or getting frustrated about users being content with “good enough results” or dismissing Discovery services and Google for dumbing down the research process, that we try to empathize with these expectations and behaviors The part of the research process that should be (and is) challenging is the analyzing, evaluating and synthesizing of information sources not the finding/searching.
- Libraries need to focus more resources and lobby for smarter, more intuitive and more adaptive systems. Many libraries have come up with In house solutions: For instance is California State University at Fullerton came up with a “scrubbing program” that identifies and then “cleans” APA formatted citations entered into their discovery layer, stripping out everything except the title which as we saw are the most successful search types Leaning on vendors - to improve overall relevancy and help features field weighting and term proximity “How close together are the query terms found in the record? If closer together, then a record is more relevant” providing better “Did you mean?” suggestions adding query reformulation suggestions for searches that get no results Out of 39 queries with user errors a “Did You Mean” suggestion showed up 7 times but only 2 of those times did the link take you to the known item <<DOUBLE CHECK THIS “this system is stupid. there are some systems that learn, Google learns” [43:40] - Jeff Edmunds Video; Discovery tools should adapt so that search results will not suffer no matter how many metadata elements are included learn how to better deal with 2 or more metadata elements Smarter systems Google is smart and adaptable DIscovery vendors need to smarter and more adaptable
- Lastly we want to leave you with what we realize may be potentially controversial suggestion. Everyone identifies Google’s simple interface and relevance algorithm as the keys to its success. But since 2009 it has been incorporating personalization features into its presentation of results which are further shaping user expectations. More and more of our users—including faculty and librarians—are calling on libraries to harness the vast amounts of user data we collect to provide results that are tailored to individual research interests and needs, recognizing that presenting the same relevancy ranked results to everyone no longer works. Maybe this is missing link libraries need to “cautiously and conscientiously explore” in order to provide users with “expected” results for all searches. Make “serendipitous discovery work” - from yesterday’s “Limited by search” session (“relationship methodologies - relationships between content). One recent proposal, made by David Weinberger from Harvard’s Berkman Center for Internet and Society (October 2014 issue of Chronicle of higher education), was to create a “stackscore” which would signify “how relevant an item is to the library’s patrons as measured by how they’ve used it.” (footnote) 68 He lists numerous datasets that are either already being collected or that could be easily obtained, which could be factored into developing this score, from renewals and recalls to readings listed on a syllabus. Commercial interests motivated the initial push towards offering personalized services, but bringing personalization technologies into libraries holds the promise of enhancing the breadth, depth, and reach of scholarship and scholarly communication in new and exciting ways. Part of Google’s success is due to its use of personal data to enhance the relevancy of search results for each individual user. Libraries will never succeed in providing a truly Google-like search experience without moving in this direction. By offering personalized search systems, libraries will be better able to serve their users, not just in leading them to relevant content, but in anticipating and meeting their future information needs. ----- Meeting Notes (3/28/15 08:04) ----- fix this